mobileInsight-core Part-I: Monitor

In this series, we will review the codes in mobileinsight-core, the core modules and desktop version of MobileInsight. This code review has two parts. In this article (Part-I), we will introduce the big picture of mobileinsight-core, and its cellular message monitor. In Part-II, we will introduce analyzer, the cellular protocol analytics module.

What is mobileinsight-core?

mobileinsight-core implements the core features and the desktop-version Python module. It provides the following functions - Core modules in MobileInsight: monitor and analyzer - Cellular message configuration manager and parser: dm_collector_c - Wireshark helper (for message parsing) ws_dissector - Desktop-version packaging and installation MANIFEST.in and setup.py - Other utilities, such as example codes, GUI, test scripts, etc.

mobileinsight-core is written in Python-2.7 (monitors, analyzers, examples, test scripts, GUI) and C++ (ws_dissector and dm_collector_c). This makes the cross-platform support (Android/OS X/Win/Linux) possible.

Code Structure: A Big Picture

You can download mobileinsight-core from our code repository:

git clone https://github.com/mobile-insight/mobileinsight-core.git
cd mobileinsight-core

If you run ls mobileinsight-core, you will see files below:

./mobileinsight-core
├── mobile_insight # Core modules, including monitors and analyzers
├── dm_collector_c # Cellular message configuration manager and parser
├── ws_dissector # Wireshark helper (for message parsing)
├── MANIFEST.in # manifest file (used for packaging desktop-version modules)
├── setup.py # External package function for MobileInsight installation
├── GUI  # Desktop-version GUI (LogViewer)
├── docs # Documentations
├── examples # Example codes for MobileInsight tutorial
├── unit-test # Unit test scripts
├── LICENSE # software license description
└── README.md # readme file

The core modules in MobileInsight, Monitors and Analyzers, are located in mobileinsight-core/mobile_insight:

./mobileinsight-core/mobile_insight
├── __init__.py # Python packaging script, which imports monitors and analyzers
├── analyzer # Analyzers folder
├── element.py # Base class of monitors and analyzers
├── monitor # Monitors folder
└── utils.py # Some utility functions

How to compile and install?

As a developer, you should know how to compile AND install MobileInsight. This differs from normal MobileInsight users that only need to install, because dm_collector_c and ws_dissector have been pre-compiled before distribution. The following are concrete steps:

  1. Make sure your desktop has installed Python2.7, Cython and Wireshark source code.

  2. Compile dm_collector_c: run the following command

    python setup.py build_ext --inplace
    

    This will compile dm_collector_c and copy it to mobile_insight/monitor/dm_collector.

  3. Compile ws_dissector: run the following command

    cd ws_dissector
    make
    

    You should change ``Makefile`` before this step. The Makefile includes path of Wireshark source code, which should be modified based on your desktop’s local path.

  4. Installation: run the following command

    cd .. # mobileinsight-core root directory
    sudo python setup.py install
    

Base class of monitors and analyzers: Element

We next elaborate how Monitors and Analyzers are implemented in MobileInsight. For more details of their high-level features and functionalities, we recommend you to read our Mobicom’16 paper.

Before implementing Monitors and Analyzers, MobileInsight first declares a base class called Element. The rationale is that, both Monitors and Analyzers require some common features, including - Abstraction of cellular messages (as an event); - Event-driven callback mechanisms (i.e., MI-DEV API); - Interface with other mobile applications (i.e, MI-APP API); - Logging functions

MobileInsight abstracts these common features into Element. The following shows a simplified code snippet of it (the complete code is in mobile_insight/element.py):

class Event(object):
    '''The event is used to trigger the analyzer and perform some actions.
    The event can be raised by a trace collector (a message) or an analyzer.
    An event is a triple of (timestamp, type, data).
    '''
    def __init__(self,timestamp,type_id,data):
        self.timestamp = timestamp
        self.type_id = type_id
        self.data = data

class Element(object):
    '''
    The parent class to derive trace collectors and analyzers.
    '''

    def __init__(self):
        """
        Initialize logging function, and empty list of callbacks
        """

    """
    MI-DEV API: implements send/recv-based callback mechanism (for events)
    """

    def send(self,event):
        """
        Raise an event to all Analyzers in from_analyzer_list

        :param event: the event to be sent
        """

    def recv(self,module,event):
        """
        Upon receiving an event from module, trigger associated callbacks

        This method should be overwritten by the analyzer and monitor

        :param module: the module who raises the event
        :param event: the event to be received
        """

    """
    MI-APP API: interface between mobile apps and MobileInsight
    reference: http://mobileinsight.net/mi-app.html
    """

    def broadcast_info(self, method, msg_dict):
        """
        (Mobile-version only) Broadcast monitor/analyzer information to other Android apps.
        This method is the interface between MobileInsight and Android apps requiring cellular info.
        It leverages Android's intent-based broadcast mechanism.

        The intent is per-analyzer/monitor based, and the action is named as follows:
        `MobileInsight.ANALYZER/MONITOR-NAME.method`

        where ANALYZER/MONITOR-NAME is the class name of the monitor/analyzer,
        and method is analyzer/monitor-specific method to notify the Android apps

        :param method: analyzer/monitor-specific methods for broadcast action
        :type method: string
        :param msg_dict: A dictionary that lists the information to be broadcasted.
        :type msg_dict: string->string dictionary
        """

    """
    The following are logging functions
    Reference: https://docs.python.org/2/library/logging.html
    """

    def set_log(self,logpath,loglevel=logging.INFO):
        """
        Set the logging in analyzers.
        All the analyzers share the same logger.
        """
    def log_info(self, msg):
        # ...
    def log_debug(self, msg):
        # ...
    def log_warning(self, msg):
        # ...
    def log_error(self, msg):
        # ...
    def log_critical(self, msg):
        # ...

Monitors

MobileInsight monitors extract raw cellular messages, parse them as human-readable format, and triggers the further analysis for Analyzers. We have realized various monitors, including online and offline fashions, and desktop and in-device versions. All monitors are included in mobile_insight/monitor, as shown below:

./mobileinsight-core/mobile_insight/monitor/
├── __init__.py # Python packaging function
├── monitor.py # Base class for all monitors
├── android_dev_diag_monitor.py # Android's in-device, runtime monitor using diag_revealer
├── android_qmdl_monitor.py # (Depreciated) Android's in-device, runtime monitor using diag_mdlog
├── dm_collector # Desktop-version runtime monitor (through serial port in USB)
├── offline_replayer.py # Offline mi2log log replayer
└── online_monitor.py # A wrapper of cross-platform (Android/Desktop) runtime monitor

Despite their diverse functions, their implementations share the same structure. We next elaborate it.

Base class: Monitor

All MobileInsight monitors should inherit from the base class Monitor, which defines the basic interface for a monitor: - Enable user-specified cellular message types; - Bind analyzers to this monitor; - Start the monitoring process. Each monitor should overload these interfaces based on its specific environment. For example, to start the monitoring process (run()), the offline replayer OfflineReplayer loads the mi2log from the storage, while other online monitor AndroidDiagMonitor extract cellular messages from in-device diagnostic port.

The following code snippet summaries the abstract interfaces (complete code in mobile_insight/monitor/monitor.py):

from ..element import Element, Event


class Monitor(Element):
    """
    An abstraction for mobile network monitors
    """

    def __init__(self):
        # No source for Monitor
        Element.__init__(self)

        self._skip_decoding = False

        self._save_log_path = None
        self._save_file = None

    def available_log_types(self):
        """
        Return available log types
        """
        return None

    def save_log_as(self,path):
        """
        Save the log as a mi2log file (for offline analysis)

        :param path: the file name to be saved
        :type path: string
        :param log_types: a filter of message types to be saved
        :type log_types: list of string
        """
        pass



    def set_skip_decoding(self, decoding):
        """
        Configure whether deferred message decoding is enabled

        :param decoding: if True, only the message header would be decoded, otherwise the entire packet would be decoded
        :type decoding: Boolean
        """
        self._skip_decoding = decoding


    # Add an analyzer that needs the message
    def register(self,analyzer):
        """
        Register an analyzer driven by this monitor

        :param analyzer: the analyzer to be added
        :type analyzer: Analyzer
        """
        if analyzer not in self.to_list:
            self.to_list.append(analyzer)

    def deregister(self,analyzer):
        """
        Deregister an analyzer driven by this monitor

        :param analyzer: the analyzer to be removed
        :type analyzer: Analyzer
        """
        if analyzer in self.to_list:
            self.to_list.remove(analyzer)

    def enable_log(self, type_name):
        """
        Enable the messages to be monitored.

        :param type_name: the message type(s) to be monitored
        :type type_name: string or list
        """
        pass

    def enable_log_all(self):
        """
        Enable all supported logs
        """
        pass

    def run(self):
        """
        Start monitoring the mobile network. This is usually the entrance of monitoring and analysis.

        This method should be overloaded in every subclass.
        """
        pass

Helpers for a Monitor: DMLogPacket and dm_collector_c

In implementing above common interfaces, MobileInsight also defines several common facilities to implement a monitor. There are two major helpers:

  • ``DMLogPacket``: it abstracts the cellular message parser. It facilitates a monitor to decode cellular messages without knowing its format. The most common features are two-fold (the complete code is in mobileinsight-core/mobile_insight/monitor/dm_collector/dm_endec/dm_log_packet.py):

  • Initialization: specify the path of ws_dissector and necessary libraries (e.g., `libwireshark);

  • Message decoding: it provides methods to decode a message as dictionary/XML/JSON.

  • ``dm_collector_c``: it abstracts the cellular message configuration and raw message extractions. Different from other modules, this module is written in C++ and compiled as a Cython binary module. It implements the following methods (the complete code is in mobileinsight-core/dm_collector_c/dm_collector_c.cpp):

  • Enable/disable cellular message types;

  • Enable runtime filtering of cellular messages;

  • Generate log configuration file (DIAG.cfg);

  • Feed raw cellular logs for decoding.

we next use a simple OfflineReplayer to exemplify how to use both helpers to implement a monitor, as shown below (complete code in mobileinsight-core/mobile_insight/monitor/offline_replayer.py):

from monitor import Monitor, Event
from dm_collector import dm_collector_c, DMLogPacket, FormatError

class OfflineReplayer(Monitor):
    """
    A log replayer for offline analysis.
    """

    def __init__(self):
        Monitor.__init__(self)

        # DMLogPacket is initialized here (with ws_dissector and libwireshark path)
        if is_android:
            libs_path = "./data"
            prefs={"ws_dissect_executable_path": os.path.join(libs_path,"android_pie_ws_dissector"),
                   "libwireshark_path": libs_path}
        else:
            prefs={}
        DMLogPacket.init(prefs)

        #...

    def enable_log(self, type_name):
        """
        Enable the messages to be monitored. Refer to cls.SUPPORTED_TYPES for supported types.

        If this method is never called, the config file existing on the SD card will be used.
        """

        # dm_collector_c: Get available cellular message types
        SUPPORTED_TYPES = set(dm_collector_c.log_packet_types)
        cls = self.__class__
        if isinstance(type_name, str):
            type_name = [type_name]
        for n in type_name:
            if n not in cls.SUPPORTED_TYPES:
                self.log_warning("Unsupported log message type: %s" % n)
            if n not in self._type_names:
                self._type_names.append(n)
                self.log_info("Enable "+n)
        # dm_collector_c: Enable filtering of messages
        dm_collector_c.set_filtered(self._type_names)

    def enable_log_all(self):
        """
        Enable all supported logs
        """
        cls = self.__class__
        self.enable_log(cls.SUPPORTED_TYPES)

    def set_input_path(self, path):
        """
        Set the replay trace path

        :param path: the replay file path. If it is a directory, the OfflineReplayer will read all logs under this directory (logs in subdirectories are ignored)
        :type path: string
        """
        dm_collector_c.reset()
        self._input_path = path

    def run(self):
        """
        Start monitoring the mobile network. This is usually the entrance of monitoring and analysis.
        """

        # Open log file
        self.log_info("Loading "+path)
        self._input_file = open(path, "rb")

        # Reset dm_collector_c
        dm_collector_c.reset()
        while True:
            s = self._input_file.read(64)
            if not s:   # EOF encountered
                break

            # dm_collector_c: feed binaries for parsing
            dm_collector_c.feed_binary(s)
            decoded = dm_collector_c.receive_log_packet(self._skip_decoding,
                                                        True,   # include_timestamp
                                                        )
            if decoded:
                try:
                    packet = DMLogPacket(decoded[0])
                    d = packet.decode()

                    if d["type_id"] in self._type_names:
                        event = Event(  timeit.default_timer(),
                                        d["type_id"],
                                        packet)
                        self.send(event)

                except FormatError, e:
                    # skip this packet
                    print "FormatError: ", e

        self._input_file.close()

Next Step

In the next article, we will introduce analyzers, another core module in MobileInsight for cellular protocol analytics.

mobileinsight-core Part-II: Analyzer

References

How to use MI-APP?

Introduction of dm_collector_c

Python logging facilities

Writing the Python Setup Script