mobileInsight-core
Part-I: Monitor¶
In this series, we will review the codes in mobileinsight-core
,
the core modules and desktop version of MobileInsight
. This code
review has two parts. In this article (Part-I), we will introduce the
big picture of mobileinsight-core
, and its cellular message
monitor
. In Part-II, we will
introduce analyzer
, the cellular protocol analytics module.
Contents
What is mobileinsight-core
?¶
mobileinsight-core
implements the core features and the
desktop-version Python module. It provides the following functions -
Core modules in MobileInsight: monitor
and analyzer
- Cellular
message configuration manager and parser: dm_collector_c
- Wireshark
helper (for message parsing) ws_dissector
- Desktop-version
packaging and installation MANIFEST.in
and setup.py
- Other
utilities, such as example codes, GUI, test scripts, etc.
mobileinsight-core
is written in Python-2.7
(monitors,
analyzers, examples, test scripts, GUI) and C++
(ws_dissector and
dm_collector_c). This makes the cross-platform support (Android/OS
X/Win/Linux) possible.
Code Structure: A Big Picture¶
You can download mobileinsight-core
from our code repository:
git clone https://github.com/mobile-insight/mobileinsight-core.git
cd mobileinsight-core
If you run ls mobileinsight-core
, you will see files below:
./mobileinsight-core
├── mobile_insight # Core modules, including monitors and analyzers
├── dm_collector_c # Cellular message configuration manager and parser
├── ws_dissector # Wireshark helper (for message parsing)
├── MANIFEST.in # manifest file (used for packaging desktop-version modules)
├── setup.py # External package function for MobileInsight installation
├── GUI # Desktop-version GUI (LogViewer)
├── docs # Documentations
├── examples # Example codes for MobileInsight tutorial
├── unit-test # Unit test scripts
├── LICENSE # software license description
└── README.md # readme file
The core modules in MobileInsight, Monitors
and Analyzers
, are
located in mobileinsight-core/mobile_insight
:
./mobileinsight-core/mobile_insight
├── __init__.py # Python packaging script, which imports monitors and analyzers
├── analyzer # Analyzers folder
├── element.py # Base class of monitors and analyzers
├── monitor # Monitors folder
└── utils.py # Some utility functions
How to compile and install?¶
As a developer, you should know how to compile AND install
MobileInsight. This differs from normal MobileInsight users that only
need to install, because dm_collector_c
and ws_dissector
have
been pre-compiled before distribution. The following are concrete steps:
Make sure your desktop has installed
Python2.7
,Cython
andWireshark
source code.Compile dm_collector_c: run the following command
python setup.py build_ext --inplace
This will compile
dm_collector_c
and copy it tomobile_insight/monitor/dm_collector
.Compile ws_dissector: run the following command
cd ws_dissector make
You should change ``Makefile`` before this step. The
Makefile
includes path ofWireshark
source code, which should be modified based on your desktop’s local path.Installation: run the following command
cd .. # mobileinsight-core root directory sudo python setup.py install
Base class of monitors and analyzers: Element
¶
We next elaborate how Monitors
and Analyzers
are implemented in
MobileInsight. For more details of their high-level features and
functionalities, we recommend you to read our Mobicom’16 paper.
Before implementing Monitors
and Analyzers
, MobileInsight first
declares a base class called Element
. The rationale is that, both
Monitors
and Analyzers
require some common features, including -
Abstraction of cellular messages (as an event); - Event-driven callback
mechanisms (i.e., MI-DEV API
); - Interface with other mobile
applications (i.e, MI-APP API
); - Logging functions
MobileInsight abstracts these common features into Element
. The
following shows a simplified code snippet of it (the complete code is in
mobile_insight/element.py
):
class Event(object):
'''The event is used to trigger the analyzer and perform some actions.
The event can be raised by a trace collector (a message) or an analyzer.
An event is a triple of (timestamp, type, data).
'''
def __init__(self,timestamp,type_id,data):
self.timestamp = timestamp
self.type_id = type_id
self.data = data
class Element(object):
'''
The parent class to derive trace collectors and analyzers.
'''
def __init__(self):
"""
Initialize logging function, and empty list of callbacks
"""
"""
MI-DEV API: implements send/recv-based callback mechanism (for events)
"""
def send(self,event):
"""
Raise an event to all Analyzers in from_analyzer_list
:param event: the event to be sent
"""
def recv(self,module,event):
"""
Upon receiving an event from module, trigger associated callbacks
This method should be overwritten by the analyzer and monitor
:param module: the module who raises the event
:param event: the event to be received
"""
"""
MI-APP API: interface between mobile apps and MobileInsight
reference: http://mobileinsight.net/mi-app.html
"""
def broadcast_info(self, method, msg_dict):
"""
(Mobile-version only) Broadcast monitor/analyzer information to other Android apps.
This method is the interface between MobileInsight and Android apps requiring cellular info.
It leverages Android's intent-based broadcast mechanism.
The intent is per-analyzer/monitor based, and the action is named as follows:
`MobileInsight.ANALYZER/MONITOR-NAME.method`
where ANALYZER/MONITOR-NAME is the class name of the monitor/analyzer,
and method is analyzer/monitor-specific method to notify the Android apps
:param method: analyzer/monitor-specific methods for broadcast action
:type method: string
:param msg_dict: A dictionary that lists the information to be broadcasted.
:type msg_dict: string->string dictionary
"""
"""
The following are logging functions
Reference: https://docs.python.org/2/library/logging.html
"""
def set_log(self,logpath,loglevel=logging.INFO):
"""
Set the logging in analyzers.
All the analyzers share the same logger.
"""
def log_info(self, msg):
# ...
def log_debug(self, msg):
# ...
def log_warning(self, msg):
# ...
def log_error(self, msg):
# ...
def log_critical(self, msg):
# ...
Monitors¶
MobileInsight monitors extract raw cellular messages, parse them as
human-readable format, and triggers the further analysis for Analyzers.
We have realized various monitors, including online and offline
fashions, and desktop and in-device versions. All monitors are included
in mobile_insight/monitor
, as shown below:
./mobileinsight-core/mobile_insight/monitor/
├── __init__.py # Python packaging function
├── monitor.py # Base class for all monitors
├── android_dev_diag_monitor.py # Android's in-device, runtime monitor using diag_revealer
├── android_qmdl_monitor.py # (Depreciated) Android's in-device, runtime monitor using diag_mdlog
├── dm_collector # Desktop-version runtime monitor (through serial port in USB)
├── offline_replayer.py # Offline mi2log log replayer
└── online_monitor.py # A wrapper of cross-platform (Android/Desktop) runtime monitor
Despite their diverse functions, their implementations share the same structure. We next elaborate it.
Base class: Monitor
¶
All MobileInsight monitors should inherit from the base class
Monitor
, which defines the basic interface for a monitor: - Enable
user-specified cellular message types; - Bind analyzers to this monitor;
- Start the monitoring process. Each monitor should overload these
interfaces based on its specific environment. For example, to start the
monitoring process (run()
), the offline replayer OfflineReplayer
loads the mi2log from the storage, while other online monitor
AndroidDiagMonitor
extract cellular messages from in-device
diagnostic port.
The following code snippet summaries the abstract interfaces (complete code in mobile_insight/monitor/monitor.py):
from ..element import Element, Event
class Monitor(Element):
"""
An abstraction for mobile network monitors
"""
def __init__(self):
# No source for Monitor
Element.__init__(self)
self._skip_decoding = False
self._save_log_path = None
self._save_file = None
def available_log_types(self):
"""
Return available log types
"""
return None
def save_log_as(self,path):
"""
Save the log as a mi2log file (for offline analysis)
:param path: the file name to be saved
:type path: string
:param log_types: a filter of message types to be saved
:type log_types: list of string
"""
pass
def set_skip_decoding(self, decoding):
"""
Configure whether deferred message decoding is enabled
:param decoding: if True, only the message header would be decoded, otherwise the entire packet would be decoded
:type decoding: Boolean
"""
self._skip_decoding = decoding
# Add an analyzer that needs the message
def register(self,analyzer):
"""
Register an analyzer driven by this monitor
:param analyzer: the analyzer to be added
:type analyzer: Analyzer
"""
if analyzer not in self.to_list:
self.to_list.append(analyzer)
def deregister(self,analyzer):
"""
Deregister an analyzer driven by this monitor
:param analyzer: the analyzer to be removed
:type analyzer: Analyzer
"""
if analyzer in self.to_list:
self.to_list.remove(analyzer)
def enable_log(self, type_name):
"""
Enable the messages to be monitored.
:param type_name: the message type(s) to be monitored
:type type_name: string or list
"""
pass
def enable_log_all(self):
"""
Enable all supported logs
"""
pass
def run(self):
"""
Start monitoring the mobile network. This is usually the entrance of monitoring and analysis.
This method should be overloaded in every subclass.
"""
pass
Helpers for a Monitor: DMLogPacket
and dm_collector_c
¶
In implementing above common interfaces, MobileInsight also defines several common facilities to implement a monitor. There are two major helpers:
``DMLogPacket``: it abstracts the cellular message parser. It facilitates a monitor to decode cellular messages without knowing its format. The most common features are two-fold (the complete code is in
mobileinsight-core/mobile_insight/monitor/dm_collector/dm_endec/dm_log_packet.py
):Initialization: specify the path of
ws_dissector
and necessary libraries (e.g., `libwireshark);Message decoding: it provides methods to decode a message as dictionary/XML/JSON.
``dm_collector_c``: it abstracts the cellular message configuration and raw message extractions. Different from other modules, this module is written in
C++
and compiled as aCython
binary module. It implements the following methods (the complete code is inmobileinsight-core/dm_collector_c/dm_collector_c.cpp
):Enable/disable cellular message types;
Enable runtime filtering of cellular messages;
Generate log configuration file (
DIAG.cfg
);Feed raw cellular logs for decoding.
we next use a simple OfflineReplayer
to exemplify how to use both
helpers to implement a monitor, as shown below (complete code in
mobileinsight-core/mobile_insight/monitor/offline_replayer.py
):
from monitor import Monitor, Event
from dm_collector import dm_collector_c, DMLogPacket, FormatError
class OfflineReplayer(Monitor):
"""
A log replayer for offline analysis.
"""
def __init__(self):
Monitor.__init__(self)
# DMLogPacket is initialized here (with ws_dissector and libwireshark path)
if is_android:
libs_path = "./data"
prefs={"ws_dissect_executable_path": os.path.join(libs_path,"android_pie_ws_dissector"),
"libwireshark_path": libs_path}
else:
prefs={}
DMLogPacket.init(prefs)
#...
def enable_log(self, type_name):
"""
Enable the messages to be monitored. Refer to cls.SUPPORTED_TYPES for supported types.
If this method is never called, the config file existing on the SD card will be used.
"""
# dm_collector_c: Get available cellular message types
SUPPORTED_TYPES = set(dm_collector_c.log_packet_types)
cls = self.__class__
if isinstance(type_name, str):
type_name = [type_name]
for n in type_name:
if n not in cls.SUPPORTED_TYPES:
self.log_warning("Unsupported log message type: %s" % n)
if n not in self._type_names:
self._type_names.append(n)
self.log_info("Enable "+n)
# dm_collector_c: Enable filtering of messages
dm_collector_c.set_filtered(self._type_names)
def enable_log_all(self):
"""
Enable all supported logs
"""
cls = self.__class__
self.enable_log(cls.SUPPORTED_TYPES)
def set_input_path(self, path):
"""
Set the replay trace path
:param path: the replay file path. If it is a directory, the OfflineReplayer will read all logs under this directory (logs in subdirectories are ignored)
:type path: string
"""
dm_collector_c.reset()
self._input_path = path
def run(self):
"""
Start monitoring the mobile network. This is usually the entrance of monitoring and analysis.
"""
# Open log file
self.log_info("Loading "+path)
self._input_file = open(path, "rb")
# Reset dm_collector_c
dm_collector_c.reset()
while True:
s = self._input_file.read(64)
if not s: # EOF encountered
break
# dm_collector_c: feed binaries for parsing
dm_collector_c.feed_binary(s)
decoded = dm_collector_c.receive_log_packet(self._skip_decoding,
True, # include_timestamp
)
if decoded:
try:
packet = DMLogPacket(decoded[0])
d = packet.decode()
if d["type_id"] in self._type_names:
event = Event( timeit.default_timer(),
d["type_id"],
packet)
self.send(event)
except FormatError, e:
# skip this packet
print "FormatError: ", e
self._input_file.close()
Next Step¶
In the next article, we will introduce analyzers
, another core
module in MobileInsight for cellular protocol analytics.