mobileInsight-mobile
Overview¶
In this article, we will review the codes in mobileInsight-mobile
,
the mobile version of MobileInsight
.
Contents
What is mobileInsight-mobile
?¶
mobileInsight-mobile
implements the mobile version of MobileInsight.
By compiling it, we will get the MobileInsight-x.y.z.apk
, which
provides the following functionalities: - Built-in MobileInsight
monitor
and analyzer
modules; - Three built-in plugins:
NetLogger
, RrcAnalysis
and NasAnalysis
. Users can directly
run them; - Support for 3rd-party plugins (running as Android service);
- Other utilities, including - In-app *.mi2log
log viewer; -
Customized setting panels (per plugin); - Automatic software update
checker; - Crash bug report.
Different from normal Android apps, mobileInsight-mobile
is written
in Python and complied with python-for-android
. The reasons is that,
MobileInsight’s monitor/analyzer modules are developed in Python. For
cross-platform features, we use Python for all platforms (mobile and
desktop). It depends on mobileinsight-core
for the core
MobileInsight monitor/analyzer modules. Its UI is developed in kivy
language (under Python). To compile it, you need also install
Android sdk-v19
, and Android ndk-r10e
. We will elaborate how to
compile later.
mobileInsight-mobile
VS. mobileinsight-core
¶
As we know, MobileInsight provides both mobile-version apk and
desktop-version Python modules. So clearly, mobileInsight-mobile
corresponds to the apk, while mobileinsight-core
corresponds to
the Python modules. However, these two repositories are not orthogonal.
As we will see below, ``mobileInsight-mobile`` depends on
``mobileinsight-core`` for the monitor/analyzer modules. To this
sense, mobileinsight-core
includes the core features required by
all other repositories for MobileInsight.
Code Structure: A Big Picture¶
You can get the complete repository using the command below:
git clone https://github.com/mobile-insight/mobileinsight-mobile.git
If you run ls mobileInsight-mobile
, you will see files below:
./mobileInsight-mobile
├── Makefile # Makefile for compiling the app, building python-for-android distribution
├── README.md # Readme file, which elaborates how to compile this code
├── config # Configurations for the compilation (manual changes required for correct Android SDK/NDK path)
├── demo_app # The main directory for MobileInsight apk
├── deploy.py # Utilities to facilitate compilation
├── diag_revealer # In-device raw cellular message extracter
├── docs # Documentation
├── internal_app # Some experimental MobileInsight plugins
└── resources # Icons, welcome screens, etc.
Note that the major source codes for MobileInsight-x.y.z.apk are
located in ``mobileInsight-mobile/demo_app``. If you run
ls demo_app
in the shell, you will see the files below, all of which
are source codes for MobileInsight-x.y.z.apk:
./mobileInsight-mobile/demo_app
├── README.md # Readme file
├── app # Built-in plugins (NetLogger, RrcAnalysis and NasAnalysis)
├── check_update.py # Module for automatic update
├── crash_app.py # module for crash/bug report
├── data # It includes libraries and binaries (e.g., libwireshark, diag_revealer, android_ws_dissector) for message extraction and parsing.
├── log_viewer_app.py # In-app log viewer
├── main.py # Entry point of MobileInsight apk
├── main_ui.kv # Kivy UI description for main UI
├── main_utils.py # Utilities functions for MobileInsight apk
├── service # Codes for launching MobileInsight plugins based on Android service. This directory is mandatory for python-for-android.
└── settings.json # default settings for MobileInsight
Dependency of Code Files¶
The following figure shows the dependency of different code files. The
demo_app/main.py
is the entry point of the overall Apk. It
integrates other code files for different functions, including UI,
plugin service, setting panel, log viewer, crash report and automatic
update. In this figure, A<-B
means file A calls methods in file B.
demo_app/main.py
<- demo_app/settings.json # setting panel description for main UI
<- demo_app/main_ui.kv # main UI description
<- demo_app/service/main.py # plugin based on Android service
<-demo_app/service/mi2app_utils.py # utilities functions in Android service
<- demo_app/check_update.py #Automatic software update
<-demo_app/main_utils.py # utilities functions in main UI
<- demo_app/log_viewer.py #In-app log viewer
<-demo_app/main_utils.py # utilities functions in main UI
<- demo_app/crash_app.py # Bug/crash report
<-demo_app/main_utils.py # utilities functions in main UI
<- demo_app/main_utils.py # utilities functions in main UI
<- demo_app/data/* #main.py will access files in this directory
<- demo_app/app/* #main.py will load these built-in plugins
Where are MobileInsight monitor/analyzer
located?¶
From above codes, you would not find any files related to
monitors/analyzers
(except diag_revealer
), which are the core
features of MobileInsight. Where are they located? The answer is,
``monitor/analyzer`` are included during the compilation as built-in
Python modules.
To understand this, first we should know that python-for-android
supports customize the Python modules to be included by defining a
recipe. In MobileInsight, we have customized our own
``python-for-android` <https://github.com/mobile-insight/python-for-android.git>`__
for MobileInsight, which includes a recipe for MobileInsight in
pythonforandroid/recipes/mobileinsight
. The following code shows how
monitor/analyzer are compiled at runtime (in
pythonforandroid/recipes/mobileinsight/__init__.py
):
class MobileInsightRecipe(Recipe):
# change line 9 and 10 before installing!
local_debug = False
mi_local_src = '/Users/Dale/Workspace/mobileInsight/mobileinsight-core'
mi_git = 'https://github.com/mobile-insight/mobileinsight-core.git'
mi_branch = 'master'
version = '2.4'
toolchain_version = 4.8 # default GCC toolchain version we try to use
depends = ['python2'] # any other recipe names that must be built before this one
# ...
def prebuild_arch(self, arch):
# ...
info("clone MobileInsight sources from {}".format(self.mi_git))
shprint(sh.git,
'clone', '-b',
self.mi_branch,
'--depth=1',
self.mi_git,
tmp_dir,
_tail = 20,
_critical = True)
# ...
As you can see, at compilation time, our customized
python-for-android
will download mobileinsight-core
and
compiles its monitor/analyzer modules. This creates a recipe with
built-in MobileInsight modules. By compiling mobileInsight-mobile
with this recipe, all monitor/analyzer
modules will be built-in
inside the apk.
How to compile?¶
First install python-for-android
from
$ git clone https://github.com/mobile-insight/python-for-android.git
$ cd python-for-android
$ sudo python setup.py install
**NOTE: PLEASE use our customized version of python-for-android
.
As explained above, this version includes a special recipe to include
MobileInsight monitor/analyzer
modules.
Then clone this repository
$ git clone https://github.com/mobile-insight/mobileinsight-mobile.git
$ cd mobileInsight-mobile
$ make config
After the configuration, please modify
mobileInsight-mobile/config/config.yml
to specify Android SDK/NDK
and Python-for-android paths, build version, etc.
Then compile the distribution (only run it first time)
$ make dist
Last compile the apk
$ make apk_debug/release
Note that, to make release version, you are required for the release key (please contact Yuanjie or Zengwen for the key).
So far, you should have successfully built the debug version of MobileInsight2 app. Install it on Android phone is easy:
$ adb install -r MobileInsight-<ver>-debug.apk
To build an APK for release, use make apk_release
instead.
Entry Point: main.py
¶
We next review the major source codes in mobileInsight-mobile
. We
start from main.py
, the entry point of the apk. In this file, there
are two major classes, MobileInsightApp
and MobileInsightScreen
.
MobileInsightApp
integrates major functions of the entire APK
(plugin, setting, crash report, log viewer, etc.), while
MobileInsightScreen
defines the main UI and integrates specific
functions for plugin service. MobileInsightScreen
is called by
MobileInsightApp
.
The entry point code is straightforward:
if __name__ == "__main__":
try:
"""
Run MobileInsight normally
"""
MobileInsightApp().run()
except Exception, e:
"""
Bug/error occurs when running MobileInsight.
We will launch crash report feature, and ask user if they want to upload bug report to our server
"""
import traceback,crash_app
print str(traceback.format_exc())
crash_app.CrashApp().run()
How is MobileInsightApp
defined? The following code summarizes
functions in it (details of implementation can be found in
here):
class MobileInsightApp(App):
def build_settings(self, settings):
"""
Load default setting panels for MobileInsight (using kivy's settings.add_json_panel feature).
This is a built-in function in kivy.App
"""
def create_app_settings(self,config,settings):
"""
Load setting panels for each plugin.
In MobileInsight, each plugin can define its own setting panel by defining a settings.json.
More details can be found in this tutorial:
http://metro.cs.ucla.edu/mobile_insight/tutorial-plugin.html
At runtime, MobileInsight will launch settings.json in each plugin, and synthesis a uniformed setting pannel.
"""
def build_config(self, config):
"""
Load default values of the default settings for MobileInsight.
This is a built-in function in kivy.App
"""
def create_app_default_config(self, config):
"""
Load default values of the default settings for each plugin.
"""
def build(self):
"""
This is a built-in function in kivy.App.
In this function, we specify the major kivy.Screen UI.
Currently we load two UIs (subclass of kivy.Screen):
- MobileInsightScreen: this screen encapsulates MobileInsight's major functions
- LogViewerScreen: this screen specifies in-app log viewer
Then we set MobileInsightScreen as the main screen.
"""
def on_pause(self):
"""
Built-in function in kivy.App.
It will be called when users switch to other apps.
In this function, we did cleanup tasks and tricks to prevent some python-for-android bugs.
Reference: https://kivy.org/docs/api-kivy.app.html#kivy.app.App.on_pause
"""
def check_update(self):
"""
Check if new MobileInsight update is available.
It will launch check_update function
"""
def on_start(self):
"""
Built-in function in kivy.App.
It will be called when MobileInsight is launched.
In this function, we will check for software update (if user specifies so in the setting panel)
Reference: https://kivy.org/docs/api-kivy.app.html#kivy.app.App.on_start
"""
From above code, you will see that although MobileInsightApp
integrates most functions, the core MobileInsight features are
encapsulated in ``MobileInsightScreen``, which is defined as follows:
class MobileInsightScreen(Screen):
def __init__(self,name):
"""
Initialization function. We will do the following task (in order):
1. Check if the device is rooted
2. Initialize necessary libs required by MobileInsight (e.g., libwireshark)
3. Check if Android's security policy allows MobileInsight to access diagnostic mode.
This is mainly caused by SELinux
4. Create necessary folders on SDcard (e.g., /sdcard/mobile_insight/, /sdcard/mobile_insight/log/)
5. Load built-in and 3rd-party plugins (located in /sdcard/mobile_insight/apps/)
6. Check if the diagnostic mode is enabled
7. Load configurations from the setting panel (configs stored in /sdcard/.mobileinsight.ini)
"""
def __check_security_policy(self):
"""
Update SELinux policy.
For Nexus 6/6P, the SELinux policy may forbids the log collection.
"""
def __check_diag_mode(self):
"""
Check if diagnostic mode is enabled.
"""
def __init_libs(self):
"""
Initialize libs required by MobileInsight.
These libs will be copied to /system/lib and /system/bin, respectively
"""
def start_service(self, app_name):
"""
Launch the plugin selected by the user.
app_name specifies the name of the plugin selected by user.
In this code, it will use the code in demo_app/service to launch the plugin as Android service.
"""
def stop_service(self):
"""
Stop the Android service if user clicks "Stop" button
"""
def about(self):
"""
Popup the "About" panel when user clicks "About" button
"""
"""
The following functions are used for on-screen logging purpose with different levels.
"""
def log_info(self, msg):
def log_warning(self, msg):
def log_error(self, msg):
def append_log(self, s):
def show_log(self):
These are all about main.py
, but a question still remains: how does
the application know which method should it call, when user clicks a
button or selects a plugin? The answer is that, the UI
widget-callback binding is defined in ``demo_app/main_ui.kv``, which
is exemplified below (full implementation can be found
here)
# Main screen
<MobileInsightScreen>:
GridLayout:
Button:
id: run_plugin
text: 'Run selected plugin'
on_release: root.start_service(root.ids.checkbox_app.selected)
Button:
id: stop_plugin
text: 'Stop %s' % root.default_app_name
on_release: root.stop_service()
Button:
text: 'LogViewer'
on_release: app.manager.current = 'LogViewerScreen'; app.log_viewer_screen.onOpen()
...
main_ui.kv
specifies the layout of the MobileInsight UI. For each
button declared here, there is a on_release
attribute, which
specifies the callbacks to be used once user clicks this button. This is
where we can bind our methods to the UI.
Plugin Service Implementation¶
MobileInsight lets users to wrap their monitor/analyzer code as a plugin, and run it on the phone. For how to use the plugin, we recommend you to read the following tutorial:
[Running MobileInsight code on the phone] (http://metro.cs.ucla.edu/mobile_insight/tutorial-plugin.html)
Here we will focus on how mobileInsight-mobile
runs the user-defined
plugins. When the user clicks the “Run” button on UI, the
MobileInsightScreen.start_service
method will be called, which will
run the selected plugin as an Android service. Inside
MobileInsightScreen.start_service
, we launch the Android service as
follows (in demo_app/main.py
):
def start_service(self, app_name):
if platform == "android" and app_name in self.app_list:
if self.service:
# stop the running service
self.stop_service()
from android import AndroidService
self.error_log = "Running " + app_name + "..."
self.service = AndroidService("MobileInsight is running...", app_name)
self.service.start(app_name+":"+self.app_list[app_name][0]) # app name
self.default_app_name = app_name
else:
self.error_log = "Error: " + app_name + "cannot be launched!"
You can see the secret lies in the self.service.start()
method,
which accepts the plugin’s code and runs it as an Android service. How
does it work? This is actually features provided by
python-for-android
: it will launch demo_app/service/main.py
, in
which we use Python’s built-in function execfile()
to run the
plugin’s code. The basic logic of demo_app/service/main.py
is as
follows:
# In mobileInsight-mobile/demo_app/service/main.py
if __name__ == "__main__":
arg = os.getenv("PYTHON_SERVICE_ARGUMENT") # get the argument passed
tmp = arg.split(":")
if len(tmp)<2:
raise AssertionError("Error: incorrect service path:"+arg)
app_name = tmp[0]
app_path = tmp[1]
namespace = {"service_context": mi2app_utils.get_service_context()}
#Load configurations as global variables
config = ConfigParser()
config.read('/sdcard/.mobileinsight.ini')
# Set global variable plugin_config for the plugin
# This way, users can get configurations by calling plugin_config[PARA_NAME]
namespace["plugin_config"] = plugin_config
execfile(app_file, namespace)
Other Functionalities¶
Besides the main functions, mobileInsight-mobile
also has some other
modules for different purposes. We briefly review each below.
Crash report helper crash_app.py
¶
When MobileInsight crashes due to some bugs, the CrashApp
will be
triggered for the bug report. The bug report to be uploaded is the
logcat
log, with the traceback logs in Python.
The CrashApp
is defined in crash_app.py
, whose code structure is
similar to MobileInsightApp
. When it is triggered, a popup is
created to ask user. If user accepts to upload, CrashApp
will dump
the logcat
into
/sdcard/mobile_insight/crash_report/crash_report_PHONE-INFO_TIMESTAMP.txt
,
and upload to metro server by calling the PHP script
http://metro.cs.ucla.edu/mobile_insight/upload_crash_log.php
.
Automatic update helper check_update.py
¶
MobileInsight supports automatic software update, which is implemented
in check_update.check_update()
. The logical procedure of this method
is as follows:
Check the latest version by retrieving
http://metro.cs.ucla.edu/mobile_insight/update_meta.json
, whose structure is as follows:
{
Version: "2.1.1",
URL: "http://metro.cs.ucla.edu/mobile_insight/update.apk",
Description: "- Various bug fix;"
}
It includes the latest version, URL of the latest version, and explanations of the update.
With above info,
check_update.check_update()
further compares current version on the phone and the latest one. If the current version is behind the latest one, a popup will be created to notify the user, and ask him/her whether to update;If user chooses to update,
check_update.check_update()
will download the latest version from the specified URL, and launches Android’s installation service (by constructing theIntentClass.ACTION_VIEW
intent).
In-app log viewer log_viewer_app.py
¶
log_viewer_app.py
implements the in-device *.mi2log
log viewer.
When user clicks the “In-App Logviewer” button, a
log_viewer_app.LogViewerScreen
will be launched as the main UI, and
the following tasks will be done:
Browse all files in
/sdcard/mobile_insight/log/
, filters out all*.mi2log
files under this directories, show them on screen and ask user to select one;Once user selects one
*.mi2log
, it will useOfflineReplayer
to load the log, and useLogAnalyzer
to parse the logs. Once it is done, per-message view will be generated on screen. Note that in the current implementation, the message will be shown on screen until all logs are parsed. This causes long waiting time for user, and should be avoided in future version.When user selects one message, a popup is created to show the decoded message contents (in xml format).
More details can be found in this article:
[Code Review: LogViewer
] (logviewer)
Various utilities: main_utils.py
and mi2app_utils.py
¶
In implementing mobileInsight-mobile
, there are several functions
that would be repetitively called, such as accessing built-in folders
like /sdcard/mobile_insight/log
. To facilitate development, we have
offered several utility functions in demo_app/main_utils.py
and
demo_app/service/mi2app_utils.py
. They provide same utilities, but
they should be used in different scenarios. More specifically, - If
you want to use the utility functions outside MobileInsight plugin,
you should use main_utils.py
; - If you want to use them inside
MobileInsight plugin, you should use mi2app_utils.py
.
To use them, you need to add the following statement in the code
import main_utils #Outside MobileInsight plugin
# OR
import mi2app_utils #Inside MobileInsight plugin codes
Some utilities are declared here:
def get_cur_version():
"""
Get current apk version string
"""
def is_rooted():
"""
Check if the phone has been rooted
"""
def run_shell_cmd(cmd, wait = False):
"""
Run android shell command with root privilege
"""
def get_sdcard_path():
"""
Return the sdcard path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_path():
"""
Return the root path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_log_path():
"""
Return the log path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_analysis_path():
"""
Return the analysis result path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_log_decoded_path():
"""
Return the decoded log path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_log_uploaded_path():
"""
Return the uploaded log path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_cfg_path():
"""
Return the configuration path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_db_path():
"""
Return the database path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_plugin_path():
"""
Return the plugin path of MobileInsight, or None if not accessible
"""
def get_mobile_insight_crash_log_path():
"""
Return the plugin path of MobileInsight, or None if not accessible
"""
def detach_thread():
"""
If you are using multi-threading in Python, please call it after the thread
"""
def get_cache_dir():
"""
Get the APK's cache directory
"""
def get_files_dir():
"""
Get the APK's file directory
"""
diag_revealer
¶
diag_revealer
extracts raw low-level cellular messages from the
mobile phone’s cellular chipset. We have a separate documents about how
it works. For details, please read