mobileInsight-mobile Overview

In this article, we will review the codes in mobileInsight-mobile, the mobile version of MobileInsight.

What is mobileInsight-mobile?

mobileInsight-mobile implements the mobile version of MobileInsight. By compiling it, we will get the MobileInsight-x.y.z.apk, which provides the following functionalities: - Built-in MobileInsight monitor and analyzer modules; - Three built-in plugins: NetLogger, RrcAnalysis and NasAnalysis. Users can directly run them; - Support for 3rd-party plugins (running as Android service); - Other utilities, including - In-app *.mi2log log viewer; - Customized setting panels (per plugin); - Automatic software update checker; - Crash bug report.

Different from normal Android apps, mobileInsight-mobile is written in Python and complied with python-for-android. The reasons is that, MobileInsight’s monitor/analyzer modules are developed in Python. For cross-platform features, we use Python for all platforms (mobile and desktop). It depends on mobileinsight-core for the core MobileInsight monitor/analyzer modules. Its UI is developed in kivy language (under Python). To compile it, you need also install Android sdk-v19, and Android ndk-r10e. We will elaborate how to compile later.

mobileInsight-mobile VS. mobileinsight-core

As we know, MobileInsight provides both mobile-version apk and desktop-version Python modules. So clearly, mobileInsight-mobile corresponds to the apk, while mobileinsight-core corresponds to the Python modules. However, these two repositories are not orthogonal. As we will see below, “mobileInsight-mobile“ depends on “mobileinsight-core“ for the monitor/analyzer modules. To this sense, mobileinsight-core includes the core features required by all other repositories for MobileInsight.

Code Structure: A Big Picture

You can get the complete repository using the command below:

git clone https://github.com/mobile-insight/mobileinsight-mobile.git

If you run ls mobileInsight-mobile, you will see files below:

./mobileInsight-mobile
├── Makefile # Makefile for compiling the app, building python-for-android distribution
├── README.md # Readme file, which elaborates how to compile this code
├── config # Configurations for the compilation (manual changes required for correct Android SDK/NDK path)
├── demo_app # The main directory for MobileInsight apk
├── deploy.py # Utilities to facilitate compilation
├── diag_revealer # In-device raw cellular message extracter
├── docs # Documentation
├── internal_app # Some experimental MobileInsight plugins
└── resources # Icons, welcome screens, etc.

Note that the major source codes for MobileInsight-x.y.z.apk are located in “mobileInsight-mobile/demo_app“. If you run ls demo_app in the shell, you will see the files below, all of which are source codes for MobileInsight-x.y.z.apk:

./mobileInsight-mobile/demo_app
├── README.md # Readme file
├── app # Built-in plugins (NetLogger, RrcAnalysis and NasAnalysis)
├── check_update.py # Module for automatic update
├── crash_app.py # module for crash/bug report
├── data # It includes libraries and binaries (e.g., libwireshark, diag_revealer, android_ws_dissector) for message extraction and parsing.
├── log_viewer_app.py # In-app log viewer
├── main.py # Entry point of MobileInsight apk
├── main_ui.kv  # Kivy UI description for main UI
├── main_utils.py # Utilities functions for MobileInsight apk
├── service # Codes for launching MobileInsight plugins based on Android service. This directory is mandatory for python-for-android.
└── settings.json # default settings for MobileInsight

Dependency of Code Files

The following figure shows the dependency of different code files. The demo_app/main.py is the entry point of the overall Apk. It integrates other code files for different functions, including UI, plugin service, setting panel, log viewer, crash report and automatic update. In this figure, A<-B means file A calls methods in file B.

demo_app/main.py
    <- demo_app/settings.json # setting panel description for main UI
    <- demo_app/main_ui.kv # main UI description
    <- demo_app/service/main.py # plugin based on Android service
           <-demo_app/service/mi2app_utils.py # utilities functions in Android service
    <- demo_app/check_update.py #Automatic software update
           <-demo_app/main_utils.py  # utilities functions in main UI
    <- demo_app/log_viewer.py #In-app log viewer
           <-demo_app/main_utils.py  # utilities functions in main UI
    <- demo_app/crash_app.py # Bug/crash report
           <-demo_app/main_utils.py  # utilities functions in main UI
    <- demo_app/main_utils.py  # utilities functions in main UI
    <- demo_app/data/* #main.py will access files in this directory
    <- demo_app/app/* #main.py will load these built-in plugins

Where are MobileInsight monitor/analyzer located?

From above codes, you would not find any files related to monitors/analyzers (except diag_revealer), which are the core features of MobileInsight. Where are they located? The answer is, “monitor/analyzer“ are included during the compilation as built-in Python modules.

To understand this, first we should know that python-for-android supports customize the Python modules to be included by defining a recipe. In MobileInsight, we have customized our own “python-for-android` <https://github.com/mobile-insight/python-for-android.git>`__ for MobileInsight, which includes a recipe for MobileInsight in pythonforandroid/recipes/mobileinsight. The following code shows how monitor/analyzer are compiled at runtime (in pythonforandroid/recipes/mobileinsight/__init__.py):

class MobileInsightRecipe(Recipe):
    # change line 9 and 10 before installing!
    local_debug       = False
    mi_local_src      = '/Users/Dale/Workspace/mobileInsight/mobileinsight-core'
    mi_git            = 'https://github.com/mobile-insight/mobileinsight-core.git'
    mi_branch         = 'master'
    version           = '2.4'
    toolchain_version = 4.8          # default GCC toolchain version we try to use
    depends           = ['python2']  # any other recipe names that must be built before this one

    # ...
    def prebuild_arch(self, arch):
        # ...
        info("clone MobileInsight sources from {}".format(self.mi_git))
        shprint(sh.git,
                    'clone', '-b',
                    self.mi_branch,
                    '--depth=1',
                    self.mi_git,
                    tmp_dir,
                    _tail     = 20,
                    _critical = True)
        # ...

As you can see, at compilation time, our customized python-for-androidwill download mobileinsight-core and compiles its monitor/analyzer modules. This creates a recipe with built-in MobileInsight modules. By compiling mobileInsight-mobile with this recipe, all monitor/analyzer modules will be built-in inside the apk.

How to compile?

First install python-for-android from

$ git clone https://github.com/mobile-insight/python-for-android.git
$ cd python-for-android
$ sudo python setup.py install

**NOTE: PLEASE use our customized version of python-for-android. As explained above, this version includes a special recipe to include MobileInsight monitor/analyzer modules.

Then clone this repository

$ git clone https://github.com/mobile-insight/mobileinsight-mobile.git
$ cd mobileInsight-mobile
$ make config

After the configuration, please modify mobileInsight-mobile/config/config.yml to specify Android SDK/NDK and Python-for-android paths, build version, etc.

Then compile the distribution (only run it first time)

$ make dist

Last compile the apk

$ make apk_debug/release

Note that, to make release version, you are required for the release key (please contact Yuanjie or Zengwen for the key).

So far, you should have successfully built the debug version of MobileInsight2 app. Install it on Android phone is easy:

$ adb install -r MobileInsight-<ver>-debug.apk

To build an APK for release, use make apk_release instead.

Entry Point: main.py

We next review the major source codes in mobileInsight-mobile. We start from main.py, the entry point of the apk. In this file, there are two major classes, MobileInsightApp and MobileInsightScreen. MobileInsightApp integrates major functions of the entire APK (plugin, setting, crash report, log viewer, etc.), while MobileInsightScreen defines the main UI and integrates specific functions for plugin service. MobileInsightScreen is called by MobileInsightApp.

The entry point code is straightforward:

if __name__ == "__main__":
    try:
        """
        Run MobileInsight normally
        """
        MobileInsightApp().run()
    except Exception, e:
        """
        Bug/error occurs when running MobileInsight.
        We will launch crash report feature, and ask user if they want to upload bug report to our server
        """
        import traceback,crash_app
        print str(traceback.format_exc())
        crash_app.CrashApp().run()

How is MobileInsightApp defined? The following code summarizes functions in it (details of implementation can be found in here):

class MobileInsightApp(App):

    def build_settings(self, settings):
        """
        Load default setting panels for MobileInsight (using kivy's settings.add_json_panel feature).
        This is a built-in function in kivy.App
        """

    def create_app_settings(self,config,settings):
        """
        Load setting panels for each plugin.
        In MobileInsight, each plugin can define its own setting panel by defining a settings.json.
        More details can be found in this tutorial:
        http://metro.cs.ucla.edu/mobile_insight/tutorial-plugin.html

        At runtime, MobileInsight will launch settings.json in each plugin, and synthesis a uniformed setting pannel.
        """

    def build_config(self, config):
        """
        Load default values of the default settings for MobileInsight.
        This is a built-in function in kivy.App
        """

    def create_app_default_config(self, config):
        """
        Load default values of the default settings for each plugin.
        """

    def build(self):

        """
        This is a built-in function in kivy.App.
        In this function, we specify the major kivy.Screen UI.
        Currently we load two UIs (subclass of kivy.Screen):
            - MobileInsightScreen: this screen encapsulates MobileInsight's major functions
            - LogViewerScreen: this screen specifies in-app log viewer
        Then we set MobileInsightScreen as the main screen.
        """

    def on_pause(self):
        """
        Built-in function in kivy.App.
        It will be called when users switch to other apps.
        In this function, we did cleanup tasks and tricks to prevent some python-for-android bugs.
        Reference: https://kivy.org/docs/api-kivy.app.html#kivy.app.App.on_pause
        """

    def check_update(self):
        """
        Check if new MobileInsight update is available.
        It will launch check_update function
        """

    def on_start(self):
        """
        Built-in function in kivy.App.
        It will be called when MobileInsight is launched.
        In this function, we will check for software update (if user specifies so in the setting panel)
        Reference: https://kivy.org/docs/api-kivy.app.html#kivy.app.App.on_start
        """

From above code, you will see that although MobileInsightApp integrates most functions, the core MobileInsight features are encapsulated in “MobileInsightScreen“, which is defined as follows:

class MobileInsightScreen(Screen):

    def __init__(self,name):
        """
        Initialization function. We will do the following task (in order):
            1. Check if the device is rooted
            2. Initialize necessary libs required by MobileInsight (e.g., libwireshark)
            3. Check if Android's security policy allows MobileInsight to access diagnostic mode.
            This is mainly caused by SELinux
            4. Create necessary folders on SDcard (e.g., /sdcard/mobile_insight/, /sdcard/mobile_insight/log/)
            5. Load built-in and 3rd-party plugins (located in /sdcard/mobile_insight/apps/)
            6. Check if the diagnostic mode is enabled
            7. Load configurations from the setting panel (configs stored in /sdcard/.mobileinsight.ini)
        """

    def __check_security_policy(self):
        """
        Update SELinux policy.
        For Nexus 6/6P, the SELinux policy may forbids the log collection.
        """

    def __check_diag_mode(self):
        """
        Check if diagnostic mode is enabled.
        """

    def __init_libs(self):
        """
        Initialize libs required by MobileInsight.
        These libs will be copied to /system/lib and /system/bin, respectively
        """

    def start_service(self, app_name):
        """
        Launch the plugin selected by the user.
        app_name specifies the name of the plugin selected by user.
        In this code, it will use the code in demo_app/service to launch the plugin as Android service.
        """

    def stop_service(self):
        """
        Stop the Android service if user clicks "Stop" button
        """

    def about(self):
        """
        Popup the "About" panel when user clicks "About" button
        """

    """
    The following functions are used for on-screen logging purpose with different levels.
    """
    def log_info(self, msg):
    def log_warning(self, msg):
    def log_error(self, msg):
    def append_log(self, s):
    def show_log(self):

These are all about main.py, but a question still remains: how does the application know which method should it call, when user clicks a button or selects a plugin? The answer is that, the UI widget-callback binding is defined in “demo_app/main_ui.kv“, which is exemplified below (full implementation can be found here)

# Main screen
<MobileInsightScreen>:
    GridLayout:

        Button:
            id: run_plugin
            text: 'Run selected plugin'
            on_release: root.start_service(root.ids.checkbox_app.selected)

        Button:
            id: stop_plugin
            text: 'Stop %s' % root.default_app_name
            on_release: root.stop_service()

        Button:
            text: 'LogViewer'
            on_release: app.manager.current = 'LogViewerScreen'; app.log_viewer_screen.onOpen()
        ...

main_ui.kv specifies the layout of the MobileInsight UI. For each button declared here, there is a on_release attribute, which specifies the callbacks to be used once user clicks this button. This is where we can bind our methods to the UI.

Plugin Service Implementation

MobileInsight lets users to wrap their monitor/analyzer code as a plugin, and run it on the phone. For how to use the plugin, we recommend you to read the following tutorial:

[Running MobileInsight code on the phone] (http://metro.cs.ucla.edu/mobile_insight/tutorial-plugin.html)

Here we will focus on how mobileInsight-mobile runs the user-defined plugins. When the user clicks the “Run” button on UI, the MobileInsightScreen.start_service method will be called, which will run the selected plugin as an Android service. Inside MobileInsightScreen.start_service, we launch the Android service as follows (in demo_app/main.py):

def start_service(self, app_name):
    if platform == "android" and app_name in self.app_list:
        if self.service:
            # stop the running service
            self.stop_service()

        from android import AndroidService
        self.error_log = "Running " + app_name + "..."
        self.service = AndroidService("MobileInsight is running...", app_name)
        self.service.start(app_name+":"+self.app_list[app_name][0])   # app name
        self.default_app_name = app_name

    else:
        self.error_log = "Error: " + app_name + "cannot be launched!"

You can see the secret lies in the self.service.start() method, which accepts the plugin’s code and runs it as an Android service. How does it work? This is actually features provided by python-for-android: it will launch demo_app/service/main.py, in which we use Python’s built-in function execfile() to run the plugin’s code. The basic logic of demo_app/service/main.py is as follows:

# In mobileInsight-mobile/demo_app/service/main.py

if __name__ == "__main__":
        arg = os.getenv("PYTHON_SERVICE_ARGUMENT")  # get the argument passed

        tmp = arg.split(":")
        if len(tmp)<2:
            raise AssertionError("Error: incorrect service path:"+arg)
        app_name = tmp[0]
        app_path = tmp[1]

        namespace = {"service_context": mi2app_utils.get_service_context()}

        #Load configurations as global variables
        config = ConfigParser()
        config.read('/sdcard/.mobileinsight.ini')

       # Set global variable plugin_config for the plugin
       # This way, users can get configurations by calling plugin_config[PARA_NAME]
        namespace["plugin_config"] = plugin_config

        execfile(app_file, namespace)

Other Functionalities

Besides the main functions, mobileInsight-mobile also has some other modules for different purposes. We briefly review each below.

Crash report helper crash_app.py

When MobileInsight crashes due to some bugs, the CrashApp will be triggered for the bug report. The bug report to be uploaded is the logcat log, with the traceback logs in Python.

The CrashApp is defined in crash_app.py, whose code structure is similar to MobileInsightApp. When it is triggered, a popup is created to ask user. If user accepts to upload, CrashApp will dump the logcat into /sdcard/mobile_insight/crash_report/crash_report_PHONE-INFO_TIMESTAMP.txt, and upload to metro server by calling the PHP script http://metro.cs.ucla.edu/mobile_insight/upload_crash_log.php.

Automatic update helper check_update.py

MobileInsight supports automatic software update, which is implemented in check_update.check_update(). The logical procedure of this method is as follows:

  1. Check the latest version by retrieving http://metro.cs.ucla.edu/mobile_insight/update_meta.json, whose structure is as follows:
{
Version: "2.1.1",
URL: "http://metro.cs.ucla.edu/mobile_insight/update.apk",
Description: "- Various bug fix;"
}

It includes the latest version, URL of the latest version, and explanations of the update.

  1. With above info, check_update.check_update() further compares current version on the phone and the latest one. If the current version is behind the latest one, a popup will be created to notify the user, and ask him/her whether to update;
  2. If user chooses to update, check_update.check_update() will download the latest version from the specified URL, and launches Android’s installation service (by constructing the IntentClass.ACTION_VIEWintent).

In-app log viewer log_viewer_app.py

log_viewer_app.py implements the in-device *.mi2log log viewer. When user clicks the “In-App Logviewer” button, a log_viewer_app.LogViewerScreen will be launched as the main UI, and the following tasks will be done:

  1. Browse all files in /sdcard/mobile_insight/log/, filters out all *.mi2log files under this directories, show them on screen and ask user to select one;
  2. Once user selects one *.mi2log, it will use OfflineReplayer to load the log, and use LogAnalyzer to parse the logs. Once it is done, per-message view will be generated on screen. Note that in the current implementation, the message will be shown on screen until all logs are parsed. This causes long waiting time for user, and should be avoided in future version.
  3. When user selects one message, a popup is created to show the decoded message contents (in xml format).

More details can be found in this article:

[Code Review: LogViewer] (logviewer)

Various utilities: main_utils.py and mi2app_utils.py

In implementing mobileInsight-mobile, there are several functions that would be repetitively called, such as accessing built-in folders like /sdcard/mobile_insight/log. To facilitate development, we have offered several utility functions in demo_app/main_utils.py and demo_app/service/mi2app_utils.py. They provide same utilities, but they should be used in different scenarios. More specifically, - If you want to use the utility functions outside MobileInsight plugin, you should use main_utils.py; - If you want to use them inside MobileInsight plugin, you should use mi2app_utils.py.

To use them, you need to add the following statement in the code

import main_utils #Outside MobileInsight plugin
# OR
import mi2app_utils #Inside MobileInsight plugin codes

Some utilities are declared here:

def get_cur_version():
    """
    Get current apk version string
    """

def is_rooted():
    """
    Check if the phone has been rooted
    """


def run_shell_cmd(cmd, wait = False):
    """
    Run android shell command with root privilege
    """

def get_sdcard_path():
    """
    Return the sdcard path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_path():
    """
    Return the root path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_log_path():
    """
    Return the log path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_analysis_path():
    """
    Return the analysis result path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_log_decoded_path():
    """
    Return the decoded log path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_log_uploaded_path():
    """
    Return the uploaded log path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_cfg_path():

    """
    Return the configuration path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_db_path():
    """
    Return the database path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_plugin_path():
    """
    Return the plugin path of MobileInsight, or None if not accessible
    """

def get_mobile_insight_crash_log_path():
    """
    Return the plugin path of MobileInsight, or None if not accessible
    """

def detach_thread():
    """
    If you are using multi-threading in Python, please call it after the thread
    """

def get_cache_dir():
    """
    Get the APK's cache directory
    """

def get_files_dir():
    """
    Get the APK's file directory
    """

diag_revealer

diag_revealer extracts raw low-level cellular messages from the mobile phone’s cellular chipset. We have a separate documents about how it works. For details, please read

Code Review: diag_revealer