Code Review: diag_revealer

In this page, we will review diag_revealer, the core of the in-device of MobileInsight.

What is diag_revealer?

Simply put, diag_revealer extracts raw low-level cellular messages from the mobile phone’s cellular chipset. This is the first step for MobileInsight forward in-device cellular monitoring and analysis. Each time you run [OnlineMonitor] (https://github.com/mobile-insight/mobileinsight-core/blob/master/examples/online-analysis-example.py) in MobileInsight apk, the first step is to launch the diag_revealer deamon.

Please note that, diag_revealer is NOT a MobileInsight monitor, although it is called by MobileInsight’s OnlineMonitor (mobile version). A MobileInsight monitor does more than just collecting raw logs, such as parsing and filtering the logs. diag_revealer is only developed to support the monitor’s functions. The relations between diag_revealer and monitors will be elaborated below.

Currently diag_revealer supports Qualcomm Snapdragon chipsets. We plan to extend it to support more cellular chipsets (e.g., MediaTek, Intel XMM series).

Logic of diag_revealer: A Big Picture

[Here] (https://github.com/mobile-insight/mobileinsight-mobile/blob/master/diag_revealer/qcom/jni/diag_revealer.c) is the complete code of diag_revealer. Here we only discuss its high-level logics. The following figure shows the architecture of diag_revealer:

+---------------+  Raw binaries   +-------------+ Raw binary logs (via diag_revealer_fifo) +------------------+
|Diagnostic port| --------------> |diag_revealer| ---------------------------------------> |AndroidDiagMonitor|
|  (/dev/diag)  | <-------------- | (raw logs)  | <--------------------------------------- | (OnlineMonitor)  |
+---------------+    DIAG.cfg     +-------------+         DIAG_CFG_PATH FIFO_PATH          +------------------+
                                                        [OUTPUT_DIR] [LOG_CUT_SIZE]

As shown above, diag_revealer is implemented as a proxy daemon between in-device diagnostic port (/dev/diag) and MobileInsight monitor. It extracts raw binaries from the diagnostic port, filters out cellular logs and push them into a monitor-specified FIFO (i.e., a [Linux pipe] (http://man7.org/linux/man-pages/man2/pipe.2.html)). Then the monitor can pull the raw cellular logs from FIFO and performs further parsing/analysis.

The overall life cycle of diag_revealer includes three steps:

  1. activate the diagnostic port;
  2. configure the cellular messages to be collected;
  3. extract raw cellular messages.

We next present some key implementation details for each step.

Step 1: activate the diagnostic port

In this step, diag_revealer opens the diagnostic port, and configures it with the standard Linux ioctl function. The configurations include activation of the log collection, clearing the buffers, and optionally real-time settings for the buffer. The following snapshots show how this is achieved:

int fd = open("/dev/diag", O_RDWR);
if (fd < 0) {
    perror("open diag dev");
    return -8002;
}

//...

/*
 * Enable logging mode
 * Reference: https://android.googlesource.com/kernel/msm.git/+/android-6.0.0_r0.9/drivers/char/diag/diagchar_core.c
 */
ret = ioctl(fd, DIAG_IOCTL_SWITCH_LOGGING, (char *) &mode);
if (ret < 0) {
    LOGD("ioctl SWITCH_LOGGING fails, with ret val = %d\n", ret);
    perror("ioctl SWITCH_LOGGING");
    // Yuanjie: the following works for Samsung S5
    ret = ioctl(fd, DIAG_IOCTL_SWITCH_LOGGING, (char *) mode);
    if (ret < 0) {
        LOGD("Alternative ioctl SWITCH_LOGGING fails, with ret val = %d\n", ret);
        perror("Alternative ioctl SWITCH_LOGGING");
    }
}

In activating log, the key parameter for ioctl is the request string (e.g., DIAG_IOCTL_SWITCH_LOGGING), which determines the command to be used. A very natural question would arise: how can I know the request string and the parameters associated with it? The short answer is they are defined in the driver codes of /dev/diag. The following code defines the macros for these request strings (the complete driver source code can be found `here <https://android.googlesource.com/kernel/msm.git/+/android-6.0.0_r0.9>`__):

/* Different IOCTL values */
#define DIAG_IOCTL_COMMAND_REG      0
#define DIAG_IOCTL_SWITCH_LOGGING   7
#define DIAG_IOCTL_GET_DELAYED_RSP_ID   8
#define DIAG_IOCTL_LSM_DEINIT       9
#define DIAG_IOCTL_DCI_INIT     20
#define DIAG_IOCTL_DCI_DEINIT       21
#define DIAG_IOCTL_DCI_SUPPORT      22
#define DIAG_IOCTL_DCI_REG      23
#define DIAG_IOCTL_DCI_STREAM_INIT  24
#define DIAG_IOCTL_DCI_HEALTH_STATS 25
#define DIAG_IOCTL_DCI_LOG_STATUS   26
#define DIAG_IOCTL_DCI_EVENT_STATUS 27
#define DIAG_IOCTL_DCI_CLEAR_LOGS   28
#define DIAG_IOCTL_DCI_CLEAR_EVENTS 29
#define DIAG_IOCTL_REMOTE_DEV       32
#define DIAG_IOCTL_VOTE_REAL_TIME   33
#define DIAG_IOCTL_GET_REAL_TIME    34
#define DIAG_IOCTL_PERIPHERAL_BUF_CONFIG    35
#define DIAG_IOCTL_PERIPHERAL_BUF_DRAIN     36

Among these driver codes, some critical ones are listed here for reference:

  • includes/linux/diagchar.h: this file defines the ioctl parameters. By reading this file, you may find some useful parameters such as clearing the buffers, improving the real-time behaviors, registering callbacks for the messages, etc.
  • drivers/char/diag/diagchar_core.c: this file implements how ioctl options are handled (in diagchar_compat_ioctl and diagchar_ioctl). By reading this file, you can better understand the meaning and required parameter types for each ioctl request strings. This helps you better extend diag_revealer.

Step 2: configure the cellular messages to be collected

The next step is to configure the diagnostic port on the message types to be collected. To do this, the OnlineMonitor first generates file called DIAG.cfg, which specifies the cellular message types (the details of DIAG.cfg will be discussed in the upcoming tutorials). Then diag_revealer reads this file, and write it directly into /dev/diag:

BinaryBuffer buf_write = read_diag_cfg(argv[1]); //… ret = write_commands(fd, &buf_write); //fd points to /dev/diag //…

In diag_revealer, this step is implemented in write_commands, as shown below:

static int
write_commands (int fd, BinaryBuffer *pbuf_write)
{
// Write DIAG.cfg directly into /dev/diag
}

Step 3: extract raw cellular messages

The last step is to collect raw cellular logs. To do so, diag_revealer first opens the Linux FIFO pipe specified by the monitor, and set the pipe size large enough (128MB in current implementation). This prevents the overflow of buffer when the cellular log volume is huge (e.g., all logs enabled)

// Messages are output to this FIFO pipe
int fifo_fd = open(argv[2], O_WRONLY);  // block until the other end also calls open()
if (fifo_fd < 0) {
    perror("open fifo");
    return -8005;
} else {
    // LOGD("FIFO opened\n");
}
int pipesize = DIAG_FIFO_PIPE_SIZE; //128MB in current implementation
fcntl(fifo_fd, F_SETPIPE_SZ, pipesize);

Then diag_revealer will pool the dev/diag and read raw binaries. Each time it reads binaries, it parses the very basic header, and check if the binaries are real cellular logs (USER_SPACE_DATA_TYPE, defined in includes/linux/diagchar.h). If so, it pushes the binaries into the FIFO pipe. If any failures happen in writing logs into FIFO, diag_revealer will exit safely.

while (1) {
    int read_len = read(fd, buf_read, sizeof(buf_read));
    if (read_len > 0) {
        if (*((int *)buf_read) == USER_SPACE_DATA_TYPE) {
            // Read further binaries and push them into the FIFO
            // ...
        }
}

FAQs

Q: Does diag_revealer support all Android-Qualcomm phones?

A: Currently diag_revealer cannot support Nexus 5/6 (Issue #107). The phenomenon is that, step 1-3 works correctly, but /dev/diag does not push binaries with type USER_SPACE_DATA_TYPE. Note that other types of binaries are observed, which implies that /dev/diag works correctly. We suspect it is caused by some missing steps we haven’t implemented in diag_revealer.

Q: Is it possible to see the debugging messages of diagnostic drivers?

A: Yes! You can run dmesg | grep diag in adb shell, which filters the in-kernel debugging messages of diag driver. Then you can search them in the driver code. This provides more hints on how diag driver works.