nDPI - Quick Start Guide

Scalable LGPLv3 open deep packet inspection library

image.png

table of Contents

nDPI - Quick Start Guide
Open Source Scalable LGPLv3 deep packet inspection library
1, nDPI introduced
1.1, download the source
2, nDPI library
2.1, the compiler nDPI source
2.2, compiling the sample ndpiReader source
2.3, ndpiReader command line options
2.4 protocol file
3, example
3.1 , real-time capture mode
3.2, pcap capture mode
3.3 protocol file
4, API nDPI
5, nDPI develop custom protocol
5.1 introduction
5.2, create a new protocol
5.3, add your agreement to nDPI

1, nDPI Introduction

nDPI is based OpenDPI of DPI library, currently maintained by ntop.

To give you a DPI to experience cross-platform, we not only support the Unix / Linux, but also supports Windows. Not only that, we have by modifying nDPI, making it more suitable for traffic monitoring applications, we by disabling those features for network traffic monitoring unnecessary, this function has been achieved, and therefore also accelerated the speed DPI engine.

nDPI allow detection of application layer protocol, and what ports regardless of using. This means that a non-standard port monitor known protocols (for example, in addition to port 80 of the port to detect the HTTP protocol) or vice versa (eg Skype traffic detection port 80) have become possible. This is because in today's concept of "port = application" is no longer applicable.

In the past few months, we have also added several new features, including:

1. Application Example ndpiReader speed / support functions and packaging have been strengthened. (For example, you can now analyze the GTP tunnel traffic)
2. NDPI can be compiled into the Linux kernel, so you can use it to develop efficient kernel-based modules.
3. Strengthen multifaceted speed, nDPI now faster than his predecessor (prior to version).
4. Add a number of agreements (so far we support about 200 agreements), including SAP and Citrix and other such "business" agreement, it does have Dropbox and Spotify "desktop" protocol.
5. Able to define protocol detection based on the port (and port range) in order to come and you can use the classic supplemental agreement based on the detection of the port detection.
6. In order to allow nDPI support encrypted link, we've SSL (includes server and client) certificate added to the decoder. Therefore, we can understand the use of encryption certificates to clarify the agreement. This will also allow us to identify as Citrix Online and Apple ICloud other general agreement can not be detected.
7. Based on the matching string by using the sub-protocol support.

1.1, download the source

When you build ntop and nProbe nDPI will automatically be downloaded. Of course, nothing can stop you use it as a standalone DPI library. You can start https://github.com/ntop/nDPI download the source code.

2, nDPI library

2.1, the compiler source code nDPI

NDPI start using the library is very simple. To compile this library, you must meet certain prerequisites such as:

GUN autotools/libtool 
gawk 
gcc

To achieve this, you need to use the following command to install them:

image.png

Once you have completed the above installation, you can use the following command to compile the source code nDPI:

cd <nDPI source code directory>
./autogen.sh
./configure
make

2.2, the compiler source code examples ndpiReader

Started ndpiReader is also very simple. To compile it you must use the following command:

cd <nDPI source code directory>/example 
make

2.3, ndpiReader command-line options

This application example can be used in ndpiReader speed / functional analysis of package and support. In particular, ndpiReader can specify a number of command line options.

Here are brief overview command options and each option can be used:

image.png

image.png

-i <file.pcap |device>

This command is used to specify a pcap file to read the data packets, or to specify a device for real-time capture. And only one of the two of them.

-f <BPF filter>

Specify a BPF filter for filtering the selected flow rate. NDPI and only allows those receiving packets that match the filter (if specified).

-s <duration>

定义了捕获持续时间，以秒为单位，仅用于实时流量捕获。

-p <file.protos>

指定了一个协议文件（例如protos.txt）来扩展对子协议和基于端口协议检测的支持。尤其需要小心的是，协议一旦被定义到protos文件中就会覆盖之前的已经存在的协议。

-l <num loops>

检测循环数（仅用与测试）

-d

这个标志会禁用nDPI协议猜测，仅使用DPI。

-t

打印ndpiReader帮助

-v <1|2>

使用这个标志时，ndpiReader会生成详细的输出，这个输出能被用于调试其性能。数字1是显示带有“未知协议”数据包的最低等级，而2级会显示地更加详细，两者只能指定其一。

image.png

2.4、协议文件

nDPI能够通过基于字符串的匹配支持子协议。这是由于许多新的子协议比如Apple iCloud/iMessage，WhatsApp和许多其他的使用HTTP（s）的协议能够通过解码SSL证书主机或者HTTP“Host:”被检测到。因此我们决定将一个基于流行的Aho-Corasick 算法的高效字符匹配库嵌入到nDPI中，用于数十万子字符的高效匹配（即在普通硬件上能足够快地支持10Gb的流量）。

你可以在运行时通过使用一个用以下格式的协议文件来指定子协议：

image.png

除此之外，你也能使用以下格式来指定一个基于端口的检测：

image.png

你可以使用ndpiReader这个应用（使用 –p 选项）来测试你的自定义配置。或者你可以使用ndpi_load_protocls_file() nDPI API调用来增强你自己的应用。

3、示例

在这一节我们会展示一些ndpiReader的使用范例。

3.1、实时捕获模式

下面这个例子将会展示ndpiReader的实时捕获模式。使用参数-i指定接口设备，参数-s 指定实时捕获持续时间。

image.png

3.2、pcap 捕获模式

创建一个pcap文件最简单的方法就是通过使用tcpdump命令，就像下面这个例子：

image.png

一旦pcap文件被创建，你就能启动ndpiReader，使用参数-I :

image.png

image.png

3.3、协议文件

为了阐明协议文件的特性，我们现在将解释怎样让你识别来自ntop.org的流。

我们可以通过编辑protos.txt文件来实现它：

ntop$ echo 'host:"ntop.org"@nTop'> protos.txt

一旦这个协议文件已经被修改，你可以启动ndpiReader，使用参数 –p：

image.png

4、API nDPI

在这一节中nDPI API是重点。

示例ndpiReader现在有一个基础的例子用来展示如何初始化这个库。需要一个已经编译好的库和一个合适配置的Makefile（即示例Makefile）。

想要在你的应用中开始使用nDPI的API，除了你自己的包含外，必须也添加下面这个包含文件：

#include"dpi_main.h"

这个库可以像下面这样初始化：

1、声明协议位掩码，初始化检测模块：

NDPI_PROTOCOL_BITMASK all;
ndpi_struct = ndpi_init_detection_module(detection_ticks_resolution,malloc_wrapper,
            free_wrapper,debug_printf);

这个函数允许你初始化检测模块。字段有如下含义：

U_int32_t ticks_per_second

每秒时间戳分辨率（比如1000每毫秒分辨率）。

void*(*_ndpi_malloc)(unsigned long size);

指向一个内存分配器的函数指针。

void*(*_ndpi_free)(void*prt);

指向一个调试输出函数的函数指针，在生产环境下置空（NULL）

2、通过适当的宏并将他们设置到检测模块中来启用所有协议（需要注意的是，如果你愿意你也可以启用这些协议的子集）。

// enable all protocols
NDPI_BITMASK_SET_ALL(all);
ndpi_set_protocol_detection_bitmask2(ndpi_struct, &all);

这个函数能让你设置已经定义到检测模块中的协议位掩码。

3、为了加载一个存在的协议文件，你必须使用如下函数：

dpi_load_protocols_file(ndpi_struct, _protoFilePath);

4、每当从你的pcap文件或者入口设备中捕获到流时，它们可以通过使用以下函数分析：

protocol = (const u_int32_t)ndpi_detection_process_packet(
            ndpi_struct,ndpi_flow,iph ? (uint8_t *)iph : (uint8_t *)if,
            ipsize,
            time,
            src,
            dst);

这些字段有以下的含义：

struct ndpi_detection_module_struct*ndpi_struct;

检测模块。

 struct ndpi_flow_struct*flow;

指向连接状态机的流空指针。

const unsigned char*packet;

这个packet作为一个无符号的字符指针，长度为packetlen。这个指针必须指向第三层（IP报头）。

const unsigned short packetlen;

packetlen包的长度。

const U_int32_t current_tickt;

数据包的当前时间戳。

struct ndpi_id_struct*src;

指向源订阅状态机的空指针。

struct ndpi_id_struct*dst;

指向目的订阅状态机的空指针。

5、一旦流被分析后，有必要通过下面的函数来摧毁检测模块。

 ndpi_exit_detection_module(ndpi_struct, free_wrapper);

这些字段有以下含义：

 struct ndpi_detection_module_struct*ndpi_struct;

需要被消除的检测模块。

void*(*_ndpi_free)(void*prt);

指向一个内存释放函数的函数指针。

如需进一步的信息，我们建议可以阅读以下文件：

nDPI/example/ndpiReader.c,
nDPI/src/include/ndpi_structs.h,
nDPI/src/include/ndpi_public_functions.h
nDPI/src/ndpi_main.c

协议解析器文件被包含在nDPI/src/protocols 目录下。

5、开发nDPI自定义协

在这一节中，我们将展示将你的协议包含在nDPI中的方法。

5.1、介绍

Each nDPI protocol entry will be implemented as a function, it is used at runtime by nDPI. nDPI There are several protocols that can be used as an example in this regard. Below, we've listed some of the main concepts, if you intend to develop a nDPI agreement, which is that you need to know.

5.2, create a new protocol

Each protocol must have a corresponding definition file header in the following:

<nDPI source code directory>/src/include/ndpi_protocols_osdpi.h

E.g:

#define NDPI_PROTOCOL_MY_PROTOCOL     171

NDPI_PROTOCOL_MY_PROTOCOL is the name of this agreement, 171 is the ID of this agreement, and must be unique.

Once you have defined protocol, you must create a new protocol source files, such as

<nDPI source code directory>/src/lib/protocols/my_protocol.c

It includes the following:

#include "ndpi_utils.h"
#ifdef NDPI_PROTOCOL_MY_PROTOCOLS
.....
#endif

On the inside, to define a entry function, such as:

void ndpi_search_my_protocol(
            struct ndpi_detection_module_struct *ndpi_struct,
            struct ndpi_flow_struct *flow)
{
      struct ndpi_packet_struct *packet = &flow->packet;
      NDPI_LOG(NDPI_PROTOCOL_MY_PROTOCOL, ndpi_struct,NDPI_LOG_DEBUG, "my protocol detection...\n");
/* skip marked packets by checking if the detection protocol stack */
    if (packet->detected_protocol_stack[0] != NDPI_PROTOCOL_MY_PROTOCOL) {
        ndpi_check_my_protocol(ndpi_struct, flow);
    }
}

And detecting a core function, to process a packet stream, comprising the following:

static void ndpi_check_my_protocol(struct ndpi_detection_module_struct *ndpi_struct,
                                   struct ndpi_flow_struct *flow)
{
    struct ndpi_packet_struct *packet = &flow->packet;
    u_int32_t payload_len = packet->payload_packet_len;
    .....
    .....
    if("Found Protocol") {
        NDPI_LOG(NDPI_PROTOCOL_MY_PROTOCOL, ndpi_struct,
                 NDPI_LOG_DEBUG, "Found my protocol.\n");
        ndpi_int_my_protocol_add_connection(ndpi_struct, flow);
        return;
    }
    /*Exclude Protocol*/
    NDPI_LOG(NDPI_PROTOCOL_MY_PROTOCOL, ndpi_struct, NDPI_LOG_DEBUG,
             "exclude my protocol.\n");
    NDPI_ADD_PROTOCOL_TO_BITMASK(
                flow->excluded_protocol_bitmask,
                NDPI_PROTOCOL_MY_PROTOCOL);
    }
}

There is also a specific function is used to report on the correct protocol identification information, such as:

static void ndpi_int_my_protocol_add_connection(
        struct ndpi_detection_module_struct *ndpi_struct,
        struct ndpi_flow_struct *flow,
        u_int8_t due_to_correlation)
{
    ndpi_int_add_connection(ndpi_struct, flow,
                            NDPI_PROTOCOL_MY_PROTOCOL,
                            /*Choose the type of your protocol*/
                            NDPI_CORRELATED_PROTOCOL or NDPI_REAL_PROTOCOL);
}

5.3 Add your agreement to nDPI

After the agreement is created, you must declare your entry function in the following header file:

<nDPI source code directory>/src/include/ndpi_protocols.h

It includes the following:

/* my protocol entry */
void ndpi_search_my_protocol(
        struct ndpi_detection_module_struct *ndpi_struct,
        struct ndpi_flow_struct *flow);

Each agreement must be associated with a NDPI_SELECTION_BITMASK. For a complete list NDPI_SELECTION_BITMASK is contained in the file:

<nDPI source code directory>/src/include/ndpi_define.h

After selecting a certain bit mask for your protocol, you must inform the existence nDPI new agreement, by editing the file:

<nDPI source code directory>/src/lib/ndpi_main.c

You need to add your agreement to the following functions:

void ndpi_set_protocol_detection_bitmask2(
        struct ndpi_detection_module_struct *ndpi_struct,
        const NDPI_PROTOCOL_BITMASK * dbm)
.....
.....
.....
#ifdef NDPI_PROTOCOL_MY_PROTOCOL
ndpi_set_bitmask_protocol_detection(ndpi_struct,detection_bitmask,a,
                                    NDPI_PROTOCOL_MY_PROTOCOL,
                                    ndpi_search_my_protocol,
                                    NDPI_SELECTION_BITMASK_MY_PROTOCOL,
                                    SAVE_DETECTION_BITMASK_AS_UNKNOW,
                                    ADD_TO_DETECTION_BITMASK);
/* Update callback_buffer index */
a++;
#endif
.....
.....
.....
ndpi_struct->callback_buffer_size = a;
NDPI_LOG(NDPI_PROTOCOL_UNKNOWN, ndpi_struct, NDPI_LOG_DEBUG,
         "callback_buffer_size is %u\n", ndpi_struct->callback_buffer_size);