Plugin summary files

Introduction

When all data is processed or T2 is interrupted by Ctrl+C or a signal, then the end report is printed. At the same time certain global information can be printed by your plugin.

Getting started

Create folders for your data and results

If you have not created a separate data and results directory yet, please do it now. This will greatly facilitate your workflow:

mkdir ~/data ~/results

Reset tranalyzer2 and the plugins configuration

If you have followed the other tutorials, you may have modified some of the core and plugins configuration. To ensure your results match those in this tutorial, make sure to reset everything:

t2conf -a --reset

You can also clean all build files:

t2build -a -c

Empty the plugin folder

To ensure we are not left with some unneeded plugins or plugins which were built using different core configuration, it is safer to empty the plugins folder:

Are you sure you want to empty the plugin folder '/home/user/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied

Download the PCAP file

The PCAP file used in this tutorial can be downloaded here:

• annoloc2.pcap

Please save it in your ~/data folder:

wget --no-check-certificate -P ~/data https://tranalyzer.com/download/data/annoloc2.pcap

Build tranalyzer2 and the required plugins

For this tutorial, we will need to build the core (tranalyzer2) and the following plugin:

• basicFlow

We do not need flow output, so no sink plugin is needed.

As you may have modified some of the automatically generated files, it is safer to use the -r and -f options.

...

BUILDING SUCCESSFUL

Source code

In this tutorial, we will extend tcpWin06.tar.gz, the final version of the previous tutorial (Plugin Packet Mode).

If you are impatient, you can download the final versions of the tcpWin plugin we will develop in this tutorial.

To use one of those plugins, just unpack it in the plugins folder of your T2 installation.

tranpl

tar -xf ~/Downloads/tcpWin07.tar.gz

And let t2_aliases know about it:

source "$T2HOME/scripts/t2_aliases"

Summary files

Every plugin can produce a summary of traffic content. Let’s generate a file which lists all IPs with their maximal winThCnt, so we could find the IP having the largest L4 trouble.

In order to produce just summary files, no t2OnLayer2(...)/t2OnLayer4(...) callback is needed. You could do everything in a packet callback, but that would be inefficient, as you go over the code for every packet. If it is possible, it is always good practice to move heavy processing to t2OnNewFlow(...)/t2OnFlowTerminate(...) callbacks, as it is only called once per flow.

For the sake of simplicity, we just have arrays and a global index as defined in the global section of the plugin.

So move to the tcpWin plugin and open tcpWin.h. Add the maximal size of the global IP/window size count array and the appendix of the global file. Just add the lines marked by // <-- (including the typedef gwz_t)

tcpWin

vi src/tcpWin.h

...

/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */

#define TCPWIN_THRES      1 // TCP window size threshold undershoot flag
#define TCPWIN_MINPKTS   50 // <-- Summary file: minimal TCP packets seen to start saving process
#define TCPWIN_MAXWSCNT 100 // <-- Summary file: maximal number of window size threshold count array elements

#define TCPWIN_SUFFIX "_tcpwin.txt"  // <-- Summary file: file name suffix

/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */


// plugin defines

// tcpWinStat status variable
#define TCPWIN_STAT_THU 0x01 // TCP window size threshold undershoot


typedef struct {                      // <--
    ipAddr_t wzip  [TCPWIN_MAXWSCNT]; // <-- IP address array
    uint32_t tcpCnt[TCPWIN_MAXWSCNT]; // <-- TCP packet count
    float    wzCnt [TCPWIN_MAXWSCNT]; // <-- window size count
    int      wzi;                     // <-- window size index
} gwz_t;                              // <--


// Plugin structure

...

Note that we only select flows which contain more than 50 packets, to avoid small flow clutter.

So you are all set for the header file. Let’s now move to tcpWin.c, where we first need to define all necessary variables global to the plugin.

vi src/tcpWin.c

...

/*
 * Plugin variables that may be used by other plugins (MUST be declared in
 * the header file as 'extern tcpWinFlow_t *tcpWinFlows;'
 */

tcpWinFlow_t *tcpWinFlows;


/*
 * Static variables are only visible in this file
 */

// window size counts
static gwz_t  gwz;                      // <-- global window size structure

static uint32_t pktTcpCnt;              // Aggregated TCP packet count
static uint32_t winThCntG, winThCntG0;  // Aggregated win threshold count and variable for the last threshold count
static uint8_t  tcpWinStat;             // Aggregated status

...

Now go to the t2OnFlowTerminate(...) callback. First we need the pointer to the flow structure. To show the principal we just do a linear search whether we already have the IP stored and check whether its counter is greater than the potential existing, then we store the new data. Yes, it can be more elegant, using a hash or a tree. We will discuss T2 support for that matter in tutorial block T2 Kung Fu. Note that the following code is only for IPv4, as we want to start simple.

vi src/tcpWin.c

...
void t2OnFlowTerminate(unsigned long flowIndex, outputBuffer_t *buf) {
    const tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];

    ...

    // <-- Add all the code below

    const flow_t * const flowP = &flows[flowIndex];
    if (!FLOW_IS_IPV4(flowP)) return;  // IPv4 only

    if (tcpWinFlowP->winThCnt == 0 || tcpWinFlowP->pktTcpCnt < TCPWIN_MINPKTS) return;

    const int wzi = gwz.wzi; // store element count in const local variable, makes the compiler happy
    if (wzi >= TCPWIN_MAXWSCNT) return;  // If array full, stop saving

    int i;
    for (i = 0; i < wzi; i++) {
        if (gwz.wzip[i].IPv4.s_addr == flowP->srcIP.IPv4.s_addr) break; // does IP exist?
    }

    if (f > gwz.wzCnt[i]) {                     // only update if count is greater than the previous one
        gwz.tcpCnt[i] = tcpWinFlowP->pktTcpCnt; // update TCP packet count
        gwz.wzCnt[i] = f;                       // update relative count
        if (i == wzi) {                         // new one?
            gwz.wzip[i] = flowP->srcIP;         // save new IP
            gwz.wzi++;                          // increment global window size counter
        }
    }
}
...

The t2Finalize() callback not only serves to free flow memory but also to produce all kinds of global output into specialized files. The code below stores the IP and its count. Just add it after the free(...) command.

...
void t2Finalize() {
    free(tcpWinFlows); // free the tcpWinFlows

    // open tcpWin statistics file
    FILE *fp = t2_fopen_with_suffix(baseFileName, TCPWIN_SUFFIX, "w");
    if (UNLIKELY(!fp)) return;  // return if file could not be opened

    fprintf(fp, "# IP\tpktTcpCnt\twinRelThCnt\n"); // print header

    char srcIP[INET_ADDRSTRLEN];
    for (int i = 0; i < gwz.wzi; i++) {
        T2_IP_TO_STR(gwz.wzip[i], 4, srcIP, INET_ADDRSTRLEN);                     // convert IP to string
        fprintf(fp, "%s\t%" PRIu32 "\t%f\n", srcIP, gwz.tcpCnt[i], gwz.wzCnt[i]); // print in file
    }

    fclose(fp);
}

After you edited the skeleton code you should compare your implementation with tcpWin07.tar.gz.

Compile and run t2:

t2build tcpWin

...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed      flows: 17100 (17.10 K)
Number of processed L2   flows: 99 [0.58%]
Number of processed IPv4 flows: 16937 (16.94 K) [99.05%]
Number of processed IPv6 flows: 64 [0.37%]
Number of processed A    flows: 9719 (9.72 K) [56.84%]
Number of processed B    flows: 7381 (7.38 K) [43.16%]
Number of request        flows: 9676 (9.68 K) [56.58%]
Number of reply          flows: 7424 (7.42 K) [43.42%]
Total   A/B    flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed   packets/flows: 71.29
Number of processed A packets/flows: 58.05
Number of processed B packets/flows: 88.71
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B   packets/s: 48859.83 (48.86 K)
Number of processed A     packets/s: 22615.05 (22.61 K)
Number of processed   B   packets/s: 26244.78 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Open your results cmd window and look at what t2 produced

annoloc2_tcpwin.txt

So the *_tcpwin.txt* file contains the summary of all IP and their counts being below the window size threshold. If you are interested in the IP with the worst connection sort the *_tcpwin.txt* file according to the 3rd column.

tawk -s '#' 't2sort(winRelThCnt)' ~/results/annoloc2_tcpwin.txt | tcol

# IP             pktTcpCnt  winRelThCnt
138.212.187.203  72         1.000000
36.152.156.46    79         0.962025
216.237.125.166  429        0.489510
216.217.165.245  52         0.269231
138.212.186.191  251        0.247012
138.212.185.150  195        0.246154
138.212.186.160  205        0.229268
138.212.186.52   162        0.203704
200.44.192.225   315        0.196825
201.98.31.61     388        0.164948
216.56.159.22    95         0.157895
138.212.186.60   140        0.150000
201.9.136.60     197        0.142132
193.87.5.62      235        0.140426
201.9.140.14     194        0.123711
19.112.1.129     78         0.115385
138.212.185.186  74         0.094595
212.88.230.156   131        0.045802
201.123.124.98   340        0.041176
193.86.108.236   847        0.035419
138.212.185.98   133        0.030075
200.50.55.138    87         0.022989
200.50.132.98    111        0.018018
193.87.239.57    365        0.016438
83.42.68.176     62         0.016129
200.32.26.254    250        0.016000
19.123.222.7     72         0.013889
138.212.188.204  73         0.013699
193.87.97.162    843        0.011862
138.212.191.84   92         0.010870
133.26.84.187    2079       0.009620
209.132.7.75     114        0.008772
193.104.31.16    133        0.007519
193.87.112.223   957        0.005225
209.147.223.37   244        0.004098
138.212.188.66   249        0.004016
192.224.45.42    719        0.002782
70.5.118.83      415        0.002410
83.128.136.224   977        0.002047
36.89.79.225     538        0.001859
138.212.184.48   1081       0.001850
201.98.215.67    618        0.001618
219.127.165.87   670        0.001493
138.212.188.99   1134       0.000882
201.9.148.42     1332       0.000751
138.212.190.117  2309       0.000433

So in all packets the host 138.212.187.203 asked the opposite IP to stop sending packets. If you increase TCPWIN_MINPKTS what will happen?

Then add the destination address. Does that change much? Look at the flow asymmetry. Why not improve the measure by adding the TCP WS option. These are your jobs now. Let yourself be inspired by the tcpFlags plugin of T2.

Conclusion

You can download the final version of the tcpWin plugin.

The next tutorial will teach you geo-whois-labeling.

Have fun writing plugins!

See also

• Plugin programming cheatsheet
• The basics: your first flow plugin
• Plugin end report
• Plugin monitoring
• Plugin packet mode
• Plugin geo labeling
• Plugin dependencies
• Plugin alarm mode
• Plugin force mode
• Plugin pcap extraction
• Plugin flow timeout
• Plugin sink
• Developing Tranalyzer plugins in C++
• Developing Tranalyzer plugins in Rust