Plugin end report

plugin development

Introduction

When all data is processed or T2 is interrupted by Ctrl+C or a signal, then the end report is printed.

Getting started

Create folders for your data and results

If you have not created a separate data and results directory yet, please do it now. This will greatly facilitate your workflow:

mkdir ~/data ~/results

Reset tranalyzer2 and the plugins configuration

If you have followed the other tutorials, you may have modified some of the core and plugins configuration. To ensure your results match those in this tutorial, make sure to reset everything:

t2conf -a --reset

You can also clean all build files:

t2build -a -c

Empty the plugin folder

To ensure we are not left with some unneeded plugins or plugins which were built using different core configuration, it is safer to empty the plugins folder:

t2build -e -y 

Are you sure you want to empty the plugin folder '/home/user/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied

Download the PCAP file

The PCAP file used in this tutorial can be downloaded here:

Please save it in your ~/data folder:

wget --no-check-certificate -P ~/data https://tranalyzer.com/download/data/annoloc2.pcap

Build tranalyzer2 and the required plugins

For this tutorial, we will only need to build the core (tranalyzer2). As we do not have dependencies and do not need flow output, no plugin is required.

As you may have modified some of the automatically generated files, it is safer to use the -r and -f options.

t2build -r -f tranalyzer2 

...

BUILDING SUCCESSFUL

Source code

In this tutorial, we will extend tcpWin02.tar.gz, the final version of the previous tutorial (The basics: your first flow plugin).

If you are impatient you can download the final version.

To use one of those plugins, just unpack it in the plugins folder of your T2 installation.

tranpl

tar -xf ~/Downloads/tcpWin03.tar.gz

And let t2_aliases know about it:

source "$T2HOME/scripts/t2_aliases"

Implementing the end report

If your plugin wants to contribute to the end report, then global variables and a void t2PluginReport(FILE *stream) callback have to be added. So open tcpWin.c in an editor and add two global variables after the tcpWinFlows definition. Look for the // <-- markers.

tcpWin

vi src/tcpWin.c 

...

/*
 * Plugin variables that may be used by other plugins MUST be declared in
 * the header file as 'extern tcpWinFlow_t *tcpWinFlows;'
 */

tcpWinFlow_t *tcpWinFlows;


/*
 * Static variables are only visible in this file
 */

static uint32_t pktTcpCnt;   // <-- Aggregated TCP packet count
static uint32_t winThCntG;   // <-- Aggregated win threshold count
static uint8_t  tcpWinStat;  // <-- Aggregated status

...

In the t2OnFlowTerminate(...) callback, add the lines marked by // <-- to aggregate pktTcpCnt, tcpWinStat and winThCnt of all flows.

vi src/tcpWin.c 

...
void t2OnFlowTerminate(unsigned long flowIndex, outputBuffer_t *buf) {
    const tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];

    tcpWinStat |= tcpWinFlowP->stat;       // <-- Aggregate all packet flags
    pktTcpCnt  += tcpWinFlowP->pktTcpCnt;  // <-- Count all TCP packets
    winThCntG  += tcpWinFlowP->winThCnt;   // <-- Sum all winThCnt packet counts

    OUTBUF_APPEND_U8(buf, tcpWinFlowP->stat);
    OUTBUF_APPEND_U8(buf, tcpWinFlowP->ttl);
    OUTBUF_APPEND_U32(buf, tcpWinFlowP->tcpWinInit);
    OUTBUF_APPEND_U32(buf, tcpWinFlowP->winThCnt);

    float f = 0.0;
    if (tcpWinFlowP->pktTcpCnt) f = (float)tcpWinFlowP->winThCnt/(float)tcpWinFlowP->pktTcpCnt;

    OUTBUF_APPEND_FLT(buf, f);
}

...

Implementing the t2PluginReport() callback

Now search for the void t2PluginReport(FILE *stream) callback after t2OnFlowTerminate(). There we print the status and the aggregated winThCnt.

For printing aggregated status, you can use the T2_FPLOG_AGGR_HEX() macro, which uses the name and the size of the variable to produce a properly formatted message (Refer to Reporting Aggregated Bitfields in the Plugin Programming Cheatsheet.

For printing numbers, use one of the T2_FPLOG_NUM() or T2_FPLOG_NUMP() macros. Both print the numbers in raw, e.g., 2415, and in a human readable form, e.g., 2.42 K. The second macro reports the percentage as well (and thus requires a normalization factor). (Refer to Reporting Numbers in the Plugin Programming Cheatsheet.

For more advanced cases, refer to the sections titled Reporting … in the Plugin Programming Cheatsheet.

The plugin_name variable is automatically created for you in T2_PLUGIN_INIT() and T2_PLUGIN_INIT_WITH_DEPS() That way you don’t have to type tcpWin multiple times and risk typos and it helps you identify your output in the end report. All macros accept standard C printf() formatting.

vi src/tcpWin.c 

...

/*
 * This function is used to report information regarding the plugin.
 * This will appear in the final report.
 */
void t2PluginReport(FILE *stream) {
    if (winThCntG) {
        T2_FPLOG_AGGR_HEX(stream, plugin_name, tcpWinStat);
        T2_FPLOG_NUMP(stream, plugin_name,
                "Number of TCP winsize packets below threshold " STR(TCPWIN_THRES),
                winThCntG, pktTcpCnt);
    }
}

...

For more complex output, you could use the T2_FPLOG() macro. As an example the reporting in the callback above could have been written as follows:

if (winThCntG) {
    T2_FPLOG(stream, plugin_name, "Aggregated tcpWinStat=0x%02" B2T_PRIX8, tcpWinStat);

    char hrnum[64];
    T2_CONV_NUM(winThCntG, hrnum); // Convert winThCntG to human readable string, e.g., 2415 -> 2.42 K
    T2_FPLOG(stream, plugin_name, "Number of TCP winsize packets below threshold %d: %" PRIu32 "%s [%.2f%%]",
            TCPWIN_THRES, winThCntG, hrnum, 100.0*(float)winThCntG/pktTcpCnt);
}

Note, that the core can be configured to display hex numbers small or capital. To be consistent with that option, you should use one of the B2T_PRIX{8,16,32,64} macro instead of the normal PRIx{8,16,32,64} or %02x constant. So you do not need to worry about these nitty-gritty things anymore.

Recompile tcpWin and execute T2

t2build tcpWin

t2 -r ~/data/annoloc2.pcap -w ~/results/ 

================================================================================
Tranalyzer 0.9.0 (Anteater), Cobra. PID: 17582, SID: 666
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: tcpWin, 0.9.0
[INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481503 (481.50 K)
[INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41497 (41.50 K)
Processing file: /home/wurst/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Snapshot length: 66
Dump start: 1022171701.691172000 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398000 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226000 sec
Finished processing. Elapsed time: 0.263058000 sec
Finished unloading flow memory. Time: 0.352172000 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pad bytes: 8591685635 (8.59 G)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218588 (1.22 M) [99.96%]
Number of IPv6 packets: 180 [0.01%]
Number of A packets: 564228 (564.23 K) [46.29%]
Number of B packets: 654787 (654.79 K) [53.71%]
Number of A bytes: 29447896 (29.45 M) [45.95%]
Number of B bytes: 34634830 (34.63 M) [54.05%]
Average A packet load: 52.19
Average B packet load: 52.89
--------------------------------------------------------------------------------
tcpWin: Aggregated tcpWinStat=0x01
tcpWin: Number of TCP winsize packets below threshold 1: 2415 (2.42 K) [0.25%]
--------------------------------------------------------------------------------
Headers count: min: 2, max: 5, average: 3.01
Number of ARP packets: 247 [0.02%]
Number of GRE packets: 20 [0.00%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of TCP bytes: 52643546 (52.64 M) [82.15%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of UDP bytes: 11234272 (11.23 M) [17.53%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed      flows: 17100 (17.10 K)
Number of processed L2   flows: 99 [0.58%]
Number of processed IPv4 flows: 16937 (16.94 K) [99.05%]
Number of processed IPv6 flows: 64 [0.37%]
Number of processed A    flows: 9719 (9.72 K) [56.84%]
Number of processed B    flows: 7381 (7.38 K) [43.16%]
Number of request        flows: 9676 (9.68 K) [56.58%]
Number of reply          flows: 7424 (7.42 K) [43.42%]
Total   A/B    flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed   packets/flows: 71.29
Number of processed A packets/flows: 58.05
Number of processed B packets/flows: 88.71
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B   packets/s: 48859.83 (48.86 K)
Number of processed A     packets/s: 22615.05 (22.61 K)
Number of processed   B   packets/s: 26244.78 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 685.39
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270268480 b/s (270.27 Mb/s)
Max number of flows in memory: 17100 (17.10 K) [6.52%]
Memory usage: 0.06 GB [0.09%]
Aggregated flowStat=0x0c0098fa0222d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 payload length > framing length
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] Stop dissecting: Clipped packet, unhandled protocol or subsequent fragment
[INF] Layer 2 flows
[INF] IPv4 flows
[INF] IPv6 flows
[INF] ARP
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] GTP tunnel
[INF] SSDP/UPnP

Do you notice your end report output starting with tcpWin?


...
--------------------------------------------------------------------------------
tcpWin: Aggregated tcpWinStat=0x01
tcpWin: Number of TCP winsize packets below threshold 1: 2415 (2.42 K) [0.25%]
--------------------------------------------------------------------------------
...

Wasn’t so difficult, right?

As we didn’t load a sink plugin, no flows are produced, although the buffers are filled. If you do not want that to happen try to run: t2conf tranalyzer2 -D BLOCK_BUF=1 && t2build -R. Then all plugins stop filling the flow output buffers.

Conclusion

You can add any output you want by adding more T2_FPLOG*() macros. For more advanced cases and examples, refer to the sections titled Reporting … in the Plugin Programming Cheatsheet.

You can download the final version of the tcpWin plugin.

The next tutorial will teach you how to add plugin monitoring report.

Have fun writing plugins!

See also