Plugin dependencies
Contents
Introduction
In order to avoid reinventing the wheel, T2 provides the means to access other plugins results.
Getting started
Create folders for your data and results
If you have not created a separate data and results directory yet, please do it now. This will greatly facilitate your workflow:
mkdir ~/data ~/results
Reset tranalyzer2 and the plugins configuration
If you have followed the other tutorials, you may have modified some of the core and plugins configuration. To ensure your results match those in this tutorial, make sure to reset everything:
t2conf -a --reset
You can also clean all build files:
t2build -a -c
Empty the plugin folder
To ensure we are not left with some unneeded plugins or plugins which were built using different core configuration, it is safer to empty the plugins folder:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/user/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Download the PCAP file
The PCAP file used in this tutorial can be downloaded here:
Please save it in your ~/data folder:
wget --no-check-certificate -P ~/data https://tranalyzer.com/download/data/annoloc2.pcap
Build tranalyzer2 and the required plugins
For this tutorial, we will need to build the core (tranalyzer2) and the following plugins:
As you may have modified some of the automatically generated files, it is safer to use the -r and -f options.
...
BUILDING SUCCESSFUL
Source code
In this tutorial, we will extend tcpWin08.tar.gz, the final version of the previous tutorial (Plugin Geo Labeling).
If you are impatient, you can download the final versions of the tcpWin plugin we will develop in this tutorial.
To use one of those plugins, just unpack it in the plugins folder of your T2 installation.
tranpl
tar -xf ~/Downloads/tcpWin09.tar.gz
And let t2_aliases know about it:
source "$T2HOME/scripts/t2_aliases"
Accessing core internal information
If you need the number of TCP packets or any other global counts defined in t2stats.h, which is included in t2Plugin.h every plugin includes, then no dependency is needed.
tranalyzer2
vi src/t2stats.h
...
// counter and monitoring statistics absolute/diff mode
extern uint64_t bytesOnWire, bytesOnWire0;
extern uint64_t bytesProcessed, bytesProcessed0;
extern uint64_t corrReplFlws, corrReplFlws0;
extern uint64_t maxNumFlows; //, maxNumFlows0;
extern uint64_t maxNumFlowsPeak; //, maxNumFlowsPeak0;
extern uint64_t numABytes, numABytes0;
extern uint64_t numAlarmFlows, numAlarmFlows0;
extern uint64_t numAlarms, numAlarms0;
extern uint64_t numAPackets, numAPackets0;
extern uint64_t numAYIYAPackets, numAYIYAPackets0;
extern uint64_t numBBytes, numBBytes0;
extern uint64_t numBPackets, numBPackets0;
#if FORCE_MODE == 1
extern uint64_t numForced, numForced0;
#endif // FORCE_MODE == 1
extern uint64_t numFragV4Packets, numFragV4Packets0;
extern uint64_t numFragV6Packets, numFragV6Packets0;
extern uint64_t numGREPackets, numGREPackets0;
extern uint64_t numLAPDPackets, numLAPDPackets0;
extern uint64_t numLLCPackets, numLLCPackets0;
extern uint64_t numPackets, numPackets0;
extern uint64_t numTeredoPackets, numTeredoPackets0;
extern uint64_t numV4Packets, numV4Packets0;
extern uint64_t numV6Packets, numV6Packets0;
extern uint64_t numVxPackets, numVxPackets0;
extern uint64_t padBytesOnWire, padBytesOnWire0;
extern uint64_t rawBytesOnWire, rawBytesOnWire0;
extern uint64_t totalAFlows, totalAFlows0;
extern uint64_t totalBFlows, totalBFlows0;
extern uint64_t totalFlows, totalFlows0;
extern uint64_t totalIPv4Flows, totalIPv4Flows0;
extern uint64_t totalIPv6Flows, totalIPv6Flows0;
extern uint64_t totalL2Flows, totalL2Flows0;
// global L2 protocols
extern uint64_t numBytesL2[65536], numBytes0L2[65536];
extern uint64_t numPacketsL2[65536], numPackets0L2[65536];
// global L3 protocols
extern uint64_t numBytesL3[256], numBytes0L3[256];
extern uint64_t numPacketsL3[256], numPackets0L3[256];
...So if you need any of these variables, which denote the aggregated counts over all packets up to the time of request, just use them. BUT NEVER EVER WRITE TO THEM !!! We give you as developer the freedom to access core variables and function as we do not try to treat programmers as airbus treats pilots. So we give you access, please treat the anteater respectfully.
Accessing other plugins information
For plugins, it is a bit differentl. Hence, the following rules apply:
- the plugin supplying data must have a number lower than yours
- the header file (pluginName.h) of the plugin must be included at the beginning of your plugin
- Replace
T2_PLUGIN_INIT(...)withT2_PLUGIN_INIT_WITH_DEPS(...)(and add the dependencies) - the path of the plugin must be added in your meson.build, CMakeLists.txt and src/Makefile.am files
- variables you are interested in must be declared as
extern ... __attribute__((weak));
It may sound complicated, but it actually is not. Just stick to the plugin coding rules and you’ll be fine.
In this example, we will add a dependency to the basicStats plugin, to access the number of transmitted packets numTPkts.
Open your tcpWin.c and replace T2_PLUGIN_INIT(...) with T2_PLUGIN_INIT_WITH_DEPS(..., "basicStats").
Just add the lines marked by // <--
tcpWin
vi src/tcpWin.h
...
#include "tcpWin.h"
#include "basicStats.h" // <-- TODO
/*
* Plugin variables that may be used by other plugins MUST be declared in
* the header file as 'extern tcpWinFlow_t *tcpWinFlows;'
*/
tcpWinFlow_t *tcpWinFlows;
/*
* Variables from dependencies, i.e., other plugins, MUST be declared weak,
* in order to prevent dlopen() from trying to resolve them. If the symbols
* are missing, it means the required dependency was not loaded. The error
* will be reported by loadPlugins.c when checking for the dependencies
* listed in the t2Dependencies() or T2_PLUGIN_INIT_WITH_DEPS() function.
*/
extern bSFlow_t *bSFlow __attribute__((weak)); // <-- TODO
/*
* If the dependency is optional, it MUST be defined with the following two
* statements and the dependency MUST NOT be listed in t2Dependencies()
*/
//extern pktSIAT_t *pktSIAT_trees __attribute__((weak));
//pktSIAT_t *pktSIAT_trees;
...
// Tranalyzer functions
/*
* This describes the plugin name, version, major and minor version of
* Tranalyzer required and dependencies
*/
T2_PLUGIN_INIT_WITH_DEPS("tcpWin", "0.9.0", 0, 9, "basicStats"); // <-- TODO
...So we can access the basicStats flow struct now, as shown below:
basicStats
vi src/basicStats.h
typedef struct {
uint64_t numTPkts; // Number of packets transmitted.
uint64_t numTBytes; // Number of bytes transmitted (depends on PACKETLENGTH)
uint64_t totTBytes; // Number of bytes transmitted (total rawLen)
#if BS_STATS == 1
#if BS_XCLD > 0
uint64_t numTPkts0; // Number of packets transmitted pktlen > 0
#endif // BS_XCLD > 0
struct timeval lst;
float avePktSzf;
#if BS_VARSTD > 0
float varPktSz;
float varIATSz;
#endif // BS_VARSTD > 0
float aveIATSzf;
float minIAT;
float maxIAT;
uint16_t minL3PktSz; // Smallest L3 packet size detected
uint16_t maxL3PktSz; // Largest L3 packet size detected
#endif // BS_STATS == 1
} bSFlow_t;
// plugin struct pointer for potential dependencies
extern bSFlow_t *bSFlow;Open tcpWin.c and add the lines marked by // <-- in the t2OnFlowTerminate(...) callback function.
There we define a pointer to the struct of that very flowIndex.
Then, we replace tcpWinFlowP->pktTcpCnt with bSFlowP->numTPkts.
We will also replace our global pktTcpCnt with numPacketsL3[L3_TCP] defined in t2stats.h, so you can
delete the global static uint32_t pktTcpCnt;!
You can also delete the line where tcpWinFlowP->pktTcpCnt++; in t2OnLayer4().
And finally, the uint32_t pktTcpCnt; field in gwz_t in tcpWin.h is not needed anymore, so delete it as well.
Note that we optimized the code a little bit, by adding a test for PROTO_IS_TCP().
That way, we only do calculations when we have to (IPv4 and TCP).
tcpWin
vi src/tcpWin.c
void t2OnFlowTerminate(unsigned long flowIndex, outputBuffer_t *buf) {
tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];
bSFlow_t * const bSFlowP = &bSFlow[flowIndex]; // <-- TODO
float f = 0.0;
if (bSFlowP->numTPkts) { // <-- TODO
f = (float)tcpWinFlowP->winThCnt/(float)bSFlowP->numTPkts; // <-- TODO
}
OUTBUF_APPEND_U8(buf, tcpWinFlowP->stat);
OUTBUF_APPEND_U8(buf, tcpWinFlowP->ttl);
OUTBUF_APPEND_U32(buf, tcpWinFlowP->tcpWinInit);
OUTBUF_APPEND_U32(buf, tcpWinFlowP->winThCnt);
OUTBUF_APPEND_FLT(buf, f);
const flow_t * const flowP = &flows[flowIndex];
if (!FLOW_IS_IPV4(flowP) || !PROTO_IS_TCP(flowP)) return; // IPv4 and TCP only // <-- TODO
if (tcpWinFlowP->winThCnt == 0 || bSFlowP->numTPkts < TCPWIN_MINPKTS) return; // <-- TODO
const int wzi = gwz.wzi; // store element count in const local variable, makes the compiler happy
if (wzi >= TCPWIN_MAXWSCNT) return; // If array full, stop saving
int i;
for (i = 0; i < wzi; i++) {
if (gwz.wzip[i].IPv4.s_addr == flowP->srcIP.IPv4.s_addr) break; // does IP exist?
}
if (f > gwz.wzCnt[i]) { // only update if count is greater than the previous one
gwz.tcpCnt[i] = bSFlowP->numTPkts; // update TCP packet count <-- TODO
gwz.wzCnt[i] = f; // update relative count
if (i == wzi) { // new one?
gwz.wzip[i] = flowP->srcIP; // save new IP
#if SUBNET_ON != 0 && (AGGREGATIONFLAG & SUBNET) == 0
gwz.sID[i] = flowP->subnetNrSrc; // save subnetID from core
#endif // SUBNET_ON != 0 && (AGGREGATIONFLAG & SUBNET) == 0
gwz.wzi++; // increment global window size counter
}
}
}
...That’s it for the code part. Now we need to tell the compiler what we want.
For meson add the depending plugin in the inc statement in meson.build
tcpWin
vi meson.build
...
inc = include_directories(
join_paths('..', '..', 'utils'),
join_paths('..', '..', 'tranalyzer2', 'src'),
join_paths('..', 'basicStats', 'src'), # <-- TODO
)
...For cmake in the target_include_directories of CMakeLists.txt
vi CMakeLists.txt
...
target_include_directories(${_plugin_name}
PRIVATE
../../utils
../../tranalyzer2/src
../../basicStats/src # <-- TODO
)
...And for autotools, open src/Makefile.am and add the path to the source of the
basicStats
plugin in libtcpWin_la_CFLAGS as outlined below
vi src/Makefile.am
...
libtcpWin_la_CFLAGS = \
-I$(top_srcdir)/../../utils \
-I$(top_srcdir)/../../tranalyzer2/src \ # <-- Do not forget to add a backslash!! TODO
-I$(top_srcdir)/../../basicStats/src # <-- TODO
if APPLE
libtcpWin_la_CFLAGS += -D_DARWIN_C_SOURCE # macOS specific flags
else
libtcpWin_la_CFLAGS += -D_GNU_SOURCE
endif
...Then your plugin does its job for all build methods. Fine, now try to build your changes and see whether it complains.
After you edited the skeleton code you should compare your implementation with tcpWin09.tar.gz.
If all is good run t2 on the pcap, otherwise, debug. If you have difficulties, please don’t hesitate to contact the Anteater.
t2build tcpWin
t2 -r ~/data/annoloc2.pcap -w ~/results================================================================================ Tranalyzer 0.9.0 (Anteater), Cobra. PID: 6508, SID: 666 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.9.0 02: basicStats, 0.9.0 03: tcpStates, 0.9.0 04: tcpWin, 0.9.0 05: txtSink, 0.9.0 [INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481503 (481.50 K) [INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41497 (41.50 K) Processing file: /home/wurst/data/annoloc2.pcap Link layer type: Ethernet [EN10MB/1] Snapshot length: 66 Dump start: 1022171701.691172000 sec (Thu 23 May 2002 16:35:01 GMT) [WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500 ... -------------------------------------------------------------------------------- basicStats: Biggest L2 flow talker: 00:d0:02:6d:78:00: 57 [0.00%] packets basicStats: Biggest L2 flow talker: 00:d0:02:6d:78:00: 2622 (2.62 K) [0.00%] bytes basicStats: Biggest L3 flow talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets basicStats: Biggest L3 flow talker: 138.212.189.38 (JP): 33731054 (33.73 M) [52.64%] bytes tcpStates: Aggregated tcpStatesAFlags=0xdf tcpWin: Aggregated tcpWinStat=0x01 tcpWin: Number of TCP winsize packets below threshold 1: 2415 (2.42 K) [0.25%] tcpWin: IP: 138.212.187.203, country: jp, org: ASAHI KASEI CORPORATION -------------------------------------------------------------------------------- ...
tcpWin reports the same values as before, nice!
Change to your results window and look at the flow file. You will notice that the relative measure matches the ones from previous tutorials. At least record displayed might convince you that plugin dependency works.
tcol ~/results/annoloc2_flows.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType vlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpStatesAFlags tcpWinStat tcpWinIpTTL tcpInitWinSz tcpWinThCnt tcpWinSzThRt
A 265 0x0400000000004000 1022171701.709116000 1022171701.709116000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:fc:0e:21:56 0x0800 209.171.12.143 ca "TELUS Communications Inc" 4987 138.212.185.230 jp "ASAHI KASEI CORPORATION" 41250 6 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0xc3 0x01 117 0 1 1
A 447 0x0400000000004000 1022171701.721366000 1022171701.721366000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:fc:3b:62:78 0x0800 217.41.129.13 gb "BT Infrastructure Layer" 58872 138.212.187.186 jp "ASAHI KASEI CORPORATION" 80 6 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0xc3 0x01 44 0 1 1
A 392 0x0400000000004000 1022171701.716998000 1022171701.716998000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:bf:59:85:48 0x0800 36.242.181.230 jp "SoftBank Corp" 4685 138.212.188.67 jp "ASAHI KASEI CORPORATION" 1214 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x03 0x00 105 64240 0 0
B 392 0x0400000000004001 1022171701.732313000 1022171701.732313000 0.000000000 1 3 eth:ipv4:tcp 00:50:bf:59:85:48 00:d0:02:6d:78:00 0x0800 138.212.188.67 jp "ASAHI KASEI CORPORATION" 1214 36.242.181.230 jp "SoftBank Corp" 4685 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x43 0x01 128 0 1 1
A 906 0x0400000000004000 1022171701.816638000 1022171701.816638000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:60:08:69:80:dd 0x0800 161.135.53.11 us "Federal Express Corp" 5001 138.212.191.94 jp "ASAHI KASEI CORPORATION" 80 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x03 0x00 236 32768 0 0
B 906 0x0400000000004001 1022171701.817195000 1022171701.817195000 0.000000000 1 3 eth:ipv4:tcp 00:60:08:69:80:dd 00:d0:02:6d:78:00 0x0800 138.212.191.94 jp "ASAHI KASEI CORPORATION" 80 161.135.53.11 us "Federal Express Corp" 5001 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x43 0x01 128 0 1 1
A 1027 0x0400000000004000 1022171701.872817000 1022171701.872817000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:80:48:b3:13:27 0x0800 146.162.158.230 gb "Norwich Union Insurance Limite" 2849 138.212.184.193 jp "ASAHI KASEI CORPORATION" 6346 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x03 0x00 115 16384 0 0
B 1027 0x0400000000004001 1022171701.873426000 1022171701.873426000 0.000000000 1 3 eth:ipv4:tcp 00:80:48:b3:13:27 00:d0:02:6d:78:00 0x0800 138.212.184.193 jp "ASAHI KASEI CORPORATION" 6346 146.162.158.230 gb "Norwich Union Insurance Limite" 2849 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x43 0x01 255 0 1 1
A 1154 0x0400000000004000 1022171701.939627000 1022171701.939627000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:bf:59:85:48 0x0800 193.133.224.57 gb "UK PA route" 3286 138.212.188.67 jp "ASAHI KASEI CORPORATION" 1214 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x03 0x00 111 16384 0 0
B 1154 0x0400000000004001 1022171701.947575000 1022171701.947575000 0.000000000 1 3 eth:ipv4:tcp 00:50:bf:59:85:48 00:d0:02:6d:78:00 0x0800 138.212.188.67 jp "ASAHI KASEI CORPORATION" 1214 193.133.224.57 gb "UK PA route" 3286 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x43 0x01 128 0 1 1
A 867 0x0400000200004000 1022171701.805350000 1022171701.805350000 0.000000000 1 3 eth:ipv4:tcp 00:60:b0:b5:da:10 00:d0:02:6d:78:00 0x0800 138.212.184.48 jp "ASAHI KASEI CORPORATION" 6666 36.74.248.27 id "PT Telekomunikasi Indonesia" 1108 6 1 1 137 0 137 137 137 0 0 0 0 0 0 0 0 1 0x03 0x00 64 5840 0 0
B 867 0x0400000000004001 1022171702.012658000 1022171702.012658000 0.000000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:60:b0:b5:da:10 0x0800 36.74.248.27 id "PT Telekomunikasi Indonesia" 1108 138.212.184.48 jp "ASAHI KASEI CORPORATION" 6666 6 1 1 0 137 0 0 0 0 0 0 0 0 0 0 0 -1 0x43 0x01 112 0 1 1
A 864 0x0400000200004000 1022171701.805329000 1022171702.066438000 0.261109000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:60:b0:ec:34:27 0x0800 19.54.241.65 us "MAINT-APNIC-AP" 6667 138.212.191.209 jp "ASAHI KASEI CORPORATION" 45891 6 3 3 324 0 0 185 108 59.8867 0 0.214542 0.08703634 0.06619374 11.48946 1240.861 0 1 0x03 0x00 56 33580 0 0
B 864 0x0400000000004001 1022171701.806695000 1022171702.066682000 0.259987000 1 3 eth:ipv4:tcp 00:60:b0:ec:34:27 00:d0:02:6d:78:00 0x0800 138.212.191.209 jp "ASAHI KASEI CORPORATION" 45891 19.54.241.65 us "MAINT-APNIC-AP" 6667 6 3 3 0 324 0 0 0 0 0 0.206711 0.08666233 0.06270833 11.53904 0 0 -1 0x43 0x01 64 62780 1 0.3333333
...As an exercise, try to use more information from the
basicStats
plugin structure.
If you use variables which can be influenced by a reconfiguration of
basicStats
you need to apply the same compiler pragmas as in bSFlow_t.
Make your plugin a dependency
As an exercise, in addition to share tcpWinFlows, we will make a pointer to the gwz structure available to other plugins.
We will then use this dependency in the Plugin Sink tutorial.
This is very easy, open tcpWin.h and look at the end of the file:
tcpWin
vi src/tcpWin.h
...
// Plugin structure
typedef struct { // always large variables first to limit memory fragmentation
uint32_t tcpWinInit; // initial window size
uint32_t winThCnt; // win undershoot count
uint8_t ttl; // TTL
uint8_t stat; // plugin flow status
} tcpWinFlow_t;
// plugin struct pointer for potential dependencies
extern tcpWinFlow_t *tcpWinFlows;
extern gwz_t *gwzP; // <--
#endif // T2_TCPWIN_H_INCLUDEDThe extern tcpWinFlow_t *tcpWinFlows; statement assures that whoever includes your tcpWin.h file
will have access to the tcpWinFlow_t flow memory. Any plugin global variable or structure you define
here as extern is in principle available for other plugins to access.
The real pointers to the whole struct flow memory have to be defined in your tcpWin.c file as global variables.
vi src/tcpWin.c
...
/*
* Plugin variables that may be used by other plugins MUST be declared in
* the header file as 'extern tcpWinFlow_t *tcpWinFlows;'
*/
tcpWinFlow_t *tcpWinFlows;
gwz_t *gwzP; // <--
...
void t2Init() {
...
gwzP = &gwz; // <-- Global pointer for plugin dependencies
}So the tcpWinFlows structure of tcpWin was already eligible as a dependency.
Conclusion
You can download the final version of the tcpWin plugin.
The next tutorial will teach you how to implement the alarm mode.
Have fun writing plugins!
See also
- Plugin programming cheatsheet
- The basics: your first flow plugin
- Plugin end report
- Plugin monitoring
- Plugin packet mode
- Plugin summary files
- Plugin geo labeling
- Plugin alarm mode
- Plugin force mode
- Plugin pcap extraction
- Plugin flow timeout
- Plugin sink
- Developing Tranalyzer plugins in C++
- Developing Tranalyzer plugins in Rust