Regex the flow
pcre regex
Contents
PCRE regex
In this tutorial we will show you how transform T2 into a regex based IDS, or a flow labeller for AI training. The plugin regex_pcre implements a full PCRE regex machine where rule trees can be constructed which traverse the flow boundary. In order to enhance performance L3/4 header parameters can be preselected before a regex is applied.
Preparation
First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Then compile the core (tranalyzer2) and the following plugins:
t2build tranalyzer2 basicFlow tcpStates regex_pcre txtSink
...
BUILD SUCCESSFUL
If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:
mkdir ~/data ~/results
The sample PCAP used in this tutorial can be downloaded here: faf-exercise.pcap.
Please save it in your ~/data folder.
Now you are all set!
regex_pcre plugin
The regex plugin produces flow based output if a rule matches. It also implements the ALARM mode, only releasing flows when rules match. It can be one rule or a collection of rules operating on many flows forming a tree.
The configuration of regex_pcre is essentially controlled by two .h files located in the src/ folder.
- regfile_pcre.h
- regex_pcre.h
regfile_pcre.h defines the ingredients of the regfile.txt, containing all rules.
regex_pcre
vi src/regfile_pcre.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define RULE_OPTIMIZE 1 // 0: No opt rules allocated
// 1: Allocate opt rule structure & compile regex
#define REGEX_MODE PCRE_DOTALL // regex compile time options
#define PREIDMX 4 // Max number of node predecessors
/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */
/* No env / runtime configuration flags available for regex_pcre */
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
// local defines
#define HDRSELMX 5 // max dimension of the header features to select from packet, flowdir, prot, srcIP, dstIP + select bits < 12
...So currently only 4 predecessors are allowed in a rule set, you may increase it, if needed, but you really need to know what you do here. You then need to extend the number of rule columns.
vi src/regex_pcre.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define EXPERTMODE 0 // 0: only display the most severe class,
// 1: display all matched classes plus some extra information
#define PKTTIME 0 // whether or not to display the time at which a rule was matched
#define AGGR 0 // 1: Aggregate Alarms
#define SALRMFLG 0 // 1: enable sending FL_ALARM for pcapd
// defines Regex
#define OVECCOUNT 3 // value % 3
#define MAXREGPOS 30 // Maximal # of matches stored / flow
/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */
#define RGX_POSIX_FILE "regexfile.txt" // regexfile name under .tranalyzer/plugins
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...regexfile
REXPOSIX_FILE defines the regex file containing all rules to be tested against every packet of a flow.
The rule trees which can be built, are very mighty but also confusing for the uninitiated,
so let’s have a look at some examples of different rule types depicted below:
tcol scripts/regfile.txt
#ID PreID Flags ClassID Severity Sel Regexmode FlwStat Proto srcPort dstPort offset Regex
# standalone rule: Alarm, start L7, Regexmode: default, select FlwStat: Req; Proto, dstPort
1 0 0x10 15 3 0x8000000d 0x0000000 0x00000000 6 0 80 0 (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT)[^\r\n]*\/u7avi*\.bin
# standalone rule: Alarm, disabled, start L7, select Regexmode: (PCRE_CASELESS|PCRE_DOTALL), FlwStat: Teredo, IPv6, Vlan, Repl; Proto, srcPort
3 0 0x10 15 3 0x0800000e 0x0000005 0x00088101 6 80 0 0 \x31\xDB\x8D\x43\x0D\xCD\x80\x66.*\x31
# standalone rule: Alarm, start L7, Regexmode: default, FlwStat: IPv4, Rply
4 0 0x10 15 3 0x8000000c 0x0000000 0x00004001 6 80 0 20 \x38\x55\x42\x66\xe2\xb5\x34.*\xb5\x95\xbb
# standalone rule, Alarm, start L7, select Regexmode: (PCRE_CASELESS|PCRE_DOTALL)
100 0 0x10 1 0 0x88000000 0x0000005 0x00000000 6 0 80 0 ^http/1.0
# root rules to following tree, Reset if leaf fires
202 0 0x40 10 4 0x80000000 0x0000000 0x00000001 6 0 80 0 (GET|PUT).*update/u7avi1777u1705ff.bin
203 202,4 0x41 20 4 0x88000000 0x0000005 0x00000001 6 0 80 0 302 (?i)Found
# successors and predecessors, Reset if leaf fires
204 202,203 0x41 43 5 0x80000000 0x0000000 0x00000001 6 0 21 0 (?i)\.exe
# successors 206 & 205 to 204 AND ruleset, don't reset tree if 205 fires
205 204 0x16 40 4 0x80000002 0x0000000 0x00000000 6 0 20 0 ^get .*porno.*
206 204 0x56 35 6 0x8000000c 0x0000000 0x00000001 6 0 21 0 igfxzoom\.exe
...t2build invokes the regconv script (located in the scripts/ folder) to transform regfile.txt into a T2 compatible regexfile.txt and
copies it under the plugin directory.
After changing regfile.txt, ALWAYS invoke t2build -f.
Each rule has an ID which not necessary needs to be unique, so that it can be linked by the predecessor preD. The latter denotes
that a rule only fires if the predecessor ID also fired. The Flags define the modes of operation
The regfile.txt file reflects the following rule tree:
1 3 4
|
202:RST - 203:202&4,RST
\ /
\ /
204:1&2,RST
/ \
/ \
205:1&2,RST 206:1&2The Flags define the modes of operation, internal states of the PCRE engine and action on alarm in the flow shown below:
Flags
| code | description |
|---|---|
0x00 |
solitary node |
0x01 |
and(pred1, pred2, …) |
0x02 |
or(pred1, pred2, …) |
0x03 |
xor(pred1, pred2, …) |
0x04 |
leaf |
0x08 |
- |
0x10 |
Print alarm to flow file |
0x20 |
future: rule active only in flow boundary |
0x40 |
Reset REG_F_MTCH tree if match |
0x80 |
Internal: regex match |
The first 2 bits define the operation on the predecessors, such as AND, OR, XOR. Hence, a specific rule with predecessors can only fire if the operation on the results of its predecessors results true.
ClassID and Severity describe the class and severity of an alarm.
You may choose these numbers at your discretion. By default, they will be displayed in the
flow output.
The Sel column controls the activation of the following fields in the selection packet process:
Dir, Proto, srcPort, dstPort
and in which layer the application of the regex rule starts.
Sel
| code | object |
|---|---|
0x0001 |
Activate flowStat |
0x0002 |
Activate l4Proto |
0x0004 |
Activate srcPort |
0x0008 |
Activate dstPort |
0x0010 |
Activate - |
0x0020 |
Activate - |
0x0040 |
Activate - |
0x0080 |
Activate - |
0x0100 |
Activate - |
0x0200 |
Activate - |
0x0400 |
Activate - |
0x0800 |
Activate - |
0x1000 |
Offset start L2 header |
0x2000 |
Offset start L3 header |
0x4000 |
Offset start L4 header |
0x8000 |
Offset start L7 header |
flowStat denotes the first 16 bit of the flow status, hence
the requesting or replying flow could be selected or IPv4/6 etc.
max 12 different parameters can be selected for the Sel columns.
If you add columns in the regfile.txt file, HDRSELMX in regfile_pcre.h
has to be increased accordingly.
Default output
Run t2 on the pcap in default configuration and look at the end report and flow file.
t2 -r ~/data/faf-exercise.pcap -w ~/results/ -s
regex_pcre reports 4 flows with 6 alarms in 72 flows. If you look at the flow file, ID 4, 100 and 206 produce these alarms, which are exactly the ones who have the print bit on.
Look in your results window.
tawk '$rgxCnt > 0' ~/results/faf-exercise_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto tcpStates rgxCnt rgxRID_cType_sev
B 3 0x0400000000004001 1258544216.915576 1258544217.008019 0.092443 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 0x00 1 4_15_3
B 33 0x0400000000004001 1258587444.873221 1258587445.638482 0.765261 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 0x02 2 100_1_0;100_1_0
B 34 0x0400000000004001 1258587445.998250 1258587446.047471 0.049221 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 0x02 1 100_1_0
B 36 0x0400000000004001 1258594163.487027 1258594185.427506 21.940479 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 0x02 2 206_35_6;206_35_6Selecting $rgxCnt > 0 in the packet mode shows all content and rules which matched:
tawk '$rgxCnt > 0' ~/results/faf-exercise_packets.txt | tcol
%pktNo flowInd flowStat time pktIAT pktTrip flowDuration numHdrs hdrDesc ethVlanID srcMac dstMac ethType srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto pktLen l7Len rgxCnt rgxRID_cType_sev l7Content
17 2 0x0400000000004000 1258544216.554751 0.002991 0.003438 0.169381 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 Private network 1259 77.67.44.206 gb Akamai Technologies 80 6 380 322 1 202_10_4 GET /softw/90/update/u7avi1777u1705ff.bin HTTP/1.1\r\nUser-Agent: AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA=\r\nHost: backup.avg.cz\r\nAccept: */*\r\nAccept-Encoding: identity,deflate,gzip\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nx-avg-id:78-175947826\r\n\r\n
26 3 0x0400000000004000 1258544216.929764 0.013738 0.014188 0.021480 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 Private network 1260 198.189.255.75 us California State University 80 6 377 319 1 202_10_4 GET /softw/90/update/u7avi1777u1705ff.bin HTTP/1.1\r\nUser-Agent: AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA=\r\nHost: aa.avg.com\r\nAccept: */*\r\nAccept-Encoding: identity,deflate,gzip\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nx-avg-id:78-175947826\r\n\r\n
48 3 0x0400000000004001 1258544216.960826 0.000009 0.007320 0.045250 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1260 6 1434 1380 1 4_15_3 ]8&..[...x.K.P...A...]...-~<.M..8UBf..4G*.S3.........bOTf.,.l...0...\t....*\\t..).....?X....N...a....MEp[:...L.....<..M....Z.d......M[...LC...G.]\e.6g.G.N.q...~9...y..n.."P8..:0..e.@.....\f...V...=5@..K..:..wj.\tg.c.\t.....c.....*..7......r-.m.7..*GW/..>I....[.*.\b.N.+`.Z&..5M..,.+....k.]... <FE.....wK...../.N..\n\r.2....\n..U...H..d7.;.Gt^...U..>+.\eB......) ..I...<...G.\e'....u[.".g...&.iY.....%R..`.O.=%....2...@...0R....c \b......H!1\r ...k.."1\n..c..om...".Y.P.......AK.Q'.+}>M!.....;\r...n...R}.........MB.P..c.."..=...rP.......;..i...i.......\rp.4/.....{q...K..>.i...5H.1.K...W....Y...O.....&. .... t.q....W2.Z(h>yv.\r).b.YG.I.9.@........r.../.h......W].=).$X3q..D..1@<....[.E|.4@.,.;J.L.H.E.<..\n...Xe.p..O... G.T....<..\n...\ns.n....z.ec\e.4...G.N..f%....W%...]...0hI...W.$......D.b[.\=#...B\f..........b.N.*,1.v..J.....r.Q.z.@.....G..n.T.t.....BT_ly...=.'.\eC~t...>!.).....n.M..^.n...OV...".~..`.?..........Zo.M.-..v.V.em............./.&.`\r3c....X..uA...?.K..)1.a.3.\n.U%g..^p1J....~.!e... ....#..p.dTT;|..+I...tv.r%8..G.Vy....f..w~8a)\f..E..\t..:\t0,....Q.1....R~.m.b........k.!..`......././..x.i*....{_a.......K.m!....'/\fLz\Uv.t...u\e..'..Q.N\r...5.....7n..-...fH.T.....+......~{.6...(^(..p)j...?.]._........../.A.^..H.^....&h!I.C.n.5...f-.V...6.]..........:........P}.9..]....X..H}.rQ\r._z....O..6\t... ...y..6].h......HR....L.l;.w.....v..V.....2.f/\n...p.i..z.......y...._R..[}..|P3...2*\bS(..9%..
118 4 0x0400000000004001 1258544217.346549 0.177128 0.166440 0.177128 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb Akamai Technologies 80 192.168.1.104 07 Private network 1261 6 520 466 1 203_20_4 HTTP/1.1 302 Found\r\nDate: Wed, 18 Nov 2009 11:39:50 GMT\r\nServer: Apache\r\nLocation: http://aa.avg.com/softw/90/update/u7iavi2511u2510ff.bin\r\nContent-Length: 239\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>302 Found</title>\n</head><body>\n<h1>Found</h1>\n<p>The document has moved <a href="http://aa.avg.com/softw/90/update/u7iavi2511u2510ff.bin">here</a>.</p>\n</body></html>\n
162 6 0x0400000000004001 1258544217.752541 0.177889 0.166436 0.177889 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb Akamai Technologies 80 192.168.1.104 07 Private network 1263 6 514 460 1 203_20_4 HTTP/1.1 302 Found\r\nDate: Wed, 18 Nov 2009 11:39:50 GMT\r\nServer: Apache\r\nLocation: http://aa.avg.com/softw/90/update/x8xplsb2_118c8.bin\r\nContent-Length: 236\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>302 Found</title>\n</head><body>\n<h1>Found</h1>\n<p>The document has moved <a href="http://aa.avg.com/softw/90/update/x8xplsb2_118c8.bin">here</a>.</p>\n</body></html>\n
184 8 0x0400000000004001 1258544218.127308 0.175146 0.166190 0.175146 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb Akamai Technologies 80 192.168.1.104 07 Private network 1265 6 520 466 1 203_20_4 HTTP/1.1 302 Found\r\nDate: Wed, 18 Nov 2009 11:39:51 GMT\r\nServer: Apache\r\nLocation: http://aa.avg.com/softw/90/update/x8xplsc_149d148c8.bin\r\nContent-Length: 239\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>302 Found</title>\n</head><body>\n<h1>Found</h1>\n<p>The document has moved <a href="http://aa.avg.com/softw/90/update/x8xplsc_149d148c8.bin">here</a>.</p>\n</body></html>\n
205 10 0x0400000000004001 1258562467.761692 0.000252 0.006059 0.007003 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us Mozilla Corporation 80 192.168.1.104 07 Private network 1379 6 498 444 1 203_20_4 HTTP/1.1 302 Found\r\nServer: Apache/2.2.3 (CentOS)\r\nX-Powered-By: PHP/5.1.6\r\nLocation: http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/\r\nVary: Accept-Language,Accept-Encoding\r\nContent-Encoding: gzip\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 20\r\nX-Varnish-IP: 10.200.74.11\r\nDate: Wed, 18 Nov 2009 16:44:02 GMT\r\nX-Varnish: 62432869 62419064\r\nAge: 17\r\nVia: 1.1 varnish\r\nConnection: keep-alive\r\n\r\n..\b.................
1146 33 0x0400000000004001 1258587444.924436 0.033404 0.040517 0.051215 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1908 6 1434 1380 1 100_1_0 HTTP/1.0 200 OK\r\nServer: Apache\r\nETag: "9bd8e3a274d8ada852bc3d9736116bf6:1258575646"\r\nLast-Modified: Wed, 18 Nov 2009 19:47:42 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 95423\r\nContent-Type: application/octet-stream\r\nDate: Wed, 18 Nov 2009 23:40:20 GMT\r\nConnection: close\r\n\r\nMZ AVG7 UpdateBin grp(avi:1778)dep(avi:1705)tm(0911182047)pri(2)..7TW;I...qb.....Z.\b.J7..*...os...N.1....u.jx.\n.n..%........8......c....X.c.sO..M............Y.7|..\e...q........w/mb.D#...:.`.H|..(.:\e..wjA ...u....C{.]..7.....~.\e\nxX\r(+...L.k..U>&te...-.....a..\f`..n. h.....0.......9Ig.s..7^.)..,....,.....R..+...f ...xg..xq..1........|....)..*..~.%.I\bo.*......)...P...7.W?V...7A).`..~....ox......K.d......r...\ev......9]5\b..Ly7.!.0......5ELd........L..\f...\eq.<&!D.\r.]......>A?.......6..5i..W.....a...pj._.i..x...M.5."K.o.......Z5l.\t.p\b....\t^D....d.j).......m...I.T3.....d.2.....a.5.m..1. 1x...".k+\nz.#@.w;...\r.<...{.....sVv.......u.....Gx..I.\r.Wy.q$M.S....\b........uS.=.....,.O.sl<..\f.;NX............w......2\n......ky...tvf.gzio.U.\rr^....x.D$h...{\bx....!o...lE..C7p.\f.2....=..D..8B./.....P..\..XJ.((i...cR#.NC..Fn..... (...C./.e62...*\n...>..51.Ge.\b.,..s...}........).E....&..n.c.bw...Ne.Z..WTq..n.j.).w.......#<..Sj.....Ck........h.c.iq.....0.{at]....5.|......\f..g..'.[xx.{.'(.B8.@.r..u.E..mG8.........R.Yp>..n<.3..^.%\f'...=.]....hU...Y..(.rd.x.6..Zt...\A..0+...0.7..M...;...x..kt1.._\f.T.[.q.....z..J..I..Pu.
1151 33 0x0400000000004001 1258587445.498378 0.573915 0.573488 0.625157 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1908 6 1434 1380 1 100_1_0 HTTP/1.0 200 OK\r\nServer: Apache\r\nETag: "9bd8e3a274d8ada852bc3d9736116bf6:1258575646"\r\nLast-Modified: Wed, 18 Nov 2009 19:47:42 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 95423\r\nContent-Type: application/octet-stream\r\nDate: Wed, 18 Nov 2009 23:40:20 GMT\r\nConnection: close\r\n\r\nMZ AVG7 UpdateBin grp(avi:1778)dep(avi:1705)tm(0911182047)pri(2)..7TW;I...qb.....Z.\b.J7..*...os...N.1....u.jx.\n.n..%........8......c....X.c.sO..M............Y.7|..\e...q........w/mb.D#...:.`.H|..(.:\e..wjA ...u....C{.]..7.....~.\e\nxX\r(+...L.k..U>&te...-.....a..\f`..n. h.....0.......9Ig.s..7^.)..,....,.....R..+...f ...xg..xq..1........|....)..*..~.%.I\bo.*......)...P...7.W?V...7A).`..~....ox......K.d......r...\ev......9]5\b..Ly7.!.0......5ELd........L..\f...\eq.<&!D.\r.]......>A?.......6..5i..W.....a...pj._.i..x...M.5."K.o.......Z5l.\t.p\b....\t^D....d.j).......m...I.T3.....d.2.....a.5.m..1. 1x...".k+\nz.#@.w;...\r.<...{.....sVv.......u.....Gx..I.\r.Wy.q$M.S....\b........uS.=.....,.O.sl<..\f.;NX............w......2\n......ky...tvf.gzio.U.\rr^....x.D$h...{\bx....!o...lE..C7p.\f.2....=..D..8B./.....P..\..XJ.((i...cR#.NC..Fn..... (...C./.e62...*\n...>..51.Ge.\b.,..s...}........).E....&..n.c.bw...Ne.Z..WTq..n.j.).w.......#<..Sj.....Ck........h.c.iq.....0.{at]....5.|......\f..g..'.[xx.{.'(.B8.@.r..u.E..mG8.........R.Yp>..n<.3..^.%\f'...=.]....hU...Y..(.rd.x.6..Zt...\A..0+...0.7..M...;...x..kt1.._\f.T.[.q.....z..J..I..Pu.
1244 34 0x0400000000004001 1258587446.016254 0.000482 0.007553 0.018004 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1910 6 1434 1380 1 100_1_0 HTTP/1.0 200 OK\r\nServer: Apache\r\nETag: "a8e1ef490967ef7eb6641bef9eed4003:1258575654"\r\nLast-Modified: Wed, 18 Nov 2009 19:48:29 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 21359\r\nContent-Type: application/octet-stream\r\nDate: Wed, 18 Nov 2009 23:40:21 GMT\r\nConnection: close\r\n\r\nMZ AVG7 UpdateBin grp(iavi:2512)dep(iavi:2511)tm(0911182047)pri(2)...8..{R.8F].9{\r...G...hJ..\n.b|n..>H.1.".........%....n...:......c....X.c.sN..M............Y.7|..\e...@'..N....w/mb.D#...:..hH|..(.:\e..F]D-7..u....C{.]D.GO..\e.X..b.8SO.`].8.k..U>&te...-.....&..H<.\6.V......0.......9Ig.s..7^.)..c.....o+.....0...c. ...R......j..([...-%|....)..*..~.%.I\bo.*......)...P...(1\b.L..5.._\r!..../..._G......cN..........vo...d{..tJZ....$*ef.=.(@.I1D..3.O....O.1.\b..)._..\.B..B.a.x.)..9.f....!jX..=.PQ|.P=Ts[.\rS...0\e.m\f..\e5..\e....0.p'N. .69.J.................A...........>...!.......G.gC.[....|Bk.}..l....3NP,....D.D'6...Xz...A....3....X+...........\..Ns.\t..0.z...^q\b...+.>..'#4.P3t..2.!.......Q...91..l..i.....IEA./_.U....Sd...b.|..u^.._.y8.`.....m\e..aY<..l..,.q...q..m.-..^5\eHL.E\e...V.....>.\t.....>nb\f.+.\e...#0...r..L .<.{..L4..".Z?.d...9.U8.=9..z...e.-oH..|.D..a....Me...u......('n...in>..$.Q.....L\n.... ..u.g..2.....-..\n...F9..l<@!."%..8(j..4.1[..`...Br..9.m.-.H.^DG<o.^..Mc.Ul..`)..^.M...e........_..u.q.TT&.g!.....rX..X.[.g2...Y....uo\n{. .d3\r..@.....5e.s0... .nQ....N.Q..I.......<..j...\P.\...P...<-.......\t.(..".
1283 35 0x0400000000004000 1258594163.487490 0.159645 0.079908 0.559148 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 Private network 49329 143.166.11.10 us Dell 21 6 82 24 1 204_43_5 SIZE /video/R79733.EXE\r\n
1285 35 0x0400000000004000 1258594163.566694 0.079204 0.000704 0.638352 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 Private network 49329 143.166.11.10 us Dell 21 6 82 24 1 204_43_5 RETR /video/R79733.EXE\r\n
1384 36 0x0400000000004001 1258594164.127154 0.010002 0.009564 0.640127 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 rial version from Xceed Software's web site at:.About.......http://www.xceedsoft.com/...notepad.notepad ........"]A."]A... . . . . . . . . .(.(.(.(.(. . . . . . . . . . . . . . . . . . .H............................................................................................................................................................................................. ........................................................................................................................................................................................................................................................................................................................................................com.....bat.....cmd.....exe...../\.?*..p\nC.....p\nC.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
1542 36 0x0400000000004001 1258594166.553127 0.000014 0.076766 3.066100 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 ....-.. ].]. ..R............,k.r~\b&.q2lE.....v\t.4.a.\r.QB`.Sg.G.XWk~..J.4DU..R...^e....R...*......o.=....._....8T..4..q....\..\rl..,<..r."\n.3....;-.."H@G\r]<9.7.u..8.I....6....*..(.q/&S}D......../..Q*.?..dC.~9..L.......1......T...1S\r2.H.v.i9`..tc...0i...bz.0......J..... LO.%....`..[@Lj....Yu..%a2...S.N...Z..z...&........L..OS........L.?....9.$...&..\fb.^..0.c..1mE.,I[..6'L..<..6#L8.p=..!`:<X......?.*./...d.c.?.L..U..=d.j-.L[A&N.\..6.L..[.3.p.\t)...B..R.%d....3..L&..6fz\n2)......:a..ut....../.P.<.......}..{BvN.6.g.[F..aD6[=..J+A...`..E...0\.(-.g....Zk-.....+.*..ZO....Q.4...+-....E.&iy..u....[.x...3......&|.PM+G..K.......5+.T..9.G..#"...c\f.iZ...@E.8.......}?. ...h....0..I.,...._g6..._N.#.....:cuvK.Mgl.k'i.MC.....J......b*).#\e..-.3.\t..w.A.\r\nX....L\t.m6)n....H}.s....V...=P%<NZ..W..0o....Y>...cq.u;q2q.+.I.........w....+.....n,|..b.n.../..X....2...QGfn..vz..4..H.%.o,......Og.Ln_r~V.S*.>...-..Q...F...G.AY.Dr......".- ..'T<...8.N:.....0.Q..5.....1.;.8.......u..u..^w.7PK......\b..~.0..tx.....@..\t...Setup.exe.[}t.U..N*M...h#...qK.$ ks.hE'...c...tTH.Wd..E .......+o....8..\b.xT\....Y....."(xv.c.?G+...8..S....;..9.1...tU.....~.{..\r..Z/d.. .c...M0[.p..&\bc&.a..u....lU{'.....Y....Y~...{.^..A....z...x.[.).[.y..E.N...%[(.y.O._~w.7.9..[.-x...f...z...........9o.......<_.....I..5.K.;./.]}.=!.{!.|..Pe....V...>..l.mvAX\b.{...?...g).H:N./ZkRoaa.9....N..L.;...!.\t.....\n...:.d]xz.z.J....e3\tZh\eH
1767 36 0x0400000000004001 1258594167.808855 0.000013 0.000329 4.321828 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 p3|.r.O...p...F......W.?.{..z\p...\n.t.|...\t.B.o].#W...x^...J.E.....c.[[..V..=.[...\b.I.M...F../..6.Mc....8+..l....e.Jx.N.k.m.........v.v..@/."X$.f1,..!.....X2#..U.RX*Kc.h>..8.e.....,....E...`Ve..U.J.....5h\fu`n....8.N.....9.Q.d....8.hN.h.k.Z.....9]...\e...\tn...a.HC..p..l7....n).^.V.3.zi^...E..%.T..2UX..bU..T\r.b.j.n....."t...I`....3u.....RW.z..w.V..{td... =.\b... ...EAIP.T`\e.\tj...!h\n...xcGp2.F.D.D.l.Q&..+.d...6..\b-..T.w..ZSo..I6.V....4.B..%.D.Xx^"I&.(.F:.G..!a.O....\nREjH..H.ZZ.9I..X+.RV..c..&........)+......4;.....)U....n.[...&...c..,.C.}.....Z(...../.......t..W...l..p.h'.8*`..N.....2..\nH.:.Nc...n\f..}....jt.F...t.h......9.\n.?..b..1.t.@.l.M|.r.\e-"^(..]8\e\r.D..\n..5.V.zD.8+"d$Ze..G.L.......}.LV"o.e..B..F.H...<..zS..\n.\nxN...5y../.g.\b..z..."1.$..$..$.".L\n..$mN....7..x\n...........<o.>.\V.tp1.^.V..c\t.X....Zp.A5..[.........O@(tV......(...}.d..L...v..ig...7.\b...Bl..v.]j.a...+.y.]..k....f..Fp`'X..n.....w.].I.....4.F...K.h<M.CH._..O..?.........O..?........7PK......\b.o^J0y.i.............Win2000/hkcmd.exe.[}x.E.....I..\bQQ.6j....$..#g"t.\n..orq&p....*....\f\e..h:E....~.......A.3.";8.$< .m...{.{9-.....0.Y....g.@.@...c.....z....~T...7.I....L...b.b..E.\b...o.....^.!,~........\..?.^.....?...j.?4.....y.....U.|...\egO...k.8..`.........&Tm.....J.mR..../\nU?........o....}... <O.s+..Yu.M...w..?..B.fxV.......h.1..(2!.....wcq...q\n.B...\e.e..>.?.....%....9.'!)V.U......./;V\t.y.....A.z..lI....j.Z..
3605 36 0x0400000000004001 1258594175.999760 0.000008 0.000131 12.512733 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 ....r.......+..3........w..y.-...Yt.u....bm...............,.3.:..`.....Y..../...._PVp. ....~...}y....E7..Ztg..\b.Z.-....To.j......./LZs..n^k.d~..#.a......e....w.L........3.r.......%/.4......r.s?..}ne...#y'.....+\n.M..3.4{.....~......s..T[..\b....k...7.!C.~...,S..F...g*0......J..MkM...S.i.i..........id..@&.......\eJ...uP.......P.\t(.mz.t.t.t..c:a:e:m.0\r..M!..)l.........jN3g......\l.3......5.jX.Mf......|.9.gE..g&....v...n....u.k6.r.H....F_...q3.4.....n...PK......\b..^J0UM...\t..........Win2000/igfxcfg.exe.:.tS...&....$h+E+.7..\nb....j.n..R.!...)\b..W@.)....as....?..y..p.......=.bM\e_.*X..x......R..c1@.....I[.....?;..;.|...o..fB.....q.....q..^.......7..w'po.;2....d.".c\e..oX.f..'\nW.X.v.R.hm.........{.U...U.3sr..\r...*{m..-K.}.{m..m.K..Y.....e....X[..v].3...?..3._Qf5h..o..:......_U...B./e.:....J..|l..i.i...q.y.{a....`}....39.1...y.........`h=...v..j..Y.....mF......Bk3......\n....zZ....L.v.......B.....R\b...a.\ne..=z...>h.#J9.?SG..?...v..v4^l...\eVb...s..v.*.rn....2V..X.+ce....2V.._U\....nr.|...=..!.\e..9....M.;..|.W.<t.|zIWM"...O~..p7.a.N.....x.;y3T..........+....4..a...7.o..st..@.?.A...'6....Hs.o...d.\tz.\nQ..H. CC........\n.\n..p<..\A.o.........jA.K.Z.. F..B.z.5ZP...=...h..s?k.[..p...D...\f.8...0.a...\bZ.......F`....)qd.......e...F.y."...P.H..0{..UI,...~\r.V%PF.....fm>..\\e..1.1... .........`A.\..-"e.??'.j.%.....\...534.Y...>'..C.L.\r........9.*...a].Rx-an...S..QnZ..@.78.Z...g+8.E.O.C=
3952 36 0x0400000000004001 1258594177.414366 0.000018 0.000868 13.927339 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 .\b{.y....^d%.w....<....g......Ve......v..<...r...C>...8/.\t^.%^..|...U..7....{....*?.g.._.K~.k..1#cd...7.....1n...Q6f.9c.......dl\e;..qh.....qn\.5..\D\$DR...H..x(.EAL.....bQ,.e."V..X.\ebSl...'...8.Uq!..1cf.L.I..L.i3c...f..4..q.`..Y.b......l......nn.{.........ye......V....V....V.zh.X...U..V.Z....k.Z...\rk........u`.....uf]Y5+n'...k............o..E{.^.W.M{.....}..>.....j..g..}a_.Wv.&..........1..\t...2%.2#.dVR)d...19..rB.eI......."...\.+rU..u.!7.........{r_.H.1...\t...I.KS4Ks.RA.i...9:..".;t.........<..b.....\n[........K.^.T....y>\b\yD!I.|.ok.5..\e...vh.vh.vh.vh.vh.vh..Q..PK......\b..^J0.N/h/....P......Win2000/igfxdiag.exe.Z}t.U..JW...I7.@.\bQ.L..F..ClF..B..C..n".(B&..Q.*.]..t.M...f6..;..g....=.]..!.C.|....N.8\e.._..F..!R{..........:..W...~.........;8..q...u..s.S.].q...1..\fno.;...E....j...u.....G~...#.=....jM.z.....e/X........JOO.1e.{j..)..h..[1...4x....p....P..O....x/..dx.y.w.7...?...a2.W.;...6..w.....m..).{.{..G.p.X.d.....w ...X].g.'.).W.\n.\eu.S.....x.la.a.#>.7W.d....~...#^....9\r<....I\......x.'].y..f..._o6...G..'.....~.#.#.W..\f.......\nf.......i...._d..\r...2..tf.....\.r.....g......g......g.........r....J.$z.......y.."!..\n9....9."gkQ&foZQvKQ..*...!.....\.4.SL...,iY.ZKG........`.".:.\t|.[..~...\b8..7Z.6...2.....k.8hL.U.N..hR^..... t%G\fT\n\k.....G9...RTG...QI....5t9..Pc./.U.......z....J.9L1....q....&.k..f$...(VY.........z.J.f..e...dA.4".z)...]I.@.R$m./...".?.U..,....A.HP3.Sb<.
4155 36 0x0400000000004001 1258594178.045213 0.000009 0.000106 14.558186 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 .j..A...fD.-..^...0.B\r.{(]..E."U......z....j..F.ta[.U(.D...$.J`.... ...e.. ......W....-..4B...i>..D....V`..\r...Q.......c...OU.nD...i..{h\f2....}.|.\r...l..Q".|W`...}"9Cz..M.'.c.a.w.~?....a.}.L..2.,.N.?..".........Wf...:Y5ldc./j..|.Y.N...,\e\ei6Z`...B...A.r.BQ.[.n.O...W<$.H...\e.c.&9...bi..R..z.Yj.6K.$YR.a.....3...7.\t.y....tNzE..~&.)]...w...G..M...l.Z.W.6...\n.G......xJ<-.._./.o...eqJ."N.W.k.u.0.B.-.K.._z.#\bJ.R..#.J}RX....a....?#.E...I.2z."M..k.u..tKr{*<k<.........\fz...'....xF=.=.=G<.<.=g=.=....k...\e.[...;....................o.w.w...o.7.....].]........\f.N.U.X*..n.\.............x\<.{~K.-..~.a.Ei......,....,....,.......PK......\b..^J0.._r.....p......Win2000/igfxext.exe.Yot[Ev.....G.....b@...ICX..Q...).\e9..K1...\t.Q.....\eg\r.Zx.....{N...]J.....vk..p"...%...=.../L.....6....<YI.=....sl...;w...s......T)I........-.../.G.j/x.Vzv....xV.j...;.........|W....l..\b..9.....cK.qM[...7m^r..5.%..>.9....D..G....D]..#.'...7P........ b....L.4..o.WD.E.>.....#.....>Q/.dPG..P.[.....N.%.K.j.,-...2lTR=.=....=.....\r...E......t-....C...Z.....t...=.....H.u......=..x...~..?.H.C......%i....n6n....../-...\b...h...M..x.........m..m.W....._..H3e....2Sf.L.)3e....2S.(.W..X.?.l.%.;.]..^.g..q~..I..u/.".oG.....Z.Q.9..\f....(k..{c.4...<...}.2.-4.^i~`z.h.X..+7.....n......;\r).".._\n.\r.....\r...6.!.!..._`...ec...@Yl.N...\r/...|Eh0....@.#....Ugc....."R.N...M.......p.h....T.)..G.G..3.....2%..lD'...E!.v.s.}\n.1.g..N..X
5726 36 0x0400000000004001 1258594184.409416 0.000009 0.000299 20.922388 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 204_43_5 ........D.w..\f\e..j...Pl./.R.?.,......fu.a..Ug.Uv5.G-.F\e....Y.S\n]~...D..$.....x.......JC...L\f(....T...e....).......?n......5.Z...@n".....r...<@..G.:..<E.!.&......+.U.:y.. ......c.)..|M.#...$.....#.X:.N.'.I.d:..F\rjQ.z.Lz6.I....s..z!.......:..')@..@=.H.....\r.!.....4..u...&kE.&...^..^..^..^..^..^..^..^..^..^..^..^..^..^..^...f.'PK......\b..^J0)Z[wo....`......Win2000/igfxtray.exe.Z.tTU~.3.2y.\t.4.......R5...0.&qw\r>..1b..$.....G..0.dl^......V.G..S{..v.....x..* .....lY.89k..'2....o....=..............M.n{...c\n~,..^f7?..[.....~1........\e......m..m...6.yO.]wn.t.^.........7U..1Xq..\e[....U.a.:...yh.........v|...s.r<......................i,. ...s"wg..........r/.[...+.._.sr|..wE....Q.0v.Ca.^r..|..+qLw..K9h ......s._.....\....N.........wb..N..F.l.:.8..*..k........-........r.I.Z.cw,...N.N...0.w..;.a.......\n..N{..y.....m..B_..L^V.....T.jSm.M..6...T.jSm.M..S...u....".;.1......xZq...H.........i1V,..}.GL>..'..N5l.....,.2.K..3.~A.[H.,...R.'/..L..;...5.. ..-\nv.......t....e.*....\tp.Q..Jvz......M.~w.al...5.............l.....\e.y....\e.bbr.D:.t.B.,+~...C...B.9.z.....CXh..f.P.%..Xd....h.gU\n\bL7.k.G......+.x....T..E..\f.0...zM...Ef...h.,..GD)6.`.A"...D`..~D.].'."..&bb...L?G.8\t..\r..c.1S.[...1.........N...1Aj.w. .v....G.ob.R..X.7.,.\n3......|'3.{....M.\fK...\r..@Cb1i'..V^.....I.+Ig.....m..\rg..a.a.?":.K....\RY.E.U..4@.|F...6..C..P9.J.R.$..\r.;.i.rq=H......Y[f..\..i...I.VM...$........f.S*..\b...Rdm.
5830 36 0x0400000000004001 1258594184.878377 0.000012 0.000564 21.391350 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 2 204_43_5;206_35_6 .Xi....@i+~..P:.!.A..J....4.["aK.:...V.jX\r.a5......PK......\b..^J0M...{....`......Win2000/igfxzoom.exe.[}pTU....I......(A[}h4....l..l..........\b..m[Dd:...\nv:.q.........afX..U7.P.!.:....1......\b;..,.P..9....:..Q5...................u.,B..^. ....K........|.......|o_Q.....u.....w=.....k..x.....e......;....j.w....(.....^../..|......../..~...........cu$_%.#....(......v..1..k\b.....2.W..B|..,..Z..7F\R....B...{..#.{.S^..9>..!Y-.2xl.+.....I.9}.....Jd.e./........".n.@..(.\f..t....4..7......n.e....x.."^.L,B:.A.\b........{ .d...=oF=/.yf..g..yf..g..yf..g..y.O...`.@7.......G....~G".........RZ..A.0.......M]..^.j..c.C...c%.!.W.2...>."L.a.&..(.*7@............$X_.nc.FQ.....hDQ.:.\b^......&...[..\n..a%...........Q....0...B......;.*D!.u....r.z.\t;..brt.dga..F.\rxDn!.. ....Y#.E.+.b.Q....Th!....8.;!B.Ch.G../?Y..`|....P:.B|2vvPr....}...\b.....F.@......M....N...%.E....J.u\n \e.|\n\f\e... k5...q=....v/.n.H.\n..UD........VB.J .MCZF.SH..H{&!..*RH..i9\riY -.!-[...HWq....@Z.H...e.t......t..z........\f. ...H\.c.T.@.l\f.iNX\n............N.O...0.Y..7a<..a)o....M$....\t.E.E3...Ea....Q7'+...*'..F2;<.J1\t$s... K.E....:....<Q=/....5.v.k./@.:h...."......>>.)(.Q....7..........|...{.@Y"......../A.6.!.>..K...M.i...C...\e..[..K..=...=l.......p...M..y.!....;.).Vc.m...r.)...w....j...L...k..%.{.7...\f..*.j...\\bUYoz........N.I.b.0\t5. .m...p.\t..)u..+ .....^.\t.R\r]Bl.....l.........v\r..\t.Z...TT..Wf.:X...4...+..m..7n..R.z8R.M. .
5890 36 0x0400000000004001 1258594185.348818 0.000011 0.077507 21.861792 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us Dell 64334 192.168.1.105 07 Private network 49330 6 1434 1380 1 206_35_6 .?B.....`............ ...g.9.Win2000/igfxrjpn.lrcPK........\b.\r_J0"..a.....`............ .....:.Win2000/igfxrkor.lrcPK........\b.._J0.~ZC.................. ....1;.Win2000/igfxrptb.lrcPK........\b.._J00...<....p............ .....;.Win2000/igfxrtha.lrcPK........\b.e^J0....d....0............ ...'.<.Win2000/igfxsrvc.dllPK........\b..^J0)Z[wo....`............ .....>.Win2000/igfxtray.exePK........\b..^J0M...{....`............ ...^.?.Win2000/igfxzoom.exePK........\b..^J02....]................ ....d@.Win2000/oemdspif.dllPK....\n......M./...................... .....@.autorun.infPK........\b.iY30...w/...1............. ...>.@.Install.cfgPK........\b..S.2)P..j...U.....F....... .....@.Version.txt\n. ..............f.......f.......f..NU..NUCX....V.e.r.s.i.o.n...t.x.t.PK......O.O.R...Q.@..................0....\t*.H..\r......q0..m...1.0\f.\b*.H..\r....0g.\n+.....7....Y0W03.\n+.....7...0%.... .....<.<.<.O.b.s.o.l.e.t.e.>.>.>0 0\f.\b*.H..\r.........D.@3\fC.6...6N....0...0..........m.z.).......Gn;i0\r.\t*.H..\r.....0_1.0\t..U....US1.0...U.\n..VeriSign, Inc.1705..U....Class 3 Public Primary Certification Authority0..\r011203000000Z.\r111202235959Z0..1.0...U.\n..VeriSign, Inc.1.0...U....VeriSign Trust Network1;09..U...2Terms of use at https://www.verisign.com/rpa (c)011.0,..U...%VeriSign Class 3 Code Signing 2001 CA0..0\r.\t*.H..\r.........0.........u.....p.vb.b..s?Q....)..........b)..X..e\t<...g....]M.^g..A6D..DExpert mode, Timestamp, Alarm aggregation
Switch on aggregation of alarms, expert mode and the output of the
timestamp just to see all the info provided to you.
Recompile regex_pcre and run t2 on the pcap.
t2conf regex_pcre -D AGGR=1 -D EXPERTMODE=1 -D PKTTIME=1 && t2build regex_pcre
t2 -r ~/data/faf-exercise.pcap -w ~/results/... -------------------------------------------------------------------------------- tcpStates: Aggregated tcpStatesAFlags=0x4a regex_pcre: 4 alarms in 4 flows [5.56%] with max severity 6 -------------------------------------------------------------------------------- ... Aggregated flowStat=0x0400000000004000 [WRN] 4 alarms in 4 flows [5.56%] [INF] IPv4 flows
As we aggregate, duplicate alarms are suppressed, that explains the reduction by two alarms, but still 4, 100 and 206.
tawk '$rgxCnt' ~/results/faf-exercise_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto tcpStates rgxCnt rgxRID_cType_sev_pktN_bPos_time
B 3 0x0400000000004001 1258544216.915576 1258544217.008019 0.092443 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 0x00 1 4_15_3_18_12_1258544216.000960
B 33 0x0400000000004001 1258587444.873221 1258587445.638482 0.765261 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 0x02 1 100_1_0_1_0_1258587444.000924
B 34 0x0400000000004001 1258587445.998250 1258587446.047471 0.049221 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 0x02 1 100_1_0_1_0_1258587446.000016
B 36 0x0400000000004001 1258594163.487027 1258594185.427506 21.940479 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 0x02 1 206_35_6_3058_89_1258594184.000878The first three numbers are the same as in the default case. The new following ones are packet number, byte position in the packet and the time stamp. In the packet mode the expert info is contained in columns provided by the core, so the output does not change.
Regex based pcap extraction
As described in the pcap extraction tutorial, the regex_pcre plugin has also the capability to extract
packets on an alarm basis.
The pcapd plugin acts on the FL_ALARM bit set by a firing regex rule in flowStat if SALRMFLG is enabled.
Recompile regex_pcre and run t2 on the pcap.
t2conf regex_pcre -D SALRMFLG=1 && t2build regex_pcre pcapd
t2 -r ~/data/faf-exercise.pcap -w ~/results/... -------------------------------------------------------------------------------- tcpStates: Aggregated tcpStatesAFlags=0x4a regex_pcre: 4 alarms in 4 flows [5.56%] with max severity 6 pcapd: number of packets extracted: 3233 (3.23 K) [54.78%] -------------------------------------------------------------------------------- ... [WRN] 4 alarms in 4 flows [5.56%] [INF] IPv4 flows [INF] IPAlarm
If you look into your ~/results/ directory, you also see faf-exercise_pcapd.pcap created by pcapd.
ls ~/results
faf-exercise_flows.txt faf-exercise_headers.txt faf-exercise_pcapd.pcap
Now run t2 now on faf-exercise_pcapd.pcap but unload pcapd or switch off SALRMFLG to prevent
creating the same pcap again.
t2build -u pcapd
t2 -r ~/results/faf-exercise_pcapd.pcap -w ~/results/... -------------------------------------------------------------------------------- tcpStates: Aggregated tcpStatesAFlags=0x03 regex_pcre: 3 alarms in 3 flows [27.27%] with max severity 5 -------------------------------------------------------------------------------- Headers count: min: 3, max: 3, average: 3.00 Number of TCP packets: 3233 (3.23 K) [100.00%] Number of TCP bytes: 4572148 (4.57 M) [100.00%] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of processed flows: 11 Number of processed A flows: 10 [90.91%] Number of processed B flows: 1 [9.09%] Number of request flows: 4 [36.36%] Number of reply flows: 7 [63.64%] ... Aggregated flowStat=0x0402000000004000 [WRN] 3 alarms in 3 flows [27.27%] [INF] IPv4 flows [INF] IPAlarm
Only three alarms. The flow file shows that now ID 206 is missing, Why? Look into your results folder:
tawk '$rgxCnt' ~/results/faf-exercise_pcapd_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto tcpStates rgxCnt rgxRID_cType_sev_pktN_bPos_time
B 2 0x0402000000004001 1258544216.960826 1258544217.008019 0.047193 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 0x01 1 4_15_3_1_12_1258544216.000960
A 7 0x0402000000004001 1258587444.924436 1258587445.638482 0.714046 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 0x03 1 100_1_0_1_0_1258587444.000924
A 8 0x0402000000004001 1258587446.016254 1258587446.047471 0.031217 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 0x03 1 100_1_0_1_0_1258587446.000016If you look for the FL_ALARM bit in flowStat, all flows which produced an alarm including the ones where
no alarm is printed in the flow file.
tawk 'bitsanyset($flowStat, 0x0002000000000000)' ~/results/faf-exercise_pcapd_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto tcpStates rgxCnt rgxRID_cType_sev_pktN_bPos_time
A 1 0x0402000000004000 1258544216.554751 1258544216.723144 0.168393 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1259 77.67.44.206 gb "Akamai Technologies" 80 6 0x03 0
A 2 0x0402000000004000 1258544216.929764 1258544217.008468 0.078704 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1260 198.189.255.75 us "California State University" 80 6 0x01 0
B 2 0x0402000000004001 1258544216.960826 1258544217.008019 0.047193 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 0x01 1 4_15_3_1_12_1258544216.000960
A 3 0x0402000000004001 1258544217.346549 1258544217.513942 0.167393 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1261 6 0x03 0
A 4 0x0402000000004001 1258544217.752541 1258544217.919686 0.167145 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1263 6 0x03 0
A 5 0x0402000000004001 1258544218.127308 1258544218.294696 0.167388 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1265 6 0x03 0
A 6 0x0402000000004001 1258562467.761692 1258562509.653962 41.892270 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla Corporation" 80 192.168.1.104 07 "Private network" 1379 6 0x03 0
A 7 0x0402000000004001 1258587444.924436 1258587445.638482 0.714046 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 0x03 1 100_1_0_1_0_1258587444.000924
A 8 0x0402000000004001 1258587446.016254 1258587446.047471 0.031217 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 0x03 1 100_1_0_1_0_1258587446.000016
A 10 0x0402000000004000 1258594164.127154 1258594185.427506 21.300352 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 0x03 0
A 9 0x0402000000004000 1258594163.487490 1258594185.618346 22.130856 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49329 143.166.11.10 us "Dell" 21 6 0x03 0For forensic purposes, it is useful to also extract the flow direction which did not produce an alarm, but is part of the alarm process of the opposite flow.
Extract also the opposite flows
In order to extract also the opposite flow of an alarm flow the constant
PD_OPP has to be enabled and the plugin recompiled. Then rerun t2 on
faf-exercise.pcap.
t2conf pcapd -D PD_OPP=1 && t2build pcapd
t2 -r ~/data/faf-exercise.pcap -w ~/results/... -------------------------------------------------------------------------------- tcpStates: Aggregated tcpStatesAFlags=0x4a regex_pcre: 4 alarms in 4 flows [5.56%] with max severity 6 pcapd: number of packets extracted: 4775 (4.78 K) [80.90%] -------------------------------------------------------------------------------- ... Aggregated flowStat=0x0402000000004000 [WRN] 4 alarms in 4 flows [5.56%] [INF] IPv4 flows [INF] IPAlarm
t2build -u pcapd
t2 -r ~/results/faf-exercise_pcapd.pcap -w ~/results/... -------------------------------------------------------------------------------- tcpStates: Aggregated tcpStatesAFlags=0x43 regex_pcre: 3 alarms in 3 flows [15.00%] with max severity 5 -------------------------------------------------------------------------------- Headers count: min: 3, max: 3, average: 3.00 Number of TCP packets: 4775 (4.78 K) [100.00%] Number of TCP bytes: 4699757 (4.70 M) [100.00%] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of processed flows: 20 Number of processed A flows: 10 [50.00%] Number of processed B flows: 10 [50.00%] Number of request flows: 10 [50.00%] Number of reply flows: 10 [50.00%] ... Aggregated flowStat=0x0402000000004000 [WRN] 3 alarms in 3 flows [15.00%] [INF] IPv4 flows [INF] IPAlarm
Now we have 20 flows, because also the opposite A or B flow is extracted, and all have the alarm bit set, as you can see if you look at the flow file below.
tcol ~/results/faf-exercise_pcapd_flows.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto tcpStates rgxCnt rgxRID_cType_sev_pktN_bPos_time
A 1 0x0402000000004000 1258544216.554751 1258544216.723144 0.168393 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1259 77.67.44.206 gb "Akamai Technologies" 80 6 0x01 0
B 1 0x0400000000004001 1258544216.720958 1258544216.888595 0.167637 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1259 6 0x01 0
A 2 0x0402000000004000 1258544216.929764 1258544217.008468 0.078704 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1260 198.189.255.75 us "California State University" 80 6 0x01 0
B 2 0x0402000000004001 1258544216.936827 1258544217.008019 0.071192 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 0x01 1 4_15_3_18_12_1258544216.000960
A 3 0x0400000000004000 1258544217.347008 1258544217.348506 0.001498 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1261 77.67.44.206 gb "Akamai Technologies" 80 6 0x03 0
B 3 0x0402000000004001 1258544217.346549 1258544217.513942 0.167393 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1261 6 0x03 0
A 4 0x0400000000004000 1258544217.753003 1258544217.754495 0.001492 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1263 77.67.44.206 gb "Akamai Technologies" 80 6 0x03 0
B 4 0x0402000000004001 1258544217.752541 1258544217.919686 0.167145 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1263 6 0x03 0
A 5 0x0400000000004000 1258544218.127768 1258544218.129260 0.001492 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1265 77.67.44.206 gb "Akamai Technologies" 80 6 0x03 0
B 5 0x0402000000004001 1258544218.127308 1258544218.294696 0.167388 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1265 6 0x03 0
A 6 0x0400000000004000 1258562467.900050 1258562509.633370 41.733320 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1379 63.245.221.11 us "Mozilla Corporation" 80 6 0x01 0
B 6 0x0402000000004001 1258562467.761692 1258562509.653962 41.892270 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla Corporation" 80 192.168.1.104 07 "Private network" 1379 6 0x01 0
A 7 0x0400000000004000 1258587444.924890 1258587445.631435 0.706545 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1908 198.189.255.75 us "California State University" 80 6 0x03 0
B 7 0x0402000000004001 1258587444.924436 1258587445.638482 0.714046 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 0x03 1 100_1_0_1_0_1258587444.000924
A 8 0x0400000000004000 1258587446.016701 1258587446.040428 0.023727 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1910 198.189.255.75 us "California State University" 80 6 0x03 0
B 8 0x0402000000004001 1258587446.016254 1258587446.047471 0.031217 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 0x03 1 100_1_0_1_0_1258587446.000016
A 10 0x0402000000004000 1258594164.127154 1258594185.427506 21.300352 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 0x03 0
B 10 0x0400000000004001 1258594164.127586 1258594191.015208 26.887622 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49330 143.166.11.10 us "Dell" 64334 6 0x43 0
A 9 0x0402000000004000 1258594163.487490 1258594185.618346 22.130856 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49329 143.166.11.10 us "Dell" 21 6 0x03 0
B 9 0x0400000000004001 1258594163.565990 1258594491.683288 328.117298 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 21 192.168.1.105 07 "Private network" 49329 6 0x43 0So now you have the basics of the regex_pcre plugin. Create your own rules and test them on your own traffic.
Conclusion
You can also use the findexer plugin instead of pcapd. Refer to the pcap extraction & upscaling tutorial.
Do not forget to reset the configuration of regex_pcre:
t2conf regex_pcre --reset && t2build -R
Have fun!