PCAP extraction and upscaling
Contents
- Introduction
- Preparation
- How to select relevant flows for pcap extraction
- Extract flows with pcapd plugin
- Extract pcaps using the FL_ALARM bit
- The superior way: findexer plugin
- findexer and Wireshark export
- Management of humongous flow files: tawk ffsplit
- Working on findexed pcaps: fextractor
- fextractor as flow based pcap slicer
- pcap generation and upscaling
- Conclusion
Introduction
This tutorial describes the reduction of pcaps to the very significant packets to answer a specific question. Did it happen to you that your pcap was in the TByte range and you had no clue what’s in it and loading it into Wireshark is already at 1GB cumbersome. Or did you have a pcap recorded with 10GBit/s and you need a pcap with the same content and packet/flow statistics with 10 or even 100GBit/s? Then you are right here.
So one task is to reduce the pcap to the very significant part, hence downsize it to a manageable size. This is what we do everyday, so I had to find a way to solve that problem. There a two options:
pcapd is older and extracts
packets into a new pcap according to flow indices in different
operational modes of the Anteater. E.g. if T2 is in alarm mode pcaps are only extracted
if an alarm in an internal signalling block globalWarn is set. It was designed for
maximum flexibility to enable the user to configure T2 into an intelligent flow based
IDS. So mostly running on an interface. pcapd works on pcap and pcap-ng. Since version 0.9.0lmw2
it is a factor 100 faster.
Forensic guys might have several pcaps, and always different questions, then
pcapd
has to be invoked every time different flows have to be extracted.
This is avoided by findexer.
This plugin indexes all packets in the pcap. Currently it can only process pcaps.
Whenever your question changes you can select flows and store them into a pcap without
running T2 again, hence the drill down process is much faster.
The other task of pcap generation can only be achieved by pcapd. We come back to that topic later.
Preparation
First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Then compile the following plugins:
t2build tranalyzer2 basicFlow basicStats tcpFlags tcpStates protoStats txtSink
...
BUILD SUCCESSFUL
If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:
mkdir ~/data ~/results
The sample PCAPs used in this tutorial can be downloaded here: faf-exercise.pcap.
Please save it in your ~/data folder.
Note that pcapd and findexer were made for reduction of really large pcaps, but we cannot supply here TBytes for you. So imagine the sample pcap is very, very big. Or just use your own if you need something larger.
In addition, sniffing directly from the interface will be required, so have your sudo password ready.
Now you are all set for pcap extraction experiments.
How to select relevant flows for pcap extraction
In order to find interesting flows it is always good practice to look first at the end report and protocol statistics.
The rest of the standard plugins also provide pertinent information in the end report (if PLUGIN_REPORT in tranalyzer.h
is set to 1, which it is by default).
t2 -r ~/data/faf-exercise.pcap -w ~/results/
Only IPv4 TCP flows from 2009. All symmetric, all bandwidth measure the same, so we have all the packets. Good. Not much traffic, average packet load indicates downloads, and who is responsible? Probably the biggest talker from USA. Let’s look into the Anomaly flags.
tawk -V tcpAnomaly=0x02cc -V tcpFStat=0x4ff1The tcpAnomaly column with value 0x02cc is to be interpreted as follows: bit | tcpAnomaly | Description ============================================================================= 2 | 0x0004 | SEQ Fast retransmission 3 | 0x0008 | Duplicate ACK 6 | 0x0040 | Sequence number out-of-order 7 | 0x0080 | Sequence mess, rather spurious Retransmission 9 | 0x0200 | Previous packet not captured The tcpFStat column with value 0x4ff1 is to be interpreted as follows: bit | tcpFStat | Description ============================================================================= 0 | 0x0001 | Packet good for inter-distance assessment 4 | 0x0010 | Window state-machine initialized 5 | 0x0020 | Window update 6 | 0x0040 | Win 0 probe 7 | 0x0080 | Win 0 probe ACK 8 | 0x0100 | Min Window detected 9 | 0x0200 | WS used 10 | 0x0400 | Window full 11 | 0x0800 | Window state-machine count up(1)/down(0) 14 | 0x4000 | TCP Selective ACK Option
There are definitely problems with flow control, some retries, and the window size hit 0 for 3 times, nevertheless not a security problem.
Looking into the *_protocols.txt* file from the protoStats plugin reveals three well known ports: FTP, SMTP and HTTP.
cat ~/results/faf-exercise_protocols.txt
# Total packets: 5902 (5.90 K)
# Total bytes: 4993414 (4.99 M)
# L2/3 Protocol Packets Bytes Description
0x0800 5902 [100.00%] 4993414 [100.00%] Internet Protocol version 4 (IPv4)
# Total IPv4 packets: 5902 (5.90 K) [100.00%]
# Total IPv6 packets: 0 [0.00%]
# L4 Protocol Packets Bytes Description
6 5902 [100.00%] 4993414 [100.00%] Transmission Control Protocol (TCP)
# Total TCP packets: 5902 (5.90 K) [100.00%]
# Total TCP bytes: 4993414 (4.99 M) [100.00%]
# TCP Port Packets Bytes Description
21 22 [ 0.37%] 2595 [ 0.05%] File Transfer [Control]
25 894 [ 15.15%] 148980 [ 2.98%] Simple Mail Transfer Protocol (SMTP)
80 371 [ 6.29%] 304381 [ 6.10%] World Wide Web HTTP
64334 4615 [ 78.19%] 4537458 [ 90.87%]
# Total UDP packets: 0 [0.00%]
# Total UDP bytes: 0 [0.00%]Unencrypted protocols, not uncommon in 2009. Let’s assume FTP and HTTP is unusual for our network, we lately transferred to SFTP and HTTPS internally.
So we would like to load the plugins ftpDecode and httpSniffer. The mail we ignore for now, homework for the reader.
t2build ftpDecode httpSniffer
And execute t2 with that configuration:
... -------------------------------------------------------------------------------- basicStats: Biggest L3 talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets basicStats: Biggest L3 talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes tcpFlags: Aggregated ipFlags=0x0044 tcpFlags: Aggregated tcpFStat=0x4ff1 tcpFlags: Aggregated tcpFlags=0x071f tcpFlags: Aggregated tcpAnomaly=0x02cc tcpFlags: Number of TCP SYN retries, seq retries: 0, 27 tcpFlags: Number WinSz below 1: 3 [0.05%] tcpStates: Aggregated tcpStatesAFlags=0x4a ftpDecode: Aggregated ftpStat=0x0b ftpDecode: Number of FTP control packets: 22 [0.37%] ftpDecode: Number of FTP-DATA packets: 4612 (4.61 K) [78.14%] smtpDecode: Aggregated smtpStat=0x01 smtpDecode: Number of SMTP packets: 894 [15.15%] smtpDecode: Number of SMTP files: 3 httpSniffer: Aggregated httpStat=0x002c httpSniffer: Aggregated httpAFlags=0x5000 httpSniffer: Aggregated httpCFlags=0x0010 httpSniffer: Aggregated httpHeadMimes=0x0045 httpSniffer: Number of files img_vid_aud_msg_txt_app_unk: 3_0_0_0_10_7_0 httpSniffer: Number of HTTP packets: 3821 (3.82 K) [64.74%] httpSniffer: Number of HTTP GET requests: 16 [0.42%] -------------------------------------------------------------------------------- ...
If you want to decode the hex flags, use tawk -V option.
If you do not want to write the full name of the variable, just start
with, e.g., tcp and hit tab two times to trigger the autocompletion:
The ftpStat column with value 0x0b is to be interpreted as follows:
bit | ftpStat | Description
=============================================================================
0 | 0x01 | FTP control port found
1 | 0x02 | FTP passive parent flow
3 | 0x08 | FTP active parent flow
Not abnormal.
And the HTTP anomaly flags (httpAFlags):
The httpAFlags column with value 0x5000 is to be interpreted as follows: bit | httpAFlags | Description ============================================================================= 12 | 0x1000 | Possible EXE download 14 | 0x4000 | HTTP 1.0
So there is HTTP/1.0, very odd for 2009, and we download an executable. Not good!
What type of HTTP content are we facing here?
tawk -V httpHeadMimes=0x0045
The httpHeadMimes column with value 0x0045 is to be interpreted as follows:
bit | httpHeadMimes | Description
=============================================================================
0 | 0x0001 | Application
2 | 0x0004 | Image
6 | 0x0040 | Text
So applications, images and text. Let’s look at some pictures. Security guys should be interested now. And we have passive FTP around. According to company policy we should only see SSH and SFTP. So let’s dig into the flows.
So this is our question in tawk to the flow file:
tawk '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) { print } else { split($httpImg_Vid_Aud_Msg_Txt_App_Unk, A, "_"); if (A[1] > 0) print } }’ ~/results/faf-exercise_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipToS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpISeqN tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpBFlgtMx tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpMPTBF tcpMPF tcpMPAID tcpMPdssF tcpTmS tcpTmER tcpEcI tcpUtm tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStatesAFlags ftpStat ftpCDFindex ftpCC ftpRC ftpNumUser ftpUser ftpNumPass ftpPass ftpNumCP ftpCP httpStat httpAFlags httpMethods httpHeadMimes httpCFlags httpGet_Post httpRSCnt httpRSCode httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim httpImg_Vid_Aud_Msg_Txt_App_Unk httpHosts httpURL httpMimes httpCookies httpImages httpVideos httpAudios httpMsgs httpAppl httpText httpPunk httpBdyURL httpUsrAg httpXFor httpRefrr httpVia httpLoc httpServ httpPwr
B 3 0x0400000000004001 1258544216.915576 1258544217.008019 0.092443 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 73 18 95603 319 0 1380 1309.63 274.7284 0 0.021251 0.001266342 0.003059902 789.6758 1034183 0.6043956 0.9933488 0x0811 1 42177 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4272559795 73 95603 0 73 319 0 16560 5840 6432 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.00045 7.8e-05 0.014188 0.001192556 0.002669854 0.00849205 0.002675539 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_u7avi1777u1705ff.bin_3_B_1_0" "Apache"
B 5 0x0400000000004001 1258544217.357036 1258544217.413505 0.056469 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1262 6 26 9 30820 320 0 1380 1185.385 425.1664 0 0.017243 0.002171885 0.00369694 460.4296 545786.2 0.4857143 0.9794477 0x0811 1 20242 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4278713148 26 30820 0 26 320 0 9660 5840 6431.921 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000462 0.000214 0.010205 0.001545667 0.002568446 0.008867128 0.002571311 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_u7iavi2511u2510ff.bin_5_B_1_0" "Apache"
B 7 0x0400000000004001 1258544217.763049 1258544217.791016 0.027967 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1264 6 7 6 5268 317 0 1380 752.5714 567.7107 0 0.017745 0.003995285 0.005150571 250.295 188364.9 0.07692308 0.8864816 0x0811 1 53026 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4269177574 7 5268 0 7 317 0 5268 5840 6362.352 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000445 0.000423 0.010686 0.0025205 0.003097677 0.009895643 0.003103809 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_x8xplsb2_118c8.bin_7_B_1_0" "Apache"
B 9 0x0400000000004001 1258544218.137543 1258544218.165782 0.028239 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1266 6 5 6 1690 320 0 1380 338 441.2029 0 0.017993 0.005647799 0.005497161 177.0601 59846.31 -0.09090909 0.681592 0x0811 1 30104 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4270113443 5 1690 0 5 320 0 1690 5840 6289.861 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000466 0.000456 0.010956 0.002583333 0.003173773 0.009871334 0.003179277 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_x8xplsc_149d148c8.bin_9_B_1_0" "Apache"
B 11 0x0400000000004001 1258562478.266384 1258562509.653978 31.387594 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla Corporation" 80 192.168.1.104 07 "Private network" 1384 6 8 7 4184 476 0 1380 523 579.3142 0 23.04 3.923449 7.138864 0.2548778 133.3011 0.06666667 0.7957082 0x0811 59415 6116 52 52 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 531087527 8 4184 0 8 476 0 2760 5840 6383.246 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000411 0.000406 8.196325 1.171333 2.655198 4.101608 7.522519 0x00 0x00 0 0 0 0x0068 0x0000 0x00 0x0004 0x0010 0_0 1 200 0_1_0_1_0_0_0_0_0_1 1_0_0_0_0_0_0 "image/png" "_en-US_thunderbird_2.0.0.23_start_thunderbird_startpage.png_11_B_1_0" "1.1 varnish" "Apache/2.2.3 (CentOS)"
B 10 0x0400000000004001 1258562467.754689 1258562509.653962 41.899273 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla Corporation" 80 192.168.1.104 07 "Private network" 1379 6 18 14 15606 1801 0 1380 867 568.1608 0 22.97829 2.327737 5.515566 0.4296017 372.4647 0.125 0.7930717 0x0811 9916 55605 52 52 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3267889532 18 15606 0 18 1801 0 4140 5840 9616.309 5840 9648 0 4 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000207 0.000186 9.952202 1.338241 2.93412 2.651285 5.881808 0x00 0x00 0 0 0 0x0068 0x0000 0x00 0x0044 0x0010 0_0 2 302;200 0_1_1_1_1_0_0_0_0_2 2_0_0_0_2_0_0 "text/html";"image/jpeg" "_style_dalvay_bg-header-small.jpg_10_B_4_0";"_style_dalvay_main-feature.jpg_10_B_5_1" "_thunderbird_2.0.0.23_start__10_B_1_0";"_en-US_thunderbird_2.0.0.23_start__10_B_2_1" "1.1 varnish" "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/" "Apache/2.2.3 (CentOS)" "PHP/5.1.6"
A 33 0x0400000000004000 1258587444.865917 1258587445.631435 0.765518 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1908 198.189.255.75 us "California State University" 80 6 24 74 319 97078 0 319 13.29167 53.98931 0 0.574143 0.03189658 0.1023528 31.35132 416.7113 -0.5102041 -0.9934495 0x0011 1 2 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3626872219 23 319 0 23 95699 1 319 65535 65535 65535 65535 0 0 0 0 0x011b 0x0008 1 4 0x00000016 1460 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007304 0.007047 0.573488 0.01674978 0.06118462 0.007744 0 0x02 0x00 0 0 0 0x006c 0x4000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/u7avi1778u1705z7.bin" "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 33 0x0400000000004001 1258587444.873221 1258587445.638482 0.765261 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 74 24 97078 319 0 1380 1311.865 268.8814 0 0.573915 0.01034136 0.06162005 96.69904 126856.1 0.5102041 0.9934495 0x0811 33497 31968 57 57 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2668953963 72 99839 0 74 319 0 8280 5840 6432 5840 6432 0 1 1 0 0x031b 0x00c0 1 4 0x00000016 1380 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.00044 0.000183 0.010698 0.0008634167 0.001742813 0.01761319 0.06120944 0x02 0x00 0 0 0 0x0068 0x5000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_2_0 "application/octet-stream" "_softw_90_update_u7avi1778u1705z7.bin_33_B_1_0";"_softw_90_update_u7avi1778u1705z7.bin_33_B_5_1" "Apache"
A 34 0x0400000000004000 1258587445.990733 1258587446.040428 0.049695 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1910 198.189.255.75 us "California State University" 80 6 8 19 320 21634 0 320 40 88.41758 0 0.00999 0.006211875 0.002938672 160.982 6439.28 -0.4074074 -0.9708481 0x0011 1 1 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3544152804 7 320 0 7 21635 0 320 65535 65535 65535 65535 0 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007517 0.007043 0.007573 0.007338999 0.0001176775 0.007978 0 0x02 0x00 0 0 0 0x006c 0x4000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/u7iavi2512u2511z7.bin" "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 34 0x0400000000004001 1258587445.998250 1258587446.047471 0.049221 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 19 8 21634 320 0 1380 1138.632 455.4652 0 0.017522 0.002590579 0.003994345 386.0141 439527.8 0.4074074 0.9708481 0x0811 40938 24582 57 57 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2671895030 19 21635 0 19 320 0 5520 5840 6431.036 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000461 0.000402 0.010451 0.00174075 0.00275091 0.00907975 0.002753426 0x02 0x00 0 0 0 0x0068 0x5000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_u7iavi2512u2511z7.bin_34_B_1_0" "Apache"
A 36 0x0400000000004000 1258594163.408285 1258594191.015208 27.606923 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49330 143.166.11.10 us "Dell" 64334 6 1514 3101 0 4268858 0 0 0 0 0 5.58724 0.01823444 0.1478493 54.84132 0 -0.3438787 -1 0x49b1 1 223 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3392384642 1511 0 0 1511 4255056 437 0 8192 45402 0 64860 253 194 254 0.001981506 0x0416 0x0008 511 1536 0x0000003e 1460 4 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.078742 1e-06 0.67088 0.03865783 0.04056465 0.07920299 0 0x42 0x02 35 0 0 1 "/video/R79733.EXE" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
B 36 0x0400000000004001 1258594163.487027 1258594185.427506 21.940479 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 3101 1514 4268858 0 0 1380 1376.607 60.23097 0 0.67109 0.007075286 0.02745561 141.3369 194565.4 0.3438787 1 0x0c51 1 6365 111 111 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3177226489 2866 5233476 27 3099 0 0 28980 8192 64860 8192 64860 0 1 1 0 0x031b 0x02c4 1 4 0x00000016 1380 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000461 2e-06 5.587702 0.004777641 0.1447472 0.04343547 0.1503238 0x02 0x02 35 0 0 1 "/video/R79733.EXE" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
A 35 0x0400000000004000 1258594162.928342 1258594185.618346 22.690004 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49329 143.166.11.10 us "Dell" 21 6 11 11 92 1231 0 24 8.363636 8.41835 0 21.78007 2.062728 5.945361 0.484795 4.054649 0 -0.8609222 0x0811 1 2328 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2427598871 10 92 0 10 1231 0 24 8192 62176.56 8192 64860 8 1 9 0 0x001a 0x0000 1 4 0x00000016 1460 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.080252 0.077494 306.0649 29.85102 83.48595 0.08095 0 0x02 0x09 36 USER;PASS;TYPE;PASV;SIZE;RETR 1 "anonymous" 1 "IEUser@" 2 "I";"/video/R79733.EXE" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
B 35 0x0400000000004001 1258594163.008594 1258594491.683288 328.674694 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 21 192.168.1.105 07 "Private network" 49329 6 11 11 1231 92 0 950 111.9091 232.9224 0 306.2558 29.87952 83.53862 0.03346774 3.745345 0 0.8609222 0x0811 2732 26560 239 239 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 365320932 11 1231 0 11 92 0 971 4140 4214.603 4140 4232 0 6 1 0 0x061e 0x0000 1 2 0x00000014 1380 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000698 0.00045 0.194089 0.04303964 0.07018124 29.89405 83.48597 0x42 0x09 36 220;331;230;200;227;213;125;226 0 0 1 "125 Data connection already open; Transfer startin" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0Clearly unencrypted. And look at flow 35 and 36 an executable R79733.EXE downloaded. Damn bad. Imagine your pcap is 20TB large, then the flow files become large as well, in the GB range.
So you might be interested in reducing the pcap size first to the very interesting protocols. And then go into more detail.
This is what we do in the next section.
Extract flows with pcapd plugin
Extracting flow based information from the pcap to produce a downsized pcap can be achieved by the pcapd plugin. It functions in various modes, which can be configured in pcap.h.
pcapd
vi src/pcapd.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define PD_MODE_IN 0 // 0: extract flows listed in input file (if -e option was used),
// or extract flows if alarm bit is set (if -e option was not used)
// 1: dump all packets
#define PD_EQ 1 // whether to dump matching (1) or non-matching (0) flows
#define PD_OPP 0 // extract also the opposite flow, 0: don't
#define PD_MODE_OUT 0 // 0: one pcap,
// 1: one pcap per flow
#define PD_SPLIT 1 // whether or not to split output file (-W option)
#define PD_FORMAT 0 // Format of the input file (-e option):
// 0: flow index only,
// 1: flow file format
#define PD_TSHFT 0 // Timeshift
#define PDTTSFTS 0 // add by sec
#define PDTTSFTNMS 1 // add usec or nsec, depends on TSTAMP_PREC
#define PD_MACSHFT 0 // Mac shift last byte
#define PDMACSSHFT 1 // add to src mac
#define PDMACDSHFT 1 // add to dst mac
#define PD_VLNSHFT 0 // vlan shift
#define PDVLNSHFT 1 // add value inner vlan
#define PD_IPSHFT 0 // add to src, dst IP
#define PDIP4SHFTN 0x01000000 // add IP4 32 bit network order
#define PDIP6SHFTN 0x0100000000000000 // add IP6 shift last 64 bit network order
/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */
#define PD_MAX_FD 128 // Maximum number of simultaneously open file descriptors
#define PD_SUFFIX "_pcapd.pcap" // extension for generated pcap file
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...The constant PD_SPLIT controls the size of the resulting pcap files if
the -W option is selected in the T2 command line. Not important here,
so leave it at default setting.
PD_MODE_IN controls the basic mode of operation.
The idea behind the first mode (0) and the -e option is to use tawk to extract flows of interest
and then the pcapd plugin to create a PCAP with all those flows.
The first column must be the flow index, the rest is ignored
Lines starting with %, #, a space or a tab are ignored, along with empty lines.
Flows whose index appears in the -e file will be written to a file named PREFIX_PD_SUFFIX, where PREFIX is the value given to T2 in the -e option.
Note that if PD_EQ is set to 0, the flows whose index does NOT appear in the file will be dumped.
If the FL_ALARM bit is set by any plugin, such as regex_pcre or dnsDecode,
PD_OPP defines whether also the opposite flow is extracted, where
the very alarm did not occur. This is important for forensics and IDS mode purposes if you want to see the
full history of an alarm. It is also active when reading a flow index file, so we will switch it on.
If PD_MODE_IN is set to 1 and the -e option is NOT provided, a special alarm mode is activated
where any plugin can trigger pcapd to dump a flow. Will be discussed in the alarm mode tutorial.
PD_MAX_FD controls the number of active file handles, so that the number of open files does not exceed the OS limits.
This comes in handy when PD_MODE_OUT is set to 1. Then all packets of each flow are stored in a separate file.
We leave everything at default, because we work now on a pcap and generated a flow index file: ~/results/nudel.
Hence, we need the -e option in the T2 command line later on.
In order to accelerate the extraction process it is recommended to build pcapd
under another directory, so that only this plugin can be loaded. Hence, if
another question has to be applied to the pcap only the plugin folder needs be
selected by the -p option. Because pcapd is only needed temporarily, it
can be built under /tmp. Sure, you can build it with the other plugins, and
it would be easier, but then you execute all the other plugins loaded… a waste of time!
So for the sake of the exercise I’ll show you how different configurations are handled in parallel. Hence, we build an extra pcapd and configure it to extract both parts of a flow.
t2conf pcapd -D PD_OPP=1
t2build -p /tmp tranalyzer2 pcapd
Move to your results directory window now and produce a file
which contains all interesting flow indexes selected earlier by tawk. Just add
the variable $flowInd in the print statement as shown below:
tawk -H '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) { print $flowInd } else { split($httpImg_Vid_Aud_Msg_Txt_App_Unk, A, "_"); if (A[1] > 0) print $flowInd } }' ~/results/faf-exercise_flows.txt | uniq > ~/results/nudel
cat ~/results/nudel
3
5
7
9
11
10
33
34
36
35And tell pcapd that it should load the flow index file from the results directory with the -e option:
================================================================================ Tranalyzer 0.9.0(Anteater), Tarantula. PID: 6853 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: pcapd, 0.9.0 [INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K) [INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K) [INF] pcapd: 10 flow indices ... -------------------------------------------------------------------------------- pcapd: number of packets extracted: 4958 (4.96 K) [84.01%] -------------------------------------------------------------------------------- ...
As you can see T2 produces a new pcap called ~/results/nudel_pcapd.pcap. Rerun T2 on the new pcap and now we can also produce a packet file, because it now contains only flows of interest.
ls ~/results
faf-exercise_flows.txt faf-exercise_headers.txt faf-exercise_packets.txt faf-exercise_protocols.txt
nudel nudel_pcapd.pcap
t2 -r ~/results/nudel_pcapd.pcap -w ~/results/nudel -s
20 flows instead of 72. If you look at the resulting flow file, you actually see more flows than in
our tawk request on the original flow file above. Because not in every flow direction certain flags are set.
So you also get the return or originating flow now, which preserves context of a communication and avoids
unnecessary further tawk questions.
tcol ~/results/nudel_flows.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipToS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpISeqN tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpBFlgtMx tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpMPTBF tcpMPF tcpMPAID tcpMPdssF tcpTmS tcpTmER tcpEcI tcpUtm tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStatesAFlags ftpStat ftpCDFindex ftpCC ftpRC ftpNumUser ftpUser ftpNumPass ftpPass ftpNumCP ftpCP httpStat httpAFlags httpMethods httpHeadMimes httpCFlags httpGet_Post httpRSCnt httpRSCode httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim httpImg_Vid_Aud_Msg_Txt_App_Unk httpHosts httpURL httpMimes httpCookies httpImages httpVideos httpAudios httpMsgs httpAppl httpText httpPunk httpBdyURL httpUsrAg httpXFor httpRefrr httpVia httpLoc httpServ httpPwr
A 1 0x0400000000004000 1258544216.908284 1258544217.008468 0.100184 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1260 198.189.255.75 us "California State University" 80 6 18 73 319 95603 0 319 17.72222 61.73868 0 0.013738 0.005565778 0.00363383 179.6694 3184.141 -0.6043956 -0.9933488 0x0011 1 2 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 1942699277 17 319 0 17 95603 0 319 65535 65535 65535 65535 0 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007292 0.006796 0.007834 0.007299494 0.0001743189 0.007742 0 0x00 0x00 0 0 0 0x006c 0x0000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/u7avi1777u1705ff.bin" "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 1 0x0400000000004001 1258544216.915576 1258544217.008019 0.092443 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 73 18 95603 319 0 1380 1309.63 274.7284 0 0.021251 0.001266342 0.003059902 789.6758 1034183 0.6043956 0.9933488 0x0811 1 42177 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4272559795 73 95603 0 73 319 0 16560 5840 6432 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.00045 7.8e-05 0.014188 0.001192556 0.002669854 0.00849205 0.002675539 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_u7avi1777u1705ff.bin_1_B_1_0" "Apache"
A 2 0x0400000000004000 1258544217.349751 1258544217.413719 0.063968 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1262 198.189.255.75 us "California State University" 80 6 9 26 320 30820 0 320 35.55556 84.1992 0 0.009743 0.007107555 0.002012674 140.6953 5002.501 -0.4857143 -0.9794477 0x0011 1 2 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 998473205 8 320 0 8 30820 0 320 65535 65300.4 65075 65535 1 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007285 0.007036 0.00757 0.007321462 0.0001213493 0.007747 0 0x00 0x00 0 0 0 0x006c 0x0000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/u7iavi2511u2510ff.bin" "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 2 0x0400000000004001 1258544217.357036 1258544217.413505 0.056469 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1262 6 26 9 30820 320 0 1380 1185.385 425.1664 0 0.017243 0.002171885 0.00369694 460.4296 545786.2 0.4857143 0.9794477 0x0811 1 20242 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4278713148 26 30820 0 26 320 0 9660 5840 6431.921 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000462 0.000214 0.010205 0.001545667 0.002568446 0.008867128 0.002571311 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_u7iavi2511u2510ff.bin_2_B_1_0" "Apache"
A 3 0x0400000000004000 1258544217.755746 1258544217.791475 0.035729 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1264 198.189.255.75 us "California State University" 80 6 6 7 317 5268 0 317 52.83333 98.10986 0 0.010241 0.005954833 0.002865114 167.9308 8872.345 -0.07692308 -0.8864816 0x0011 1 2 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 935235267 5 317 0 5 5268 0 317 65535 65535 65535 65535 0 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007303 0.007039 0.007577 0.007375143 0.0001950087 0.007748 0 0x00 0x00 0 0 0 0x006c 0x0000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/x8xplsb2_118c8.bin" "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 3 0x0400000000004001 1258544217.763049 1258544217.791016 0.027967 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1264 6 7 6 5268 317 0 1380 752.5714 567.7107 0 0.017745 0.003995285 0.005150571 250.295 188364.9 0.07692308 0.8864816 0x0811 1 53026 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4269177574 7 5268 0 7 317 0 5268 5840 6362.352 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000445 0.000423 0.010686 0.0025205 0.003097677 0.009895643 0.003103809 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_x8xplsb2_118c8.bin_3_B_1_0" "Apache"
A 4 0x0400000000004000 1258544218.130258 1258544218.166240 0.035982 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1266 198.189.255.75 us "California State University" 80 6 6 5 320 1690 0 320 53.33333 99.03834 0 0.01049 0.005997 0.00290767 166.75 8893.335 0.09090909 -0.681592 0x0011 1 1 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 1909828099 5 320 0 5 1690 0 320 65535 65535 65535 65535 0 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007285 0.007037 0.007543 0.007288 0.0001869964 0.007751 0 0x00 0x00 0 0 0 0x006c 0x0000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/x8xplsc_149d148c8.bin" "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 4 0x0400000000004001 1258544218.137543 1258544218.165782 0.028239 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1266 6 5 6 1690 320 0 1380 338 441.2029 0 0.017993 0.005647799 0.005497161 177.0601 59846.31 -0.09090909 0.681592 0x0811 1 30104 57 57 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 4270113443 5 1690 0 5 320 0 1690 5840 6289.861 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000466 0.000456 0.010956 0.002583333 0.003173773 0.009871334 0.003179277 0x00 0x00 0 0 0 0x0068 0x1000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_x8xplsc_149d148c8.bin_4_B_1_0" "Apache"
A 6 0x0400000000004000 1258562478.204828 1258562509.633367 31.428539 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1384 63.245.221.11 us "Mozilla Corporation" 80 6 7 8 476 4184 0 476 68 138.7919 0 23.04004 4.489791 7.379 0.2227275 15.14547 -0.06666667 -0.7957082 0x0011 1 109 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 654559040 6 476 0 6 4184 0 476 65535 65535 65535 65535 0 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.061556 0.020611 23.03958 2.930275 7.038339 0.061967 0 0x00 0x00 0 0 0 0x006c 0x0000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_1_0_0 0_0_0_0_0_0_0 "www.mozillamessaging.com" "/en-US/thunderbird/2.0.0.23/start/thunderbird_startpage.png" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23" "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"
B 6 0x0400000000004001 1258562478.266384 1258562509.653978 31.387594 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla Corporation" 80 192.168.1.104 07 "Private network" 1384 6 8 7 4184 476 0 1380 523 579.3142 0 23.04 3.923449 7.138864 0.2548778 133.3011 0.06666667 0.7957082 0x0811 59415 6116 52 52 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 531087527 8 4184 0 8 476 0 2760 5840 6383.246 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000411 0.000406 8.196325 1.171333 2.655198 4.101608 7.522519 0x00 0x00 0 0 0 0x0068 0x0000 0x00 0x0004 0x0010 0_0 1 200 0_1_0_1_0_0_0_0_0_1 1_0_0_0_0_0_0 "image/png" "_en-US_thunderbird_2.0.0.23_start_thunderbird_startpage.png_6_B_1_0" "1.1 varnish" "Apache/2.2.3 (CentOS)"
A 5 0x0400000000004000 1258562467.749142 1258562509.633370 41.884228 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1379 63.245.221.11 us "Mozilla Corporation" 80 6 14 18 1801 15606 0 455 128.6429 183.7502 0 22.97809 2.991731 6.053773 0.3342547 42.99948 -0.125 -0.7930717 0x0811 1 91 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3115250060 13 1801 0 13 15606 0 455 65535 65497.7 64737 65535 2 2 2 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.005547 7.1e-05 22.97788 1.313044 5.097706 0.005754 0 0x00 0x00 0 0 0 0x006c 0x0000 0x02 0x0000 0x0010 4_0 0 4_0_0_0_0_1_0_1_0_0 0_0_0_0_0_0_0 "www.mozillamessaging.com" "/thunderbird/2.0.0.23/start/";"/en-US/thunderbird/2.0.0.23/start/";"/style/dalvay/bg-header-small.jpg";"/style/dalvay/main-feature.jpg" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23" "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"
B 5 0x0400000000004001 1258562467.754689 1258562509.653962 41.899273 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla Corporation" 80 192.168.1.104 07 "Private network" 1379 6 18 14 15606 1801 0 1380 867 568.1608 0 22.97829 2.327737 5.515566 0.4296017 372.4647 0.125 0.7930717 0x0811 9916 55605 52 52 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3267889532 18 15606 0 18 1801 0 4140 5840 9616.309 5840 9648 0 4 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000207 0.000186 9.952202 1.338241 2.93412 2.651285 5.881808 0x00 0x00 0 0 0 0x0068 0x0000 0x00 0x0044 0x0010 0_0 2 302;200 0_1_1_1_1_0_0_0_0_2 2_0_0_0_2_0_0 "text/html";"image/jpeg" "_style_dalvay_bg-header-small.jpg_5_B_4_0";"_style_dalvay_main-feature.jpg_5_B_5_1" "_thunderbird_2.0.0.23_start__5_B_1_0";"_en-US_thunderbird_2.0.0.23_start__5_B_2_1" "1.1 varnish" "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/" "Apache/2.2.3 (CentOS)" "PHP/5.1.6"
A 7 0x0400000000004000 1258587444.865917 1258587445.631435 0.765518 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1908 198.189.255.75 us "California State University" 80 6 24 74 319 97078 0 319 13.29167 53.98931 0 0.574143 0.03189658 0.1023528 31.35132 416.7113 -0.5102041 -0.9934495 0x0011 1 2 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3626872219 23 319 0 23 95699 1 319 65535 65535 65535 65535 0 0 0 0 0x011b 0x0008 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007304 0.007047 0.573488 0.01674978 0.06118462 0.007744 0 0x02 0x00 0 0 0 0x006c 0x4000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/u7avi1778u1705z7.bin" "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 7 0x0400000000004001 1258587444.873221 1258587445.638482 0.765261 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1908 6 74 24 97078 319 0 1380 1311.865 268.8814 0 0.573915 0.01034136 0.06162005 96.69904 126856.1 0.5102041 0.9934495 0x0811 33497 31968 57 57 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2668953963 72 99839 0 74 319 0 8280 5840 6432 5840 6432 0 1 1 0 0x031b 0x00c0 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.00044 0.000183 0.010698 0.0008634167 0.001742813 0.01761319 0.06120944 0x02 0x00 0 0 0 0x0068 0x5000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_2_0 "application/octet-stream" "_softw_90_update_u7avi1778u1705z7.bin_7_B_1_0";"_softw_90_update_u7avi1778u1705z7.bin_7_B_5_1" "Apache"
A 8 0x0400000000004000 1258587445.990733 1258587446.040428 0.049695 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1910 198.189.255.75 us "California State University" 80 6 8 19 320 21634 0 320 40 88.41758 0 0.00999 0.006211875 0.002938672 160.982 6439.28 -0.4074074 -0.9708481 0x0011 1 1 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3544152804 7 320 0 7 21635 0 320 65535 65535 65535 65535 0 0 0 0 0x011b 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.007517 0.007043 0.007573 0.007338999 0.0001176775 0.007978 0 0x02 0x00 0 0 0 0x006c 0x4000 0x02 0x0000 0x0010 1_0 0 1_0_0_0_0_1_0_0_0_0 0_0_0_0_0_0_0 "aa.avg.com" "/softw/90/update/u7iavi2512u2511z7.bin" "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="
B 8 0x0400000000004001 1258587445.998250 1258587446.047471 0.049221 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1910 6 19 8 21634 320 0 1380 1138.632 455.4652 0 0.017522 0.002590579 0.003994345 386.0141 439527.8 0.4074074 0.9708481 0x0811 40938 24582 57 57 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2671895030 19 21635 0 19 320 0 5520 5840 6431.036 5840 6432 0 1 1 0 0x031b 0x0000 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000461 0.000402 0.010451 0.00174075 0.00275091 0.00907975 0.002753426 0x02 0x00 0 0 0 0x0068 0x5000 0x00 0x0001 0x0010 0_0 1 200 0_0_0_1_0_0_0_0_0_1 0_0_0_0_0_1_0 "application/octet-stream" "_softw_90_update_u7iavi2512u2511z7.bin_8_B_1_0" "Apache"
A 10 0x0400000000004000 1258594163.408285 1258594191.015208 27.606923 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49330 143.166.11.10 us "Dell" 64334 6 1514 3101 0 4268858 0 0 0 0 0 5.58724 0.01823444 0.1478493 54.84132 0 -0.3438787 -1 0x49b1 1 223 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3392384642 1511 0 0 1511 4255056 437 0 8192 45402 0 64860 253 194 254 0.001981506 0x0416 0x0008 511 1536 0x0000003e 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.078742 1e-06 0.67088 0.03865783 0.04056465 0.07920299 0 0x42 0x02 9 0 0 1 "/video/R79733.EXE" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
B 10 0x0400000000004001 1258594163.487027 1258594185.427506 21.940479 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 3101 1514 4268858 0 0 1380 1376.607 60.23097 0 0.67109 0.007075286 0.02745561 141.3369 194565.4 0.3438787 1 0x0c51 1 6365 111 111 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3177226489 2866 5233476 27 3099 0 0 28980 8192 64860 8192 64860 0 1 1 0 0x031b 0x02c4 1 4 0x00000016 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000461 2e-06 5.587702 0.004777641 0.1447472 0.04343547 0.1503238 0x02 0x02 9 0 0 1 "/video/R79733.EXE" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
A 9 0x0400000000004000 1258594162.928342 1258594185.618346 22.690004 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49329 143.166.11.10 us "Dell" 21 6 11 10 92 1231 0 24 8.363636 8.41835 0 21.78007 2.062728 5.945361 0.484795 4.054649 0.04761905 -0.8609222 0x0811 1 2328 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2427598871 10 92 0 10 1231 0 24 8192 62176.56 8192 64860 8 1 9 0 0x001a 0x0000 1 4 0x00000016 1460 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.080252 0.077494 21.58924 2.22962 6.122048 0.08095 0 0x02 0x09 10 USER;PASS;TYPE;PASV;SIZE;RETR 1 "anonymous" 1 "IEUser@" 2 "I";"/video/R79733.EXE" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
B 9 0x0400000000004001 1258594163.008594 1258594185.427515 22.418921 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 21 192.168.1.105 07 "Private network" 49329 6 10 11 1231 92 0 950 123.1 241.7142 0 21.78333 2.241892 6.179659 0.4460518 54.90898 -0.04761905 0.8609222 0x0811 2732 26560 239 239 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 365320932 10 1207 0 10 92 0 971 4140 4207.147 4140 4232 0 6 1 0 0x021a 0x0000 1 2 0x00000014 1380 0 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000698 0.00045 0.194089 0.04303964 0.07018124 2.27266 6.122451 0x02 0x09 10 220;331;230;200;227;213;125;226 0 0 1 "125 Data connection already open; Transfer startin" 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0As we ran t2 with the -s option, a packet file was also created.
tcol ~/results/nudel_packets.txt
%pktNo flowInd flowStat time pktIAT pktTrip flowDuration numHdrs hdrDesc ethVlanID srcMac dstMac ethType srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto pktLen l7Len ipToS ipID ipIDDiff ipFrag ipTTL ipHdrChkSum ipCalChkSum l4HdrChkSum l4CalChkSum ipFlags ip6HHOptLen ip6HHOpts ip6DOptLen ip6DOpts ipOptLen ipOpts seq ack seqMax seqDiff ackDiff seqLen ackLen seqFlowLen ackFlowLen tcpMLen tcpBFlgt tcpFStat tcpFlags tcpAnomaly tcpWin tcpWS tcpMSS tcpTmS tcpTmER tcpOptLen tcpOpts tcpStatesAFlags ftpStat l7Content
1 1 0x0400000000004000 1258544216.908284 0.000000 0.000000 0.000000 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 Private network 1260 198.189.255.75 us California State University 80 6 66 0 0x00 1571 0 0x4000 128 0x6c8b 0x6c8b 0x44f1 0x44f1 0x0040 0 0 0 1942699277 0 1942699277 0 0 0 0 0 0 0 0 0x0010 0x0002 0x0000 65535 0 1460 0 0 8 0x02;0x04;0x05;0xb4;0x01;0x01;0x04;0x02 0x00 0x00
2 1 0x0400000000004001 1258544216.915576 0.000000 0.007292 0.000000 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1260 6 62 0 0x00 0 0 0x4000 57 0xb9ae 0xb9ae 0x1902 0x1902 0x0040 0 0 0 4272559795 1942699278 4272559795 0 0 0 0 0 0 0 0 0x0010 0x0212 0x0000 5840 0 1380 0 0 8 0x02;0x04;0x05;0x64;0x01;0x01;0x04;0x02 0x00 0x00
3 1 0x0400000000004000 1258544216.916026 0.007742 0.000450 0.007742 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 Private network 1260 198.189.255.75 us California State University 80 6 64 0 0x00 1572 1 0x4000 128 0x6c92 0x6c92 0x5c46 0x5c46 0x0040 0 0 0 1942699278 4272559796 1942699278 1 0 0 0 0 0 0 0 0x0011 0x0010 0x0000 65535 0 1460 0 0 0 0x00 0x00
4 1 0x0400000000004000 1258544216.929764 0.013738 0.014188 0.021480 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 Private network 1260 198.189.255.75 us California State University 80 6 377 319 0x00 1573 1 0x4000 128 0x6b52 0x6b52 0x4f1a 0x4f1a 0x0040 0 0 0 1942699278 4272559796 1942699278 0 0 0 0 0 0 319 319 0x0011 0x0018 0x0000 65535 0 1460 0 0 0 0x00 0x00 GET /softw/90/update/u7avi1777u1705ff.bin HTTP/1.1\r\nUser-Agent: AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA=\r\nHost: aa.avg.com\r\nAccept: */*\r\nAccept-Encoding: identity,deflate,gzip\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nx-avg-id:78-175947826\r\n\r\n
5 1 0x0400000000004001 1258544216.936827 0.021251 0.007063 0.021251 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1260 6 54 0 0x00 42177 42177 0x4000 57 0x14f5 0x14f5 0x41e7 0x41e7 0x0040 0 0 0 4272559796 1942699597 4272559796 1 319 0 319 0 319 0 0 0x0811 0x0010 0x0000 6432 0 1380 0 0 0 0x00 0x00
6 1 0x0400000000004001 1258544216.937559 0.000732 0.007795 0.021983 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us California State University 80 192.168.1.104 07 Private network 1260 6 1434 1380 0x00 42178 1 0x4000 57 0x0f90 0x0f90 0x266d 0x266d 0x0040 0 0 0 4272559796 1942699597 4272559796 0 0 0 0 0 319 1380 1380 0x0811 0x0010 0x0000 6432 0 1380 0 0 0 0x00 0x00 HTTP/1.1 200 OK\r\nServer: Apache\r\nETag: "0210a9516dd34abc481683f877bd8680:1258533754"\r\nLast-Modified: Wed, 18 Nov 2009 07:55:25 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 95323\r\nContent-Type: application/octet-stream\r\nDate: Wed, 18 Nov 2009 11:39:50 GMT\r\nConnection: keep-alive\r\n\r\nMZ AVG7 UpdateBin grp(avi:1777)dep(avi:1705)tm(0911180855)pri(2)..7TW;........"....m.b...YbU..&..6.P.B.....jx.\n.n..%....g...8......c....X.c.sO..M............Y.7|......q........w/mb.D#...:.`.H|..(.:...wjA/...u....C{.]..7.y..8..v....n.5..L.k..U>&te...-.....a...`..n. h.....0.......9Ig.s..7^.)..,........ .R..+...f ...xg..xq....;1...F.|....)..*..~.%.I.o.*......)...P...w.V.q....41....h...w%o..,Ha;.~}..#!.p....{..w.=A.0...8..IB.;.*...]..w.@..%F[L9(.. ..`..Iq...'......4.&..........Gz0S}`...s.....s...6\).4(..x.J..[do...w./..m..[.X.D...z\.. ..F...\nA[....O_...."..te..|b..".......\t..e-..i.q....<&h....SKz.gR.+.<1....n........|...-...B..?..".../.g.I@..m[s....iu3$.t\tL...`...D$..eff..7(.L.\V_..HR!.X.........A#....=...K.[.>..CO.2J...R...k.k.p..ME...\}.v..l_.D\t...D...;c......0~3:A......i..7X&..].@.......k?..Qn........,c.`..K.\t.B.M........~\.....>..|._. ...W.YP.....N...u.....s@:..Z.z..n.."B..Q.M.9..D[.c.z.l...z.G....l..6.yPJ.8.........Q.eE.....oPK.'.s. ..(....+..3........."q...d.....v....@......q..+. _YK.`.Zn.c..a..E.q...cI......c....\r0..\n.... ]p..Z=.{./Iz..'..<.d...9...]:...P.}v<...9.h...T9cf../<..U.L
...Have a look into it and see what you can find. Have fun!
Another method is the alarm support mode and the findexer which we will discuss in the following chapters.
Extract pcaps using the FL_ALARM bit
If t2 -e option is not present, pcapd extracts all packets if a plugin
sets an internal flag in flow status using the macro:
T2_SET_STATUS(flowP, FL_ALARM)We will have an example in the Alarm controlled packet extraction chapter of the Alarm mode tutorial and one in the regex based pcap extraction tutorial.
In general, findexer is the fastest and most efficient way to extract flows to pcaps,
as you only need to run it once, and then you can change your extraction rule as
often as you want. pcapd always requires a rerun of t2.
The superior way: findexer plugin
If you do not want to build an alarming tool for anomalies which extracts packets from on plugin request or if you have a stack of questions to a large pcap, and you do not want to rerun T2 all over again several times, the findexer plugin is the tool of choice.
So compile findexer in your plugin repository and rerun T2 on the original pcap.
t2build findexer
t2 -r ~/data/faf-exercise.pcap -w ~/results/================================================================================ Tranalyzer 0.9.0(Anteater), Tarantula. PID: 7098 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: protoStats, 0.9.0 02: basicFlow, 0.9.0 03: basicStats, 0.9.0 04: tcpFlags, 0.9.0 05: tcpStates, 0.9.0 06: ftpDecode, 0.9.0 07: httpSniffer, 0.9.0 08: txtSink, 0.9.0 09: findexer, 0.9.0 [INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481438 (481.44 K) [INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41486 (41.49 K) Processing file: /home/wurst/data/faf-exercise.pcap Link layer type: Ethernet [EN10MB/1] ...
findexer produces a binary ~/results/faf-exercise_flows.xer file, which contains all references between flow indexes and packets.
Move to your results bash window or move to the results directory and do a ls then you spot it.
faf-exercise_flows.txt faf-exercise_flows.xer faf-exercise_headers.txt faf-exercise_icmpStats.txt faf-exercise_protocols.txt nudel nudel_pcapd.pcap nudel_flows.txt
nudel_headers.txt nudel_icmpStats.txt nudel_protocols.txt
Now invoke the tawk command which selects the flows from the pcapd section and add the -x option to produce a pcap.
[INFO] Processing findexer : /home/wurst/results/faf-exercise_flows.xer [INFO] Processing pcap 1/1 : /home/wurst/data/faf-exercise.pcap [INFO] Extracted 4959 packets.
And ls will show you the knoedel.pcap file.
faf-exercise_flows.txt faf-exercise_flows.xer faf-exercise_headers.txt faf-exercise_icmpStats.txt faf-exercise_protocols.txt knoedel.pcap nudel nudel_pcapd.pcap nudel_flows.txt
nudel_headers.txt nudel_icmpStats.txt nudel_protocols.txt
You may now run T2 on knoedel.pcap and you will see only the selected flows as in the pcapd section.
The main difference is that you can run another tawk command with a different question and you will get right away
a new pcap without re-invoking T2. E.g. a troubleshooting question:
Select all flows where the window size was reported to 0, so the receiver could not process more packets anymore.
Use the TCP Window size threshold ratio variable: tcpWinSzThRt from tcpFlags (threshold is 1 by default).
I leave this exercise to the reader.
findexer and Wireshark export
There are people who like to work with Wireshark or tshark instead of t2 packet files.
Fair enough, a bit more complicated if you have a lot of data.
So tawk comes to the rescue. Simply invoke tawk with the same question as above, but this time, add the -k option:
[INFO] Processing findexer : /home/wurst/results/faf-exercise_flows.xer [INFO] Processing pcap 1/1 : /home/wurst/data/faf-exercise.pcap ** (wireshark:9617) 14:26:09.519872 [Capture MESSAGE] -- Capture Start ... [INFO] Extracted 4959 packets. ** (wireshark:9617) 14:26:09.770420 [Capture MESSAGE] -- Capture started ** (wireshark:9617) 14:26:09.770512 [Capture MESSAGE] -- File: "/tmp/wireshark_-B89ZG1.pcapng" ** (wireshark:9617) 14:26:09.917255 [Capture MESSAGE] -- Capture stopped.
If Wireshark complains that you do not have rights as a user then use the following command
sudo gpasswd -a YOUR_USERNAME wireshark
And log off and on again. Then try the tawk above again and you will have
Wireshark open with knoedel.pcap.
How cool is that?
Management of humongous flow files: tawk ffsplit
Very large pcaps produce large flow files, if you do not aggregate enough or if
you have too many plugins. Right! There is a way to split these files.
One way is to use the -W option in T2 which produces many smaller flow files
instead of one large file.
If you are interested in segregating several protocols, that is possible
with tawk and the ffsplit function:
ffsplit(s, k, h):
Split the input file into smaller more manageable files.
The files to create can be specified as argument to the function (one comma
separated string) (See `s`).
If no argument is specified, creates one file per column whose name ends
with Stat, e.g., dnsStat, and one for pwxType (pw) and covertChannels (cc)
`s` Valid arguments for 's' are:
- arp - bgp - dhcp - dns - ftp - http
- icmp - igmp - irc - modbus - nat - ntp
- ospf - pop - radius - sctp - smb - smtp
- ss - stp - syslog - tftp - voip - vrrp
- pw (cleartext passwords)
- cc (covert channels)
`h` Valid arguments for 'h' are:
- if h is empty or omitted: keep the default columns
default = "dir,flowInd,timeFirst,timeLast,duration,ethVlanID,
srcIP,srcIP4,srcIP6,dstIP,dstIP4,dstIP6,srcPort,dstPort,l4Proto"
- if h is a non-existent column, e.g., 0: keep only relevant fields
- if h starts with '+', e.g., "+flowStat,hdrDesc", keep default fields
and the ones specified
- else keep fields specified in h, e.g., "srcIP,dstIP"
Parameters:
- s: a comma separated list of files to create (see `s`).
- k: 0: keep all columns, 1: keep only relevant columns
- h: comma separated list of fields to keep in each file (see `h`)
Dependencies:
- None
Examples:
- tawk 'ffsplit()' file.txt
- tawk 'ffsplit("dns")' file.txt
- tawk 'ffsplit("cc,dns,http")' file.txt
- tawk 'ffsplit("", 1)' file.txt
- tawk 'ffsplit("dns,http", 1, "+flowStat,hdrDesc")' file.txt
- tawk 'ffsplit("", 1, "srcIP,dstIP")' file.txt
Let’s try the simplest way first, extract all protocols in separate flow files:
tawk 'ffsplit()' ~/results/faf-exercise_flows.txt
ls ~/results
faf-exercise_flows_ftp.txt faf-exercise_flows_icmp.txt faf-exercise_flows.xer faf-exercise_icmpStats.txt knoedel.pcap nudel_pcapd.pcap
faf-exercise_flows_http.txt faf-exercise_flows.txt faf-exercise_headers.txt faf-exercise_protocols.txt nudel
So you see that now for each existing protocol a different flow file is produced. Where is the SMTP? Right, we didn’t load the smtpDecode plugin. Was your home work, remember? Let’s look into FTP:
tcol ~/results/faf-exercise_flows_ftp.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipToS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpISeqN tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpBFlgtMx tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpMPTBF tcpMPF tcpMPAID tcpMPdssF tcpTmS tcpTmER tcpEcI tcpUtm tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStatesAFlags ftpStat ftpCDFindex ftpCC ftpRC ftpNumUser ftpUser ftpNumPass ftpPass ftpNumCP ftpCP smtpStat smtpCC smtpRC smtpUsr smtpPW smtpSANum smtpESANum smtpERANum smtpSA smtpESA smtpERA httpStat httpAFlags httpMethods httpHeadMimes httpCFlags httpGet_Post httpRSCnt httpRSCode httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim httpImg_Vid_Aud_Msg_Txt_App_Unk httpHosts httpURL httpMimes httpCookies httpImages httpVideos httpAudios httpMsgs httpAppl httpText httpPunk httpBdyURL httpUsrAg httpXFor httpRefrr httpVia httpLoc httpServ httpPwr
A 36 0x0400000000004000 1258594163.408285 1258594191.015208 27.606923 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49330 143.166.11.10 us "Dell" 64334 6 1514 3101 0 4268858 0 0 0 0 0 5.58724 0.01823444 0.1478493 54.84132 0 -0.3438787 -1 0x49b1 1 223 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3392384642 1511 0 0 1511 4255056 437 0 8192 45402 0 64860 253 194 254 0.001981506 0x0416 0x0008 511 1536 0x0000003e 1460 4 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.078742 1e-06 0.67088 0.03865783 0.04056465 0.07920299 0 0x42 0x02 35 0 0 1 "/video/R79733.EXE" 0x00 0 0 0 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
B 36 0x0400000000004001 1258594163.487027 1258594185.427506 21.940479 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 64334 192.168.1.105 07 "Private network" 49330 6 3101 1514 4268858 0 0 1380 1376.607 60.23097 0 0.67109 0.007075286 0.02745561 141.3369 194565.4 0.3438787 1 0x0c51 1 6365 111 111 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 3177226489 2866 5233476 27 3099 0 0 28980 8192 64860 8192 64860 0 1 1 0 0x031b 0x02c4 1 4 0x00000016 1380 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000461 2e-06 5.587702 0.004777641 0.1447472 0.04343547 0.1503238 0x02 0x02 35 0 0 1 "/video/R79733.EXE" 0x00 0 0 0 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
A 35 0x0400000000004000 1258594162.928342 1258594185.618346 22.690004 1 3 eth:ipv4:tcp 00:08:74:38:01:b4 00:19:e3:e7:5d:23 0x0800 192.168.1.105 07 "Private network" 49329 143.166.11.10 us "Dell" 21 6 11 11 92 1231 0 24 8.363636 8.41835 0 21.78007 2.062728 5.945361 0.484795 4.054649 0 -0.8609222 0x0811 1 2328 128 128 0 0x00 0x0040 0 0x00_0x00000000 0_0 0x00000000_0x00000000 2427598871 10 92 0 10 1231 0 24 8192 62176.56 8192 64860 8 1 9 0 0x001a 0x0000 1 4 0x00000016 1460 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.080252 0.077494 306.0649 29.85102 83.48595 0.08095 0 0x02 0x09 36 USER;PASS;TYPE;PASV;SIZE;RETR 1 "anonymous" 1 "IEUser@" 2 "I";"/video/R79733.EXE" 0x00 0 0 0 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0
B 35 0x0400000000004001 1258594163.008594 1258594491.683288 328.674694 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:08:74:38:01:b4 0x0800 143.166.11.10 us "Dell" 21 192.168.1.105 07 "Private network" 49329 6 11 11 1231 92 0 950 111.9091 232.9224 0 306.2558 29.87952 83.53862 0.03346774 3.745345 0 0.8609222 0x0811 2732 26560 239 239 0 0x00 0x0044 0 0x00_0x00000000 0_0 0x00000000_0x00000000 365320932 11 1231 0 11 92 0 971 4140 4214.603 4140 4232 0 6 1 0 0x061e 0x0000 1 2 0x00000014 1380 1 0x0000 0x00 0 0x00 0 0 0 0.000000 0.000000 0.000698 0.00045 0.194089 0.04303964 0.07018124 29.89405 83.48597 0x42 0x09 36 220;331;230;200;227;213;125;226 0 0 1 "125 Data connection already open; Transfer startin" 0x00 0 0 0 0x0000 0x0000 0x00 0x0000 0x0010 0_0 0 0_0_0_0_0_0_0_0_0_0 0_0_0_0_0_0_0Yep all FTP. Does that help you? Give us feedback please.
If somebody is interested in why it is so easy to produce all kinds of functionality with
tawk, the answer is in the fextractor which does all the work of extracting all
packets belonging to a given flow, using the .xer file.
Working on findexed pcaps: fextractor
Imagine you have only a distinct selection of pcaps to analyze, and they are big,
so that you do not want to run Tranalyzer every time you change your question,
or you select certain flows. And imagine you want to script everything
yourself without using tawk -k because you want to do your own thing. Fair enough,
then fextractor is your tool of choice. Note that tawk -k uses fextractor as well.
Let’s have a look at the help first
fextractor -h
Usage: fextractor -r INPUT[:start][,end] (-w OUTPUT | -n) [OPTIONS]... \
[[DIR@]FLOWINDEX[:start][,end]]...
Extract the flows FLOWINDEX using the _flows.xer INTPUT generated by Tranalyzer2 findexer plugin.
Alternatively use a list of findexer files generated by Tranalyzer2 -W option from index start
to end. The extracted flows are written to the OUTPUT pcap.
An optional packet range can be provided on each command line FLOWINDEX to only extract packets
in the range [start, end] of this flow. If start or end are omitted, they are replaced by,
respectively, the first and the last available packets in the flow. The FLOWINDEX can also
optionally be prefixed with a direction A or B, by default both directions are extracted.
OPTIONS:
-r INPUT[:start][,end]
either read packet indexes from a single _flows.xer file named INPUT
or read packet indexes from multiple _flows.xer files prefixed by INPUT
and with suffix in range [start, end]. If start or end are omitted,
they are replaced by, respectively, first and last available XER files.
-w OUTPUT write packets to pcap file OUTPUT
OUTPUT "-" means that the PCAP is written to stdout.
-f overwrite OUTPUT if it already exists
-n print oldest PCAP still available, its first packet timestamp and exit
-h print this help message
-i FILE read flow indexes from FILE. FILE can either be in _flows.txt format
(flow index in 2nd tab-separated column), or have one flow index per line.
FILE "-" means that flows are read from stdin.
-b by default when FILE is in _flows.txt format, only directions present in
it are extracted, this option force both directions to be extracted even if
only the A or B direction is present in the flow file.
-s N skip the first N PCAPs
-p DIR search pcaps in DIR
should only be set if pcaps were moved since Tranalyzer2 was runThe -r option allows you to process any pcap which is indexed by findexer. If you have several
files which are indexed, e.g. nudel1, nudel2, … then use the colon notation.
The -w option denotes the location and name where the extracted pcap should be written.
The -i option denotes the flow index file. From our earlier experiment with pcapd above we
still have our flow index file: nudel. So move to your results folder and invoke
fextractor using the already generated findexer faf-exercise_flows.xer file.
The -b option enables to extract also the opposite flow, which might not match
the extraction condition. Nevertheless, this is very useful for forensics purposes
to have the full traffic being involved in one pcap.
fextractor -i ~/results/nudel -r ~/results/faf-exercise_flows.xer -w ~/results/nudelxer.pcap
ls ~/results
faf-exercise_flows.txt faf-exercise_flows.xer faf-exercise_headers.txt faf-exercise_icmpStats.txt faf-exercise_protocols.txt
nudel nudel.pcap nudelxer.pcap
Now you may change the ~/results/nudel file and re-invoke the same fextractor command without
running T2 ever again, if you are only interested in the same pcap.
fextractor as flow based pcap slicer
If your pcap is really large, and you want to split it into smaller chunks we advice to use the -c option
of tcpdump. But if you want to sort the packets according to their flow affiliation into separate
packet files the fextractor is your friend. Below is a script which creates small pcaps, where each
pcap contains one flow.
#!/usr/bin/env bash
if [ -z "$1" ]; then
echo "Usage: ./t2pef numFlows"
exit 1
fi
shopt -s expand_aliases
source "$HOME/tranalyzer2/scripts/t2_aliases"
for ((i = 1; i <= $1; i++)) do
echo $i > ~/results/nudel
fextractor -i ~/results/nudel -r ~/results/faf-exercise_flows.xer -w ~/results/nudelxer$i.pcap
doneNote that a substantial number of pcap files will be created which might disturb your OS. The pcapd plugin might be faster here.
pcap generation and upscaling
So you have a pcap at low speed and want to convert it to high speed with the same statistic properties. pcapd will come to the rescue.
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define PD_MODE_PKT 0 // 0: all packets, 1: activate packet range selection
#define PD_STRTPKT 1 // Packet at which processing starts (PD_MODE_PKT=1)
#define PD_ENDPKT 10 // 0: End of flow, > 0: Packet at which processing ends (PD_MODE_PKT=1)
#define PD_MODE_IN 0 // 0: extract flows listed in input file (if -e option was used),
// or extract flows if alarm bit is set (if -e option was not used)
// 1: dump all packets
#define PD_LBSRCH 1 // search 1: binary, 0: linear
#define PD_EQ 1 // whether to dump matching (1) or non-matching (0) flows
#define PD_OPP 0 // if PD_MODE_IN extract also the opposite flow, 0: don't
#define PD_MODE_OUT 0 // 0: one pcap,
// 1: one pcap per flow
#define PD_SPLIT 1 // whether or not to split output file (-W option)
#define PD_TSHFT 0 // Timeshift
#define PD_TTSFTS 0 // Add value to secs
#define PD_TTSFTNMS 1 // Add value to usec/nsec (depends on TSTAMP_PREC in tranalyzer.h)
#define PD_MACSHFT 0 // MAC shift last byte
#define PD_MACSSHFT 1 // Add value to src MAC
#define PD_MACDSHFT 1 // Add value to dst MAC
#define PD_VLNSHFT 0 // VLAN ID shift
#define PD_VLNISHFT 1 // Add value to inner VLAN ID
#define PD_IPSHFT 0 // Add to src/dst IP
#define PD_IP4SHFT 0x00000001 // Add value to IPv4 32 bit
#define PD_IP6SHFT 0x0000000000000001 // Add value to IPv6 shift last 64 bit
#define PD_TTLSHFT 0 // 0: no TTL change, 1: TTL shift, 2: random shift
#define PD_TTL 8 // sub value from TTL
#define PD_TTLMOD 128 // TTL modulo
#define PD_CHKSUML3 0 // Correct checksum in IPv4 header
...
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...Conclusion
That’s it! Now reset the configuration of core and all plugins for later tutorials:
t2conf -a --reset
Have fun experimenting!