t2py: control and operate T2 with Python

Python

Introduction

t2py is a python library which can be used to control and operate Tranalyzer2. It can be used as an alternative to t2conf, t2build and other scripts.

Caveats

This library is still experimental and at an early stage of development. As such, the API may be subject to change.

Bug reports, feature requests, feedback and suggestions are welcome and can be addressed directly to Andy.

Dependencies

Required dependencies

python3 -m pip install multipledispatch

Optional dependencies

The following dependencies are only required for specific operations:

pandas is only required for the T2.to_pandas() and T2Utils.to_pandas() functions.
pdoc3 is only required to generate the API documentation as HTML.

python3 -m pip install pandas pdoc3

Getting started

Setup

If you installed Tranalyzer with the setup.sh script (or are sourcing t2_aliases in your ~/.bashrc file), then you should be ready to use t2py:

t2py

Otherwise, you must tell Python where to find the t2py module. This is achieved with the PYTHONPATH environment variable:

Make sure $T2HOME exists:

echo $T2HOME
```
/home/user/tranalyzer2-0.9.2/
```
If it does not exist, make sure to set it:

export T2HOME="/home/user/tranalyzer2-0.9.2/"

echo $T2HOME
```
/home/user/tranalyzer2-0.9.2/
```
In order to be able to import the t2py module in your current terminal, run the following command:

export PYTHONPATH="$PYTHONPATH:$T2HOME/scripts/"
If you want this change to be permanent, add the following line to your ~/.bashrc (or ~/.zshrc, …):

export PYTHONPATH="$PYTHONPATH:/home/user/tranalyzer2-0.9.2/scripts/"

Now you are all set to start using t2py!

Available modules

t2py provides the following modules:

T2Utils: provide wrappers around Tranalyzer scripts and utilities
T2Plugin: represent a Tranalyzer2 plugin
T2: manage a session (set of plugins, configuration changes, flow file, …)

Import

The simplest way to get started is to use the t2py alias, which will start an interactive Python session and import all the modules:

t2py

Welcome to t2py, Python interface to Tranalyzer2!
Type "help(T2Utils)", "help(T2Plugin)" or "help(T2)" for more information.
>>>

If you want to work on specific plugins, you can give the plugin names to t2py and the requested T2Plugin will be automatically created for you:

t2py myPlugin1 myPlugin2

>>> myPlugin1

<t2py.T2Plugin.T2Plugin object at 0x123456789>

>>> myPlugin2

<t2py.T2Plugin.T2Plugin object at 0x123456789>

If you want to do it the hard way, first start an interactive Python session, then import the T2Plugin module and finally create your plugins as T2Plugin objects:

python3

>>> from t2py import T2Plugin

>>> myPlugin1 = T2Plugin(\'myPlugin1\')

>>> myPlugin2 = T2Plugin(\'myPlugin2\')

You can import each module separately, several or all of them at the same time:

>>> from t2py import *                      # Import all modules
>>> from t2py import T2, T2Plugin, T2Utils  # Import all modules
>>> from t2py import T2                     # Import the T2 module
>>> from t2py import T2Plugin               # Import the T2Plugin module
>>> from t2py import T2Utils                # Import the T2Utils module
>>> from t2py import T2, T2Plugin           # Import the T2 and T2Plugin modules

The following sections simply list the available variables and methods. For more details and examples, refer to the t2py API documentation.

T2Utils.py: simple wrapper around Tranalyzer2 scripts and utilities

For more details, refer to the t2py.T2Utils API.

>>> from t2py import T2Utils

# Read-Only Properties

>>> T2Utils.T2HOME    # -> str
>>> T2Utils.T2PLHOME  # -> str

>>> T2Utils.T2BUILD   # -> str
>>> T2Utils.T2CONF    # -> str
>>> T2Utils.T2FM      # -> str
>>> T2Utils.T2PLUGIN  # -> str
>>> T2Utils.TAWK      # -> str

# Static Functions

>>> T2Utils.apply_config(
...     plugin: str,
...     infile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.build(
...     plugin: Union[str, List[str]],
...     plugin_folder: str = None,
...     force_rebuild: bool = False,
...     debug: bool = False,
...     verbose: bool = False
... )
>>> T2Utils.clean(
...     plugin: Union[str, List[str]],
...     verbose: bool = False
... )
>>> T2Utils.create_pcap_list(
...     pcaps: List[str],
...     outfile: str = None
... )
>>> T2Utils.create_plugin_list(
...     plugins: List[str],
...     outfile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.follow_stream(
...     filename: str,
...     flow: int,
...     output_format: Union[int, str] = 2,
...     direction: str = None,
...     payload_format: Union[int, str] = 4,
...     reassembly: bool = True,
...     colors: bool = True
... ) -> Union[str, List[str], List[bytearray], List[Dict[str, Any]]]
>>> T2Utils.generate_config(
...     plugin: str,
...     outfile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.get_config(
...     plugin: str,
...     name: str,
...     infile: str = None
... ) -> Any
>>> T2Utils.get_config_from_source(
...     plugin: str,
...     name: str
... ) -> Any
>>> T2Utils.get_default(
...     plugin: str,
...     name: str
... ) -> Any
>>> T2Utils.list_config(
...     plugin: str
... ) -> List[str]
>>> T2Utils.list_plugins(
...     infile: str = None
... ) -> List[str]
>>> T2Utils.load_plugins(
...     plugin: Union[str, List[str]] = None
... )
>>> T2Utils.network_interfaces() -> List[str]
>>> T2Utils.plugin_description(
...     plugin: str
... ) -> str
>>> T2Utils.plugin_number(
...     plugin: str
... ) -> str
>>> T2Utils.plugins(
...     category: Union[str, List[str]] = None
... ) -> List[str]
>>> T2Utils.reset_config(
...     plugin: Union[str, List[str]],
...     name: Union[str, List[str]] = None,
...     outfile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.run_tranalyzer(
...     pcap: str = None,
...     iface: str = None,
...     pcap_list: Union[str, List[str]] = None,
...     output_prefix: str = None,
...     log_file: bool = False,
...     monitoring_file: bool = False,
...     packet_mode: bool = False,
...     plugin_folder: str = None,
...     loading_list: str = None,
...     plugins: List[str] = None,
...     bpf: str = None,
...     t2_exec: str = None,
...     timeout: int = None,
...     verbose: bool = False
... )
>>> T2Utils.set_config(
...     plugin: str,
...     name: str,
...     value: Any,
...     outfile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.set_config(
...     plugin: str,
...     name_val: Dict[str, Any],
...     outfile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.set_default(
...     plugin: Union[str, List[str]],
...     name: Union[str, List[str]] = None,
...     outfile: str = None,
...     verbose: bool = False
... )
>>> T2Utils.t2_exec(
...     debug: bool = False
... ) -> str
>>> T2Utils.tawk(
...     program: str = None,
...     filename: str = None,
...     options: List[str] = None
... ) -> str
>>> T2Utils.to_json_array(
...     infile: str,
...     delimiter: str = '\t'
... ) -> List[Dict[str, Any]]
>>> T2Utils.to_pandas(
...     infile: str,
...     delimiter: str = None
... ) -> pandas.core.frame.DataFrame
>>> T2Utils.to_pdf(
...     pcap: str = None,
...     flow_file: str = None,
...     prefix: str = None,
...     config: bool = True,
...     reset_config: bool = True,
...     open_pdf: bool = True,
...     verbose: bool = False
... )
>>> T2Utils.unload(
...     plugin: Union[str, List[str]],
...     plugin_folder: str = None,
...     verbose: bool = False
... )
>>> T2Utils.valid_plugin_names() -> List[str]

T2Plugin.py: represent a plugin

For more details, refer to the t2py.T2Plugin API.

>>> from t2py import T2Plugin

# Constructor

>>> myPlugin = T2Plugin(
...     name: str,
...     config: str = None
... )

# Read-Only Properties

>>> myPlugin.changes      # -> Dict[str, Any]
>>> myPlugin.default      # -> Dict[str, Any]
>>> myPlugin.description  # -> str
>>> myPlugin.flags        # -> List[str]
>>> myPlugin.name         # -> str
>>> myPlugin.number       # -> str

# Read/Write Properties

>>> myPlugin.config_file  # -> str, default: None

# Functions

>>> myPlugin.apply_changes(
...     outfile: str = None,
...     verbose: bool = False
... )
>>> myPlugin.build(
...     plugin_folder: str = None,
...     force_rebuild: bool = False,
...     debug: bool = False,
...     verbose: bool = False
... )
>>> myPlugin.clean(
...     verbose: bool = False
... )
>>> myPlugin.diff(
...     base: str = None
... ) -> Dict[str, Any]
>>> myPlugin.discard_changes()
>>> myPlugin.generate_config(
...     outfile: str = None,
...     verbose: bool = False
... )
>>> myPlugin.get_default(
...     name: str
... ) -> Any
>>> myPlugin.list_config() -> List[str]
>>> myPlugin.load_config(
...     config: str
... )
>>> myPlugin.reset(
...     name: Union[str, List[str]] = None
... )
>>> myPlugin.save_config(
...     outfile: str = None,
...     verbose: bool = False
... )
>>> myPlugin.set_default(
...     name: Union[str, List[str]] = None
... )
>>> myPlugin.status()
>>> myPlugin.unload(
...     plugin_folder: str = None,
...     verbose: bool = False
... )

# In addition, configuration flags can be accessed as follows:

>>> myPlugin.MY_PLUGIN_FLAG
>>> myPlugin.MY_PLUGIN_FLAG = myNewValue

T2.py: manage several plugins and run T2, convert/display flow file, …

For more details, refer to the t2py.T2 API.

>>> from t2py import T2

# Constructor

>>> t2 = T2(
...     pcap: str = None,
...     iface: str = None,
...     pcap_list: Union[str, List[str]] = None,
...     output_prefix: str = None,
...     monitoring_file: bool = False,
...     packet_mode: bool = False,
...     plugin_folder: str = None,
...     loading_list: str = None,
...     plugins: List[str] = None,
...     output_format: Union[str, List[str]] = None,
...     bpf: str = None,
...     streaming: bool = False
... )

# Read-Only Properties

>>> t2.default_plugin_folder  # -> str
>>> t2.plugins                # -> Dict[str, T2Plugin]
>>> t2.t2_exec                # -> str
>>> t2.tranalyzer2            # -> T2Plugin

# Read/Write Properties

>>> t2.plugin_folder          # -> str
>>> t2.loading_list           # -> str
>>> t2.streaming              # -> bool

# Functions

>>> t2.add_output_format(
...     extension: Union[str, List[str]]
... )
>>> t2.add_plugin(
...     plugin: str
... )
>>> t2.add_plugins(
...     plugins: List[str]
... )
>>> t2.apply_changes(
...     verbose: bool = False
... )
>>> t2.build(
...     plugin: Union[str, List[str]] = None,
...     plugin_folder: str = None,
...     force_rebuild: bool = False,
...     debug: bool = False,
...     verbose: bool = False
... )
>>> t2.clean(
...     plugin: Union[str, List[str]] = None,
...     verbose: bool = False
... )
>>> t2.clear_plugins()
>>> t2.create_plugin_list(
...     plugins: List[str] = None,
...     outfile: str = None,
...     verbose: bool = False
... )
>>> t2.discard_changes()
>>> t2.flow_file() -> str
>>> t2.flow_file_json() -> str
>>> t2.flow_file_txt() -> str
>>> t2.flows() -> List[Dict[str, Any]]
>>> t2.flows_json() -> List[Dict[str, Any]]
>>> t2.flows_txt(
...     delimiter: str = None
... ) -> List[Dict[str, Any]]
>>> t2.follow_stream(
...     flow: int,
...     output_format: Union[int, str] = 2,
...     direction: str = None,
...     payload_format: Union[int, str] = 4,
...     reassembly: bool = True,
...     colors: bool = True
... ) -> Union[str, List[str], List[bytearray], List[Dict[str, Any]]]
>>> t2.headers() -> str
>>> t2.headers_file() -> str
>>> t2.list_plugins()
>>> t2.log() -> str
>>> t2.log_file() -> str
>>> t2.monitoring() -> List[Dict[str, Any]]
>>> t2.monitoring_file() -> str
>>> t2.packet_file() -> str
>>> t2.packets() -> List[Dict[str, Any]]
>>> t2.print_flows()
>>> t2.print_flows_json()
>>> t2.print_flows_txt()
>>> t2.print_headers()
>>> t2.print_log()
>>> t2.print_monitoring()
>>> t2.print_packets()
>>> t2.print_report()
>>> t2.remove_plugin(
...     plugin: str
... )
>>> t2.remove_plugins(
...     plugins: List[str]
... )
>>> t2.report() -> str
>>> t2.reset()
>>> t2.run(
...     pcap: str = None,
...     iface: str = None,
...     pcap_list: Union[str, List[str]] = None,
...     output_prefix: str = None,
...     monitoring_file: bool = False,
...     packet_mode: bool = False,
...     plugins: List[str] = None,
...     plugin_folder: str = None,
...     loading_list: str = None,
...     bpf: str = None,
...     rebuild: bool = False,
...     streaming: bool = False,
...     timeout: int = None,
...     verbose: bool = False
... )
>>> t2.set_plugins(
...     plugins: List[str] = None
... )
>>> t2.status()
>>> t2.stream() -> Iterator[Dict[str, Any]]
>>> t2.to_pandas(
...     infile: str = None,
...     delimiter: str = None
... ) -> pandas.core.frame.DataFrame
>>> t2.unload(
...     plugin: Union[str, List[str]] = None,
...     plugin_folder: str = None,
...     verbose: bool = False
... )

# In addition, each plugin is accessible as a T2Plugin object:

>>> t2.myPlugin

API documentation

For the complete documentation with examples, refer to the t2py API documentation.

Getting help

To list the variables and functions available, run one of the following commands:

>>> dir(T2)
>>> dir(T2Plugin)
>>> dir(T2Utils)

The documentation for each module can be accessed as follows:

>>> help(T2)
>>> help(T2Plugin)
>>> help(T2Utils)

Sample sessions using t2py

Let’s see how to use t2py for real!

Simple operations with the T2Utils module

In this example, we will be configuring, building and performing some simple operations with the T2Utils module.

First, open a terminal and start a t2py session:

t2py

Ok, let’s start by configuring and building the basicStats plugin:

>>> T2Utils.list_config('basicStats')
['BS_AGGR_CNT', 'BS_REV_CNT', 'BS_MOD', 'BS_STATS', 'BS_PL_STATS', 'BS_IAT_STATS', 'BS_VAR', 'BS_STDDEV', 'BS_XCLD', 'BS_XMIN', 'BS_XMAX']
>>> T2Utils.get_config('basicStats', 'BS_AGGR_CNT')
0
>>> T2Utils.set_config('basicStats', 'BS_AGGR_CNT', 1)
>>> T2Utils.get_config('basicStats', 'BS_AGGR_CNT')
1

>>> T2Utils.build('basicStats')


Plugin 'basicStats'

...


basicStats successfully built


Plugin basicStats copied into /home/user/.tranalyzer/plugins


BUILDING SUCCESSFUL

Let’s build tranalyzer2 and add the basicFlow, tcpStates and txtSink plugins, so we produce something useful!

>>> T2Utils.build(['tranalyzer2', 'basicFlow', 'tcpStates', 'txtSink'])


'Tranalyzer2'

...


Plugin 'basicFlow'


Plugin 'tcpStates'


Plugin 'txtSink'



BUILDING SUCCESSFUL

We are now ready to run tranalyzer2 against a PCAP file! We just have to tell T2 which plugins to load, to activate the packet mode and to instruct T2 to save the results in the ~/results folder:

>>> T2Utils.run_tranalyzer(
...     pcap='/home/user/data/annoloc2.pcap',
...     output_prefix='/home/user/results/',
...     plugins=['basicFlow', 'basicStats', 'tcpStates', 'txtSink'],
...     packet_mode=True)


================================================================================
Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 4414
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.14
    02: basicStats, 0.8.14
    03: txtSink, 0.8.14
...

Let’s reset the configuration of the basicStats plugin:

>>> T2Utils.reset_config('basicStats')

Plugin configuration with the T2Plugin module

In this example, we will be working with the basicStats plugin.

First, open a terminal and start a t2py session:

t2py basicStats

Let us make sure the plugin was successfully created:

>>> basicStats
<t2py.T2Plugin.T2Plugin object at 0x7f1136c07550>

First of all, let us inspect some basic properties of the plugin, namely its name, description and number:

>>> basicStats.name
'basicStats'
>>> basicStats.description
'Basic statistics'
>>> basicStats.number
'120'

Second, let us inspect the available flags and their default values:

>>> basicStats.flags
['BS_AGGR_CNT', 'BS_REV_CNT', 'BS_MOD', 'BS_STATS', 'BS_PL_STATS', 'BS_IAT_STATS', 'BS_VAR', 'BS_STDDEV', 'BS_XCLD', 'BS_XMIN', 'BS_XMAX']
>>> basicStats.default
{'BS_AGGR_CNT': 'no', 'BS_REV_CNT': 'yes', 'BS_MOD': 0, 'BS_STATS': 1, 'BS_PL_STATS': 1, 'BS_IAT_STATS': 1, 'BS_VAR': 'no', 'BS_STDDEV': 'yes', 'BS_XCLD': 0, 'BS_XMIN': 1, 'BS_XMAX': 'UINT16_MAX'}```

Let us inspect and then toggle the BS_AGGR_CNT and BS_VAR and BS_STDDEV flags:

>>> basicStats.BS_AGGR_CNT
0
>>> basicStats.BS_AGGR_CNT = 1
>>> basicStats.BS_AGGR_CNT
1
>>> basicStats.BS_VAR
0
>>> basicStats.BS_VAR = 1
>>> basicStats.BS_VAR
1
>>> basicStats.BS_STDDEV
1
>>> basicStats.BS_STDDEV = 0
>>> basicStats.BS_STDDEV
0

Note that we could have used the special values 'yes' and 'no' instead of 0 and 1:

>>> basicStats.BS_VAR
1
>>> basicStats.BS_VAR = 'yes'
>>> basicStats.BS_VAR
1

Let us inspect our (pending) changes:

>>> basicStats.changes
{'BS_AGGR_CNT': 1, 'BS_VAR': 1, 'BS_STDDEV': 0}

Looks good! Let’s apply our changes!

>>> basicStats.apply_changes()

Let us now inspect our pending changes again:

>>> basicStats.changes
{}

OK, no more pending changes! But let us make sure the changes have been applied:

>>> basicStats.diff()
{'BS_AGGR_CNT': 1, 'BS_VAR': 1, 'BS_STDDEV': 0}

Excellent, now we can build the plugin:

>>> basicStats.build()


Plugin 'basicStats'

...


basicStats successfully built


Plugin basicStats copied into /home/user/.tranalyzer/plugins


BUILDING SUCCESSFUL

Now we can either run T2 with the T2 module, the T2Utils module or via a standard shell command:

t2 -r /home/user/data/annoloc2.pcap -w /home/user/results/

When we do not need the basicStats plugin anymore, we can unload it (remove it from the plugin folder) and clean up the temporary building files:

>>> basicStats.unload()


Plugin 'basicStats'


UNLOADING SUCCESSFUL

>>> basicStats.clean()


...

CLEANING SUCCESSFUL

It is a good habit to restore a plugin configuration to its default once finished:

>>> basicStats.reset()

Let us inspect the changes using the status() function this time:

>>> basicStats.status()


3 changes pending:
    BS_AGGR_CNT = 0
    BS_STDDEV = 1
    BS_VAR = 0

OK, let us now apply the changes:

>>> basicStats.apply_changes()

Tranalyzer2 execution session with the T2 module

In this example, we will be working with Tranalyzer2, a set of plugins and a pcap file.

First, open a terminal and start a t2py session:

t2py

All the parameters can be specified when the T2 object is created or later by using the appropriate properties or functions as illustrated in this example.

>>> t2 = T2()
>>> t2
<t2py.T2.T2 object at 0x7fb6965c0ca0>

Let us start by selecting some plugins:

>>> t2.plugins
{}
>>> t2.add_plugins(['basicFlow', 'basicStats', 'tcpFlags'])
>>> t2.plugins
{'basicFlow': <t2py.T2Plugin.T2Plugin object at 0x102cd0be0>, 'basicStats': <t2py.T2Plugin.T2Plugin object at 0x102cd0e80>, 'tcpFlags': <t2py.T2Plugin.T2Plugin object at 0x102d4f8b0>}
>>> t2.add_plugin('tcpStates')
>>> t2.list_plugins()
['basicFlow', 'basicStats', 'tcpFlags', 'tcpStates']

OK, now it is time to choose an appropriate output format:

>>> t2.add_output_format(['json', 'txt.gz'])
>>> t2.list_plugins()
['basicFlow', 'basicStats', 'jsonSink', 'tcpFlags', 'tcpStates', 'txtSink']

Nice, the required sink plugins were automatically added!

Each plugin is represented by a T2Plugin object, and can be independently configured as discussed in the previous Plugin Configuration with the T2Plugin Module section.

As a reminder, let us activate BS_AGGR_CNT in basicStats:

>>> t2.basicStats.BS_AGGR_CNT = 1
>>> t2.basicStats.status()


1 change pending:
    BS_AGGR_CNT = 1

Let us now configure tranalyzer2 itself! It is also available in the T2 object as a T2Plugin object:

>>> t2.tranalyzer2
<t2py.T2Plugin.T2Plugin object at 0x1030b5e20>
>>> t2.tranalyzer2.SCTP_ACTIVATE = 1
>>> t2.tranalyzer2.changes
{'SCTP_ACTIVATE': 1}

OK, time to start thinking about running tranalyzer! We want to select a PCAP file and output the flow file in the ~/results/ folder:

>>> t2.pcap = '/home/user/data/annoloc2.pcap'
>>> t2.output_prefix = '/home/user/results/'

Now we are almost set! We still have to apply our changes and then run tranalyzer.

But before that, let us inspect all the changes pending, using the status() function on the T2 object:

>>> t2.status()


Tranalyzer2 [1 change pending]:
    SCTP_ACTIVATE = 1

Plugins:
    1: basicFlow
    2: basicStats [1 change pending]
            BS_AGGR_CNT = 1
    3: tcpFlags
    4: tcpStates
    5: txtSink [1 change pending]
            TFS_GZ_COMPRESS = 1
    6: jsonSink

Tranalyzer options:
    -r /home/user/data/annoloc2.pcap
    -w /home/user/results
    -l

Looks good! Let us apply the changes and build everything:

>>> t2.apply_changes()
>>> t2.status()


Plugins:
    1: basicFlow
    2: basicStats
    3: tcpFlags
    4: tcpStates
    5. txtSink
    6. jsonSink

Tranalyzer options:
    -r /home/user/data/annoloc2.pcap
    -w /home/user/results
    -l

>>> t2.build()

Time to run tranalyzer! Let us do a run with the packet mode and a BPF filter:

>>> t2.run(packet_mode=True, bpf='icmp or tcp or udp')

Now it is time to analyze the output! Let’s start with the header and log files:

>>> t2.headers()
'# Date: 1628090651.486362 sec (Wed 04 Aug 2021 17:24:11 CEST)\n# Tranalyzer 0.8.14 (Anteater), Tarantula.\n# Core configuration: L2, IPv4, IPv6, SCTP\n# SensorID: 666\n# PID: 57975\n# Command line: /home/user/.tranalyzer/plugins/bin/tranalyzer -l -r /home/user/data/annoloc2.pcap -w /home/user/results/ -b /tmp/plugins.load -s icmp or tcp or udp\n# HW info:...\n'

>>> t2.print_headers()

# Date: 1628090651.486362 sec (Wed 04 Aug 2021 17:24:11 CEST)
# Tranalyzer 0.8.14 (Anteater), Tarantula.
# Core configuration: L2, IPv4, IPv6, SCTP
# SensorID: 666
# PID: 57975
# Command line: /home/user/.tranalyzer/plugins/bin/tranalyzer -l -r /home/user/data/annoloc2.pcap -w /home/user/results/ -b /tmp/plugins.load -s icmp or tcp or udp
...
# Plugins loaded:
#   01: basicFlow, version 0.8.14
#   02: basicStats, version 0.8.14
#   03: tcpFlags, version 0.8.14
#   04: tcpStates, version 0.8.14
#   05: txtSink, version 0.8.14
#   06: jsonSink, version 0.8.14
#
# Col No.	Type	Name	Description
1	C	dir	Flow direction
2	U64	flowInd	Flow index
3	H64	flowStat	Flow status and warnings
4	U64.U32	timeFirst	Date time of first packet
5	U64.U32	timeLast	Date time of last packet
6	U64.U32	duration	Flow duration
...

The log (report) file can be accessed with t2.log()/t2.print_log() or t2.report()/t2.print_report():

>>> t2.log()
'================================================================================\nTranalyzer 0.8.14 (Anteater), Tarantula. PID: 57975\n================================================================================\n\x1b[1;34m[INF] \x1b[0;34mCreating flows for L2, IPv4, IPv6, SCTP\x1b[0m\nActive plugins:\n    01: basicFlow, 0.8.14\n    02: basicStats, 0.8.14\n    03: tcpFlags, 0.8.14\n    04: tcpStates, 0.8.14\n    05: txtSink, 0.8.14\n    06: jsonSink, 0.8.14\n\x1b[1;34m[INF] \x1b[0;34mIPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 405798 (405.80 K)\x1b[0m\n\x1b[1;34m...'

>>> t2.print_report()


================================================================================
Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 57975
================================================================================
[INF] Creating flows for L2, IPv4, IPv6, SCTP
Active plugins:
    01: basicFlow, 0.8.14
    02: basicStats, 0.8.14
    03: tcpFlags, 0.8.14
    04: tcpStates, 0.8.14
    05: txtSink, 0.8.14
    06: jsonSink, 0.8.14
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 405798 (405.80 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 50883 (50.88 K)
Processing file: /home/user/data/annoloc2.pcap
[INF] BPF: icmp or tcp or udp
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 UTC)
...

Now the good stuff, namely the flow file:

>>> t2.flow_file()
'/home/user/results/annoloc2_flows.json'
>>> t2.flows()
[..., {'dir': 'B', 'flowInd': 1030, 'flowStat': '0x0400000200004001', 'timeFirst': '1022171701.877349', 'timeLast': '1022171726.639232', 'duration': '24.761883', 'numHdrDesc': 1, 'numHdrs': [3], 'hdrDesc': ['eth:ipv4:tcp'], 'srcMac': ['00:01:02:b4:36:56'], 'dstMac': ['00:d0:02:6d:78:00'], 'ethType': '0x0800', 'srcIP': '138.212.187.109', 'srcIPCC': 'jp', 'srcIPOrg': 'ASAHI KASEI CORPORATION', 'srcPort': 80, 'dstIP': '133.26.84.187', 'dstIPCC': 'jp', 'dstIPOrg': 'Meiji University', 'dstPort': 4766, 'l4Proto': 6, 'numPktsSnt': 2729, 'numPktsRcvd': 1692, 'numPktsRTAggr': 4421, 'numBytesSnt': 3970812, 'numBytesRcvd': 0, 'numBytesRTAggr': 3970812, 'minPktSz': 0, 'maxPktSz': 1460, 'avePktSize': 1455.0428466796875, 'stdPktSize': 70.65760803222656, 'minIAT': 0.0, 'maxIAT': 0.48004499077796936, 'aveIAT': 0.009073597379028797, 'stdIAT': 0.03961425647139549, 'pktps': 110.209716796875, 'bytps': 160359.859375, 'pktAsm': 0.23456232249736786, 'bytAsm': 1.0, 'tcpFStat': '0x0040', 'ipMindIPID': 1, 'ipMaxdIPID': 8, 'ipMinTTL': 64, 'ipMaxTTL': 64, 'ipTTLChg': 0, 'ipToS': '0x00', 'ipFlags': '0x1840', 'ipOptCnt': 0, 'ipOptCpCl_Num': ['0x00', '0x00000000'], 'ip6OptCntHH_D': [0, 0], 'ip6OptHH_D': ['0x00000000', '0x00000000'], 'tcpISeqN': 1167186881, 'tcpPSeqCnt': 2653, 'tcpSeqSntBytes': 4535488, 'tcpSeqFaultCnt': 64, 'tcpPAckCnt': 0, 'tcpFlwLssAckRcvdBytes': 0, 'tcpAckFaultCnt': 2, 'tcpInitWinSz': 6432, 'tcpAveWinSz': 6432.0, 'tcpMinWinSz': 6432, 'tcpMaxWinSz': 6432, 'tcpWinSzDwnCnt': 0, 'tcpWinSzUpCnt': 0, 'tcpWinSzChgDirCnt': 0, 'tcpWinSzThRt': 0.0, 'tcpFlags': '0x98', 'tcpAnomaly': '0xb800', 'tcpOptPktCnt': 0, 'tcpOptCnt': 0, 'tcpOptions': '0x00000000', 'tcpMSS': 0, 'tcpWS': 1, 'tcpMPTBF': '0x0000', 'tcpMPF': '0x00', 'tcpMPAID': 0, 'tcpMPdssF': '0x00', 'tcpTmS': 0, 'tcpTmER': 0, 'tcpEcI': 0.0, 'tcpUtm': 0.0, 'tcpBtm': '0.000000', 'tcpSSASAATrip': 0.0, 'tcpRTTAckTripMin': 0.0, 'tcpRTTAckTripMax': 0.4159089922904968, 'tcpRTTAckTripAve': 0.0032711897511035204, 'tcpRTTAckTripJitAve': 0.018527768552303314, 'tcpRTTSseqAA': 0.018186353147029877, 'tcpRTTAckJitAve': 0.043254632502794266, 'tcpStatesAFlags': '0x03'}]

Let’s now extract all the unique source/destination IP pairs involved in an ICMP flow!

>>> set([(flow['srcIP'], flow['dstIP']) for flow in t2.flows() if flow['l4Proto'] == 1])
{..., ('138.212.189.88', '134.16.128.111'), ('138.212.186.88', '83.136.35.130'), ('138.212.188.82', '201.121.75.109'), ('201.249.165.159', '138.212.188.105'), ('138.212.185.86', '83.216.12.156'), ('138.212.187.10', '201.116.180.70'), ('138.212.189.88', '83.17.142.210'), ('138.212.18.252', '138.212.244.18'), ('138.212.18.249', '138.212.138.130')}

Note that for the _flows.txt file, all values are treated as strings, so to match ICMP, you would need to write flow['l4Proto'] == '1' instead.

Let’s start investigating the biggest talkers in terms of flows with Panda:

>>> df = t2.to_pandas()
>>> df.srcIP.value_counts()

138.212.189.66     652
138.212.187.11     369
138.212.187.186    288
138.212.188.38     202
138.212.184.165    200
                  ...
138.212.237.213      1
19.172.52.122        1
192.201.34.255       1
138.212.184.94       1
138.212.226.85       1
Name: srcIP, Length: 5671, dtype: int64

One last example displaying the biggest talkers in terms of ICMP flows:

>>> df.srcIP.where(df.l4Proto == 1).value_counts()

138.212.187.10    85
138.212.189.88    46
138.212.18.252    40
138.212.187.74    31
138.212.185.86    31
                  ..
201.96.12.71       1
138.37.10.174      1
138.212.184.48     1
68.8.139.82        1
193.107.159.17     1
Name: srcIP, Length: 213, dtype: int64

Streaming flows with the T2 module

One last feature of the T2 module we need to look at is the so-called streaming mode. We will keep working with the session created in the previous section, so let’s quickly review the status of our session:

>>> t2.status()


Plugins:
    1: basicFlow
    2: basicStats
    3: tcpFlags
    4: tcpStates
    5: txtSink
    6: jsonSink

Tranalyzer options:
    -r /home/user/data/annoloc2.pcap
    -w /home/user/results
    -s
    -l
    BPF filter: icmp or tcp or udp

Good! Now, let us activate the streaming mode and see what changes:

>>> t2.streaming = True
>>> t2.status()


Plugins:
    1: basicFlow
    2: basicStats
    3: tcpFlags
    4: tcpStates
    5: txtSink
    6: jsonSink
    7: socketSink [1 change pending]
            SKS_CONTENT_TYPE = 2

Tranalyzer options:
    -r /home/user/data/annoloc2.pcap
    -w /home/user/results
    -s
    -l
    BPF filter: icmp or tcp or udp

Aha, the socketSink plugin has been added and configured… Let’s apply the changes and build the socketSink plugin:

>>> t2.socketSink.apply_changes()
>>> t2.socketSink.build()


Plugin 'socketSink'

...

BUILDING SUCCESSFUL

Note that we could have used t2.apply_changes() and t2.build() to rebuild everything instead.

Now, the time to run t2 again has come:

>>> t2.run()
Setting up streaming server 127.0.0.1:6666
>>>

It looks like t2.run() is already finished, but actually it is simply running in a separate thread! How do we access the flows now? Simply by using the t2.stream() generator!

First, let us import the time module to simulate slow operations:

>>> import time

Ok, now the fun part:

>>> for flow in t2.stream():
...     print(flow)    # Or better yet, do some expensive computation, like sending the flow to a DB
...     time.sleep(4)  # simulate a slow operation...

{'dir': 'A', 'flowInd': 263, 'flowStat': '0x0400000000004000', 'timeFirst': '1022171701.709116', 'timeLast': '1022171701.709116', 'duration': '0.000000', 'numHdrDesc': 1, 'numHdrs': [3], 'hdrDesc': ['eth:ipv4:tcp'], 'srcMac': ['00:d0:02:6d:78:00'], 'dstMac': ['00:50:fc:0e:21:56'], 'ethType': '0x0800', 'srcIP': '209.171.12.143', 'srcIPCC': 'ca', 'srcIPOrg': 'TELUS Communications Inc', 'srcPort': 4987, 'dstIP': '138.212.185.230', 'dstIPCC': 'jp', 'dstIPOrg': 'ASAHI KASEI CORPORATION', 'dstPort': 41250, 'l4Proto': 6, 'numPktsSnt': 1, 'numPktsRcvd': 0, 'numPktsRTAggr': 1, 'numBytesSnt': 0, 'numBytesRcvd': 0, 'numBytesRTAggr': 0, 'minPktSz': 0, 'maxPktSz': 0, 'avePktSize': 0.0, 'stdPktSize': 0.0, 'minIAT': 0.0, 'maxIAT': 0.0, 'aveIAT': 0.0, 'stdIAT': 0.0, 'pktps': 0.0, 'bytps': 0.0, 'pktAsm': 1.0, 'bytAsm': 0.0, 'tcpFStat': '0x0040', 'ipMindIPID': 65535, 'ipMaxdIPID': 0, 'ipMinTTL': 117, 'ipMaxTTL': 117, 'ipTTLChg': 0, 'ipToS': '0x00', 'ipFlags': '0x0840', 'ipOptCnt': 0, 'ipOptCpCl_Num': ['0x00', '0x00000000'], 'ip6OptCntHH_D': [0, 0], 'ip6OptHH_D': ['0x00000000', '0x00000000'], 'tcpISeqN': 1471189413, 'tcpPSeqCnt': 0, 'tcpSeqSntBytes': 0, 'tcpSeqFaultCnt': 0, 'tcpPAckCnt': 0, 'tcpFlwLssAckRcvdBytes': 0, 'tcpAckFaultCnt': 0, 'tcpInitWinSz': 0, 'tcpAveWinSz': 0.0, 'tcpMinWinSz': 0, 'tcpMaxWinSz': 0, 'tcpWinSzDwnCnt': 0, 'tcpWinSzUpCnt': 0, 'tcpWinSzChgDirCnt': 0, 'tcpWinSzThRt': 1.0, 'tcpFlags': '0xc4', 'tcpAnomaly': '0x0000', 'tcpOptPktCnt': 0, 'tcpOptCnt': 0, 'tcpOptions': '0x00000000', 'tcpMSS': 0, 'tcpWS': 1, 'tcpMPTBF': '0x0000', 'tcpMPF': '0x00', 'tcpMPAID': 0, 'tcpMPdssF': '0x00', 'tcpTmS': 0, 'tcpTmER': 0, 'tcpEcI': 0.0, 'tcpUtm': 0.0, 'tcpBtm': '0.000000', 'tcpSSASAATrip': 0.0, 'tcpRTTAckTripMin': 65535.0, 'tcpRTTAckTripMax': 0.0, 'tcpRTTAckTripAve': 0.0, 'tcpRTTAckTripJitAve': 0.0, 'tcpRTTSseqAA': 0.0, 'tcpRTTAckJitAve': -1.0, 'tcpStatesAFlags': '0xc3'}
{'dir': 'A', 'flowInd': 444, 'flowStat': '0x0400000000004000', 'timeFirst': '1022171701.721366', 'timeLast': '1022171701.721366', 'duration': '0.000000', 'numHdrDesc': 1, 'numHdrs': [3], 'hdrDesc': ['eth:ipv4:tcp'], 'srcMac': ['00:d0:02:6d:78:00'], 'dstMac': ['00:50:fc:3b:62:78'], 'ethType': '0x0800', 'srcIP': '217.41.129.13', 'srcIPCC': 'gb', 'srcIPOrg': 'BT Infrastructure Layer', 'srcPort': 58872, 'dstIP': '138.212.187.186', 'dstIPCC': 'jp', 'dstIPOrg': 'ASAHI KASEI CORPORATION', 'dstPort': 80, 'l4Proto': 6, 'numPktsSnt': 1, 'numPktsRcvd': 0, 'numPktsRTAggr': 1, 'numBytesSnt': 0, 'numBytesRcvd': 0, 'numBytesRTAggr': 0, 'minPktSz': 0, 'maxPktSz': 0, 'avePktSize': 0.0, 'stdPktSize': 0.0, 'minIAT': 0.0, 'maxIAT': 0.0, 'aveIAT': 0.0, 'stdIAT': 0.0, 'pktps': 0.0, 'bytps': 0.0, 'pktAsm': 1.0, 'bytAsm': 0.0, 'tcpFStat': '0x0040', 'ipMindIPID': 65535, 'ipMaxdIPID': 0, 'ipMinTTL': 44, 'ipMaxTTL': 44, 'ipTTLChg': 0, 'ipToS': '0x00', 'ipFlags': '0x0840', 'ipOptCnt': 0, 'ipOptCpCl_Num': ['0x00', '0x00000000'], 'ip6OptCntHH_D': [0, 0], 'ip6OptHH_D': ['0x00000000', '0x00000000'], 'tcpISeqN': 3186340143, 'tcpPSeqCnt': 0, 'tcpSeqSntBytes': 0, 'tcpSeqFaultCnt': 0, 'tcpPAckCnt': 0, 'tcpFlwLssAckRcvdBytes': 0, 'tcpAckFaultCnt': 0, 'tcpInitWinSz': 0, 'tcpAveWinSz': 0.0, 'tcpMinWinSz': 0, 'tcpMaxWinSz': 0, 'tcpWinSzDwnCnt': 0, 'tcpWinSzUpCnt': 0, 'tcpWinSzChgDirCnt': 0, 'tcpWinSzThRt': 1.0, 'tcpFlags': '0x44', 'tcpAnomaly': '0x0000', 'tcpOptPktCnt': 0, 'tcpOptCnt': 0, 'tcpOptions': '0x00000000', 'tcpMSS': 0, 'tcpWS': 1, 'tcpMPTBF': '0x0000', 'tcpMPF': '0x00', 'tcpMPAID': 0, 'tcpMPdssF': '0x00', 'tcpTmS': 0, 'tcpTmER': 0, 'tcpEcI': 0.0, 'tcpUtm': 0.0, 'tcpBtm': '0.000000', 'tcpSSASAATrip': 0.0, 'tcpRTTAckTripMin': 65535.0, 'tcpRTTAckTripMax': 0.0, 'tcpRTTAckTripAve': 0.0, 'tcpRTTAckTripJitAve': 0.0, 'tcpRTTSseqAA': 0.0, 'tcpRTTAckJitAve': -1.0, 'tcpStatesAFlags': '0xc3'}
...

Let’s re-run t2 and analyze the flows one by one:

>>> t2.run()

Setting up streaming server 127.0.0.1:6666

>>> flows = t2.stream()

>>> next(flows)

{'dir': 'A', 'flowInd': 263, 'flowStat': '0x0400000000004000', 'timeFirst': '1022171701.709116', 'timeLast': '1022171701.709116', 'duration': '0.000000', 'numHdrDesc': 1, 'numHdrs': [3], 'hdrDesc': ['eth:ipv4:tcp'], 'srcMac': ['00:d0:02:6d:78:00'], 'dstMac': ['00:50:fc:0e:21:56'], 'ethType': '0x0800', 'srcIP': '209.171.12.143', 'srcIPCC': 'ca', 'srcIPOrg': 'TELUS Communications Inc', 'srcPort': 4987, 'dstIP': '138.212.185.230', 'dstIPCC': 'jp', 'dstIPOrg': 'ASAHI KASEI CORPORATION', 'dstPort': 41250, 'l4Proto': 6, 'numPktsSnt': 1, 'numPktsRcvd': 0, 'numPktsRTAggr': 1, 'numBytesSnt': 0, 'numBytesRcvd': 0, 'numBytesRTAggr': 0, 'minPktSz': 0, 'maxPktSz': 0, 'avePktSize': 0.0, 'stdPktSize': 0.0, 'minIAT': 0.0, 'maxIAT': 0.0, 'aveIAT': 0.0, 'stdIAT': 0.0, 'pktps': 0.0, 'bytps': 0.0, 'pktAsm': 1.0, 'bytAsm': 0.0, 'tcpFStat': '0x0040', 'ipMindIPID': 65535, 'ipMaxdIPID': 0, 'ipMinTTL': 117, 'ipMaxTTL': 117, 'ipTTLChg': 0, 'ipToS': '0x00', 'ipFlags': '0x0840', 'ipOptCnt': 0, 'ipOptCpCl_Num': ['0x00', '0x00000000'], 'ip6OptCntHH_D': [0, 0], 'ip6OptHH_D': ['0x00000000', '0x00000000'], 'tcpISeqN': 1471189413, 'tcpPSeqCnt': 0, 'tcpSeqSntBytes': 0, 'tcpSeqFaultCnt': 0, 'tcpPAckCnt': 0, 'tcpFlwLssAckRcvdBytes': 0, 'tcpAckFaultCnt': 0, 'tcpInitWinSz': 0, 'tcpAveWinSz': 0.0, 'tcpMinWinSz': 0, 'tcpMaxWinSz': 0, 'tcpWinSzDwnCnt': 0, 'tcpWinSzUpCnt': 0, 'tcpWinSzChgDirCnt': 0, 'tcpWinSzThRt': 1.0, 'tcpFlags': '0xc4', 'tcpAnomaly': '0x0000', 'tcpOptPktCnt': 0, 'tcpOptCnt': 0, 'tcpOptions': '0x00000000', 'tcpMSS': 0, 'tcpWS': 1, 'tcpMPTBF': '0x0000', 'tcpMPF': '0x00', 'tcpMPAID': 0, 'tcpMPdssF': '0x00', 'tcpTmS': 0, 'tcpTmER': 0, 'tcpEcI': 0.0, 'tcpUtm': 0.0, 'tcpBtm': '0.000000', 'tcpSSASAATrip': 0.0, 'tcpRTTAckTripMin': 65535.0, 'tcpRTTAckTripMax': 0.0, 'tcpRTTAckTripAve': 0.0, 'tcpRTTAckTripJitAve': 0.0, 'tcpRTTSseqAA': 0.0, 'tcpRTTAckJitAve': -1.0, 'tcpStatesAFlags': '0xc3'}

>>> next(flows)

{'dir': 'A', 'flowInd': 444, 'flowStat': '0x0400000000004000', 'timeFirst': '1022171701.721366', 'timeLast': '1022171701.721366', 'duration': '0.000000', 'numHdrDesc': 1, 'numHdrs': [3], 'hdrDesc': ['eth:ipv4:tcp'], 'srcMac': ['00:d0:02:6d:78:00'], 'dstMac': ['00:50:fc:3b:62:78'], 'ethType': '0x0800', 'srcIP': '217.41.129.13', 'srcIPCC': 'gb', 'srcIPOrg': 'BT Infrastructure Layer', 'srcPort': 58872, 'dstIP': '138.212.187.186', 'dstIPCC': 'jp', 'dstIPOrg': 'ASAHI KASEI CORPORATION', 'dstPort': 80, 'l4Proto': 6, 'numPktsSnt': 1, 'numPktsRcvd': 0, 'numPktsRTAggr': 1, 'numBytesSnt': 0, 'numBytesRcvd': 0, 'numBytesRTAggr': 0, 'minPktSz': 0, 'maxPktSz': 0, 'avePktSize': 0.0, 'stdPktSize': 0.0, 'minIAT': 0.0, 'maxIAT': 0.0, 'aveIAT': 0.0, 'stdIAT': 0.0, 'pktps': 0.0, 'bytps': 0.0, 'pktAsm': 1.0, 'bytAsm': 0.0, 'tcpFStat': '0x0040', 'ipMindIPID': 65535, 'ipMaxdIPID': 0, 'ipMinTTL': 44, 'ipMaxTTL': 44, 'ipTTLChg': 0, 'ipToS': '0x00', 'ipFlags': '0x0840', 'ipOptCnt': 0, 'ipOptCpCl_Num': ['0x00', '0x00000000'], 'ip6OptCntHH_D': [0, 0], 'ip6OptHH_D': ['0x00000000', '0x00000000'], 'tcpISeqN': 3186340143, 'tcpPSeqCnt': 0, 'tcpSeqSntBytes': 0, 'tcpSeqFaultCnt': 0, 'tcpPAckCnt': 0, 'tcpFlwLssAckRcvdBytes': 0, 'tcpAckFaultCnt': 0, 'tcpInitWinSz': 0, 'tcpAveWinSz': 0.0, 'tcpMinWinSz': 0, 'tcpMaxWinSz': 0, 'tcpWinSzDwnCnt': 0, 'tcpWinSzUpCnt': 0, 'tcpWinSzChgDirCnt': 0, 'tcpWinSzThRt': 1.0, 'tcpFlags': '0x44', 'tcpAnomaly': '0x0000', 'tcpOptPktCnt': 0, 'tcpOptCnt': 0, 'tcpOptions': '0x00000000', 'tcpMSS': 0, 'tcpWS': 1, 'tcpMPTBF': '0x0000', 'tcpMPF': '0x00', 'tcpMPAID': 0, 'tcpMPdssF': '0x00', 'tcpTmS': 0, 'tcpTmER': 0, 'tcpEcI': 0.0, 'tcpUtm': 0.0, 'tcpBtm': '0.000000', 'tcpSSASAATrip': 0.0, 'tcpRTTAckTripMin': 65535.0, 'tcpRTTAckTripMax': 0.0, 'tcpRTTAckTripAve': 0.0, 'tcpRTTAckTripJitAve': 0.0, 'tcpRTTSseqAA': 0.0, 'tcpRTTAckJitAve': -1.0, 'tcpStatesAFlags': '0xc3'}

Ok, that’s about it for the streaming mode!

Follow streams and reconstruct payload with the T2 or T2Utils module

Let us start a new T2 session with another pcap file (which is not snapped!): faf-exercise.pcap. Download it and save it your home in a data folder (or wherever you like, but make sure to use the correct path in the following command!).

t2py -t2 -r ~/data/faf-exercise.pcap basicFlow basicStats tcpFlags -w ~/results/ -s

Welcome to t2py, Python interface to Tranalyzer2!
Type "help(T2Utils)", "help(T2Plugin)" or "help(T2)" for more information.
>>>

Let us first review what we have created:

>>> t2.status()


Plugins:
    1: basicFlow
    2: basicStats
    3: tcpFlags

Tranalyzer options:
    -r /home/user/data/faf-exercise.pcap
    -w /home/user/results/
    -s
    -l

Nice! We have one more thing to do and that is activating the hex output for the packet mode. This is controlled by the SPKTMD_PCNTH variable in Tranalyzer2 (to be faster, we could also disable the ASCII output by setting SPKTMD_PCNTC to 0).

>>> t2.tranalyzer.SPKTMD_PCNTH
0
>>> t2.tranalyzer.SPKTMD_PCNTH = 1
>>> t2.tranalyzer.SPKTMD_PCNTH
1

Now we are all set! Let us apply the changes, build Tranalyzer2 and the plugins and run Tranalyzer2:

>>> t2.tranalyzer2.apply_changes()
>>> t2.build()
>>> t2.run()

In the packet mode tutorial, we had identified flow 35 and 36 as being the FTP control and data channels respectively. Let us work with those flows again, but this time with the t2.follow_stream() function:

>>> flow35 = t2.follow_stream(35)
>>> type(flow35)
<class 'list'>

>>> flow35[0]

{'flow': 35, 'packet': 1269, 'timestamp': 1258594163.087792, 'srcIP': '143.166.11.10', 'srcPort': 21, 'dstIP': '192.168.1.105', 'dstPort': 49329, 'proto': 'TCP', 'length': 27, 'seq': 365320933, 'payload': bytearray(b'220 Microsoft FTP Service\r\n')}

The packets are sorted by sequence numbers, so we can recreate the full conversation as follows:

>>> payload = ''
>>> for packet in flow35:
...     payload = payload + packet['payload'].decode()
>>> print(payload)

220 Microsoft FTP Service
USER anonymous
331 Anonymous access allowed, send identity (e-mail name) as password.
PASS IEUser@
230-Welcome to the Dell FTP site. A service of Dell Inc., Round Rock, Texas.
    For information about DELL, call +1 800 999 3355 All transfers are logged with
    your host name and email address. If you don't like this policy please disconnect now.
    Please be advised that use constitutes consent to monitoring (Elec Comm Priv Act,
    18 USC 2701-2711). Please see the file readme.txt for disclaimers pertaining to this
    service. If your FTP client crashes or hangs shortly after login, try using a dash
    (-) as the first character of your password. This will turn off the informational
    messages which may be confusing your ftp client.
    ********IN CASE OF PROBLEMS*************************
    ** File Content: send EMAIL to dellbbs@dell.com   **
    ** FTP Server: send EMAIL to hostmaster@dell.com  **
    ** WWW Server: send EMAIL to webmaster@dell.com   **
    ****************************************************
230 User logged in.
TYPE I
200 Type set to I.
PASV
227 Entering Passive Mode (143,166,11,10,251,78)
SIZE /video/R79733.EXE
213 4255056
RETR /video/R79733.EXE
125 Data connection already open; Transfer starting.
226 Transfer complete.

Alternatively, we could also change the output format to 0 and recreate the payload as follows:

>>> flow35 = t2.follow_stream(35, output_format=0)
>>> print(''.join([packet.decode() for packet in flow35]))

220 Microsoft FTP Service
USER anonymous
331 Anonymous access allowed, send identity (e-mail name) as password.
PASS IEUser@
230-Welcome to the Dell FTP site. A service of Dell Inc., Round Rock, Texas.
    For information about DELL, call +1 800 999 3355 All transfers are logged with
    your host name and email address. If you don't like this policy please disconnect now.
    Please be advised that use constitutes consent to monitoring (Elec Comm Priv Act,
    18 USC 2701-2711). Please see the file readme.txt for disclaimers pertaining to this
    service. If your FTP client crashes or hangs shortly after login, try using a dash
    (-) as the first character of your password. This will turn off the informational
    messages which may be confusing your ftp client.
    ********IN CASE OF PROBLEMS*************************
    ** File Content: send EMAIL to dellbbs@dell.com   **
    ** FTP Server: send EMAIL to hostmaster@dell.com  **
    ** WWW Server: send EMAIL to webmaster@dell.com   **
    ****************************************************
230 User logged in.
TYPE I
200 Type set to I.
PASV
227 Entering Passive Mode (143,166,11,10,251,78)
SIZE /video/R79733.EXE
213 4255056
RETR /video/R79733.EXE
125 Data connection already open; Transfer starting.
226 Transfer complete.

But the good stuff (the nasty EXE!) is in flow 36, the data flow, direction B! We will use the output format 3 to reconstruct the data:

>>> flow36B = t2.follow_stream(36, direction='B', output_format=3)
>>> type(flow36B)
<class 'bytearray'>
>>> with open('/home/user/results/flow36B.data', 'wb') as f:
...     f.write(flow36B)
4255056
>>>

As expected, the size of the data written to flow36B.data matches that reported by the FTP control channel!

Just to be sure, let us check that the MD5 sum matches that found in the packet mode tutorial, namely 6448b03e6a8709be41e7165979a440da:

>>> from hashlib import md5
>>> checksum = md5()
>>> with open('/home/user/results/flow36B.data', 'rb') as f:
...     checksum.update(f.read())
>>> checksum.hexdigest()
'6448b03e6a8709be41e7165979a440da'
>>>

The T2Utils.follow_stream() function can be used the same way, but requires the input file to be specified:

>>> print(T2Utils.follow_stream('/home/user/results/faf-exercise_packets.txt', 35, output_format=1, payload_format='hexdump'))


================================================================================
Packet 1269 (27 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365320933
================================================================================

00000000  32 32 30 20 4d 69 63 72  6f 73 6f 66 74 20 46 54   220 Micr osoft FT
00000010  50 20 53 65 72 76 69 63  65 0d 0a                  P Servic e..

================================================================================
Packet 1270 (16 bytes): flow 35 A 192.168.1.105:49329 -> 143.166.11.10:21 TCP seq: 2427598872
================================================================================

00000000  55 53 45 52 20 61 6e 6f  6e 79 6d 6f 75 73 0d 0a   USER ano nymous..

================================================================================
Packet 1271 (72 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365320960
================================================================================

0000001B  33 33 31 20 41 6e 6f 6e  79 6d 6f 75 73 20 61 63   331 Anon ymous ac
0000002B  63 65 73 73 20 61 6c 6c  6f 77 65 64 2c 20 73 65   cess all owed, se
0000003B  6e 64 20 69 64 65 6e 74  69 74 79 20 28 65 2d 6d   nd ident ity (e-m
0000004B  61 69 6c 20 6e 61 6d 65  29 20 61 73 20 70 61 73   ail name ) as pas
0000005B  73 77 6f 72 64 2e 0d 0a                            sword...

================================================================================
Packet 1272 (14 bytes): flow 35 A 192.168.1.105:49329 -> 143.166.11.10:21 TCP seq: 2427598888
================================================================================

00000010  50 41 53 53 20 49 45 55  73 65 72 40 0d 0a         PASS IEU ser@..

================================================================================
Packet 1273 (950 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365321032
================================================================================

00000063  32 33 30 2d 57 65 6c 63  6f 6d 65 20 74 6f 20 74   230-Welc ome to t
00000073  68 65 20 44 65 6c 6c 20  46 54 50 20 73 69 74 65   he Dell  FTP site
00000083  2e 20 41 20 73 65 72 76  69 63 65 20 6f 66 20 44   . A serv ice of D
00000093  65 6c 6c 20 49 6e 63 2e  2c 20 52 6f 75 6e 64 20   ell Inc. , Round
000000A3  52 6f 63 6b 2c 20 54 65  78 61 73 2e 0d 0a 20 20   Rock, Te xas...
000000B3  20 20 46 6f 72 20 69 6e  66 6f 72 6d 61 74 69 6f     For in formatio
000000C3  6e 20 61 62 6f 75 74 20  44 45 4c 4c 2c 20 63 61   n about  DELL, ca
000000D3  6c 6c 20 2b 31 20 38 30  30 20 39 39 39 20 33 33   ll +1 80 0 999 33
000000E3  35 35 20 41 6c 6c 20 74  72 61 6e 73 66 65 72 73   55 All t ransfers
000000F3  20 61 72 65 20 6c 6f 67  67 65 64 20 77 69 74 68    are log ged with
00000103  0d 0a 20 20 20 20 79 6f  75 72 20 68 6f 73 74 20   ..    yo ur host
00000113  6e 61 6d 65 20 61 6e 64  20 65 6d 61 69 6c 20 61   name and  email a
00000123  64 64 72 65 73 73 2e 20  49 66 20 79 6f 75 20 64   ddress.  If you d
00000133  6f 6e 27 74 20 6c 69 6b  65 20 74 68 69 73 20 70   on't lik e this p
00000143  6f 6c 69 63 79 20 70 6c  65 61 73 65 20 64 69 73   olicy pl ease dis
00000153  63 6f 6e 6e 65 63 74 20  6e 6f 77 2e 0d 0a 20 20   connect  now...
00000163  20 20 50 6c 65 61 73 65  20 62 65 20 61 64 76 69     Please  be advi
00000173  73 65 64 20 74 68 61 74  20 75 73 65 20 63 6f 6e   sed that  use con
00000183  73 74 69 74 75 74 65 73  20 63 6f 6e 73 65 6e 74   stitutes  consent
00000193  20 74 6f 20 6d 6f 6e 69  74 6f 72 69 6e 67 20 28    to moni toring (
000001A3  45 6c 65 63 20 43 6f 6d  6d 20 50 72 69 76 20 41   Elec Com m Priv A
000001B3  63 74 2c 0d 0a 20 20 20  20 31 38 20 55 53 43 20   ct,..     18 USC
000001C3  32 37 30 31 2d 32 37 31  31 29 2e 20 50 6c 65 61   2701-271 1). Plea
000001D3  73 65 20 73 65 65 20 74  68 65 20 66 69 6c 65 20   se see t he file
000001E3  72 65 61 64 6d 65 2e 74  78 74 20 66 6f 72 20 64   readme.t xt for d
000001F3  69 73 63 6c 61 69 6d 65  72 73 20 70 65 72 74 61   isclaime rs perta
00000203  69 6e 69 6e 67 20 74 6f  20 74 68 69 73 0d 0a 20   ining to  this..
00000213  20 20 20 73 65 72 76 69  63 65 2e 20 49 66 20 79      servi ce. If y
00000223  6f 75 72 20 46 54 50 20  63 6c 69 65 6e 74 20 63   our FTP  client c
00000233  72 61 73 68 65 73 20 6f  72 20 68 61 6e 67 73 20   rashes o r hangs
00000243  73 68 6f 72 74 6c 79 20  61 66 74 65 72 20 6c 6f   shortly  after lo
00000253  67 69 6e 2c 20 74 72 79  20 75 73 69 6e 67 20 61   gin, try  using a
00000263  20 64 61 73 68 0d 0a 20  20 20 20 28 2d 29 20 61    dash..     (-) a
00000273  73 20 74 68 65 20 66 69  72 73 74 20 63 68 61 72   s the fi rst char
00000283  61 63 74 65 72 20 6f 66  20 79 6f 75 72 20 70 61   acter of  your pa
00000293  73 73 77 6f 72 64 2e 20  54 68 69 73 20 77 69 6c   ssword.  This wil
000002A3  6c 20 74 75 72 6e 20 6f  66 66 20 74 68 65 20 69   l turn o ff the i
000002B3  6e 66 6f 72 6d 61 74 69  6f 6e 61 6c 0d 0a 20 20   nformati onal..
000002C3  20 20 6d 65 73 73 61 67  65 73 20 77 68 69 63 68     messag es which
000002D3  20 6d 61 79 20 62 65 20  63 6f 6e 66 75 73 69 6e    may be  confusin
000002E3  67 20 79 6f 75 72 20 66  74 70 20 63 6c 69 65 6e   g your f tp clien
000002F3  74 2e 0d 0a 20 20 20 20  2a 2a 2a 2a 2a 2a 2a 2a   t...     ********
00000303  49 4e 20 43 41 53 45 20  4f 46 20 50 52 4f 42 4c   IN CASE  OF PROBL
00000313  45 4d 53 2a 2a 2a 2a 2a  2a 2a 2a 2a 2a 2a 2a 2a   EMS***** ********
00000323  2a 2a 2a 2a 2a 2a 2a 2a  2a 2a 2a 2a 0d 0a 20 20   ******** ****..
00000333  20 20 2a 2a 20 46 69 6c  65 20 43 6f 6e 74 65 6e     ** Fil e Conten
00000343  74 3a 20 73 65 6e 64 20  45 4d 41 49 4c 20 74 6f   t: send  EMAIL to
00000353  20 64 65 6c 6c 62 62 73  40 64 65 6c 6c 2e 63 6f    dellbbs @dell.co
00000363  6d 20 20 20 2a 2a 0d 0a  20 20 20 20 2a 2a 20 46   m   **..     ** F
00000373  54 50 20 53 65 72 76 65  72 3a 20 73 65 6e 64 20   TP Serve r: send
00000383  45 4d 41 49 4c 20 74 6f  20 68 6f 73 74 6d 61 73   EMAIL to  hostmas
00000393  74 65 72 40 64 65 6c 6c  2e 63 6f 6d 20 20 2a 2a   ter@dell .com  **
000003A3  0d 0a 20 20 20 20 2a 2a  20 57 57 57 20 53 65 72   ..    **  WWW Ser
000003B3  76 65 72 3a 20 73 65 6e  64 20 45 4d 41 49 4c 20   ver: sen d EMAIL
000003C3  74 6f 20 77 65 62 6d 61  73 74 65 72 40 64 65 6c   to webma ster@del
000003D3  6c 2e 63 6f 6d 20 20 20  2a 2a 0d 0a 20 20 20 20   l.com    **..
000003E3  2a 2a 2a 2a 2a 2a 2a 2a  2a 2a 2a 2a 2a 2a 2a 2a   ******** ********
000003F3  2a 2a 2a 2a 2a 2a 2a 2a  2a 2a 2a 2a 2a 2a 2a 2a   ******** ********
00000403  2a 2a 2a 2a 2a 2a 2a 2a  2a 2a 2a 2a 2a 2a 2a 2a   ******** ********
00000413  2a 2a 2a 2a 0d 0a                                  ****..

================================================================================
Packet 1274 (21 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365321982
================================================================================

00000419  32 33 30 20 55 73 65 72  20 6c 6f 67 67 65 64 20   230 User  logged
00000429  69 6e 2e 0d 0a                                     in...

================================================================================
Packet 1276 (8 bytes): flow 35 A 192.168.1.105:49329 -> 143.166.11.10:21 TCP seq: 2427598902
================================================================================

0000001E  54 59 50 45 20 49 0d 0a                            TYPE I..

================================================================================
Packet 1277 (20 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365322003
================================================================================

0000042E  32 30 30 20 54 79 70 65  20 73 65 74 20 74 6f 20   200 Type  set to
0000043E  49 2e 0d 0a                                        I...

================================================================================
Packet 1278 (6 bytes): flow 35 A 192.168.1.105:49329 -> 143.166.11.10:21 TCP seq: 2427598910
================================================================================

00000026  50 41 53 56 0d 0a                                  PASV..

================================================================================
Packet 1279 (50 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365322023
================================================================================

00000442  32 32 37 20 45 6e 74 65  72 69 6e 67 20 50 61 73   227 Ente ring Pas
00000452  73 69 76 65 20 4d 6f 64  65 20 28 31 34 33 2c 31   sive Mod e (143,1
00000462  36 36 2c 31 31 2c 31 30  2c 32 35 31 2c 37 38 29   66,11,10 ,251,78)
00000472  0d 0a                                              ..

================================================================================
Packet 1283 (24 bytes): flow 35 A 192.168.1.105:49329 -> 143.166.11.10:21 TCP seq: 2427598916
================================================================================

0000002C  53 49 5a 45 20 2f 76 69  64 65 6f 2f 52 37 39 37   SIZE /vi deo/R797
0000003C  33 33 2e 45 58 45 0d 0a                            33.EXE..

================================================================================
Packet 1284 (13 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365322073
================================================================================

00000474  32 31 33 20 34 32 35 35  30 35 36 0d 0a            213 4255 056..

================================================================================
Packet 1285 (24 bytes): flow 35 A 192.168.1.105:49329 -> 143.166.11.10:21 TCP seq: 2427598940
================================================================================

00000044  52 45 54 52 20 2f 76 69  64 65 6f 2f 52 37 39 37   RETR /vi deo/R797
00000054  33 33 2e 45 58 45 0d 0a                            33.EXE..

================================================================================
Packet 1286 (54 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365322086
================================================================================

00000481  31 32 35 20 44 61 74 61  20 63 6f 6e 6e 65 63 74   125 Data  connect
00000491  69 6f 6e 20 61 6c 72 65  61 64 79 20 6f 70 65 6e   ion alre ady open
000004A1  3b 20 54 72 61 6e 73 66  65 72 20 73 74 61 72 74   ; Transf er start
000004B1  69 6e 67 2e 0d 0a                                  ing...

================================================================================
Packet 5898 (24 bytes): flow 35 B 143.166.11.10:21 -> 192.168.1.105:49329 TCP seq: 365322140
================================================================================

000004B7  32 32 36 20 54 72 61 6e  73 66 65 72 20 63 6f 6d   226 Tran sfer com
000004C7  70 6c 65 74 65 2e 0d 0a                            plete...

================================================================================
6 client packets, 9 server packets, 12 turns.

Have fun following streams!

Subject to change

This library is still experimental and at an early stage of development. As such, the API may be subject to change.

Below is a list of functions or properties which may be removed in a future version.

Feel free to send us bug reports, feature requests, feedback and suggestions!

T2Plugin

The list_config() function may be removed as the same result can be obtained with the flags property.
The set_default() function may be removed as the same result can be obtained with the reset() function.
The get_default() function may be removed as the same result can be obtained with the default property.

T2

Only one of the log()/report() function may be kept.
Only one of the print_log()/print_report() function may be kept.
The clear_plugins() function may be removed and replaced with the remove_plugins() with plugins = None or plugins = [].
The unload() function may be modified to only allow unloading of all the plugins in the session (a single plugin can be unloaded with T2Plugin unload() function).
The flows(), flows_txt() and flows_json() may become flows(extension: str = None).
The add_plugin() and add_plugins() functions may be merged.
The remove_plugin() and remove_plugins() functions may be merged.
The behavior of the headers(), log(), and report() functions may be changed to that of print_headers(), print_log() and print_report() respectively.
The print_flows(), print_flows_json(), print_flows_txt(), print_headers(), print_log(), print_packets() and print_report() functions may be removed.

Bug reports, feature requests, feedback and suggestions

Feel free to send us bug reports, feature requests, feedback and suggestions!

t2py: control and operate T2 with Python

Contents

Introduction

Caveats

Dependencies

Required dependencies

Optional dependencies

Getting started

Setup

Available modules

Import

T2Utils.py: simple wrapper around Tranalyzer2 scripts and utilities

T2Plugin.py: represent a plugin

T2.py: manage several plugins and run T2, convert/display flow file, …

API documentation

Getting help

Sample sessions using t2py

Simple operations with the T2Utils module

Plugin configuration with the T2Plugin module

Tranalyzer2 execution session with the T2 module

Streaming flows with the T2 module

Follow streams and reconstruct payload with the T2 or T2Utils module

Subject to change

T2Plugin

T2

Bug reports, feature requests, feedback and suggestions