OLS-2005

pktgen the linux packet generator

Robert Olsson

Uppsala Universitet & SLU

Motivation

Author is involved in testing

No suitable SW tools for high-perf testing

Expensive and exclusive HW tools

Hard to do performance test chip/drivers

Should be possible to extend for new tests

Don't care much about GUI

This describes the treaded rework of pktgen

ftp://robur.slu.se://pub/Linux/net-development/pktgen-testing/

Short History

Whitehole device. The secret device in linux

kernel with hardcoded addresses :)

Kernel module pg sends skb's direct into driver	hard_xmit() attracted some developers

Alexey added fast path for packet memory

and more features. Name become pg3

More hacking. Ben Greear (product), Jens Låås

DaveM named pktgen and added to kernel

Major rework for multiple CPU's in Linux 2.6.11

Does not solve all
testing problems

In kernel

Developed on need basis

Not add to much bloat and be reasonble robust

No TCP testing.

Is a complement to other testing tools netperf, ttcp etc etc.

packet generator

Performance

PIII 733 Mhz e1000 NIC

also reported with

Itanium2 (rx2600 1 Ghz) BCM5703

870.000 packets/s (64 byte)

12 NIC's DUAL XEON w. Ht 1500 byte packets

>10 Gbit/s

Finally,

XEON and w. pathed e1000 1.488 Mpps

Performance factors

I/O and DMA latency (prefetch)

Number of PCI MMIO reads/writes per packet

I/O and memory bandwith

Memory latency

RX/TX buffers (just enough)

Interrupts

PCI bus speed

PCI vx PCI-X

PCI bridges

CPU speed

SW design

Chip design

etc etc

Bus and bridge impact

Bridge adds latency, e1000 patch adds prefetch etc

MMIO read latency

Latency in processor cycles. Reading device control

register with mmio_test on a Opteron 1.6 GHz

Getting pktgen to run/1

Enable CONFIG_NET_PKTGEN

insmod pktgen if needed

One thread per CPU

[pktgen/0],

[pktgen/1]

/proc/net/pktgen/

kpktgend_0,

kpktgend_1,

pgctrl

Getting pktgen to run/2

Adding devices to threads adds

new files in /proc/

Example:

/proc/net/pktgen/

eth0

eth1

To be configured with device info

Getting pktgen to run/3

IP addresses, src, dst

counts

MAC adresses

Delay

Default:

UDP port 9 (discard) src and dst

Much more later....

Packet memory fastpath

Pktgen can do a trick to avoid full path for kmalloc and kfree when sending identical packets this increases performance. It's controlled by clone_skb

clone_skb=1000000 givs 1 master packet followed by one million clones

Results in only one full path malloc/kfree per million packets

Delay

Gap between packets in nanoseconds.

Pktgen can insert an extra delay

For small delays pktgen busy-waits

Hard to get a specific rate

In most cases bursts are sent

Default 0

Needs experimentation

Setup Examples/1

Simple. Just send

Probably you need to keep link up

eth0

pktgen/0

eth0

pktgen/0

Setup Examples/2

Just send.

Use another NIC on some box?

Set dst_mac correct if the pkts should be seen

Emulate incoming pkts with just a single box

eth0

pktgen/0

eth1

Setup Examples/3

Send to another device local or remote

Set dst_mac accordingly

Dummy dev can be used to test forwarding

eth0

pktge/0

Setup Examples/4

Classical bridging/forwarding setup

eth0

pktgen/0

Router/

switch

sink

eth1

Setup Examples/5

Bridging/forwarding in parallel setup

Can use multiple CPU's on sender(s)

(and on multiple CPU's onrouters)

pktgen/0

Router/

switch

sink

pktgen/1

Viewing pktgen threads

/proc/net/pktgen/kpktgend_0

Name: kpktgend_0

max_before_softirq: 10000

Running:

Stopped: eth1

Result: OK: max_before_softirq=10000

Viewing pktgen devices

/proc/net/pktgen/eth1

Params: count 10000000 min_pkt_size: 60 max_pkt_size: 60

frags: 0 delay: 0 clone_skb: 1000000 ifname: eth1

flows: 0 flowlen: 0

dst_min: 10.10.11.2 dst_max:

src_min: src_max:

src_mac: 00:04:23:AC:FD:82 dst_mac: 00:04:23:AE:05:16

udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9

src_mac_count: 0 dst_mac_count: 0

Flags:

Current:

pkts-sofar: 10000000 errors: 0

started: 1119356264434801us stopped: 1119356275792478us idle: 1434226us

seq_num: 10000011 cur_dst_mac_offset: 0 cur_src_mac_offset: 0

cur_saddr: 0x10a0a0a cur_daddr: 0x20b0a0a

cur_udp_dst: 9 cur_udp_src: 9

flows: 0

Result: OK: 11357677(c9923451+d1434226) usec, 10000000 (60byte,0frags)

880461pps 422Mb/sec (422621280bps) errors: 0

Configuring/1

Get a suitable script and modify

Next the glory details

Configuring/2

echo rem_device_all > proc/net/pktgen/kpktgend_0

In our scripts it looks like below. We write to /proc via

pgset which in turn uses PGDEV

PGDEV=/proc/net/pktgen/kpktgend_0

pgset "rem_device_all"

pgset "add_device eth1"

Configuring/3

Pgcontrol commands

start	Starts sending on all threads

stop	Aborts packet injection. Ctrl-C also aborts generator.

Note: It is generally better to use count 0 (forever)

and stop the run with Ctrl-C when multiple devices

are assigned to one pktgen thread.

This avoids some devices finishing before others and

skewing the results since we are primarily interested

in packets over time, not absolute number of packets.

thread commands

add_device	Add a device to thread i.e eth0

rem_device_all	Removes all devices from this thread config

max_before_softirq	do_softirq() after sending a number of	packets

Configuring/4

Device commands

debug

clone_skb	Number of identical copies of the same packet

0 means alloc for each skb. For DoS etc we must

alloc new skb's.

clear_counters

pkt_size	Link packet size minus CRC (4)

min_pkt_size	Range pkt_size setting If < max_pkt_size, then

cycle through the port range.

max_pkt_size

frags	Number of fragments for a packet

count	Number of packets to send, zero for continious sending

Delay	Gap inserted between packets in nanoseconds

Note this is before before hard_xmit()

Configuring/5

Device commands

dst	IP destination address i.e 10.0.0.1

dst_min	Same as dst If < dst_max, then

cycle through the port range.

dst_max	Maximum destination IP. i.e 10.0.0..1

src_min	Minimum (or only) source IP. i.e 10.0.0.254 If < src_max, then cycle through the port range

src_max	Maximum source IP

dst6	IPV6 destination address i.e fec0::1

src6	IPV6 source address i.e fec0::2

dstmac	MAC destination adress 00:00:00:00:00:00

srcmac	MAC source adress. If omitted it's automatically taken from source device

src_mac_count Number of MACs we'll range through.

Minimum' MAC is what you set with srcmac.

dst_mac_count	Number of MACs we'll range through.

Minimum' MAC is what you set with dstmac.

Configuring/6

Device commands

flag [name]	Flag to modify behaviour.

IPSRC_RND	IP Source is random (between min/max),

IPDST_RND	Etc

TXSIZE_RND

UDPSRC_RND

UDPDST_RND

MACSRC_RND

MACDST_RND

udp_src_min	UDP source port min, If < udp_src_max, then

cycle through the port range.

udp_src_max	UDP source port max.

udp_dst_min	UDP destination port min, If < udp_dst_max, then

cycle through the port range.

udp_dst_max	UDP destination port max.

Configuring/6

Device commands

stop

flows	Number of concurrent flows

flowlen	Length flows

Configuring/7

pktgen.conf-1-2

A script fragment assigning eth1, eth2 to CPU on single CPU system.

PGDEV=/proc/net/pktgen/kpktgend_0

pgset "rem_device_all"

pgset "add_device eth1"

pgset "add_device eth2"

pktgen.conf-2-2

A script fragment assigning eth1 to CPU0 respectivly eth2 to CPU1.

PGDEV=/proc/net/pktgen/kpktgend_0

pgset "rem_device_all"

pgset "add_device eth1"

PGDEV=/proc/net/pktgen/kpktgend_1

pgset "rem_device_all"

pgset "add_device eth2"

Configuring/8

A script fragment for route DoS testing. Note clone_skb 0

PGDEV=/proc/net/pktgen/eth1

pgset ”clone_skb 0”

# Random address with in the min-max range

# pgset "flag IPDST_RND"

pgset "dst_min 10.0.0.0"

pgset "dst_max 10.255.255.255"

pktgen.conf-1-1-ipv6

Setting device ipv6 addresses.

PGDEV=/proc/net/pktgen/eth1

pgset "dst6 fec0::1"

pgset "src6 fec0::2"

Configuring/9

Fragment for ipv6 rdos testing. Yes!!

PGDEV=/proc/net/pktgen/eth1

pgset ”clone_skb 0”

# pgset "flag IPDST_RND"

pgset "dst6_min fec0::1"

pgset "dst6_max fec0::FFFF:FFFF"

Fragment for route flow testing. Note clone_skb 0

PGDEV=/proc/net/pktgen/eth1

pgset ”clone_skb 0”

# Random address with in the min-max range

# pgset "flag IPDST_RND"

pgset "dst_min 10.0.0.0"

pgset "dst_max 10.255.255.255"

# 8k Concurrent flows at 4 pkts

pgset "flows 8192"

pgset "flowlen 4"

Interrupt affinity/1

A device can only exist on one CPU/thread

Each pktgen thread has strict affinty to it's CPU

In most cases it's a good idea set irq affitity.

So TX- interrupts and freeing of skb's runs on the same CPU as allocation of packets and TX.

Experiment

Interrupt affinity/2

/proc/interrupts

26: 933931 0 IO-APIC-level eth0

27: 936392 0 IO-APIC-level eth1

28: 8 936457 IO-APIC-level eth2

29: 8 939310 IO-APIC-level eth3

echo 1 > /proc/irq/26/smp_affinity

echo 1 > /proc/irq/27/smp_affinity

echo 2 > /proc/irq/28/smp_affinity

echo 2 > /proc/irq/29/smp_affinity

Interrupt affinity/3

Clone skb/cache effects

Read Current (RC) is a Mckinley Bus transaction

which allows the CPU to retain ownership of a

dirty cachline. Experiment by Grant Grundler. HP

Detecting pktgen pkts in drivers or kernel stack

__u8 *data = (__u8 *) skb->data+20;

/* src and dst port 9 --> pktgen */

if(data[0] == 0 &&

data[1] == 9 &&

data[2] == 0 &&

data[3] == 9)

netdev_rx_stat[smp_processor_id()].fastroute_hit+;

Often very useful to disable portions of code

packet rate (pps) vs size(byte) for GIGE wire speed

NAPI observations & issue: fairness

Ping through a idle router

Ping through a router

under a DoS attack @ 890 kpps

Vae

Very well behaved just an increase a couple of 100 microsec !!

TCP performance

2.6.11.7 SMP kernel using one CPU driver e1000 NAPI -

no-NAPI. Opteron 1.6 GHz e1000 w 82546GB.

TCP performancewhen receiving DoS on other NIC

2.6.11.7 SMP kernel using one CPU driver e1000 NAPI -

no-NAPI. Opteron 1.6 GHz e1000 w 82546GB.

ipv6 performance

How rDoS work on sparse routing table?

Hardware for high perf. Networking

Some GIGE experiments

Pktgen sending test w. 11 GIGE interfaces

skb clone = 10.0 Gbit/s

skb alloc = 7.4 Gbit/s

SeverWorks X5DL8-GG Intel e1000

Ipv6 Forwarding performance

Fills a GIGE pipe -- starting from 512 byte pkts

ipv6 performance

How rDoS work on sparse routing table?

fib_trie performance comparison

Preroute pathes to disable route hash

32/64 bit || sizeof(sk_buff)

Gcc 3.4 x86_64 vs i686 on same HW

sizeof(sk_buff) 168 vs. 256 bytes

FIB; RW-lock vs RCU test

10 GbE early days

NAPI/GIGE production in use: uu.se

Stockholm

Stockholm

PIII 933MHz

Full Internet routing

via EBGP/IBGP

DMZ

AS 2834

UU- 1

UU- 2

Internal

UU-Net

L- uu1

L- uu2

Real World use:ftp.sunet.se

Ftp0

Ftp1

Ftp2

Stockholm

OC- 48

PIII- 933MHz

Load sharing & Redundancy

with Router Discovery

Full Internet routing

via EBGP/IBGP

AS 1653

AS 15980

Cisco

GSR

Archive- r1

Archive- r2

Switch

Cisco

brick

OC-48

Uppsala

A new network symbol has been seen...

The Penguin Has Landed

References/Current work

ftp://robur.slu.se://pub/Linux/net-development/pktgen-testing/

Related/current work:

Evaluating routing without route hash

Route hash tuning

Evaluating LC-trie algorithm (level path compression) for fast lookup i.e FIB

Production routing

Scalable network architecture for IP-forwarding

Robert Olsson

2004-08-27T10:42:32

Robert Olsson