• chengcui
• testD47474

testD47474

test D47474 in VM over emulated 1Gbps x 40ms WAN

test config (TSO is not enabled)

Virtual machines (VMs) are hosted by Bhyve in two separate physical boxs (Beelink SER5 AMD Mini PC) that are using FreeBSD 14.1 release OS.
The physical boxes (Beelink SER5 AMD Mini PCs) are connected through a 1Gbps hub, which is connected to a 1Gbps router.
In each test, only one data sender and one data receiver are used, and both are Virtual Machines (VMs).
FreeBSD VM n1fbsd  and Linux VM n1linuxvm  are sending TCP data traffic through the same physical path to the Linux VM receiver n2linuxvm . Senders don't share bandwidth.
Test TCP congestion control performance of CUBIC & NewReno in VM environment with added 40ms delay at the Linux receiver.

The minimum bandwidth delay product (BDP) is 1000Mbps x 40ms == 5 Mbytes.

root@n2linuxvm:~ # tc qdisc add dev enp0s5 root netem delay 40ms
root@n2linuxvm:~ # tc qdisc show dev enp0s5
qdisc netem 8001: root refcnt 2 limit 1000 delay 40ms
root@n2linuxvm:~ #

root@n1fbsd:~ # ping -c 5 -S 192.168.50.37 192.168.50.89
PING 192.168.50.89 (192.168.50.89) from 192.168.50.37: 56 data bytes
64 bytes from 192.168.50.89: icmp_seq=0 ttl=64 time=44.003 ms
64 bytes from 192.168.50.89: icmp_seq=1 ttl=64 time=44.837 ms
64 bytes from 192.168.50.89: icmp_seq=2 ttl=64 time=43.978 ms
64 bytes from 192.168.50.89: icmp_seq=3 ttl=64 time=43.513 ms
64 bytes from 192.168.50.89: icmp_seq=4 ttl=64 time=43.631 ms

--- 192.168.50.89 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 43.513/43.993/44.837/0.463 ms
root@n1fbsd:~ #

root@n1linuxvm:~ # ping -c 5 -I 192.168.50.154 192.168.50.89
PING 192.168.50.89 (192.168.50.89) from 192.168.50.154 : 56(84) bytes of data.
64 bytes from 192.168.50.89: icmp_seq=1 ttl=64 time=43.9 ms
64 bytes from 192.168.50.89: icmp_seq=2 ttl=64 time=44.0 ms
64 bytes from 192.168.50.89: icmp_seq=3 ttl=64 time=43.7 ms
64 bytes from 192.168.50.89: icmp_seq=4 ttl=64 time=44.0 ms
64 bytes from 192.168.50.89: icmp_seq=5 ttl=64 time=43.7 ms

--- 192.168.50.89 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4031ms
rtt min/avg/max/mdev = 43.706/43.862/44.015/0.130 ms
root@n1linuxvm:~ # 

• sender/receiver sysctl tuning

root@n1fbsd:~ # cat /etc/sysctl.conf
...
# Don't cache ssthresh from previous connection
net.inet.tcp.hostcache.enable=0
# In crease FreeBSD maximum socket buffer size up to 128MB
kern.ipc.maxsockbuf=134217728
# Increase FreeBSD Max size of automatic send/receive buffer up to 128MB
net.inet.tcp.sendbuf_max=134217728
net.inet.tcp.recvbuf_max=134217728
root@n1fbsd:~ #

root@n2linuxvm:~ # cat /etc/sysctl.conf
...
net.core.rmem_max = 134217728 
net.core.wmem_max = 134217728 
# Increase Linux autotuning TCP buffer max up to 128MB buffers
net.ipv4.tcp_rmem = 4096 131072 134217728
net.ipv4.tcp_wmem = 4096 16384 134217728
# Don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
root@n2linuxvm:~ #

• iperf3 command

iperf3 -B ${src} --cport ${tcp_port} -c ${dst} -l 1M -t 100 -i 1 -f m -VC ${name}

test result

• before the patch D47474

cc before_patch % cp ~/opensrc/review_siftr_log/review_siftr2_log .; tar zxf n1fbsd.cubic.siftr2.tgz; ./review_siftr2_log -f n1fbsd.cubic.siftr2 -s 863143062;
input file name: n1fbsd.cubic.siftr2
siftr version: 2.0
flow id list:
 id: 863143062 (192.168.50.37:54321<->192.168.50.89:5201) mss:1448 SACK:1 snd/rcv_scal:12/12 cnt:6095526

starting_time: 1731689425.397768
ending_time: 1731689526.176446
log duration: 100.78 seconds
input flow id is: 863143062
plot_file_name: plot_863143062.txt
input file has total lines: 6095528, fragment_cnt == 31                  <<<
++++++++++++++++++++++++++++++ summary ++++++++++++++++++++++++++++
  192.168.50.37:54321->192.168.50.89:5201 flowid: 863143062
    has 6095526 useful records (5359949 outputs, 735577 inputs)

this program execution time: 5.384 seconds
cc before_patch %

• after the patch D47474

cc after_patch % cp ~/opensrc/review_siftr_log/review_siftr2_log .; tar zxf n1fbsd.cubic.siftr2.tgz; ./review_siftr2_log -f n1fbsd.cubic.siftr2 -s 863143062;
input file name: n1fbsd.cubic.siftr2
siftr version: 2.0
flow id list:
 id: 863143062 (192.168.50.37:54321<->192.168.50.89:5201) mss:1448 SACK:1 snd/rcv_scal:12/12 cnt:6584657

starting_time: 1731688042.718915
ending_time: 1731688143.319492
log duration: 100.60 seconds
input flow id is: 863143062
plot_file_name: plot_863143062.txt
input file has total lines: 6584659, fragment_cnt == 1                   <<<
++++++++++++++++++++++++++++++ summary ++++++++++++++++++++++++++++
  192.168.50.37:54321->192.168.50.89:5201 flowid: 863143062
    has 6584657 useful records (5780002 outputs, 804655 inputs)

this program execution time: 5.994 seconds
cc after_patch %

testD47474 in Emulab over emulated 1Gbps x 40ms WAN

The test D43470 shows fractional TSO data trunk ((data % MSS) > 0), so this test is using the similar topology and test method.

test config (TSO is enabled)

Testbed (d710 nodes with 1Gbps links) used here are from Emulab.net: Emulab hardware

The topology is a dumbbell with two nodes (s1,s2) as traffic senders using iperf3, two nodes (rt1,rt2) as routers with a single bottleneck link, and two nodes (r1,r2) as traffic receivers.
The s1,s2 are using FreeBSD 15-CURRENT, and r1,r2 are using UBUNTU 22.04.
The rt1,rt2 are using Ubuntu Linux 18.04 with shallow L2,L3 buffers combined around 256 packets.
There is a dummynet box generating 40ms round-trip-delay (RTT) on the bottleneck link. All senders are sending data traffic to their corresponding receivers, e.g. s1 => r1 and s2 => r2.

The bottleneck link has bandwidth 1Gbps. There is background UDP traffic from s2 to r2 so that a s1's TCP traffic will encounter congestion at the rt1 node output port.

testbed:

• sender has TSO enabled

root@s1:~ # ifconfig bce2
bce2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 00:24:e8:78:2e:8d
        inet 10.1.5.3 netmask 0xffffff00 broadcast 10.1.5.255
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@s1:~ #
root@s1:~ # ping -c 5 -S s1 r1
PING r1-link5 (10.1.4.3) from s1-link1: 56 data bytes
64 bytes from 10.1.4.3: icmp_seq=0 ttl=62 time=39.881 ms
64 bytes from 10.1.4.3: icmp_seq=1 ttl=62 time=40.182 ms
64 bytes from 10.1.4.3: icmp_seq=2 ttl=62 time=40.194 ms
64 bytes from 10.1.4.3: icmp_seq=3 ttl=62 time=40.407 ms
64 bytes from 10.1.4.3: icmp_seq=4 ttl=62 time=40.135 ms

--- r1-link5 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 39.881/40.160/40.407/0.168 ms
root@s1:~ #

• iperf3 command

root@s1:~ # iperf3 -B s1 --cport 54321 -c r1 -l 1M -t 100 -i 1 -f m -VC cubic
cc@s2:~ % iperf3 -B s2 --cport 54321 -c r2 -t 6 -i 2 --udp -Vb 500m

test result

• before the patch D47474

cc before_patch % ./review_siftr2_log -f s1.cubic.siftr2 -s 2023086477
input file name: s1.cubic.siftr2
siftr version: 2.0
flow id list:
 id:2023086477 (10.1.5.3:54321<->10.1.4.3:5201) mss:1448 SACK:1 snd/rcv_scal:13/9 cnt:417082

starting_time: 1732032235.083306
ending_time: 1732032335.602907
log duration: 100.52 seconds
input flow id is: 2023086477
plot_file_name: plot_2023086477.txt
++++++++++++++++++++++++++++++ summary ++++++++++++++++++++++++++++
  10.1.5.3:54321->10.1.4.3:5201 flowid: 2023086477
input file has total lines: 417084
input flow data_pkt_cnt: 210786, fragment_cnt: 2135, fragment_ratio: 0.010
    has 417082 useful records (211162 outputs, 205920 inputs)

this program execution time: 0.425 seconds

• after the patch D47474

cc after_patch % ./review_siftr2_log -f s1.cubic.siftr2 -s 2023086477
input file name: s1.cubic.siftr2
siftr version: 2.0
flow id list:
 id:2023086477 (10.1.5.3:54321<->10.1.4.3:5201) mss:1448 SACK:1 snd/rcv_scal:13/9 cnt:409102

starting_time: 1732030980.260534
ending_time: 1732031080.642423
log duration: 100.38 seconds
input flow id is: 2023086477
plot_file_name: plot_2023086477.txt
++++++++++++++++++++++++++++++ summary ++++++++++++++++++++++++++++
  10.1.5.3:54321->10.1.4.3:5201 flowid: 2023086477
input file has total lines: 409104
input flow data_pkt_cnt: 206474, fragment_cnt: 1313, fragment_ratio: 0.006
    has 409102 useful records (206947 outputs, 202155 inputs)

this program execution time: 0.400 seconds