• DevSummit
• 201708
• Transport

Transport

Notes courtesy of Tom Jones.

Topics

• Jonathan could do his slides on transport group activity
• timer precision and locking stragies in TCP
• TCP models and testing and evaluation
• Tom upstreaming crazy transport changes
• Ilias Manus LLC Changes
• BZ's methodology (GEM5)
• Update on TCP-AO
• In kernel TLS

Jonathan's slides on transport group activity

slides

Q: It is great that work has happened, but where is the code?

A:

• Fastpath and Alt stack infrastructure are in head
• bbr and rack are not

• blackbox in review

• callout is off in the future
• pacer and timer is tied to rack and bbr
• you can ask for the code

Q: Has anyone done anything interesting to others?

A: Limelight is sponsoring fast open client, it will work for default and bbr and rack. Expected by end of 2017.

Q: Who is working on the modular stacks?

A: Jonathan is working on creating the default stack.

GNN: Are there public packetdrill tests for the default stack?

A: Yes, in a public repro.

GNN is working on better timestamps in the mbuf, GNN would like to use a time_t but it won't quite fit. Hardware timestamps would be nice, but device support is not consistent.

Tom upstreaming crazy transport changes

Tom explains what UDP Options is.

Is there interest in the project? Get it into a committable state and we can talk about it.

Update on TCP-AO

Should be fairly non interfering.

BZ's methodology (GEM5)

2 use case to use the GEM5 simulator

• locking
  • The world has changed from high throughput bidirectional streams, maybe it is time to revisit locking problems. Many FreeBSD TCP workloads are now unidirectional streams. We could do some experiments with both types of workload and see where the hot spots are. This performance experiement could be done on gem5. Two potential combinations: combine horizontally (socket read and write buffer locks), or combine vertically (socket and PCB locks). Prioritize testing the vertical combination.
• callout problems could be looked at on gem5
  • Callout precision requirements: high-precision requirements vs. low-precision requirements
  • Callout run length: quick vs. long
  • Likelihood to run: likely to run vs. likely to cancel
  • How to satisfy these requirements?
  • Where is delay introduced unexpectedly?

Q: Has anyone ever measured the ioctl overhead on their running machines?

A: NF looked at setsockopts per second, remove extraneous ones and it didn't

• give much improvement.

Timer precision and locking strategies in TCP

TCP models and testing and evaluation

10 years ago a formal model was created, but TCP is specifed in 30-40 RFC documents. A newer model has been written using dtrace to capture TCP Stack events. packetdrill is used to run tests on a tun interface.

More modern tcp features are not supported in the model.

Doesn't touch the congestion control, but will touch things like timers and IW.

This model cannot discover bugs in the stack, FreeBSD must be approximately correct. Results are still pending.

A single test suite for evaluating tcp would be interesting.

Collecting traces from production to evaluate against the model could show examples of different workloads than could be produced in a lab. blackbox could be applied to generate events for the model.

Ilias Manus LLC Changes

Sigcom paper on specialised network stacks, clean slate tcp stack. Shows that specialisation and micro architecture awareness can result in large improvements.

As disc latencies have gone down, the end to end latency between the disk and the nic has reduced. If it fits in the cache it is fast, if it isn't dram bandwidth becomes a problem.

Pretty close to optimal.

Accessing memory is the problem.

lessons

• very asynchronous network stack, end to end latency is very high
• FreeBSD could do things differently, AES in dram, eliminating the buffer cache will help.
• crypto is free, dcache is everything.

Use processor behaviour as input to the tcp congestion control, optimizing to keep things in the llc.

In kernel TLS

How does NF do in kernel TLS?

SSL pushed into kernel as much as possible, modifications to openssl. Handshake and keying as normal, then offloads into the kernel with a socket option. OpenSSL still does framing of packets; kernel stores header, trailer and a pointer to the data (ala sendfile). AESNI calls used to encrypt data in the buffer.

Big performance impact.

Q: how do you handle tls alerts

A: all input packets are delivered to userspace openssl, works for the

• asymmetric traffic workload. When necessary, openssl can temporarily deactivate kernel encryption.