• DevSummit
• 201305
• NetworkReceivePerformance

FreeBSD Developer Summit: Network Receive Performance

Overview

This working group will discuss changes necessary to the interface layers in FreeBSD to deal with current advances in high performance network interfaces including Receive Side Steering, pinning queues to CPUs and other receive side features.

Goals

A list of changes and APIs that will help facility FreeBSD better using the above listed features.

Topics

Attending

In order to attend you need register for the developer summit as well as by email for the session and be confirmed by the working group organizer. Follow the guidelines described on the main page or what you received by email. For questions or if in doubt ask the session chairs.

Please do NOT add yourself here. Your name will appear automatically once you have received the confirmation email. You need to put your name on the general developer summit attendees list though.

Notes

The discussion opened with an attempt to constrain the problem we were trying to solve, including pointing out that any KPI/API suggested needed to be achievable in the next six months.

Some of the existing solutions to the problem of talking to hardware with multiple queues, which all high end NICs currently have, were:

• Connection Groups
  • Not really a KPI
  • RSS vs. Flow Table is an issue to solve, we have things for the former, but little for the latter
  • Socket affinity is also an issue
• NAPI
  • This is an APi in Linux. It uses upcalls.
• Flow table mapping. Chelsio may have some of this.
• SRIO
• VLL Cloner

There are several ways to map flows, including: 4 tuple, MAC filter, arbitrary offset. An API that only handles offset, length, value is too simple from the standpoint of getting the right data into the hardware. We need something more rich on the kernel side of the API to that driver writers don't have to figure out our intentions.

Some methods that a good KPI/API ought to have include:

• Query Device for information about its queues, including how many exist, and how they are mapped to other resources, including CPU and memory
• Map CPUID to a Flow
• Setup RSS

• Request RxRing local memory

• Solaris Mapping API might be a way to go (http://www.oracle.com/technetwork/articles/servers-storage-admin/crossbowsetup-191326.pdf)

Some consumers of such an API include: Performance, affinity, virtualization, policy, kernel bypass, QoS, and VIMAGE.

We have two patches, for different bits, to start from including Vijay's [RobertWatson] and Randall's [RandallStewart], [GeorgeNevilleNeil]

We need quite a few things, including:

• Per connection flow table
• Describing queues in the stack such that we can expose interesting parts via netstat.
• Packet Batching. This was not overwhelmingly popular.

A straw person API includes:

• MBUF Flag
• Hash Value
• The whole thing may be used as opaque
  • Used by the stack for inpcb
• Get number of buckets
• Map bucket to RSS
• Map queue/ithread to CPU
• Get width of the hash
• RSS get CPU
• RSS get hash algo
• Pick hash inputs
• Get and set key
• Rebalance
• Software hash table
• Query queue length
• Get queue affinity
• Set mask (CPUSET) on socket
• Set policy on CPU/socket
• Queue event reporting
• Load distrubtion stats

/ComparingMutiqueueSupportLinuxvsFreeBSD

Results

(Add a list or attach slides detailing the achieved results here.)

CategoryHistorical