• SummerOfCode2018Projects
• PSPAT

PSPAT Implementation

• Developer: Sumit Lakra (sumitlakradev@gmail.com)

• Mentors:

  • Alexander Chernikov (melifaro@feebsd.org)

  • Giuseppe Lettieri (giuseppe.lettieri73@gmail.com)

PSPAT Subsystem Implementation in the FreeBSD Kernel - Overview

PSPAT is an advanced and robust packet scheduling architecture aimed at systems that can generate tens of millions of network packets per second. Most hardware and software packet schedulers in use today fail to be fully efficient under such high load conditions. At present, the client threads are responsible for en-queuing/de-queuing the packets to/from a scheduling algorithm (SA). Hence, when there are scores of clients (Virtual Machines) operating on the same system, the lock contention to use the SA among them increases significantly and this negatively impacts the system’s potential to transmit packets at the optimum possible speed.

The idea of PSPAT is to make the client threads put their packets on lock-less queues which have been implemented as IFFQ based mailboxes. Next, a separate kernel thread ‘arbiter’ picks these packets up from the client mailboxes and queues/de-queues them to/from a Scheduling Algorithm without having to hold any locks. This is expected to significantly increase the system’s throughput. The arbiter can then also go on to transmit these packets in batches to the Network Interface or a separate kernel thread ‘dispatcher’ can be used for this purpose. At present, an attempt has been made to integrate PSPAT with dummynet by intercepting the out-going packets passing through dummynet and using dummynet’s implementation of Scheduling Algorithms with the arbiter thread to queue/de-queue the packets. The mailbox implementation and the core PSPAT implementation has been tested to work properly in stages.

Tasks Completed

• Successful Implementation of mailboxes based on Improved Fast Forward Queues (IFFQs)
• Implementation of the PSPAT module which on being loaded initializes the necessary SYSCTLs required to control the behavior of PSPAT from user space
• The module also creates per-cpu mailboxes based on the above IFFQ mailbox implementation for each cpu-core present on the system
• A dispatcher mailbox is also created and initialized
• The module also creates an arbiter thread and a dispatcher thread
• The dispatcher thread is put to sleep when the arbiter is in charge of transmitting the packets out to the NIC as well
• Interception of outgoing packets from dummynet is done in dummynet_io() in sys/netpfil/ipfw/ip_dn_io.c
• The intercepted packet is pushed to a dedicated client mailbox associated with the client’s thread. This mailbox is created if not already present
• The client mailbox’s pointer is then inserted into the per-cpu mailbox of the cpu-core on which the client thread last ran
• The arbiter periodically scans all the per-cpu mailboxes to check the last active clients and then proceeds to pick up packets from those client’s mailboxes (if any)
• The arbiter next schedules the packets using a Scheduling Algorithm of choice
• The arbiter then sends those packets to the dispatcher mailbox
• In the mode where the transmission of packets to NIC is handled by the arbiter thread, the arbiter picks them up from the dispatcher queue and calls dummynet_send() to transmit them
• In the mode where the transmission of packets to NIC is handled by a dispatcher thread, the dispatcher picks them up from the dispatcher queue and calls dummynet_send() to transmit them

Future Work

• Resolve the issue which is preventing the PSPAT threads (either arbiter or a dedicated dispatcher thread) from sending the packets out of PSPAT and to NICs even when using appropriate function calls
• Switch the Scheduling Algorithm instance on the go when a new instance is used with dummynet packets. Note: While switching SA’s on the go, all the packets from the previous SA must be de-queued completely before switching to a new SA

Code

https://github.com/theGodlessLakra/freebsd-pspat/tree/projects/pspat