Locking in syscons(4), its subordinate drivers and some edge cases

Overview

In general syscons output path to video hardware and input path from keyboards are sufficiently independent except for one special case described below. The output path is initiated in interfaces like printf and db_printf which can be called from literally any context in FreeBSD kernel. As such the output path must be lock-less or, at least, it must use context-aware locking to avoid deadlocks from unexpected re-entrance. Examples of the latter include entering kdb or panic(9) on one CPU while another CPU is at an arbitrary place in the middle of the output path. Currently the output path code utilizes only spinlocks. It seems to be non-perfect from the point of view of protecting from the unexpected re-entrance. It is clear if the perfection can be achieved here and at what cost. Consider for example a re-entrance in the middle of non-atomic hardware programming. But at the very least the locking in the output path should take into account kdb_active and SCHEDULER_STOPPED.

Input path locking

Input path is the code involved into getting data from keyboards and similar input devices. The FreeBSD kernel is interested only in keyboards input, so this article is limited only to those. Currently input path is protected by the Giant in normal contexts. The special contexts will be described later. With the exception of ukbd(4) USB keyboard driver no code in syscons(4) and other keyboard drivers depends on the special properties of the Giant. So the Giant could be easily replaced with a separate "syscons lock" mutex. Peculiarities of ukbd(4) will be described later. The Giant is almost never acquired explicitly. Mostly its use is requested via the following means:

So the above cases would have to be properly converted if/when the Giant is replaced with the syscons lock.

The Giant is acquired explicitly only in sckbdevent, sckbdevent is always called as kb_callback.kc_func and it seems that in most if not all cases it is invoked either from a callout or from an interrupt handler (called via kbdd_intr). So it is possible that this use of the Giant can be replaced with mtx_assert(MA_OWNED). It seems that kb_callback.kc_func must be called with the Giant held by necessity as the code that calls it also modifies internal state of the corresponding keyboard driver. NB: adb_kbd seems to be an exception.

In normal context the kernel doesn't interact with syscons's input path. If kernel requires console input then it is either already in one of special contexts or must enter such a context. Otherwise the input path is connected to userland via the tty layer. In this mode the keyboard drivers operate in an interrupt driven mode. When the kernel uses the input path, then it always polls the syscons and via it the keyboard drivers. Internally the drivers may still be interrupt driven, but they must not report any input events upstream and must wait for being polled. The keyboards are put into the polled mode via a special method. An important thing to note here is that the kernel doesn't acquire any designated lock while polling the syscons, so there should not be any concurrent access to the syscons internal state as well as the internal state(s) of the keyboard driver(s) during kernel polling. So, once again, since the kernel doesn't acquire any lock to ensure that, then the syscons and/or the keyboard drivers must provide the necessary concurrency protection. This can be achieved either via locking or by designing the code in such a way that the polled flag prevents the concurrent code from altering the state in parallel with kernel polling. In the latter case the polling flag itself must still be lock protected.

Another point is that the kernel never has to compete with the userland for access to the syscons input path as will be shown further. Any concurrency is always between the kernel-side threads and/or interrupts.

Special contexts and locking

1. Pause after each line mode

This is an early boot context. This mode is not supported by all the keyboard drivers. Only drivers that support early initialization (before usual discovery of system resources) can support this mode. As such, the ukbd driver doesn't support this mode. In this mode interrupts are not enabled yet, there is only a single thread0 running and so the polling works naturally without any complications. It seems that at present the atkbd driver doesn't properly handle its own "non-early" initialization if this mode is active.

2. mountroot prompt

At this stage the interrupts are already enabled and there are multiple threads of execution. There are still no userland processes. This is the most complex context from the point of view of locking. The keyboard driver's interrupt handler may get invoked but it must not interfere with kernel polling. So, in this context in particular, the keyboard drivers must ensure that either their interrupt handler do not fire or that they do not modify parts of the internal state that can be concurrently accessed by the polling code path. Giant is already held in this context - it is acquired in start_init before the vfs_mountroot call.

3. KDB/DDB context

In this context the kernel is essentially single-threaded. All but one CPUs are paused and only one thread is executing. In this case there is no need for protection from concurrent access. Additionally the only executing thread must not get blocked on any locks, so it explicitly should bypass any normal locking. Polling should work out naturally in this case.

4. Panic context with SCHEDULER_STOPPED

This context is very similar to the above context by its properties. The difference is all the locks are automatically bypassed, so there is no need for explicit checks.

5. Late shutdown context

This is a context where the kernel may ask to press a key for some action. E.g. after shutdown -h the kernel asks "to press any key to reboot". At this stage the interrupts and the scheduler should still be functional. The Giant is held by the shutdown thread - it's taken in sys_reboot.

Summary

It's important to note that the normal locking is usable in all of the above contexts except the number 3. In the contexts 1, 2 and 5 it simply just works, in the context 4 the locking is auto-magically skipped. So any explicit locking in the code that can be executed in the context 3 must have an explicit check for kdb_active.

Another important note is that there are three ways to call into the syscons code and into the keyboard drivers code:

The first two are already protected by the proper Giant locking, so there is no need for additional locking in these code paths. On the other hand the entry points for the third path must be explicitly protected. That protection must obey the previous rule about kdb_active.

Additional observation is that the kernel never really competes with the userland for access to the keyboard(s). The kernel polls the console for input either when there is no userland yet or there is no userland already or either panic or kdb are entered and the userland is "suspended". Also, it can be seen that at present there is no concurrent access to the console input from the kernel side too. So the code doesn't expect this and is not equipped to properly handle such a situation yet.

One more thing to note is that at present the kernel always accesses the console input in a well-defined locking locking context. Either it is a "single-threaded"/"lock-less" context of the panic/kdb or a rather clean context of a specific system boot or shutdown stage.

Interaction between the input and output paths

The current code has one non-trivial interaction between the output path and input path. syscons tries to provide the following behavior: if a Scroll Lock mode is active and kernel prints something to a console, the the Scroll Lock must be deactivated and correspondingly a Scroll Lock LED on a keyboard must be turned off. As mentioned above, the kernel may produce output from arbitrary contexts, but turning off the LED requires calling into the keyboard drivers which require specific locking and currently use the Giant. So a direct call may result in a LOR and potentially a deadlock (provided that all the locking requirements are properly satisfied). One way to work around this problem is to execute the action asynchronously via a taskqueue.

ukbd

The problem with ukbd is that it has to interact with rather complex hardware compared to AT compatible keyboard controller, besides also unlike the AT keyboard controller the USB controller can be shared with rather non-trivial peripherals and their drivers. By necessity the underlying USB subsystem must use multiple threads in most of the contexts. Thus the ukbd driver must honor concurrency and locking strategies of the USB subsystem when the USB threads are active.

Summary. Context 1 is not applicable to the ukbd. In contexts 3 and 4 the USB threads do not run, so the ukbd doesn't have to acquire any locks (in fact in context 3 it must not acquire any locks) and also it must directly go all the way to hardware via direct calls. In context 2 the ukbd must acquire the necessary locks, the Giant currently, if it is called via routines related to polling. It's important to note that in this context the ukbd driver works in interrupt driven mode internally. The USB threads are running and they perform the actual interaction with the hardware. So the ukbd must not access the hardware directly. Also, if it has to sleep/wait for the USB threads, then it should release the Giant while doing so, so that the USB threads can interact with the ukbd under Giant protection. ukbd is detached before context 5 is entered, so it is useless there and that context is irrelevant to ukbd.

Routines for non-polling access should already be called with the proper locks, that fact can be asserted in the code.

AndriyGapon/AvgSyscons (last edited 2016-07-21T11:02:12+0000 by KubilayKocak)