Leading items

Welcome to the LWN.net Weekly Edition for December 16, 2021

This edition contains the following feature content:

• Wrangling the typing PEPs: how best to handle typing annotations in Python, whether annotations can be used for anything other than typing, and the prospect of typing becoming nearly mandatory for Python developers.
• Stochastic bisection in Git: tracking down a regression when tests can be misleading.
• Blocking straight-line speculation — eventually: the kernel may finally get protection against a Spectre vulnerability that was disclosed in mid-2020.
• Digging into the community's lore with lei: email tools for improving access to kernel mailing lists.
• Adding fs-verity support for Fedora 36?: Fedora developers consider implementing fs-verity in an upcoming distribution release.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Wrangling the typing PEPs

When last we looked in on the great typing PEP debate for Python, back in August, two PEPs were still being discussed as alternatives for handling annotations in the language. The steering council was considering the issue after deferring on a decision for the Python 3.10 release, but the question has been deferred again for Python 3.11. More study is needed and the council is looking for help from the Python community to guide its decision. In the meantime, though, discussion about the deferral has led to the understanding that annotations are not a general-purpose feature, but are only meant for typing information. In addition, there is a growing realization that typing information is effectively becoming mandatory for Python libraries.

Background

Annotations in Python are meant to allow specifying some attribute that gets associated with, originally, function parameters and their return value; eventually that mechanism was extended to variables as well. From the beginning, annotations were used to specify type information for those elements, but the language itself did not require anything other than syntactically correct Python for the value of an annotation; it would turn the value into a Python object that gets stored. Until Python 3.5 in 2015, there was not even a standard on how to specify type information for annotations; that came from the type hints effort.

The annotation information is available at run time in the __annotations__ dictionary for the annotated object, but type hints were largely meant to be used by static type checkers and other tools that never actually consult that dictionary. Instead, those kinds of tools simply parse the Python themselves. But there are libraries that actually use the annotations at run time. There may even be uses of annotations that are not tied to typing information at all, though examples of that are thin on the ground. The Python ecosystem is enormous, however, and the annotations feature was never strictly tied to typing, so Someone Out There surely could be doing their own thing.

PEP 563 ("Postponed Evaluation of Annotations") was accepted in 2017, added as an opt-in feature in Python 3.7, and was set to replace the existing implementation in Python 3.10. The problem with forward references to types that have not yet been defined was the main impetus for the PEP; it deferred the evaluation of annotations by storing them as strings and requiring anyone who wanted to use them as Python objects to call eval() or typing.get_type_hints() to evaluate them. This new behavior was gated by a __future__ import, but was set to become the only available behavior in 3.10.

PEP 563 had further implications beyond just changing how the annotations were handled by the language. In particular, the evaluation of the strings required for run-time uses of the annotation values was done in a different scope than when it was done at compile time, which led to various problems for that use case. So PEP 649 ("Deferred Evaluation Of Annotations Using Descriptors") was a late-breaking proposal that was meant to fix some shortcomings of the earlier PEP. Instead of storing the annotations as strings that need to be evaluated later when they are used, PEP 649 turned them into descriptor functions that would be run once, the first time elements of __annotations__ are accessed. Those functions would effectively operate in the same scope as the existing interpreter uses when it generates the values, so the net result would be generally the same as the existing behavior.

Back in April, given the questions surrounding the feature, the imminent Python 3.10 feature freeze, and the two competing PEPs, the steering council deferred switching the behavior to that of PEP 563. The background here is meant as a capsule summary; the articles linked to (and the links from those) will provide lots more history and details for interested readers.

More recent events

In October, PEP 563 author Łukasz Langa published a lengthy blog post discussing the two PEPs and their strengths and weaknesses. In it he noted that it would be sensible to simply adopt PEP 649 had PEP 563 never come about. Langa found much to like about PEP 649:

Looking at PEP 649 in isolation, it provides a more flexible and elegant solution to the problem. It avoids unparsing annotations from the AST [abstract syntax tree] form back into strings, which is AFAICT [as far as I can tell] unheard of in the world of programming language implementations. It avoids proliferation of strings in annotations which means that in many happy cases the user might be blissfully unaware of them. With PEP 563 they’re user-visible which is suboptimal. Clearly, as long as the PEP can be similarly performant, it represents better engineering.

But, since there is no option to go back in time to before PEP 563 was adopted, that is not the world we live in. If, for example, PEP 563 were to be deprecated in favor of PEP 649, he asked, where would that leave code that is using the new feature and that needs to be compatible with a wide range of Python versions? Depending on whether PEP 649 was added as an opt-in feature (behind a different __future__ import), as the PEP itself suggests, or if it is adopted as the language default, as has also been discussed, there are different kinds of problems for developers using the feature. Langa suggested that maybe a middle course could be found where the one specific case that led to the deferral of PEP 563 as default could use a PEP 649 technique:

This approach, while less pure than PEP 649, would entirely avoid the necessity for complex deprecations which I believe would provide a better end user experience. It would solve what made people unhappy while keeping what makes PEP 563 effective (its availability from Python 3.7 and runtime efficiency).

On November 17, Barry Warsaw posted a note to the python-dev mailing list on behalf of the steering council. It described the status of the two PEPs from the perspective of the council, while rendering a verdict (for now):

We have concluded that we lack enough detailed information to make a decision in favor of either PEP. As we see it, adopting either PEP 563 or PEP 649 as the default would be insufficient. They will not fully resolve the existing problems these PEPs intend to fix, will break some existing code, and likely don’t address all the use cases and requirements across the static and dynamic typing constituents. We are also uncertain as to the best migration path from the current state of affairs.

Defer decision on PEP 563 and 649 in 3.11

As such, at this time, the only reasonable path forward that the SC [steering council] sees is to defer the decision in Python 3.11 again, essentially keeping the 3.10 status quo. We know that this is far from ideal, but it’s also the safest path since we can’t clearly make the situation better, and we don’t have confidence that either PEP solves the problems once and for all. Pragmatically, we don’t want to make the situation worse, and we really don’t want to find ourselves back here again in a couple of releases because we overlooked an important requirement for a set of users.

The post included a call for help from the community in order to better understand the requirements for both static and dynamic typing throughout the Python ecosystem. There is also a need to rally "typing enthusiasts to help build consensus with either of the proposed PEPs" or to find an alternative, perhaps along the lines of what Langa suggested. Beyond that, the council is looking for a neutral party who can help shepherd some kind of a solution going forward.

As might be guessed, that set off a longish thread discussing the issues. One thing that was clear, at least from the wording of that announcement, is that the council was only really considering annotations for typing purposes, thus precluding other uses of annotations, by implication anyway. Christopher Barker pointed that out as something that could probably use some clarification:

Annotations can be, and are, used for other things than "typing". I just noticed that PEP 563 apparently deprecated those other uses (well, sort of: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated"), but if the SC is reconsidering PEP 563, then it would be nice to be clear about whether non-typing uses of annotations are indeed deprecated. If not, then the challenge is to come up with a way forward that not only supports both static and dynamic typing, but also other potentially arbitrary use cases.

He briefly described his use case, which will no longer work with the PEP 563 changes. His code was written after PEP 563 was approved (with the language he quoted), so it may be his "fault" that he used the annotations feature incorrectly. If that is the case, he will be a bit disappointed, but there is a larger issue at hand, he said:

But the fact is that I, among others, have been a bit uncomfortable about the focus on typing in Python for years. But when issues are raised, we have been repeatedly told that typing is, and always will remain, optional. In short, it was made clear that anyone not interested in typing could safely ignore the discussions about it. And thus, a number of PEPs came and went, and those among us that did not choose to involve ourselves in the conversation did not pay attention to the details.

And thus we didn't notice that buried in what seemed like a typing PEP, was, in fact, a [deprecation] of any non-typing uses of an existing Python feature. (also to be fair, the title of the PEP is "Postponed Evaluation of Annotations" -- so should have caught the attention of anyone interested in annotations for any use.

Paul Moore agreed with Barker that more clarity is needed:

It's becoming harder and harder for people not particularly interested in static typing to simply ignore it, and any use of annotations to affect runtime behaviour is in a weird grey area. And as a library author, I'm now finding that I'm getting requests to add typing to my code "for my users" (i.e., using types is no longer just a choice I make for my project, it's an API design issue).

Original intentions

While the non-typing users of annotations have a legitimate complaint, Stephen J. Turnbull said, he believes that the writing has been on the wall for some time. His understanding is that typing information is and always has been "considered the primary use case for annotations" by former benevolent-dictator-for-life (BDFL) Guido van Rossum, and that the council has been following that lead. Greg Ewing remembered things somewhat differently:

[...] the BDFL didn't say that, or at least didn't say it very clearly. It sounded more like type hints were just one of many possible uses, and he encouraged people to experiment. There were even discussions about coming up with a convention to manage conflicting uses of annotations in the same code. That wouldn't have happened if typing were considered the only supported use.

Van Rossum is not sure that his communication was completely clear, but said that at least in his mind annotations were always about typing:

My memory is also hazy, but I'm quite sure that *in my mind* annotations were intended as a compromise between conflicting proposals for *typing*. We didn't have agreement on the syntax or semantics, but we did know we wanted to do something with types eventually. Some folks wanted to enforce types at runtime. Others wanted to use them to generate faster code. Yet others wanted types to be checked by the compiler. The term "gradual typing" wasn't invented (or hadn't reached our community) yet, and offline static type checking wasn't something we had thought of either (I think). But it was clear that typing would have to be optional.

Warsaw remembers things a little differently as well:

My recollection of the history of annotations falls somewhere between Greg’s and Guido’s. Annotations as a feature were inspired by the typing use case (with no decision at the time whether those were to be static or runtime checks), but at the same time allowing for experimentation for other use cases. Over time, annotations-for-typing clearly won the mindset and became the predominant use case.

But Oscar Benjamin pointed to PEP 3107 ("Function Annotations") from 2006, which lists numerous use cases, many of which were not for typing, at least directly. "As I remember it, a decision about the purpose of annotations was *explicitly* not made when they were introduced." Antoine Pitrou concurred: "Annotations were purposefully use case-agnostic, and there was no stated desire to push for one use case or another."

The intent when annotations were added is not necessarily relevant 15 years later, because the language and its users have largely excluded non-typing use cases. Warsaw said that "while all the signs are there for 'annotations are for typing', this has never been explicitly or sufficiently codified". He suggested that a PEP be written to that effect, in order to remove all doubt.

[...] We can lament the non-typing use of annotations, but I think that horse is out of the barn and I don’t know how you would resolve conflicts of use for typing and non-typing annotations. It’s been a slow boil, with no definitive pronouncement, and that needs to be fixed, but I think it's just acknowledging reality. That’s my personal opinion.

Pressure

As Moore noted, there is an increasing effort to push for typing annotations in libraries throughout the Python world. Steve Dower also sees that type checking is not really optional for many projects, which is leading to some people advocating for "annotations everywhere". The "optional" typing feature is heading toward being mandatory, Moore said:

There's a subtle (maybe not so subtle, actually) and increasing pressure on projects to add typing. Often with little or no justification beyond "you should", as if having typing is a sort of "obvious best practice". Sometimes "because it will make it easier for your users who use typing" is given as a justification, but while that's fair, it's also a disturbing gradual pressure for typing to extend everywhere, manifesting by making it feel like not adding typing is somehow "not caring about your users".

Benjamin said that code editors may be behind some of that push. His students sometimes start adding type annotations even though he has deliberately avoided the topic. "I can only presume that some editor is doing this for them or telling them that they need to do this (students often can't tell the difference between editor warnings and actual errors)." He has also seen a push to add annotations to SymPy, some of which are not particularly useful from a typing perspective but make some editors work better:

Some people apparently want to add type hints that look completely useless to me like

    def f() -> Union[MyClass, Any]

As I understand it this does not give any meaningful information to a type checker but it apparently makes vscode work better

Benjamin is not exactly opposed to adding typing information to SymPy, "but it's a huge amount of work that someone would have to do and I don't want to add useless/inaccurate hints temporarily (as some have suggested to do)". Moore agreed, but said "it is awfully tempting to passive-aggressively annotate everything as Any, just to shut people up :-(". Steven D'Aprano took that joke one step further by suggesting setting up a logical conflict between PEP 8 purism and typing activism:

We could update PEP 8 to ban type annotations, then watch as the people who over-zealously apply PEP 8 to everything AND over-zealously insist on adding type annotations to everything have their heads explode.

Benjamin had suggested that he saw more value for type annotations on the internals of SymPy, but Sebastian Rittau said that users generally want this information for the API, which can be supplied from stub files. Those files, which are only consumed by type checkers, editors, and similar tools, can be created and maintained outside of the projects; they contain type information for the APIs of various libraries. He listed several resources for those who are trying to add typing information (or review pull requests that add it), including a documentation hub, "but there is not much to see at the moment".

Moore was glad to see the pointers, but said that "the most critical missing resource is a central set of typing documentation that includes examples, FAQs and best practices as well as reference materials". He continued:

TBH [To be honest], I'd quite happily not use typing if I didn't want to and stay quiet. A lot of the frustration I see being expressed here (including my own) seems to come from the fact that it's so difficult to actually take that sort of "I can ignore it if I don't use it" attitude, whether that's because of community pressure, tool requirements, or whatever.

Rittau said that the documentation hub was meant to eventually have the kinds of information Moore is looking for. Rittau also noted that the typeshed project may help provide a stepping stone:

Providing high quality stubs and the best user experience is not easy. But I believe that referring people to typeshed can help. While we of course prefer high quality stubs or type annotations shipped with the package in question, typeshed can provide a fairly low barrier of entry for projects that don't have the resources to maintain type annotations themselves. It can also be used as an "incubator", where stubs are created and improved iteratively, until they are deemed ready for inclusion in an upstream package.

The future

Those are valuable resources, obviously, but they do tend to reinforce the message that the future of Python is typed. The pressure that library developers are feeling is real and likely to increase as more and more tools—and developers—come to depend on the availability of typing annotations. While they are optional from a language perspective, they are rapidly becoming mandatory from a community and ecosystem perspective. That is precisely what the "typing-suspicious crowd" (as Turnbull called them) has been worried about and it has come to pass—or soon will.

Meanwhile, though, it seems clear that anyone using annotations for non-typing purposes should be figuring out some other way to accomplish their goals. But the resolution of the two PEPs does not seem any closer at this point. Neither Langa or PEP 649 author Larry Hastings seem inclined to change their PEPs, at least yet, and the hoped-for PEP shepherd has not appeared either (publicly, anyway).

Given that there are 16 months or so before Python 3.12 feature freeze, it might be guessed that situation will have worked itself out by then. The contours of the problem are clearer, and some of the extraneous pieces have been removed from consideration, which should hopefully clear the way for a consensus to emerge. Since "practicality beats purity", according to The Zen of Python, something like what Langa has proposed may well be the "winner". It would seem that 2022 will provide more opportunities to finally put this issue to bed.

Comments (30 posted)

Stochastic bisection in Git

Regressions are no fun; among other things, finding the source of a regression among thousands of changes can be a needle-in-the-haystack sort of problem. The git bisect command can help; it is a (relatively) easy way to sift through large numbers of commits to find the one that introduces a regression. When it works well, it can quickly point out the change that causes a specific problem. Bisection is not a perfect tool, though; it can go badly wrong in situations where a bug cannot be reliably reproduced. In an attempt to make bisection more useful in such cases, Jan Kara is proposing to add "stochastic bisection" support to Git.

Bisection looks for problem commits using a binary search. The developer identifies the latest known good commit with git bisect good and the earliest known commit showing the bug with git bisect bad. Git will then find a commit near the midpoint between the two, check out that commit, and wait for the developer to try to reproduce the bug. Another git bisect command is used to mark the commit as "good" or "bad", and the process repeats, dividing the range of commits in half each time, until only one commit remains. That commit must be the one that introduced the bug in question.

This technique can be powerful. A bug introduced in a 12,000-commit kernel merge window can be narrowed to a single commit in 14 bisect cycles, which makes the process of finding the actual bug much easier. But it works less well when dealing with bugs that are difficult to reproduce and which, thus, may not manifest in any given testing cycle. A 14-step bisection is 14 opportunities for the developer to provide an incorrect result, and it only takes one such to throw the entire process off. It is not uncommon to see nonsensical bisection results posted to mailing lists; they are often caused by just this kind of problem.

Stochastic bisection is a way of trying to adapt the bisection process to situations where it is not 100% clear whether a bug is present at a given point in the history. It can't say for sure which commit is responsible for the bug, but it can find the commit that is probably at fault. To use this feature, a developer will supply the desired confidence level (as a number between zero and one) with the --confidence option to git bisect start. The process will then continue until one commit is identified as being the most likely source of the bug with the requested confidence level.

From there, the bisection process starts as usual, picking a commit in the middle of the specified range. With each test, though, the developer will indicate their confidence in the result they report with the same --confidence flag. A commit that exhibits the bug is almost certainly bad, so a command like:

    git bisect bad --confidence 1.0

would be appropriate. Tests that fail to show the bug may be more ambiguous, though; perhaps the race condition that causes the bug to manifest just didn't trigger this time. If the developer suspects that the bug might be present even though the test indicated otherwise, they can specify a lower confidence level.

Kara included an example in the series cover letter showing how this process would work. It starts with a commit history looking like this:

    A--B--C--D-----F-----H--------K
     \     \  \-E-/     /        /
      \     \--------G-/        /
       \------------------I--J-/

The problematic commit is I, but the developer only knows that A works and K occasionally shows the problem. They go through the steps, reporting all results (some of which are incorrect) with a confidence level of 0.7; eventually Git reports that commit I is the likely culprit with the requested confidence level of 0.9.

One thing from the example will jump out at developers who have used bisection in the past. A normal bisection process will not present the same commit to the developer for testing twice; once a verdict has been rendered on any given commit, it will not change. Stochastic bisection, instead, will ask the developer to rerun tests on commits that have been seen before, sometimes multiple times in a row. The result is a rather longer bisection process. Given the 11-commit history shown above, a normal bisection would be done after four steps; the stochastic bisection process, instead, required 14 steps. The relative expansion of a bisection over a larger range of commits is likely to be significantly less, but it will still make the process longer. A longer process is better than one that yields a nonsense result, though.

Perhaps the most substantive comment in response to this posting came from Taylor Blau. Kara's patches try to share as much logic as possible between normal and stochastic bisection, which is a worthy goal. In the process of joining the two, though, Kara caused the bisection points reached in some of the Git tests to change; the new points are just as valid, but they are different, so Kara had to change the tests as well. Blau would rather see the new feature added without any behavior changes for developers who are not using it. Kara responded that he feels that his solution is the right way to do things, but he also expressed a willingness to change things if the Git developers request it. Nobody has given him an answer on that question at this point.

There are other changes that will be needed, of course, as is usually the case for a complex patch set, so there should be at least one revised version posted to the list at some point. Stochastic bisection will not be useful for all developers, but there are clearly situations where a more probabilistic approach is required. It would thus be surprising if some version of this work didn't eventually find its way into the Git mainline.

Comments (24 posted)

Blocking straight-line speculation — eventually

The Spectre class of vulnerabilities was given that name because, it was thought, these problems would haunt us for a long time. As the fourth anniversary of the disclosure of Meltdown and Spectre approaches, there is no reason to doubt the accuracy of that name. One of the more recent Spectre variants goes by the name "straight-line speculation"; it was first disclosed in June 2020, but fixes are still trying to find their way into the compilers and the kernel.

Straight-line speculation differs somewhat from the other Spectre vulnerabilities. To review, the classic Spectre problem afflicts code like this:

    if (offset < obj->array_length)
       do_something_with(obj->array[offset]);

On its face, this code is safe; it will only attempt to index into obj->array if the given offset is within bounds. A CPU running this code, though, may be unable to fetch obj->array_length from cache, meaning that it will have to wait for that value to come from memory. Rather than do nothing, the CPU can make a guess as to how the comparison will turn out and continue execution in a speculative mode; it may guess wrong and index obj->array with an out-of-bounds offset. Again, this shouldn't be a problem; once the array length shows up and it becomes clear that the branch was not correctly predicted, the speculative work will be thrown away.

The problem, of course, is that this speculative execution can leave traces elsewhere in the system (most often the memory caches) that can be used to exfiltrate data that an attacker would otherwise be unable to access. In the worst cases, Spectre vulnerabilities can be used to attack the kernel or to carry out attacks between virtual machines running on the same physical host. They are a real threat, which is why numerous mitigations have been adopted to thwart these attacks despite a high performance cost.

Straight-line speculation, which was initially disclosed in this white paper from Arm, differs in that it does not depend on erroneous branch prediction; indeed, no conditional branches are involved at all. Instead, it takes advantage of some strange behavior around unconditional control-flow changes. There are a lot of instructions that will result in a change to the program counter; on Arm, these include instructions that generate exceptions, but also unconditional direct branches and the RET instruction to return from a function call.

One would think that it would not be all that hard for a speculative-execution mode to do the right thing (stop speculation, if nothing else) when it encounters one of these instructions. But, as it turns out, on some CPUs, in some situations, the CPU can execute speculatively right past one of those instructions as if it were not there — in a straight line, as it were. If an attacker can place code that will carry out the desired access immediately after one of these instructions, that code may be executed speculatively, with the usual effects elsewhere in the system. It is a hard vulnerability to exploit (it's not clear that anybody has done it in the real world), but it is still a vulnerability that is worth fixing.

This vulnerability in Arm CPUs was assigned CVE-2020-13844. Patches were quickly added to GCC and LLVM to address the problem; the new -mharden-sls flag can be used to turn the new mitigations on. Happily, this problem is relatively easy to fix; placing a barrier instruction after the unconditional jumps will do the trick. Since the jumps are, as noted, unconditional, those barriers will not actually be executed for real, but they will be encountered during straight-line speculative execution, at which point the speculation will stop. So there is no real performance impact, other than a slight increase in executable size, from this change.

As it happens, this is not just an Arm problem; some x86 processors can speculatively execute past RET instructions as well. The solution in this case is to place an INT3 instruction after the RET to stop speculation. INT3 is a single-byte instruction normally used by debuggers to set breakpoints; once again, it will not actually be executed when used in this way.

The compiler changes are in place to deal with this vulnerability (though they do not yet appear in released versions), but the kernel has not yet been updated to match. Among other things, the difficulty in exploiting this vulnerability has made addressing it seem less than fully urgent.

Jian Cai posted a patch series in February adding protection for Arm, but that work has not been merged. Arm developer Will Deacon was opposed to the patch:

I still don't see why SLS is worth a compiler mitigation which will affect all CPUs that run the kernel binary, but Spectre-v1 is not. In other words, the big thing missing from this is a justification as to why SLS is a problem worth working around for general C code.

That patch series has not returned since.

The x86 side has seen more recent work in the form of this patch series from Peter Zijlstra. It, too, uses the new compiler options (when available) to insert the needed instructions, but (as Deacon had noted in response to the Arm patches) that is only part of the problem. There is quite a bit of assembly code in the kernel, none of which will be addressed by changes to the compilers. So Zijlstra laboriously went through that code, replacing all RET instructions with a macro that can, when the right configuration options are selected, expand to include the INT3 instruction as well. For good measure, the objtool utility is also enhanced to check that INT3 instructions have been inserted where needed as a way of preventing potential vulnerabilities from being added in the future.

Given that nearly a year and a half has passed since this vulnerability was disclosed, it is clear that nobody is feeling that it must be addressed urgently. The slow progress toward protection from straight-line speculation is a far cry from the scrambling that took place prior to the initial Meltdown and Spectre disclosures. But one thing we should have learned by now is that attackers will eventually figure out a way to make use of any opening, no matter how difficult it is to exploit. So protection from straight-line speculation vulnerabilities with little performance penalty is welcome, even if it comes late.

Comments (34 posted)

Digging into the community's lore with lei

Email is often seen as a technology with a dim future; it is slow, easily faked, and buried in spam. Kids These Days want nothing to do with it, and email has lost its charm with many others as well. But many development projects are still dependent on it, and even non-developers still cope with large volumes of mail. While development forges show one possible path away from email, they are not the only one. What if new structures could be built on top of email to address some of its worst problems while keeping the good parts that many projects depend on? The "lei" system recently launched by Konstantin Ryabitsev is a hint of how such a future might look.

One of the initial motivations for creating LWN, back in 1997, was to spare readers from the impossible task of keeping up with the linux-kernel mailing list. After all, that list was receiving an astounding 100 messages every day, and no rational human being would try to read such a thing. Some 24 years later, that situation has changed: linux-kernel now runs over 1,000 messages per day, and there are dozens of other busy, kernel-oriented mailing lists as well. It is easy to miss important messages while trying to follow that kind of traffic — and few developers even try.

While much of the traffic that appears on any mailing list is quickly forgettable, some of it has lasting value; that means that good archives are needed. For most of the kernel project's history, those archives did not exist. There were indeed archives for most lists, but they were scattered, of mixed reliability, difficult to search, and usually incomplete. It is only a few years ago that Ryabitsev put together lore.kernel.org to serve as a better solution to this problem. By using a search-friendly archiving system (public-inbox), building complete archives from pieces obtained from numerous sources, and archiving most kernel-oriented lists, Ryabitsev was able to create a resource that quickly became indispensable within the community.

Lei (which stands for "local email interface") comes out of the public-inbox community. It works nicely with lore, to the point that Ryabitsev refers to the whole system as "lore+lei". The idea behind this combination is to create a new way of dealing with email that allows developers to see interesting messages without having to subscribe to an entire list.

Public-inbox is built on some interesting ideas, including the use of Git to store the archive itself. The real key to its usefulness, though, is the use of Xapian to implement a fast, focused search capability. The "fast" part allows for nearly instantaneous searches within the millions of messages in the email archive; this query, for example, shows immediately that the term "dromedary" has been used exactly 30 times in all of the lists archived on lore.

The search mechanism differs from that found in many email clients in that it implements search terms that are useful in the context of a technical mailing list. So, while searching for "dromedary" finds occurrences of that word, "nq:dromedary", instead, only turns up occurrences that are not in quoted text being replied to. That reduces the number of hits to 13 without missing any of the original occurrences of the word. It is also possible to search for terms in a number of message headers, the names of files touched by patches, the names of functions changed by patches, and more; see this page for details.

The purpose of lei, in short, is to take advantage of the search features built into public-inbox to give developers highly filtered views of mailing-list traffic. It takes a search query and uses it to populate a mailbox, which can then be perused with the user's client of choice. As an example, take this query, which appeared in Ryabitsev's announcement of lei in November:

    lei q -I https://lore.kernel.org/all/ -o ~/Mail/floppy \
      --threads --dedupe=mid \
      '(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy \
      OR ((nq:bug OR nq:regression) AND nq:floppy)) \
      AND rt:1.month.ago..'

The idea here is to obtain all emails that might be relevant to the maintainer of the kernel's vitally important floppy driver. It looks for any of:

• patches that touches floppy.c (dfn:drivers/block/floppy.c)
• patches that change a function whose name starts with floppy_ (dfhh:floppy_*)
• messages with "floppy" in the subject (s:floppy)
• messages that mention both "floppy" and either "bug" or "regression" in non-quoted text ((nq:bug OR nq:regression) AND nq:floppy)

The search is limited to messages sent in the last month. The lei command will go to lore and execute this search on all mailing lists, collect the matched messages, and store them in the maildir folder ~/Mail/floppy. They can then use whichever tools are best for working with the messages in that folder.

Lei will remember this search, so updating the folder with new messages at a later time is a matter of a simple "lei up" command. As described in this followup post, it is also possible to instruct lei to store the retrieved messages in a remote IMAP folder rather than a local maildir folder.

The intent behind this work is clear: it lets kernel developers keep up with email traffic that is of interest to them without the need to subscribe to a set of high-volume mailing lists. No developer can subscribe to all lists where relevant messages might appear; with lore+lei, they no longer have to even try. It may also make email more reliable for many users. There is, for example, an ongoing low rumble of complaints regarding problems getting email from kernel lists delivered to Gmail users; use of lore+lei can route around such problems entirely.

In one sense, lei follows the pattern we have seen in other parts of the Internet. Those of us who watched the World Wide Web develop all those years ago will remember the extensive efforts that went into trying to index the whole thing. Yahoo got started as a hierarchical directory of the web, for example. After a while it became clear that such efforts were hopeless and that search was the right tool for the job of finding things on the net. Lei is a statement that the same is true for email conversations; organizing our discussions into a set of topic-focused lists has gotten us far, but this approach is clearly under strain now.

Maybe, the reasoning goes, it's time to forget about all those lists and just use searches instead. As a step in that direction, Ryabitsev has created a mailing list for patches — all patches, regardless of which subsystem they affect. Developers are encouraged to copy their patch postings to this list. Anybody who is interested in specific patches can use tools like Lei to filter out the rest.

What is potentially being lost here, of course, is the serendipity of finding interesting emails that one might never think to search for. If lei serves to further isolate kernel developers into their own niches, there could be an adverse effect on kernel development as a whole. Cross-subsystem discussions could become harder, and developers could lose awareness of what is happening elsewhere in the project. Filter bubbles are already a problem in the wider world; they could make it harder to maintain the cohesiveness of a large free-software project as well.

Then again, such a world sounds like fertile ground for news sites providing a broad view of what's happening in the community, so perhaps it's not an entirely bad thing.

Finally, there is one other aspect of this work that is worth thinking about. The functionality implemented in lore+lei is highly useful in its own right, but one could also argue that it's really just the database layer that will sit underneath a new generation of collaborative-development tools. The fact that those tools don't exist yet is inconvenient, but hopefully there are developers out there who are starting to think about how to fill that void. That would, in the end, finally provide a path for email-dependent free-software communities to, at least on the surface of things, move away from email and onto something better. The email-based infrastructure underlying it all could become an implementation detail that users need not worry about if it does not interest them. That could be a future worth working toward.

Comments (13 posted)

Adding fs-verity support for Fedora 36?

Adding fs-verity file-integrity information to RPM packages for Fedora 36 is the topic of a recent discussion on the Fedora devel mailing list. The feature would provide a means to install files from RPM packages as read-only files that cannot be read or otherwise operated on if the data in the files changes at any point. The proposal is mostly about making the plumbing available for use cases that are not particularly clear—which has led to some questions and skepticism among those participating in the thread.

fs-verity and RPM

Fs-verity is a kernel feature that is supported by some filesystems; it provides a way to ensure that the contents of a file cannot change on disk. It revolves around a Merkle tree that is created for each file being protected; the tree contains hashes of each data block in the file. When a file is protected by fs-verity, it is marked as read-only and every read operation checks that the block read matches the value stored in the tree; the operation fails if there is no match. In addition, the tree itself can be cryptographically signed to ensure that nothing has been changed underneath the filesystem by, say, accessing the raw block device or image file.

Fedora program manager Ben Cotton posted the Fedora change proposal to add fs-verity support on behalf of the feature owners: Davide Cavalca, Boris Burkov, Filipe Brandenburger, Michel Alexandre Salim, and Matthew Almond. There are several elements to the plan. To start with, the Koji build system needs to be able to create and sign the Merkle tree for each file that gets shipped in the RPM package. The tree itself is not added to the RPM package, just the signed top-level hash for each file.

On the other end, an optional fs-verity RPM plugin would install the Fedora key and enable fs-verity for each file it installs. The filesystem would then recreate the Merkle tree, check it against the signature in the RPM metadata, and store the tree with the file. After that, each access to the file will be checked against the tree, which means that various kinds of operations (e.g. read(), mmap(), execve(), etc.) will only proceed if the data blocks on disk have not changed.

The proposal mainly focuses on the build side of the equation: "Specifically, installing and enabling the fs-verity rpm plugin by default is explicitly considered out of scope here." The overhead of creating the Merkle tree at installation time did not "appear to meaningfully slow down package installs during empirical testing", but there is some (unspecified) cost of creating the tree for every Koji build, of course. The Merkle tree is only stored if the RPM fs-verity plugin is enabled and adds roughly 1/127th (0.8%) to the size of the installed file. All RPMs would get additional metadata, in the form of signatures, if the proposal is adopted, but even that is fairly negligible: "in the vast majority of cases we expect to see minimal to no size increase thanks to RPM header packing".

Reaction

Kevin Fenzi had a number of questions about the proposal. He wondered which keys would be used to sign the Merkle tree at build time and who would work on the changes to the RoboSignatory component that supplies signatures to Koji in order to make all of this happen. In addition, only ext4, F2FS, and, since Linux 5.15, Btrfs (commit) have support for fs-verity; what happens for other filesystems such as XFS?

Cavalca replied, noting that the feature owners plan to work on the changes to RoboSignatory, but would need some help and advice with code review, testing, and deployment. They will be looking into the possibility of using the Fedora package-signing key:

fs-verity needs a RSA key/cert pair for file signing at package signature time. At package install time, the cert needs to be loaded in the appropriate kernel keyring. We've always used a dedicated keypair during testing -- I'm not actually sure if the package signing key could be reused for this, as it's a GPG key, but this is something we should follow up on.

Beyond that, if the filesystem does not support fs-verity, the plugin will fail and there will be no protection of that sort. If it is desired, though, fs-verity support for XFS could be added, he said.

Josh Boyer asked about adding fs-verity support for Fedora when a seemingly similar feature, adding IMA signatures to RPM, was rejected due in part to its impact on RPM size. The Integrity Measurement Architecture (IMA) is a kernel-based mechanism that can be used to detect file tampering, but Salim said that the impact on RPM size is much smaller for fs-verity than for IMA. The idea is that users who do not install the RPM plugin "will be mostly unimpacted" by the change. The proposal compared the two integrity mechanisms, noting a smaller impact at run time as well:

Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.

IMA works by measuring a file as a whole and comparing its signature whenever it’s read of executed. It has a higher runtime cost than fsverity (as it needs to verify the whole file at once) and it cannot detect subsequent alternations.

The proposal also said that IMA provides a rich policy system that can be tied into other security mechanisms, such as SELinux; it might make sense to integrate the two down the road. But Boyer and others were interested in what benefit fs-verity would bring to Fedora and its use cases. Boyer said that the benefit listed in the proposal could largely be fulfilled by IMA, so he wanted to know more. Cavalca said that this particular proposal was "mostly about putting in place the necessary plumbing"; he also described one potential use case:

For example, consider an appliance-like system placed in an untrusted location where you may not be able to control who has physical access (this could be a server, but it could also be a kiosk in an internet point or a school). In this scenario, fs-verity can be one of the building blocks to ensure and maintain system trust.

As with most security solutions, Cavalca continued, fs-verity is no silver bullet, but it is a useful building block:

Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered.

Where is this useful? For example, fs-verity can help in the scenario where an attacker has out-of-band access to the storage device (say, they pull a hard drive from a colo'd server or a sdcard from an embedded device, or they boot into a liveusb, or they access a VM image directly from a host).

Let's say that happens, and the attacker changes a few blocks of /bin/ls on the device to make it run nefarious code. When you boot your system again, it would fail at exec() time because the Merkle tree wouldn't match.

But since an attacker can simply replace the files, or add new ones, there is still a missing piece, as Zbigniew Jędrzejewski-Szmek pointed out:

If fs-verity verification prevents me from successfully modifying or replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is there anything which hinders just adding /etc/systemd/system/foo.service that does whatever I want?

He asked if there was a Linux Security Module (LSM) available to enforce whatever policies are needed to ensure the integrity of the system. Beyond that, if an attacker can replace a binary like /bin/ls, they may also be able to add new keys to the kernel keyring that would allow their binaries to pass the fs-verity checks.

If the keys are loaded from the file system, can I just drop in a rogue key, similarly to what happens when new keys are distributed as part of distro upgrades?

Florian Weimer described two additional problems that he called "unsolvable". Attackers can reuse binaries from unrelated Fedora packages that have been signed properly, or they can alter the system configuration to disable the feature at boot time. Given those problems, he said, it is unlikely that anyone who actually wants this kind of a feature would use the signatures being added to the RPM files:

The combination of these two unsolvable issues suggests to me that anyone who wants to deploy this is better off with their own trust root, and that approach will include their particular version of key management as well. But this also means that pre-computed file signatures are not [particularly] useful to them. They would have to discard them anyway before deployment.

Burkov agreed that there were some missing pieces, but noted that a message from Roberto Sassu mentioned some ongoing work integrating fs-verity with IMA that might lead to a solution to some of these problems. But, in a similar vein, Lennart Poettering pointed out a number of areas where the threat model is not at all clear—the protections are not really geared to the problems that Fedora users might have:

This protects file contents, not the [metadata], right? So what about the metadata? if I see a fs-verity enabled inode with libssl.so data in it today, and it's a vulnerably version, and I make a hardlink to it, and then it gets replaced by a fixed version (with a slightly updated name) — how you intend to make sure, that i can't fool you into loading my copy of the old file but under the new name?

[...] Is there any protection against downgrades between RPM package versions? Does this in any way protect combinations of binaries/libraries? I mean, pretty much all programs we ship consist of a large number of ELF objects, and you probably need to sign the combination of them, but this model doesn't look like it offers that at all?

There are some areas that need further research, Burkov said, but the threat model is not really specified because the feature is simply meant as plumbing:

The short answer is that just this change as-is has no practical threat model, we viewed it as an enablement step that we believe has a realistic path to broadly authenticated rpm contents. Alternatively, it does give a sophisticated user an opportunity to build something like that for themselves as well.

Poettering expressed further skepticism about the proposal, noting: "Infrastructure that has no consuming feature typically doesn't work. Experience tells us that." But, even if that were overcome, the protection provided is not particularly useful: "If I can have 'ping' and 'poweroff' binaries both signed by Fedora, but then swap them without the signing being able to detect that, why do this at all?"

There were other concerns raised as well. For example, Stephen John Smoogen objected on logistical grounds. Any changes need to be in place before the mass rebuild for Fedora 36, which is currently scheduled for January 19. Given the upcoming holiday season, it is pretty unlikely that those who need to review and sign off on these kinds of changes will even be available to do so. "Even if the changes are trivial ones, this is a very short window to land things." He suggested that Fedora 37 would be a more realistic target.

The wiki version of the change proposal has been updated with further information, along with responses to many of the questions and concerns posted in the thread. But without a real use case that benefits some actual subset of Fedora users, this has the look of a feature that is going nowhere fast. Perhaps what is needed is to join forces with those who are working on IMA to come up with an integrated plan that does provide a threat model and real use cases. The IMA proposal, for example, pointed to the Fedora IoT edition as a possible consumer of that feature, which is more in line with the existing use of fs-verity in Android. Boyd made it clear that Red Hat is interested in seeing IMA support in Fedora, so there would perhaps be some synergy in a combined effort. That approach may find more traction with the Fedora community.

Comments (22 posted)