A QUIC look at HTTP/3

The Hypertext Transfer Protocol (HTTP) is a core component of the world-wide web. Over its evolution it has added features, including encryption, but time has revealed its limitations and those of the whole protocol stack. At FOSDEM 2020, Daniel Stenberg delivered a talk about a new version of the protocol called HTTP/3. It is under development and includes some big changes under the hood. There is no more TCP, for example; a new transport protocol called QUIC is expected to improve performance and allow new features.

HTTP/1 and HTTP/2

Each HTTP session requires a TCP connection which, in turn, requires a three-way handshake to set up. Once that is done, "we can send data in a reliable data stream", Stenberg explained. TCP transmits data in the clear, so everyone can read what is transferred; the same thus holds true for the non-encrypted HTTP protocol. However, 80% of requests today are using the encrypted version, called Hypertext Transfer Protocol Secure (HTTPS), according to statistics of Mozilla (Firefox users) and Google (Chrome users). "The web is getting more and more encrypted", Stenberg explained. HTTPS uses Transport Layer Security (TLS); it adds security on the top of the stack of protocols, which are (in order): IP, TCP, TLS, and HTTP. The cost of TLS is another handshake that increases the latency. In return, we get privacy, security, and "you know you're talking to the right server".

HTTP/1 required clients to establish one new TCP connection per object, meaning that for each request, the browser needed to create a connection, send the request, read the response, then close it. "TCP is very inefficient in the beginning", Stenberg explained; connections transmit data slowly just after being established, then increase the speed until they discover what the link can support. With only one object to fetch before closing the connection, TCP was never getting up to speed. In addition, a typical web page includes many elements, including JavaScript files, images, stylesheets, and so on. Fetching one object at a time is slow, so browser developers responded by creating multiple connections in parallel.

That created too many connections to be handled by the servers, so typically the number of connections for each client was limited. The browser had to choose which of its few allowed connections to use for the next object; that led to the so-called "head-of-line blocking" problem. Think of a supermarket checkout line; you might choose the one that looks shortest, only to be stuck behind a customer with some sort of complicated problem. A big TCP efficiency improvement was added for HTTP/1.1 in 1997: open TCP connections can be reused for other requests. That improved the slow-start problem, but not the head-of-line blocking issue, which can be made even worse.

HTTP/2 from 2015 uses a single connection per host, allowing TCP to get up to speed. However, the head-of-line blocking problem became even more serious at the TCP connection level. In HTTP/1 the problem was that one longer request could block others waiting for the same connection. In HTTP/2, the single connection carries hundreds of streams. In this case, when we lose one packet, "one hundred streams are waiting for that one single packet", Stenberg said. As a reminder, this is because TCP will retransmit the missing packet only when the network stack figures out that it was lost, and the network stack will only pass the data received after the gap when the missing packet arrives.

The "boxes"

Another trend Stenberg explained is protocol ossification (which LWN looked at in 2018). He explained it in the following way: the Internet is full of "boxes" (they are often called "middleboxes") such as routers and gateways. They were installed at some time and are running software to handle networking protocols as they existed at that time. The problem is that, "they know how the Internet works — at the time they were installed". For those boxes, if a given packet-header field was always zero, it is never going to be anything else. What is worse is that those boxes do not get upgraded. They are "stuck in time", he said. This is different than the servers or the web browsers, which are updated regularly.

The existence of those boxes brings limitations to the development of new versions of the HTTP protocol. An example of this is the use of TCP protocol port 80 assigned for HTTP/1.1, which is unencrypted. Currently no browser speaks HTTP/2.0 in clear text on that port. "One browser tried to do it until they figured out it doesn't work", he said. The middleboxes modified (or blocked) the traffic based on their understanding of HTTP/1, breaking HTTP/2 traffic.

Another idea to improve the protocol was to send data earlier in the TCP connection, a functionality called TCP fast open (or TFO; LWN covered it in 2012). It allowed browsers to send request data in the packets of the TCP handshake itself. Stenberg explained that it took five to seven years until all kernels supported it. Then the browsers tried it ... and it did not work. Middleboxes would just drop the TFO packets. Currently no browser enables TFO by default. A similar story happened with Brotli compression. The middleboxes only know gzip, so they break the connections using Brotli. Currently this compression is used only over HTTPS. He concluded that the introduction of new transport protocols does not work, because "your router at home will only route TCP and UDP".

The definition of HTTP/3

The difficulties with innovation in HTTP were one of the reasons for the creation of the QUIC working group at the IETF in 2016. QUIC is a name, not an acronym, Stenberg highlighted. A number of companies are interested in this development. The work of the IETF group is built on experiments with Google QUIC, a protocol deployed first in 2013 (LWN looked at it that year). The experiments used HTTP requests over UDP, with widely used client and web services. This experiment carried a fair amount of HTTP traffic, and was taken to IETF, where the working group started. Currently the IETF version is significantly different from the Google one: it includes a new transport protocol and application level.

IETF's QUIC fixes the head-of-line blocking issues and allows early data transmission like TCP fast open does. The encryption is built-in; no clear-text version of QUIC exists. HTTP/3 implemented over QUIC includes fewer clear-text messages than HTTP/2.

During the development of QUIC, the group also addressed some other modern challenges. TCP was defined with a connection tied to an IP address. Currently, devices can have multiple addresses and change them when users move around. With TCP, a new connection must be created when the interface address changes. QUIC uses a session identification separate from IP addresses to solve this problem.

QUIC uses UDP, but in a limited fashion that is more similar to the use of IP than UDP. The transport layer is in the higher layer of QUIC, above UDP; it adds connections, reliability, flow control, and security. A big difference with TCP is how QUIC handles streams within a connection. QUIC can send multiple streams in a single connection, in either direction. They are all independent, initiated by the server or the client. If a packet is lost, the implementation knows which stream is affected; only that stream will have to wait for a retransmission. The streams are internally reliable and in-order.

Applications run on top of QUIC. The protocol definition was started for HTTP; others are expected to follow, DNS for example. The definition of other application protocols is expected to start around when QUIC ships.

HTTP over QUIC is the "same but different", Stenberg said. There will still be the GET command that should be familiar to most readers, but the way the command is transmitted changes. Stenberg explained the history of HTTP: HTTP/1 was in ASCII, HTTP/2 was binary multiplexed, and HTTP/3 is binary over multiplexed QUIC, with TLS 1.3.

HTTP/3 will be faster thanks to the improved handshakes. Early numbers from the experiments showed 70% of connections with no round-trip-time (RTT) delay because the connections were already there had been established previously. The protocol allows early data, so it should improve latency even when a connection does not already exist. The independent streams should also help in low-quality networks. He noted that he could not show numbers for now, as the protocol is not finished yet. However, the expectations is that it will be "a little better to much better".

Deployment

HTTPS URLs are everywhere; they cannot be replaced without rewriting the entire web. They imply the use of TCP port 443 with TLS. The migration to HTTP/3 will thus require a connection to a legacy server. If a site supports HTTP/3, it will provide an Alt-svc header giving the server to connect to. Browsers will check that and make the second connection in the background, or they will just try both protocols at the same time. "There will be a lot of probing", he noted. There will also be support in the domain name system in the form of a new record called HTTPSSVC that will allow the provision of information on the connection parameters. In practice, it will mean asking the DNS first to check if HTTP/3 can be used.

There will be a few challenges. One difficulty may be that many companies block UDP by default as a way of blocking distributed denial-of-service attacks. With UDP, 3-7% of connections will fail due to blocking somewhere in the network. Clients need to have fallback algorithms and use them transparently. That leads to another problem: there will be no incentive to unblock UDP because the fallback will be in place.

As of today, QUIC stacks are implemented in user space to allow easy testing. "But you need to stick to one library as there are no standard APIs", Stenberg said. There are a dozen implementations right now, in many languages. Interoperability tests happen every month and the current version of the protocol as of March 2020 is draft 27.

HTTP/3 is expected to use two to three times more CPU time than the earlier versions for the same bandwidth consumption. This might delay deployment for a while. One of the reasons is that UDP is not well optimized in Linux, while "we've been polishing TCP for years", he said. Currently UDP is not made for high-volume traffic and there is no hardware offload for QUIC. In addition, performance suffers since there are also quite a few transitions between kernel and user space because the protocol stack is implemented in user space. For now, he doesn't know if QUIC will be moved into the kernel. There are some efforts to do so, but it requires a new implementation of TLS in the kernel.

TLS usage in QUIC is different, so that existing offloads will not work. The TLS protocol transmits data using "TLS records"; the records may include one or more TLS messages, and one message may span over the record boundary. In the case of TLS over TCP, both records and messages are used. Over QUIC it will send messages only, records are not needed anymore. This changes the way the TLS libraries are used and the needed APIs.

As the use of the TLS library changes between TCP and QUIC, new APIs are necessary. An OpenSSL pull request adding the QUIC APIs (PR 8797) is still being discussed; this is expected to take a while. Then, when it gets accepted, there will be another delay until it is available in a release and deployed.

Changes to the transport protocol will also force changes in the associated tools. tcpdump is not ready yet, for example. The existing tools that do understand QUIC are Wireshark and the two QUIC-specific tools qlog and qvis. Stenberg is the author of curl, which supports the latest drafts (version 25 at the time of his talk in February 2020), but without the fallback functionality; "fallback is tricky", he says. He summarized that "there is definitely a shortage" of tools and a lot of work to do.

On the browser side, nightly builds of Chrome and Firefox can have HTTP/3 enabled. For those who want to run experiments, they need to enable some specific options. In Firefox, nightly builds include HTTP/3 support; the user should go to about:config and change network.http.http3.enabled to true. Chrome Canary (not for Linux) requires specific options when launching: --enable-quic and --quic-version=h4-25 [at the time of the talk, see comments]. On the server side, an NGINX patch exists to use quiche quiche (a library implementing QUIC) for experiments. However, the other big servers, including Apache, IIS, and the official version of NGINX, do not have it yet. There is no support in the Safari browser either.

The date when the protocol will ship is not set yet, as the group prefers to do it right, not fast; he hopes for July 2020. Currently the libraries are in alpha versions; they will ship when the specification is ready. Browsers require updates of the TLS libraries. The deployment is expected to take time. He expects that it will grow more slowly than HTTP/2, but HTTP/3 is there for the long term.

Once the protocol is ready, people are waiting to add new features to QUIC, including multipath (accessing the same site using different network connections), forward error correction, and unreliable and partially reliable streams ("for video people"). Of course, other applications will also appear. QUIC development will move to version 2 after version 1 ships.

Slides [PDF] and a video of the talk are available.