A Coordinated Response to MadeYouReset HTTP/2 Protocol Attacks

A new attack on the HTTP/2 protocol, called MadeYouReset, has been disclosed and published.  Akamai's own implementation of HTTP/2 is not vulnerable to this attack, but we believe it's useful to discuss the context and vulnerability in some detail.

Responsible, coordinated vulnerability disclosure is a routine part of our participation in the larger internet ecosystem and helps us proactively protect your safety, especially when it comes to protocol level weaknesses.

When we think about the network protocols used to serve our websites, we often think about the functional case; that is, we assume that clients are talking to servers because they want to connect and use your services. But getting that right is usually the easy part. A lot of the effort in both protocol design and operation is focused on dealing with clients that aren’t functional.

They have bugs. They work in ways we didn’t expect. Or, sometimes, they’re actively malicious and trying to break the server, keeping legitimate users from reaching your service at a critical moment.

We do our best to design protocols to be resilient to unexpected or malicious peer behavior, but protocols are designed by humans and inevitably have errors. They’re implemented by humans and those implementations inevitably have bugs. And sometimes the assumptions made by the protocol differ from the way a given application actually works.

Given all of that, it’s not a surprise when we learn that something about a protocol is vulnerable to attack and we need to make a few adjustments. Fortunately, we have a process for that.

In 2023, many server operators on the internet encountered a zero-day vulnerability called HTTP/2 Rapid Reset (CVE-2023-44487) that cybercriminals had exploited to launch massive Layer 7 distributed denial-of-service (DDoS) attacks.

HTTP/2 uses the Transmission Control Protocol (TCP), which delivers information in order. While there is a delay between when the client sends data and when the server receives it, the client can rely on the server to receive everything in the order it was sent. HTTP/2 supports many simultaneous activities, carried by “streams,” which exist within the ordered stream of data provided by TCP.

Because of this ordering, HTTP/2 says that when a client or server sends something that changes the state of a stream, it can immediately consider the stream to be in the new state. Even though the other side won’t have seen the change yet, it will have seen the change before it sees anything that was sent later.

Now suppose an HTTP/2 connection is allowed to have 100 active streams. The server checks that the client doesn’t exceed this limit. But if the client has 100 active streams and closes one, it’s now at 99 and is immediately allowed to open another one. The server will see the old stream close, then the new one open — and at no point did the client exceed the limit.

Unfortunately, many server implementations begin doing work upon receipt of a request and have difficulty immediately canceling that work when the stream carrying that request is canceled. This exposes a potential weak point.

After the client makes 100 expensive requests, it can immediately cancel all 100 requests and make 100 more expensive requests. And then it can cancel those, and make 100 more expensive requests, and so on.

According to the protocol, that’s completely valid, because the client has canceled the old requests before issuing new ones. But the server is now receiving potentially infinite expensive requests and has to decide what to do with them.

Of course, Akamai and other major players on the internet moved swiftly to block these attacks. But a similar attack approach can be used to achieve the same end goal. If the attacker cannot send a stream reset message, can it receive one?

After all, in some implementations, Rapid Reset was mitigated by placing a limit on the number of stream resets received from the client. A naive implementation of that mitigation wouldn’t be alerted by server-sent resets.

In the newest iteration of this attack, the client sends deliberately invalid control messages. It knows that the specification requires servers to reset the stream if they encounter such invalid messages — hence the name MadeYouReset. And once the server has reset the stream, the client is free to open another one, even if it hasn’t actually received the server’s reset yet.

A client might use different types of invalid control messages:

• Additional data after the end of a request
• Flow control messages offering too little (zero) or too much (2³¹) flow control credit
• Messages that are longer or shorter than required

Fewer implementations were vulnerable to MadeYouReset than to Rapid Reset for various reasons, including the defenses and corrections implemented after the Rapid Reset attacks. Akamai’s own HTTP/2 implementation was not vulnerable because we used a more robust mitigation to Rapid Reset than simply counting client resets.

However, a number of other HTTP/2 stacks were impacted in different ways, and we have been tracking the coordination of patch availability for these projects as a matter of course and best practice.

It’s worth noting that no live attacks attempting to exploit this new vulnerability have been observed. The attack vector was discovered by security researchers at Tel Aviv University, reported to Akamai via our bug bounty program, and a response was simultaneously coordinated by CERT before any disruption could occur. Since the vulnerability manifests differently in each implementation, different software providers have assigned unique CVE identifiers for their products.

For more details, please see the CERT Vulnerability Note.

Attacks like this are not a peculiarity of HTTP/2. HTTP/3 is the most recent version of the protocol that runs the web. We haven’t had a vulnerability specifically in HTTP/3 in part because HTTP/3 delegates a lot of the stream management to its next-generation transport protocol, QUIC.

Since QUIC was released, we have seen the discovery of a few protocol attacks, but Akamai’s code already had effective mitigations to these forms of misbehavior.

Marten Seemann, the developer of quic-go, was also heavily involved in standardizing QUIC and HTTP/3. He discovered potential attacks on QUIC’s path validation and connection ID mechanisms. A similar process was followed: The disclosure was shared privately with many implementations and each implementation either verified that they already protected against such attacks or prepared patches before any public announcement was made.

As a result, Akamai’s customers were protected from these vulnerabilities before they could be exploited.

Akamai routinely collaborates with security researchers and other major vendors to coordinate disclosure of and response to vulnerabilities as they are discovered. This invisible ecosystem works behind the scenes of the internet you use every day to detect and prevent attacks before the public at large can be inconvenienced.

This is one of many reasons why customers choose to work with a trusted partner such as Akamai. We leverage our ability to respond to security incidents like this so our customers can focus on their business.

If you are currently under DDoS attack or threat of extortion, please reach out for 24/7 emergency DDoS  protection.