Later they began a small refactor which easily handled the loads they were "assuming" could not be handled in the way that the refactor handled, and it was wildly successful and the code was much simpler to work on.

To developers: don't over engineer. Most languages/frameworks/libraries can handle scale beyond what you'll ever get in your initial implementation. No, you entire website does NOT need to be asynchronous. It is very possible to have too many background jobs. I know this because I've seen the horror. I've also written an entire jobless/synchronous platform that serves millions of users without issue. If you run into scaling issues, that is a good problem to have. Tackle it as it happens.

Bottom line is focus on secure, quality code above all else. Don't make assumptions.

It always hurts to build something that “won’t scale” because it was framed as a negative.

Realizing that something “scales” if it meets your current needs is pretty important.

Framing scale in terms of how many people can work on it, how fast they can work on it, and how well it meets needs is often a better way of considering the “scale” of what your building.

As you said, when request per second becomes a limiting factor you can adjust your scales but doing it from start rarely makes sense (largely because req / sec already scales pretty well)

Because we don’t trust that we can keep the wheels on.

Over time the memory fades, and the confidence improves, and we get more comfortable with things being okay instead of unassailable. But it can be a rough road until then.

I'll speak to the elephant in the room: this is what happens when you have an entire developer culture so used to automatically dynamically expanding everything and not caring how big it is, that they never think about how big something can be.

This class of problems isn't necessarily restricted to HTTP/2, although its gross complexity probably contributes; it's just that in HTTP/1.x times, more developers would be used to languages like C where managing buffer lengths takes constant attention, and no one would bother to make header allocations expand limitlessly when they should be a few K in total at most for the whole request.

If at least it had been accepted to forbid it after a non-full HEADERS frame it would have been more robust but it was perceived that the encoding job itself could have been harder (byte boundaries in compressors etc).

BTW I find it funny how we "rediscover" the same stuff every 10 years. Recently it was the well-known RESET_STREAM flood, now the CONTINUATION, soon it will probably be DATA frames of length zero, then single-byte WINDOW_UPDATES, then INITIAL_WINDOW SETTINGS that cost a lot of CPU, etc. The world is just circling in this security circus, provided it's possible to assign a name and possibly a logo to a known problem...

So no fix yet, but I think all that's needed is a recompile with the latest version of Go 1.22.2

HTTP/2 (and HTTP/3) is vastly different in features (added multiplexing, windowing, HPACK etc). All this transforms a largely stateless connection (in the HTTP/1.1 case) to a stateful one. And in order to maintain the stateful connection, you need to store some data (state, configuration etc), thus all these problems.

Also, in HTTP/2, since multiplexing is added, the protection characteristics are different. For example, if the connections were generated by CDN source draws, you may just allow fewer number of connections each with a large pool of multiplex channels, but if the connections were from direct user access, you may then want to allow large number of connections, but each with fewer number of multiplex channels. In HTTP/1, protection is much simpler, since everybody looked almost the same.

A QUIC UDP server is definitely going to need to store state data to maintain a connection/session, and now you also have the good old UDP security (packet flood protection etc) mixed in. I guess time will tell.

HTTP/1 has lots of unobvious edge cases, and legacy quirks. Text format has way more flexibility than it seems from valid headers. It has obscure features like multi-line headers and old MIME features, 100-continue race conditions, custom hop-by-hop headers, GET bodies.

Fortunately new HTTP RFCs document many pitfalls. If you just implement what RFC 2616 said, you won't have a safe implementation.

Actual size of a request or response can be specified in multiple ways (at the same time, with conflicting values), and depends on a combination of several features, and values of headers with weird parsing rules needed for backwards compat, so "simple" HTTP implementations can be tricked into request smuggling.

Either way you need a robust, well-tested mature implementation.

For normal asset serving the only advantage is more assets can be loaded in parallel since HTTP/1 is limited on connections per domain. CDNs on different domains usually prevent this from being an issue.

In theory you could serve unbundled JS assets via HTTP/2, but I have never seen it in production. Likely because you still need a compilation step most of the time.