The August 17 outage, and the work ahead

Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.

I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.

  "Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "

Wow, that is some incredible growth in a really short time.

> Errors in those services triggered a client-side retry loop that increased traffic during recovery.

The worst outages I've been part of always have some version of this :(

Here's Vladimir Fedorov's GitHub contribution graph, as linked to as the author of this post:

https://imgur.com/a/zIbT0Gi

It shows zero contributions in the past year, on this account. This is a huge, huge red flag.

The comments just shows how entitled people have become. Most people use GitHub and features for free and have the audacity to complain.

The outage is due to massive load increase. In 4 months the number of commits doubled to 2.9 Billions. Anyone worked with high load systems knows that’s it’s not a normal growth and how difficult even to keep on horizontally scaling in a short time period such a complex system.

GitHub should charge at least maybe 5$ monthly fee and most of the entitled freeloaders would leave the platform and it would free up resources

Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer: we'll proactively refund you when we miss our SLA.

What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.

Notice there's nothing about paid customers, I'll add in what they are missing:

Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.

"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."

Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.

Github is ripe for disruption and I hope it is disrupted soon.

   Central US data center failed to scale with it

I'm in Europe and I experienced token failures as well.

Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had.

I don't like paying for free stuff but gh certainly worth it.

AWS CloudWatch has an option to show the trend and what it will be like after x-period.

Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem

Has to be one of the most vague outage summaries of the year

dang and friends, can you lock (some) threads? This is getting out of hand, it seems to me?

GitHub outages seem to be very normal and even more emotional these days. Yet, that’s no reason to treat each other destructively. It’s sad.

So let’s turn this energy into creating something better.

I fear to ask, how archive.org keeps up to catch all those events for archiving...

Related recently:

GitHub has alternatives, but no replacement

https://news.ycombinator.com/item?id=49135365

Why developers are ditching GitHub for Codeberg and self-hosting alternatives

https://news.ycombinator.com/item?id=48842611

and new entry:

Cursor Origin Code Hosting

https://news.ycombinator.com/item?id=49334209

well written

We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.

GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.

And another outage. [0] Looking forward to the subsequent post-mortem on that one.

You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.

6 years ahead [1] on not going all in an centralizing everything on GitHub.

[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp

[1] https://news.ycombinator.com/item?id=22867803

Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.

Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post.

"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.

I’m done.

Github down, no hard drives available, no memory available, thanks AI!

Seems like we are headed for Tech Gridlock.

They should rewrite their Ruby code to a performant language.

> What we have done and what comes next

"You've seen what we've done. The August 21st outage comes next. See you then!"

Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.

> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations

How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.