Apache Iceberg Is Brilliant (and Your 12-Person Startup Doesn't Need It)

Here's a founder/product perspective. This maps well to the skateboard => scooter => bicycle => motorcycle => rocket ship product metaphor that's often used. Each phase teaches different design patterns, constraints, and failure (and success) modes for different inflection points of a startup's journey.

But here's the reality. What got you technically to PMF may hold you back from your Series A and next steps. Technical debt is just the natural cost of growth, but (here's the kicker) optimizing tech stacks too early can lead to slower execution time. Most startups never reach exponential scale anyways. Put another way, starting with "rocket ship" does not immune the startup from rewrites, refactoring or throw away code.

The real systems and management challenge is building architectures that are intentionally temporary or modular. Simple enough that throwing them away later isn’t traumatic and rebuilds aren’t a sign of failure but success.

FTA: The design assumes S3 eventual consistency is your enemy

S3 has done better than that since December 2020. https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...:

“Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent. What you write is what you will read, and the results of a LIST will be an accurate reflection of what’s in the bucket. This applies to all existing and new S3 objects, works in all regions, and is available to you at no extra charge! There’s no impact on performance, you can update an object hundreds of times per second if you’d like, and there are no global dependencies”