What game engines know about data that databases forgot

This seems cool but man it's hard to get over the very, very obvious AI writing.

I feel like this is a good place to bring up the "Have You Tried Rubbing a Database on It?" conf, which deals with this sort of topic. If you'd like to read about the intersection between Databases and X, where X is games, robots, user interfaces, developer tools, etc. there are some good talks there, and they're only 10 minutes each so very easy to consume. Also unlike TFA they are not AI generated.

https://www.hytradboi.com

The web is a front-end environment, where users expect 60fps, but where developers violate pretty much all the rules.

> Zero-copy is the default, not the optimization:

the amount of fluffy mapping, destructuring, temp scope-creation, ... that is the norm now for JS/TS devs is excruciating. how did this become the norm? do it once it doesnt matter, but every single layer in the app doing this and things become jittery. you first take the hit at creation time, and then another time at GC. stop doing that! Pass references around to objects, not recreate them each time at some function boundary.

> Entity as pure identity.

Stop json.stringifiyng every thing! how many hashThing() implementations i have seen that are essentially Json.stringify. stop it!

> Cache locality by default.

a little less clear for web dev, much is missing in terms of primitives. but do think about it. if anything, it's good hygiene. fixed typed array does make sense, dont mix&match types in your array, ...

Save the web, think like a videogame dev!

CMU very recently had a seminar [1] with the founder of SpacetimeDB, which was very interesting. The recording should be up in the next few days I expect

[1] https://db.cs.cmu.edu/events/pg-vs-world-spacetimedb-tyler-c...

The connection between ECS and DBs wasn't lost on developers in the early days of ECS. I'm reminded of Scott Bilias's 2002 ECS presentation: A Data-Driven Game Object System[1] which clearly says, "This is a database."

In my mind, there's a pedagogical gap in OOP where ontologies are formed using classes and this almost never works in real life applications (no in real life for that matter). We almost always construct ontological relations using predicates not rigid hierarchies. For this, ECS tends to be flexible, and expressive enough to be a pragmatic choice.

1. https://www.gamedevs.org/uploads/data-driven-game-object-sys...

I'm working on an embedded project where I'm actually thinking about using ECS on a STM32H5. It's not about cache-friendly locality (waitstates on internal SRAM for MCUs is basically a rounding error compared to the DRAM latency seen on desktop or server-class hardware), but the business is so complex that the traditional inheritance/composition is getting quite messy. When you end up using virtual and diamond inheritance in C++, you know you're in trouble...

It's too bad that ECS isn't more widely known outside of gamedev. It's not just good for performance, it's also legitimately a useful architecture design on its own to solve problems.

Or in a more general sense: Get your code as close as you can to your data

This has a lot in common with a columnstore, but there are a few interesting differences.

The main one is subrow versioning. Column stores (in OLAP at least) have always used row-level versioning, which gets in the way of small updates. A single change to a row amplifies into deleting and re-inserting the whole thing, and operations that seem sensible like adding or dropping a column break previous versions. This scheme is the first I've seen that tries to fix that problem.

One other difference is a lack of compression, as it's zero-copy, so the performance gains of operating on compressed data are lost.

Serious question:

If you take a BDD/TDD approach - do technologies like this still give you something?

I've dabbled a bit into smth similar (SpacetimeDB) with the aim of creating a headless backend for a game.

But then I realized that I'd still need to define the interface that I was testing in the traditional software layer and that all the nice DB magic seemed worthless since I'd still have to orchestrate traditionally.

(In short: No matter how nice your DB magic is, you will still hide it away in the basement/engine room, right?)

kdb does two of those by default—columnar and it uses memory mapping (no serialization/deserialization)

Not entirely forgotten in the database world!

>Cache locality by default. In a traditional row store, reading all player positions means loading entire rows — names, inventories, health, everything. Most of those bytes are wasted.

Not one mention of column stores? This didn't come from ECS...

https://en.wikipedia.org/wiki/Data_orientation

I'm not sure a game needs transactions. It seems to me that entity streaming concepts are a better fit: reading from an event queue and consistently reconciling the state over time. Like the sorted top-n in the example, if it was a game and the query is generally known, it would be better to just materialize it as a fast rank, which goes beyond the IVM-style materialization they suggested.

interesting! foundationdb was created after the team was going to build a massively multiplayer game and couldn't find a database that could support it...

I know something about game engines, I know a lot more about real time graphics, and I do even more about databases and implementation of those.

While it's nice to get yourself acquainted with all, especially easy to do with AI these days.. I do have to point out few things visible outside of myopia induced when looking from one perspective into another. In this case from gamedev to data world. In Cliff's notes since I don't have much time but I also don't want to give a drive-by snark since there are hidden values to this, imo, contrary to what I'm going to say.

What gamedev perspective myopia kind of ignores is, in no particular order..

persistence vs throughput - goal of data management is not only about execution speed. ACID, WAL, etc all are core features, not an overhead per se. Sometimes you can forego these, of course. Sometimes, if you understand what you're giving way to.

columnar fallacy, for the lack of better words - DoD, SoA are not secrets game engines use which others have forgotten. This in-particular ignores existence of OLAP. Clickhouse, Snowflake, HN's darling duckDB have been using SoA for _quite awhile_. AoS in OLTP is used because updating a single record _is_ faster that way. Why one over the other - see OLAP vs OLTP

Game engines obsess over cache hierarchy (L1-L3), and in particular cache misses by avoiding those with cache line packing, prefetching. Databases operate in a different real, that of I/O boundaries; Disk and network primarily. Bottleneck of databases is often the speed of light (latency) or the bus (NVME for example). packing structs is going to give you marginal benefit if you're waiting for 10ms for a network packet or a disk read. Suggested are micro optimizations in the context of a global system. Different realms of execution flows.

ECS are cool, everyone agrees on that. Relational databases are built on top of relational algebra though. So if you're up for running the same logic over many objects, ECS is going to be cool. If you want to allow for complex, arbitrary queries and set based logic you will need some guarantees of consistency. Complex, many-to-many joins across disparate ECS ssystems without predefined _system_ and you're foregoing the promised performance. What DBs are doing is general purpose, ECS ain't that.

Finally, yes game engines manage thousands, millions, of entities in localized 3D space. Databases manage billions, trillions, of records across petabytes of distributed storage. So, what gives? Entity model does not scale to distributed systems because of CAP theorem. No such thing as instant consistency in a globally distributed system without violating physics or sacrificing availability. TBH, some middle ground, localized to a machine, might give way to the idea but at what cost?

Don't let it shoot you down though. If there is still a kernel of idea tingling after that chat with AI, go ahead and drive through that CAP wall!

When I learned about ECS, I realized that Tablizer (old Slashdot guy who insisted that OOP was a dead end and "table oriented programming" as was done in FoxPro was the way to go) was probably right. Using an ECS for my Android game was a bit more cumbersome, but paid for itself many times over with the ability to create new kinds of entities, implement familiar behaviors for them, and add new ones without code copypasta or inheritance tree entanglement.

I'm getting fatigued out by "I had a conversation with a chatbot, here's the output" blog posts.