Author here. This came out of a production OOM in a clustered MQTT broker. The main lesson was that we had been treating different kinds of distributed state as though they all required the same consistency guarantee. Happy to discuss the incident, the ownership model, or the design trade-offs.
