What is the status on continual learning for LLMs?

It depends on what you mean by continual learning. Do you just mean some form of memorization, or something more/different?

The issue remains the context window. I haven't seem any example of a "memories" system that the agent can update over time without it devolving into slop. The magic of the human brain is that we seem to have decently sophisticated heuristics parsing/saving memories as well as for letting memories that aren't accessed decay.

my initials = LLM :(