Astra can read and write weights to improve Qwen

This is insane.

Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.

I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.

No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.

> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights

HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.

I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.

Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.

We'll see if this attempt at RSI worked.

I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.

I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.

On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.

I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.

As insane as this result is, next steps are even crazier:

> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.

Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.

I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.

These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?

What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.

I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.

This survey paper from earlier this year has a lot of good related work:

A Survey of Weight Space Learning: Understanding, Representation, and Generation

https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning

Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.

Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.

almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest

Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.

I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising

Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?

This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...

I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.

We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.

What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.

So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?

I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.

Does this mean they can improve their own weights through some system, or can they only do this for much smaller models?

I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:

- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.

- the best way to improve on some task is to have interpretability into your own neurons (..?)

- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly

- therefore, models naturally put effort into 'self-interpretability'

I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?

So how would this be in the training? Does OpenAI train models on their own epochs?

I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.

Is there any chance they’ve just seen these weights before and are recognising them?

If not this is mind blowing.

However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.

Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.

This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?

I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.

Absolutely incredible regardless.

I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.

At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?

This is wild. Could Astra read and potentially audit a mapped human brain in that case?

1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?

2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.

Such an unexpected result...

Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):

* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?

* what, if any, is that extra magic sauce?

* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.

How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?

Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.

This raises so many questions.

Unless Astra was trained on other networks, how is it able to reason about the weights?

Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?

... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.

This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.

This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.

[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...

The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.

GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.

Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next

Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.

This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.

That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).

Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.

Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?

I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?

This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.

RSI was already here?!

Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?

Wild

This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights

And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?

Is it aware of projects like [1] and implementing those?

[1] https://github.com/lyramakesmusic/neuron-explorer

A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...

Great, instead of backpropagation, we now have slopagation.

Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?

I wonder if this behavior emerges when using recursive self improvement to generate new models

this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.

Did they try it with other frontier LLMs? What about other non-trivial out of distribution tasks? So many questions.

Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.

“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park

Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).

I wonder how it would do with distilling models by hand?

Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

This is fucking insane LMFAO what?

I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.

Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?

(When they say "Writing a Go algorithm," they mean the board game, not the language.)

This doesn’t look like “normal” technology to me…

At what cost? That's the part I'm curious about.

On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.

Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?

this is maybe one of the most bonkers machine learning research i've seen all year.

Does asking an LLM to not use tools guarantee it doesn't use tools?

Wow, in a way it's like CRISPR for ML models!

Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.

We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.

Whaaaaat!? This is crazy right?

I'm just blown away. Editing weights? Writing weights from scratch!?

I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster

Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.

I wonder if one could use this to efficiently abliterate models and identify guardrails ;)

This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.

Does this open the possibility of better decoding biological brains?

Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase

Friendship ended with AdamW. Astra take the wheel.

But for real this is insane. Okay I need to get on this stat.

I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.

This looks very similar to that in a way.

I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.

Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?

This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...

This could set us on a path to weights becoming thew new compilation target.

Quick, fine-tune your qwen models before they patch this out of Astra.

I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.

Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)

I am surprised that it did that and I am just left feeling technically impressed. This is insane.

The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)

I was here

I have absolutely no words. If this is real, I can't even imagine the future anymore.

This is straight black magic. llama 70b, let’s see what we can push you to

The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.

I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.

Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.

I'm still learning but this is fun as fuck.

Possibly related to https://already-clean.goodfire.com/

Too good to be true! Shocking!

talk about homoiconicity haha

lisp people will be in awe

Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.

That seems both entirely obvious in hindsight and potentially super important.

I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

Whoa. Wild. I want to try this!

kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective

Where are we heading?

It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.

I cannot say this fills me with confidence about alignment.

I think this is how RSI will come to be.

Slightly terrifying.

I can't help but be skeptical of this, but... damn.

This is shocking.

Big if true

Sorry for what is undoubtedly a low-informational comment but..

Whoa. What the absolute fuck is going on here.

They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.

It's very impressive, but also very expected.

a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability

That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.

Edit: tweets from one of the coauthor providing more context:

https://x.com/a1zhang/status/2117328657161322810

https://x.com/a1zhang/status/2117330975122235442

Can we call this a form of recursive improvement?

Now our AIs have AIs lol

Uh oh, it's iterative self improvement time, with no training runs, we're screwed