Elektrine
Log in Register

#transformers

57 posts · Last used Sep 25

Bogdan Buduroiu @budududuroiu@hachyderm.io
· 3mo ago
Morning, today we're looking at #MechanisticInterpretability , a subfield of AI Safety that attempts to understand the inner workings of artificial intelligence by analysing concrete structures, algorithms and circuits. Why do we even need to do this? Because the hope of understanding neurons as being features died on polysemanticity -- models represent more features than dimensions by assigning them to an overcomplete set of non-orthogonal directions (i.e. you can't hope that concepts can be broken down into linear combinations of features). This isn't a new field, people have been projecting intermediate GPT layers through the final layer activation and looking at the top-k most likely next tokens since GPT-2. And that's how we arrive at the Natural Language Autoencoder -- an autoencoder over residual-stream activations where the bottleneck is natural language text instead of a sparse vector. Nothing in the training objective requires the verbalisation bottleneck to be readable, faithful, or even semantically related to the intermediate layer being investigated, so while the training algorithm is typical of autoencoders, two things are distinct here: 1) both parts of the autoencoder (here the activation-verbaliser, AV, and the activation-reconstructor, AR) undergo special SFT to ensure the verbaliser and reconstructor can generate and read natural language from intermediate layer activations 2) a KL-divergence penalty is baked into the objective to make sure that, during joint training, the NLA doesn't diverge from the SFT version A side-product of this training is that we can take the current verbalisation and "desired" verbalisation and compute steering vectors from their difference. Surprisingly, Anthropic didn't find any evidence that the NLA was engaging in steganography to smuggle information to avoid human detection, but it did find that NLAs tend to confabulate a lot. #AIResearch #Transformers #TransformerCircuits #Anthropic #NLA
1
2
1
0
Karles Marín @karlesmarin@mathstodon.xyz
· 4mo ago
Boosted by @welcome@friends.deko.cloud
#introduction — I spent the last year measuring something oddly regular: in trained transformers, attention weight decays with token distance as a clean power law, \(A(d)\propto d^{-\gamma}\), with \(R^2>0.95\) across 40+ open models. The fun part: \(\gamma\) behaves like a state variable. There's a closed-form baseline from RoPE geometry + maximum entropy, \(\gamma \approx \frac{2\theta - T\sqrt{2}}{2\theta + T\sqrt{2}}\), a phase boundary at \(\gamma=1\) (the partition function \(\sum_d d^{-\gamma}\) diverges — ~36% of public LLMs sit near it), and practical corollaries for long context and KV-cache compression. The honest part: I keep a public registry of my own refuted claims — several beautiful hypotheses died on pre-registered tests, and those documents are the ones I'm proudest of. Everything is open: a bilingual interactive field guide on how transformers attend, a browser-only diagnostics tool (zero install, four languages), papers on Zenodo, code on GitHub. Links in profile. Happy to talk attention, statistical mechanics of nets, or why your 128K-context model can't actually use 128K. #math #MachineLearning #transformers #OpenScience
0
0
0
0