[{"content":"The transfer question Anthropic\u0026rsquo;s global-workspace claim In July 2026, Anthropic\u0026rsquo;s interpretability team published Verbalizable Representations Form a Global Workspace in Language Models. The paper argues that a small, privileged set of internal representations in a language model forms something like a global workspace. Anthropic uses that term in a functional sense: information in this workspace can be put into words, deliberately manipulated, and reused by otherwise different computations. The claim does not depend on the model having subjective experience.\nThe authors locate this workspace with a Jacobian lens, or J-lens. A model\u0026rsquo;s hidden state is a long vector. The lens makes part of it easier to inspect by producing a ranked list of associated words, perhaps France, country, and Paris. Each word labels a direction in the model\u0026rsquo;s activation space. The paper calls sparse combinations of these directions J-space. The fitted lens fixes the directions; the model\u0026rsquo;s current context determines how strongly each one is present.\nConsider the question:\nThe capital of the country where the Eiffel Tower stands is …\nThe answer is Paris, but a one-line answer still requires the unstated intermediate fact France. If the lens reports France while the model is solving the question, we can remove the component of the hidden state that points in the France-labelled direction and see whether the final answer changes. This example gives the intuition. In my main intervention, I did not hand-pick France. At every processed token and layer, the lens first excluded directions associated with the unmodified model\u0026rsquo;s ten most likely next tokens, then removed the ten strongest remaining directions. This guard reduced the risk of directly suppressing words the model was about to produce.\nTwo distinctions matter for everything that follows. First, each direction is labelled with a vocabulary token through the lens construction. The training data are used to fit the lens, but the label alone does not show that Qwen used that direction in this particular computation. It may carry France or 6 some other way. Second, the intervention edits activations during inference; the model\u0026rsquo;s weights stay fixed.\nAnthropic found that broad J-space deletion hurt Claude Sonnet 4.5 much more when it answered GSM8K math problems directly than when it wrote out intermediate steps. At one medium intervention strength, Claude retained 99.4% of its clean score with written reasoning, versus 86.4% when answering directly. Their interpretation was intuitive: written reasoning acts as an external scratchpad. Once a partial result is on the page, the model no longer has to carry all of it silently.\nA transfer test on Qwen3-4B I asked whether that relationship transfers to Qwen3-4B using a public, third-party lens fitted on WikiText to predict Qwen\u0026rsquo;s final-layer state. I also preregistered a harder prediction: if written steps replace only storage, their protection should weaken when the model must choose increasingly difficult next operations.\nI did not reproduce the predicted math result. I then used two-hop factual questions as a simpler check: each question requires one unstated intermediate fact, such as France in the Eiffel Tower example. On a reused set of 40 such questions, J-space deletion caused more damage than random activation changes matched to its local removal size. However, a separate experiment on 300 new questions overturned my early claim that written chain-of-thought reasoning (CoT) specifically compensated for J-space damage. Most importantly, none of the arithmetic experiments verified that the lens had found the representation Qwen was actually using for arithmetic.\nThe arithmetic results apply only to this model, this lens, and these interventions. They leave open whether Qwen uses an internal workspace for mathematics, and they do not refute Anthropic\u0026rsquo;s result on Claude.\nThe central problem is that a near-zero result from an interpretability tool has several possible causes. The mechanism may be absent; the task may avoid using it; or the tool may fail to find and change it. A larger sample can place a tighter bound on the effect of one intervention, but it cannot tell these explanations apart.\nWhy transfer matters for AI safety Suppose an internal monitor reports no hidden planning in a deployed model. Is the planning absent, or did the monitor fail to transfer to that model? The opposite mistake is also possible: a strong intervention may make a model worse at everything, creating the appearance that a particular internal concept was causal when the model was merely damaged.\nThis project does not directly measure chain-of-thought faithfulness or deception. Its safety relevance is methodological. Before trusting an internal monitor or causal intervention, we need evidence that it recognizes the target state, changes that state, and avoids simply degrading the model.\nThis sets a high bar for a successful transfer. A score drop under J-space deletion could reflect generic sensitivity to perturbation. We need controls to isolate damage caused by the selected content. We also need to distinguish broad deletion from a targeted change to a particular arithmetic state.\nWhat would count as evidence for causal transfer? Two different causal questions The source paper presents two related kinds of arithmetic evidence. First, it measures how much benchmark score survives broad J-space deletion. Second, it studies individual computations: it locates a proposed intermediate value and then patches or swaps its contextual representation. Replacing a state associated with the correct value 6 by one associated with 7 asks whether the downstream answer changes in the predicted direction.\nA token-labelled direction is a corpus-level coordinate associated with a word such as 6. A contextual state is the activation that actually carries the value in this particular computation. A lens can supply the former without having found the latter.\nMy study mainly performs deletion. It removes directions selected by the lens and asks whether accuracy falls. Deletion tests whether a direction is necessary when the model may be able to recompute its content. A counterfactual swap asks the sharper question of whether changing the proposed content causes a corresponding change downstream. The large-scale arithmetic follow-up later in this post knew the correct intermediate value, but still deleted token-labelled directions rather than swapping a localized contextual state.\nWhat changed in this transfer test Dimension Source paper This project Model Claude Sonnet/Opus 4.5 Qwen3-4B Lens fit Model-specific; main Sonnet lens predicts a penultimate-layer state Third-party Qwen3-4B lens fitted on WikiText; predicts the final-layer state Main comparison Score retained relative to clean Paired accuracy change under direct and written answers Broad intervention Delete the ten strongest J-space contents The same broad top-k deletion family, plus generic controls of comparable size Targeted arithmetic test Curated activation patches and coordinate swaps Large-scale deletion of directions labelled with known intermediate values Answer elicitation GSM8K without versus with an explicit written scratchpad A forced-direct answer format versus a separate prompt asking for concise written reasoning Several important details differed from Anthropic\u0026rsquo;s experiment. I used Qwen instead of Claude, a third-party fitted lens, and deletion in place of the paper\u0026rsquo;s targeted swaps. The experiment tested whether the broad contrast between direct and written answers survived those changes.\nThe lens transfer was the most uncertain link. The public artifact was fitted on general WikiText and trained to predict Qwen\u0026rsquo;s final-layer state, whereas the source paper\u0026rsquo;s main Sonnet lens predicted a penultimate-layer state. The lens and deletion setup could still work—the two-hop results below show a selective effect in one domain—but its coverage of Qwen\u0026rsquo;s arithmetic states had never been established. Qwen3-4B and the three math datasets were fixed by the course assignment, and fitting a new lens was outside this phase\u0026rsquo;s original compute plan.\nThe preregistered extension I also preregistered a secondary hypothesis: if written steps mainly provide memory, their protection might weaken on harder problems because the model still has to choose each next operation. Prior work on CoT as serial computation motivated this idea (Li et al., 2024; Sprague et al., 2024; Pfau et al., 2024).\nI wrote down the prediction and its rejection criteria before the main runs, then used GSM8K, five MATH-500 difficulty levels, and AIME as a rough scale. Benchmark difficulty also changes knowledge requirements and solution length, so I interpret this secondary test cautiously.\nWhat would count as J-space-specific protection? A crucial control uses a random direction in place of a J-space direction. At each token and layer, it measures how much the J-space intervention would remove, then subtracts a deterministic random vector of exactly that size. I call this the matched control. The match is recalculated at every position along each condition\u0026rsquo;s own generated sequence. If two conditions begin to generate different tokens, the total amount removed over the full answer can also differ.\nThe logic is easiest to see with a toy example. Suppose J-space deletion lowers direct-answer accuracy by 10 points, while the matched control lowers it by only 1. We now have evidence for 9 points of damage associated with the selected J-space content. If written reasoning removes most of those extra 9 points, it plausibly substituted for that content.\nNow suppose both interventions lower direct accuracy by 10 points. Even if written reasoning reduces both losses to 2, we have only shown that written answers are more robust to perturbation in general. We have not shown that text replaced J-space in particular.\nThroughout this post:\nJ-specific damage means damage beyond the matched generic control; CoT protection means that written reasoning loses less accuracy than direct answering under the same intervention; and J-specific protection is the difference between protection under J-space deletion and protection under the matched random perturbation. A positive value means that written reasoning reduced more of the J-space damage. With that standard fixed, the first experiment asked for the prerequisite to any protection claim: was there substantial arithmetic damage for written reasoning to rescue?\nI found almost no math effect for written reasoning to rescue Average effects on GSM8K and MATH-500 The main experiment crossed two answer formats—direct answers and written reasoning—with clean and J-space-deleted inference. I ran the matched and other controls separately. On the 150 GSM8K problems shared by the four main conditions, deletion changed direct accuracy by −4.0 percentage points and written-answer accuracy by −3.3 points. The estimated CoT protection was therefore only +0.7 points, with a 95% interval from −7.3 to +8.7.\nThe answer modes had very different clean baselines: 27.3% direct versus 88.7% written on the shared GSM8K items, and 31.0% versus 84.5% on MATH-500. The prompt formats help explain why. In the direct condition, I instructed the model to give only the final answer and prefilled its response with The final answer is \\boxed{. In the written condition, I asked it to solve the problem step by step and end with a boxed answer. I therefore compare the change caused by deletion within each format; the raw accuracies are not directly comparable. Even then, the formats have different room for accuracy to fall, which weakens the mechanism comparison.\nFor readability, I report CoT protection as the written-answer accuracy change minus the direct-answer accuracy change—the negative of the preregistered direct-minus-CoT contrast. Thus negative accuracy changes mean damage, while positive protection means that written reasoning lost less. Changes are in percentage points (pp), not percentages.\nDataset Direct accuracy change Written-CoT accuracy change Estimated CoT protection GSM8K, shared n=150 −4.0 pp −3.3 pp +0.7 pp [−7.3, +8.7] MATH-500, n=200 −5.0 pp −4.5 pp +0.5 pp [−7.0, +8.0] The written-reasoning runs covered 150 GSM8K questions, while the direct-answer runs covered 400. The table uses the shared 150 questions so that the two formats are paired on identical problems. Across all 400 direct-answer questions, deletion was even closer to zero: accuracy changed by −0.5 points [−4.0, +2.8].\nThe calibrated matched control also reduced accuracy. On the same 150 GSM8K questions, direct accuracy fell by 7.3 points and written accuracy by 3.3. The resulting protection estimate was +4.0 points, larger than the +0.7 under J-space deletion. I therefore had no evidence that deleting the selected J-space content caused distinct arithmetic damage.\nThe difficulty prediction was not supported AIME could not anchor the hard end of the prediction. Direct accuracy was 0/30 with or without deletion, so it had no room to fall. Written reasoning dropped from 5/30 to 3/30, while truncations increased from 13/30 to 24/30. A direct-versus-written comparison is not meaningful when the direct condition is already at zero.\nThe fitted MATH difficulty interaction pointed opposite my prediction, but it missed the preregistered Bonferroni-adjusted threshold for two confirmatory tests (p=.0265 against .025). Accuracy changes with written reasoning also jumped from −2.5 to −12.5 to 0.0 to −7.5 to 0.0 points across the five levels. Those values do not form a coherent trend in either direction.\nWhy the math result remained inconclusive The preregistered prediction was unsupported. The deeper mechanism question remained unresolved because J-space deletion had not caused more arithmetic damage than a generic perturbation. The matched controls were often at least as harmful as J-space deletion, and several registered checks of the controls failed. There was no confirmed J-space-specific effect for written reasoning to rescue.\nThree broad explanations remained:\nthe intervention might be implemented incorrectly, applied at the wrong layers, or simply too weak; the intervention might work, but the lens or automatic selector might miss Qwen\u0026rsquo;s arithmetic states; or the mechanism might genuinely not transfer to this model and task. Because the intervention itself was unvalidated, the small effect did not tell me whether arithmetic was insensitive to the edit or whether the edit had missed arithmetic entirely. I therefore moved to a domain with an obvious hidden intermediate and asked whether the intervention could selectively change behavior there.\nI narrowed the two-hop claim after replication The two-hop experiments changed my conclusion twice. The first result suggested that written reasoning specifically compensated for J-space damage. A replication on new questions overturned that interpretation. Later, after an implementation audit, I reran the original 40 validation questions under an updated protocol. That rerun supported a narrower finding: J-space deletion selectively reduced direct-answer accuracy on this particular set. Because I had already used these questions during development, this was a check that the intervention could produce a selective effect, rather than a new replication.\nRun Questions Procedure Result Initial test 150, including 40 used during development Earlier protocol; direct and written answers Suggested J-space-specific CoT protection Replication 300 entirely new Same earlier protocol; direct and written answers Rejected the protection claim Audited rerun Original 40 only Updated protocol; direct answers only Found selective damage on the reused set My first two-hop experiment suggested J-space-specific protection Two-hop questions make the hidden intermediate easy to name. To answer “the capital of the country containing Machu Picchu,” the model must recover Peru before producing Lima. If deletion costs direct answers 12 accuracy points but written answers only four, the eight-point difference is CoT protection. To claim that text replaced J-space in particular, that protection must also be larger than the protection produced by the matched generic perturbation.\nIn the early 150-question set, estimated J-specific protection was +13.3 points [+3.3, +24.0]. It looked like the mechanism I had hoped to find: written text specifically substituting for damaged J-space content.\nForty items came from a validation set that had already served as development material. On the 110 entirely new questions alone, the point estimate remained positive but uncertain: +8.2 points [−3.6, +20.0]. I treated the 150-question result as provisional and ran a fresh replication.\nI withdrew the mechanism claim after testing 300 new questions I built a new set with no entity–relation overlap with the first one and balanced it by question type. Before running it, I committed to dropping the mechanism claim if written reasoning protected the matched random perturbation as much as it protected J-space deletion. Here, replication means a new, non-overlapping question set tested with a decision rule specified in advance. It was my own follow-up, not a replication by a separate research team.\nReplication on new questions, n=300 Estimate 95% CI Protection under J-space deletion +4.3 pp [−3.0, +11.3] Protection under the matched perturbation +8.0 pp [+1.3, +14.3] J-specific protection (J-space − matched) −3.7 pp [−12.0, +4.0] The point estimates suggested protection under both interventions, but only the matched-control interval excluded zero. Estimated J-specific protection was −3.7 points and crossed zero, so I withdrew the preregistered specificity claim.\nBoth the 150-question result and the 300-question replication used the earlier protocol. The replication is enough to overturn my intermediate claim about J-space-specific protection in this Qwen two-hop setup. It does not establish a new generic-robustness effect, and I do not combine it statistically with the later results. The finding also leaves the source paper\u0026rsquo;s GSM8K result intact because the model, task, controls, and protocol differ.\nDuring the later code audit, I made random seeds item-specific, fixed how removal magnitude was tracked at each token, corrected the logic for resuming interrupted runs, and recorded which answers hit the token limit. I did not rerun the earlier 2×2 experiment. The replication invalidates the protection claim made under the earlier protocol, but it cannot tell us what the audited protocol would have done on the 300 new questions.\nFigure 1. Positive J-specific protection would favor the claim that text uniquely replaced J-space. The initial estimate was positive, but the fresh 300-question replication reversed direction and crossed zero. Both runs used the earlier two-hop protocol and are reported separately from the later audited results.\nAfter withdrawing the protection claim, I still needed a simpler check under the audited protocol: could J-space deletion selectively disrupt a hidden fact chain at all?\nUnder the audited protocol, I retained only a narrower finding I reran the same 40 validation questions in direct-answer mode under the audited protocol. The intervention covered layers 19–28 and followed the source paper\u0026rsquo;s direction construction. I compared it with three random perturbations whose size was matched at every position along their own generated sequences. I also included 20 sentiment questions and 20 text-extraction questions, which should not require the same kind of hidden fact chain. I had first tried this setup in a small pilot, then fixed the procedure for the rerun. Since the 40 two-hop questions had already been used for development, this was a positive-control check rather than an independent test of generalization.\nHere is one example of the selective failure:\nFigure 2. I reran the original 40-question validation set under the audited protocol. I chose this example after the run using a fixed rule: the lowest-numbered question for which the unmodified model and all three matched controls were correct but J-space deletion was wrong. Under deletion, the model returned the intermediate country instead of the requested capital. I did not save an item-level lens trace, so the figure shows behavior only.\nAcross the complete test set, the result was large and selective:\nAccuracy change on two-hop recall Estimate 95% CI J-space deletion minus clean −32.5 pp [−47.5, −17.5] Extra J-space loss beyond 3 matched controls −23.3 pp [−35.8, −11.7] Deletion caused 13 additional errors among 40 questions. Roughly nine remained after subtracting the average matched-control effect. All 20 sentiment and all 20 extraction items stayed correct. The intervals are wide because the sample is small, but even their conservative ends imply double-digit losses.\nThe matched perturbations show that edit size alone cannot explain the result. They removed more of the residual norm on average—14.83%, compared with 9.11% for J-space deletion—but caused much less damage to two-hop recall.\nBased on this audit, I kept only the narrower conclusion: J-space deletion had a selective direct-answer effect on the reused two-hop questions. The rerun provided no new evidence about generalization to other questions, protection from written reasoning, or the lens\u0026rsquo;s ability to find arithmetic states.\nTakeaway: After this rerun, I retained one claim: J-space deletion selectively reduced direct-answer accuracy on this fixed two-hop question set.\nI still found no detectable math effect under more targeted interventions I next checked four practical explanations for the arithmetic null. The intervention might edit too few layers; the first dataset might be too small; the automatic selector might miss the relevant intermediate; or supplying the correct intermediate directly might work better. These checks used different procedures, so I report their results separately.\nEditing more layers still did not isolate J-space-specific damage Using the earlier protocol, I expanded deletion from the main layer range to the wider L15–32 range. This caused more damage on the two-hop questions. On GSM8K, however, J-space deletion reduced accuracy by 5.7 points while the matched perturbation reduced it by 12.2 points. Editing more layers therefore still did not isolate a J-space-specific math effect. These were exploratory point estimates; I had not preregistered a formal test of how the effect should change with intervention strength.\nAcross 4,600 new questions, the estimated effect stayed within ±2 points I next used GSM-Symbolic, which instantiates familiar problem templates with new numbers. A 400-question experiment produced only 14% clean direct accuracy. Performance was too close to the floor to reveal much additional damage, so this pilot told me little about the mechanism.\nThe larger follow-up contained 5,000 questions from 100 templates. The first 400 had already motivated the follow-up, so the primary analysis used only the 4,600 new instances. Confidence intervals were computed by resampling whole templates, because variants of the same template are not independent.\nThe clean model solved only 12.1% of these fresh questions. That low baseline limits what the experiment can say about arithmetic competence. Even at that low baseline, however, the estimated effects under this procedure were precise:\nFresh GSM-Symbolic, n=4,600 Estimate 95% cluster CI J-space deletion − clean −0.52 pp [−1.65, +0.59] J-space deletion − matched +0.26 pp [−0.96, +1.46] Both intervals fell inside a ±2-point absolute margin that I specified before the outcome analysis. I set that margin after the clean run had finished and a partial, unanalyzed J-space output file existed, but before running the matched control or examining outcomes. At a 12.1% baseline, two points would still be large in relative terms: estimated score retention was 0.957 [0.875, 1.057], whose interval overlaps the source paper\u0026rsquo;s direct-retention interval [0.802, 0.930]. This result places a narrow bound on the absolute effect for this particular direct-answer procedure and question set. It cannot establish a cross-model difference, and this experiment did not include written reasoning.\nThe 32-token limit distorted the first estimate.\nAt the original 32-token limit, deletion made some answers longer and therefore more likely to be cut off. A naive analysis gave a −1.02-point J-versus-clean effect with an interval that narrowly excluded zero ([−2.00, −0.07]). Before seeing the 128-token rerun outcomes, I had registered a rule: if any of the three paired conditions was truncated, rerun all three with a 128-token limit. The corrected estimate was −0.52 points and its interval crossed zero.\nThe rerun showed that part of the small J-versus-clean effect was a measurement artifact. The broader equivalence conclusion did not depend on this correction. More generally, an output limit can affect experimental conditions differently when an intervention changes answer length.\nThe selector rarely included the correct intermediate number So far, the J-lens had chosen its own targets during intervention: at each processed token position and layer—through prompt prefill and subsequent decoding—it selected the ten strongest vocabulary-labelled directions, excluding the clean model\u0026rsquo;s ten most likely next tokens. To audit whether a known intermediate was even available to that selector, I later fixed a simpler check at just one position: the final prompt token. On a set of order-of-operations questions, the actual intermediate number appeared among the selected directions in any layer of the band on only 10.9% of questions at layers 19–28 and 20.0% at layers 23–31—about one question in ten or one in five. Known intermediates in a separate multihop set used only for this ranking analysis appeared more often: 26.9% and 38.7%.\nThis check examined only the final prompt token, so it could miss a target that appeared earlier in the prompt or during generation. The arithmetic and multihop question sets also differed. In addition, the procedure excluded the unmodified model\u0026rsquo;s ten most likely next tokens, which removed different candidates from different questions. The coverage rates are therefore descriptive; they do not establish that the raw lens favors one domain.\nFigure 3. Panel A shows a positive example selected after analysis using a fixed lowest-ID rule. “Earth” entered the top ten at layers 28 and 29. Panel B shows how often the selector included a known intermediate at the final prompt token: 26.9% or 38.7% on the multihop questions, compared with 10.9% or 20.0% on arithmetic, depending on the layer range. The two question sets differed, so this is a descriptive comparison.\nThe underlying data for Figures 2–3 include the exact two-hop outputs, source hashes, full selector trace, plotted counts, and metric definition.\nBecause coverage was low, I next tested whether automatic target selection was the problem. Other explanations remained. My preregistered plan called the selector adequate if it found the correct intermediate on at least 50% of questions. That cutoff was only a project heuristic; it was not a validated standard for lens quality. I therefore designed a task whose correct intermediate was known from the generating program and bypassed the automatic selector entirely.\nDeleting the known number directions did not establish an accuracy effect To bypass automatic selection, I generated 384 arithmetic expressions from small programs, represented as abstract syntax trees (ASTs). Each program identified one correct intermediate value used by one, two, or three later operations. For example:\nEvaluate: 2 + 4 + (8 + (7 - 3)) program node: 2 + 4 program value: 6 one-token spellings: \u0026#34;6\u0026#34;, \u0026#34;six\u0026#34;, \u0026#34; six\u0026#34; final answer: 18 This test deliberately allowed a brief visible scratchpad. On a held-out development set, a prompt requesting only the final answer produced 9 correct answers out of 96; asking for concise arithmetic equalities produced 93 out of 96. This gave the model enough baseline accuracy for the experiment, but it weakened the causal test: the model could write the target value down and then reuse or recompute it. An unmodified completion for the example above did exactly that:\n2 + 4 + (8 + (7 - 3)) = 6 + (8 + 4) = 6 + 12 = 18 \\boxed{18} The intervention bypassed the selector. For a known value such as 6, it supplied every direction labelled by a spelling that Qwen\u0026rsquo;s tokenizer treats as one token, including \u0026quot;6\u0026quot;, \u0026quot;six\u0026quot;, and \u0026quot; six\u0026quot;. It did this at every processed prompt and generation position within the chosen layer range. The primary version left a direction untouched whenever the unmodified model also ranked it among its ten most likely next tokens at that position. This reduced the risk of simply suppressing a token that the model was about to write. A later, more aggressive version removed that safeguard. In both versions, the intervention subtracted the projection onto the remaining number directions.\nThe generating program tells us that 2 + 4 equals 6. It does not tell us how Qwen represents that fact internally. The model may use a different direction, or it may never store 6 as a stable value.\nThe design specified in advance included:\n384 expressions spanning four operations, with the chosen intermediate used by one, two, or three later operations; two layer ranges selected before the formal run for different reasons: L19–28 mapped the source paper\u0026rsquo;s medium-depth intervention onto Qwen and was the primary range; L23–31 was selected earlier by a fixed rule around this Qwen lens\u0026rsquo;s autocorrelation peak, and let me retest the exact layer range used in the earlier GSM-Symbolic study; three matched random controls for each layer range; a primary masked deletion that left likely next-token directions untouched; and a more aggressive unmasked follow-up, run only after the masked test failed its preregistered criterion. The questions, token spellings, execution checks, and decision rule were fixed before the formal run. Clean accuracy was 94.0% with a 95% interval of [91.7, 96.1], so unlike GSM-Symbolic, this task gave the intervention ample accuracy to reduce. Negative values in the next table would mean that target deletion harmed accuracy. Every comparison with the unmodified model and matched perturbations was near zero:\nFamily Band Targeted deletion − clean Targeted deletion − matched mean Masked L19–28 +0.3 pp [−1.0, +1.6] +0.4 pp [−0.8, +1.6] Masked L23–31 0.0 pp [−1.3, +1.3] +0.2 pp [−1.1, +1.4] Unmasked L19–28 0.0 pp [−1.6, +1.6] +0.3 pp [−1.1, +1.6] Unmasked L23–31 0.0 pp [−1.6, +1.6] +0.2 pp [−1.3, +1.6] These intervals strongly exclude the preregistered 10-point damage required by the arithmetic positive-control criterion. Automatic target selection was therefore not enough to explain the earlier null result.\nMore fundamentally, specifying the correct AST node does not demonstrate that Qwen encoded its value in the deleted token-labelled directions. The source paper first localized contextual activity and then patched or swapped it. My experiment instead deleted fixed token-labelled directions at scale. These interventions answer different questions.\nThe result therefore distinguishes fewer hypotheses than its precision might suggest. It cannot tell apart:\nthe token-labelled numeric directions are genuinely unnecessary for these arithmetic computations; the WikiText lens failed to learn the relevant mathematical representation; the model represents values in a geometry not aligned with single-token aliases; the chosen layer bands miss the causal state; or the visible scratchpad routes around an otherwise internal dependency. These data support a limited conclusion: deleting these token-labelled number directions did not produce a detectable arithmetic effect. They do not establish how Qwen represents arithmetic.\nTakeaway: Across wider layer ranges, 4,600 new problems, and program-specified targets, estimated arithmetic effects stayed near zero. None of these experiments verified that the lens had reached the arithmetic state Qwen was using. The final test also let the model write a short scratchpad, which could allow it to recompute the deleted value.\nFigure 4. The program supplied the correct value 6, but deleting directions labelled with 6 changed accuracy by approximately zero. Supplying the value bypassed automatic selection, but did not show that Qwen represented 6 along those directions. The task also allowed a compact visible scratchpad.\nWhat the evidence supports Status What I can say What I cannot say Supported On a reused 40-question validation set, the audited intervention caused a larger two-hop accuracy drop than three random perturbations matched locally in size, while preserving sentiment and extraction That the deleted directions encoded the task\u0026rsquo;s exact intermediate fact, that this is an independent replication on new questions, or that it validates arithmetic targets Not reproduced The math experiments did not show Anthropic\u0026rsquo;s direct-versus-written protection pattern Qwen\u0026rsquo;s arithmetic is independent of an internal workspace Withdrawn The early two-hop evidence for uniquely J-specific CoT protection failed a replication with 300 new questions The data prove all CoT robustness was generic Unresolved Bypassing automatic selection did not recover an arithmetic effect Whether the third-party lens reached Qwen\u0026rsquo;s actual arithmetic representation The model, fitted lens, tasks, and targeted intervention all differed from the source paper, so these results do not refute Anthropic\u0026rsquo;s findings. They answer a narrower transfer question: I did not reproduce Claude\u0026rsquo;s direct-versus-written math pattern with this Qwen checkpoint, third-party WikiText lens, and deletion intervention. I also never established whether the lens reached Qwen\u0026rsquo;s arithmetic states.\nOnly the initial GSM8K and MATH-500 experiment tested the direct-versus-written pattern itself. The larger GSM-Symbolic and program-targeted studies instead asked whether the lens and intervention worked as intended. They were not additional CoT replications.\nFigure 5. Negative values mean that the intervention reduced accuracy. The two-hop row uses the reused 40-question validation set under the audited protocol. The GSM-Symbolic row comes from the earlier protocol, and the program-specified rows come from a separate experiment. The rows are shown together for orientation but are not statistically combined.\nFirst show that the lens finds the arithmetic state The next experiment should test whether the lens actually reaches the relevant arithmetic state. Repeating GSM-Symbolic at still larger scale with the same lens and deletion rule would tighten an estimate we already have while leaving the central ambiguity intact.\nA more diagnostic sequence My next sequence would be:\nRefit a lens for this exact Qwen checkpoint. I would use the reference implementation and a generic pretraining-like corpus, training it to predict the model\u0026rsquo;s penultimate-layer state as in the paper\u0026rsquo;s main setup. The current WikiText lens, trained against the final layer, would remain a baseline. A math-corpus lens would be a secondary analysis specified in advance and reported regardless of its downstream result. Check that the lens finds the intermediate during an unmodified run. On held-out arithmetic questions, verify that a value such as 6 becomes active when the model computes 2 + 4. The generating program alone cannot establish this alignment. Use a counterfactual causal intervention. Patch a contextual state or swap the proposed state from 6 to 7. In the example above, the strongest evidence would be a directional change from 18 to 19. Deletion only tests necessity and permits recomputation; a swap tests represented content. Only then scale the question. Once an arithmetic causal positive control exists for a given lens, compare direct and written-CoT modes across models and difficulty. I would move to larger models when the experiment is explicitly designed to answer a scaling question. Implementation details that affect the result Padding, batch order, early stopping, and resuming interrupted jobs can all change an intervention during autoregressive generation. In the final pipeline, I tied each random sequence to the question, layer, and absolute token position; froze the tokenized prompts; preserved completed outputs; and recorded the code, configuration, and seeds used for each run.\nThe matched controls were constructed differently across phases. Earlier math experiments used fixed per-layer calibration, while the two-hop and program-targeted experiments recalculated the matching magnitude along each generated sequence. I analyze these results separately, and I only claim that earlier batched fp16 runs reproduced the same answers, not identical floating-point values. The full implementation audit records these limits in detail.\nConclusion The project produced one selective effect: J-space deletion damaged two-hop recall on a reused validation set more than matched random perturbations. It also produced several arithmetic estimates close to zero. Their interpretation remains limited because I never verified that this lens found the arithmetic state Qwen actually used. Finally, the 300-question replication overturned my early claim that written reasoning specifically compensated for J-space damage.\nThe next step is therefore clear. I would first fit and validate a Qwen-specific lens on a localized arithmetic computation, then test whether changing the proposed intermediate causes the predicted change in the answer. Only after that validation would I return to the comparison between direct answers and written reasoning.\nProject artifacts.\nThis project began as Harvard CS2881R HW0. The repository is currently private, so the links below may require access. I plan to release a compact public bundle of code and data.\nRepository and reproduction instructions Concise course report Original preregistration and amendments Intervention implementation audit Two-hop replication on 300 new questions Fresh 4,600-item GSM-Symbolic analysis Program-specified arithmetic experiment: protocol, results, and released data Role and acknowledgments.\nI designed the hypotheses and controls, implemented and audited the code, and ran or supervised the experiments. I used AI systems for coding, review, and drafting assistance, and checked their outputs against the prespecified analysis scripts and saved results. The source J-lens implementation, third-party fitted weights, and public datasets are credited in the repository.\n","permalink":"https://mikotohhh.github.io/posts/jspace-cot-tradeoff/","summary":"I tested whether Anthropic\u0026rsquo;s finding—that written reasoning protects a model from J-space interventions—also holds for Qwen3-4B. I did not reproduce the math result, and follow-up experiments showed why that null result remains inconclusive.","title":"Can Chain of Thought Replace a Model's Internal Workspace?"},{"content":"In July 2026, I asked Claude why AltTab on my Mac had become slow. The debugging session turned into an incident response when a wider process inspection found an AppleScript interpreter consuming unusual CPU, a hidden root-owned watchdog, and a binary called AccountsHelper that the watchdog relaunched every second. A persistence sweep then found mdworker_shared, a cryptocurrency stealer hidden inside .com.apple.metadata.mds, a directory named to resemble Apple\u0026rsquo;s Spotlight indexing service.\nClaude surfaced the anomaly. I turned it into a structured investigation. I designed the plan, safety boundaries, decision gates, and evidence standards; AI agents performed most of the concrete process inspection, reverse engineering, analysis scripting, and report drafting. That division of labor is central to this case study.\nThe technical result matters: we recovered the behavior of a remote-access backdoor and a cryptocurrency stealer, including their command-and-control (C2) logic. C2 is the communication channel through which malware contacts an operator-controlled server to receive instructions or return results and stolen data. We did this without executing either sample during the deep-analysis phase. But the more transferable result is a method for supervising agents when they can produce technical work faster than a human can independently reconstruct every step.\nFrom a slowdown to an incident The result, in thirty seconds:\nAccountsHelper implemented remote task polling, one-shot shell execution, and an interactive terminal. mdworker_shared watched the clipboard, replaced cryptocurrency addresses, recognized wallet-secret-like content, and prepared it for submission to a command server. The agents recovered the backdoor\u0026rsquo;s hidden command infrastructure through static analysis of both its Intel and Apple Silicon code. The conclusion was accepted because saved machine-code evidence, a second architecture, and offline checks agreed—not because an agent produced a convincing report. Several plausible early claims were wrong. The investigation improved when those claims were narrowed or withdrawn, including claims about a supposed Apple-related domain, system-wide network activity, a configuration “key,” and a packet capture. The first useful signal was not a malware signature. It was a mismatch between the symptom and the process tree.\nThe high-CPU AppleScript process belonged to a hidden shell loop running as root. That loop repeatedly launched AccountsHelper as my user from a hidden directory named .com.apple.accountsd, imitating an Apple accounts service. The binary\u0026rsquo;s ad-hoc signature carried no verified developer identity or Apple Team ID, and Gatekeeper rejected it. The second persistence chain kept mdworker_shared running as root from .com.apple.metadata.mds. Apple ships a legitimate Spotlight worker with the same filename, but this copy lived in a hidden, user-writable directory and had no Apple Team ID.\nNeither XProtect nor a later ClamAV scan flagged the samples. I treated those negative results as evidence about detector coverage, not evidence that the binaries were safe.\nBrowser history and filesystem timestamps strongly associate the first installation window with a malicious Google advertisement that impersonated Claude and led to a GitLab Pages site. A configuration file appeared about thirty seconds after that visit, followed minutes later by the backdoor and its persistence mechanism. The exact terminal command or intermediate loader was not recovered, so I do not claim a precise execution mechanism or malware operator.\nMy first consequential decision was to preserve hashes, process state, persistence files, browser evidence, and timestamps before changing the system. I then prioritized three questions: was there another implant, could the malware\u0026rsquo;s C2 communication protocol be recovered safely, and which credentials had to be treated as exposed? The inspected startup and persistence locations showed no third confirmed implant. I rotated high-value credentials from a clean device and made the command-server question the focus of the reverse engineering.\nDesigning an auditable investigation Before the deep-analysis phase, I wrote an investigation contract for the agents:\nDo not contact, scan, or probe the real command servers. Do not execute a complete sample entry point; prefer static analysis or interpretation of isolated functions with system effects disabled. Keep victim-specific configuration, sessions, browser history, packet captures, and credentials off the remote analysis node. Mount samples read-only and non-executable in a container with no network and reduced operating-system privileges. Grade important claims as confirmed, high confidence, inferred, or unknown; require machine-code or data-flow evidence for high-impact conclusions. Use another architecture or analysis tool where feasible, and record sample hashes, tool versions, commands, and code addresses. There was one important exception in the chronology. Before this stricter contract existed, a diagnostic script on the affected Mac started tcpdump, a packet recorder, to save DNS queries and TCP connection attempts. It tried to configure pf, macOS\u0026rsquo;s built-in packet filter, to allow DNS but drop other outbound TCP traffic. It then ran the complete AccountsHelper program for twelve seconds, killed it, and saved the packet capture.\nA later audit found that the script loaded its rule into a named pf ruleset but never verified that the live firewall configuration actually used that ruleset. The capture contained eight SYN packets—the opening packet of a TCP connection—sent toward the command-server IP. They all had the same source port and TCP sequence number, so they were retries from one unanswered attempt, not eight connections. There was no server reply, no completed handshake, and no application data, but the capture could not prove that the firewall caused the failure.\nI therefore stopped treating it as proof of successful firewall containment or an HTTP request. The deep-analysis plan prohibited the entire run-and-capture procedure: no full sample execution and no contact with the real command servers. This correction is a concrete example of the contract working as intended: when the safety evidence was weaker than the story, the claim became narrower and the method changed.\nWhat the evidence established Two findings became the main tests of the method: where the backdoor obtained its command-and-control addresses, and whether mdworker_shared was only a clipboard replacer. Both conclusions were accepted only after the relevant machine code, a second architecture, and offline checks agreed.\nRecovering the hidden command-server addresses Ordinary string searches found no command-server address in the backdoor. A high-entropy local configuration file made one early explanation tempting: perhaps the file was a key that decrypted the server address. Direct attempts to decode the file or brute-force the binary\u0026rsquo;s constant section did not support that story.\nThe useful evidence came from small functions that built strings one byte at a time. Rather than run the malware, the agents interpreted only those functions as Ghidra P-code, a common low-level representation into which Ghidra translates both Intel and Arm instructions. The purpose-built interpreter supported only the byte and memory operations needed by the string builders. It had no implementation for the malware\u0026rsquo;s file, process, or network calls, so the interpreted code could not perform those effects. The recovered transformation was:\nP-code for a string builder -\u0026gt; decode hexadecimal-looking text through the malware\u0026#39;s lookup table -\u0026gt; XOR the bytes with a seed derived from an anti-analysis check -\u0026gt; Base64-style decoding with the malware\u0026#39;s own 64-character alphabet -\u0026gt; plaintext In plain language, each function assembled a disguised string, and the correct text appeared only when an anti-analysis check supplied the expected seed. On a real run, the backdoor would execute an AppleScript that inspected hardware details for markers such as QEMU, VMware, and KVM. That script exited with status 100 when it detected an analysis environment and 0 otherwise. The main program used that number to derive the XOR seed: status 0 produced the valid hidden strings, while status 100 produced none of the expected values. The static analysis recovered this AppleScript as text but never executed it.\nAI agents implemented the byte-level interpreter. I had proposed and approved the static route, required the two architectures to be recovered separately, kept victim data off the remote node, and reviewed the claim against the evidence standard.\nThe resulting claim can be audited as a chain rather than accepted as prose:\nStage What was checked Claim The backdoor embeds two command-server endpoint bases; the local configuration value does not decrypt or construct them. Primary evidence Isolated string builders reconstruct the endpoints, while address-level data flow sends the configuration\u0026rsquo;s first line unchanged as registration material. Cross-check Independently analyzed Intel and Apple Silicon machine code produced the same hidden strings and startup behavior. Offline checker A small checking program verifies artifact hashes, expected values, safety metadata, and agreement between the two exported result sets. It does not rerun the full reverse engineering. Remaining unknowns The client alone cannot show whether the servers were online, which commands an operator issued, or what data was historically received. The recovered endpoint bases are written here in defanged form as 45[.]94[.]47[.]204 and foto[.]gd: replacing each dot with [.] prevents browsers and security tools from turning them into live, clickable links. The same evidence set recovered the registration, task-polling, and interactive-terminal paths. In total, the two architectures agreed on the set of eighteen hidden strings used by this part of the backdoor.\nFigure 1. A concrete claim moves from machine code to a static reconstruction, an architectural cross-check, and a scoped conclusion. The red boundary keeps unanswered incident questions outside the positive result.\nThis example is what “evidence gate” means in practice. The agent\u0026rsquo;s report was not the terminal object. The terminal object was a claim linked to inspectable evidence, a different implementation of the same behavior, an automated check with a stated scope, and a list of unanswered questions.\nReclassifying mdworker_shared mdworker_shared was initially described as a relatively narrow clipboard replacer. Deeper static analysis forced a more serious conclusion.\nIts main loop watched the macOS clipboard roughly once per second and recognized many cryptocurrency address formats. Matching addresses could be replaced by built-in or remotely supplied values. Before replacement, however, the code ran a separate secret detector. It recognized phrases of 12, 15, 18, 21, or 24 words drawn from an embedded 2,048-word BIP39 list. It also recognized private keys—the secret credentials that control wallet funds—in several representations: raw hexadecimal text; WIF, a common Bitcoin key format; and extended keys such as strings beginning with xprv, which can derive an entire family of wallet addresses. Another path packaged the clipboard content together with the name of the frontmost application for submission to its command server.\nThe wording was deliberately narrow. BIP39, a widely used standard for wallet recovery phrases, also includes a checksum that helps distinguish a valid phrase from an arbitrary sequence of listed words. The code checked word counts and list membership but not that checksum. The report therefore says “BIP39-style” rather than claiming that every match would be a valid wallet mnemonic.\nThe Intel and Apple Silicon analyses agreed on the component\u0026rsquo;s hidden strings and core control flow. The Arm build contained nineteen computed jumps—jumps that choose their destination at runtime rather than naming it directly—that Ghidra could not turn into readable control flow. To produce inspectable pseudocode, the agents temporarily replaced each jump inside a disposable Ghidra project with a direct branch whose destination had already been confirmed from the original assembly. The malware file itself was never modified. For every replacement, the analysis saved the address, original and replacement bytes, destination, and untouched instruction range, so the readable result could be traced back to unmodified code. That was enough to replace “clipboard clipper” with a more accurate description: a cryptocurrency stealer combining clipboard hijacking, wallet-secret collection, remote configuration, and event reporting.\nCorrections and lessons for agent oversight The claims that failed are as informative as the capabilities that survived.\nInitial story Decisive check Revised claim Correction source apple.net was a command-server domain Read the surrounding entries in macOS\u0026rsquo;s Unified Log, its system-wide event log It was part of an Apple logging label, not a domain I challenged the attribution Thousands of recent command-line network calls (curl) were malware beacons Attribute the count to processes and time-correlated activity The system-wide count included normal software activity and could not be assigned to the sample I challenged the attribution The high-entropy configuration value decrypted the command-server addresses Trace the value from file read to its consumer It was sent unchanged during registration; the endpoints were embedded elsewhere The agent rejected this local hypothesis during its analysis loop Eight SYN packets proved eight connections and successful firewall blocking Compare source ports and sequence numbers, then inspect whether the pf ruleset was active They were retries from one uncompleted attempt; the cause of failure remained unknown Tightened during the evidence audit The pattern is more important than who noticed each error. A plausible clue had been promoted into a larger story before its attribution or data flow was established. The durable response was to make those checks part of the process, so that future claims did not depend on someone having the same moment of skepticism again.\nThose corrections changed how I think about agent oversight in three ways.\nAutonomy is not authority. The agents were useful precisely because they could execute long technical loops, revise local hypotheses, and produce structured evidence. But permission to perform work was not permission to decide what the incident record should claim.\nReview should change the evidence channel. Asking the same agent to “double-check” can preserve its original framing. Review became more useful when it meant reading the log context, following a value to its consumer, comparing another architecture, or checking frozen outputs. A new paragraph of reasoning was weaker than a new object of evidence.\nOversight should be designed for failure. The goal was not to watch every command. It was to ensure that a failed hypothesis produced a revision, not an unsafe network action or an overstated conclusion. Permissions, stop rules, saved intermediate evidence, and explicit unknowns made that possible.\nThis connects the incident to my J-space causal audit. One project concerned claims about model internals; the other concerned adversarial software. In both, the decisive research move was to withdraw an appealing interpretation when its identifying evidence failed. The common theme is building empirical methods that remain trustworthy when models, tools, or evidence can mislead.\nScope, responsibility, and next steps The core static analysis is complete for both samples and both architectures, and the stage-specific offline checks pass. The remaining work is release engineering: consolidate the evidence matrix and unknowns, write and test stable detection rules, reconcile stale top-level documents, and build a sanitized public artifact.\nThe complete incident repository is intentionally private because it contains live samples, victim-specific configuration, browser evidence, packet data, and identifying paths. A public package should contain sanitized reports, analysis scripts, manifests, selected static outputs, and defanged indicators, but no binaries, credentials, sessions, raw browser history, or victim-specific captures.\nSeveral limits remain:\nThe malicious-ad timeline is strong, but the exact initial execution command was not recovered. mdworker_shared appeared after a backdoor capable of shell execution, but delivery through that backdoor remains an inference. Static analysis and cross-architecture agreement cannot identify the operator or reconstruct a complete victim timeline. Host cleanup or replacement and the public detection package are still being closed out; this is a scoped case study, not a final incident advisory. The easy version of this story is that an AI found and reverse-engineered malware. The useful version is less cinematic: a capable agent surfaced an anomaly and performed much of the technical execution inside a human-designed system of constraints and evidence gates.\nThe recovered malware behavior is the immediate result. The reusable result is a way to make agent work auditable: constrain unsafe actions, preserve the objects behind important claims, seek disagreement through another evidence channel, and allow uncertainty to survive into the final account.\nRole and acknowledgments I was the project owner and research decision-maker. I initiated the system debugging, escalated the anomaly into a formal incident response, designed the investigation plan and safety protocol, chose the priorities and decision gates, initiated the major course corrections, reviewed the evidence, and take responsibility for this account. Claude and Codex performed most of the concrete technical execution, including process inspection, decompilation, analysis scripting, isolated-function interpretation, saved-evidence generation, and report drafting. They also proposed small refinements and corrected local hypotheses. Credential rotation and other identity-bound remediation actions were performed by me from a clean device.\n","permalink":"https://mikotohhh.github.io/posts/auditable-malware-analysis/","summary":"A real compromise became a case study in supervising capable agents: bounded autonomy, auditable evidence, and claims designed to fail closed.","title":"From a Compromised Mac to an Auditable Malware Analysis"},{"content":"I\u0026rsquo;m Luozhong Zhou, a machine learning researcher. I write here about LLMs, interpretability, and the craft of running careful experiments.\nGitHub: mikotohhh Email: luozhong [at] sakana.ai ","permalink":"https://mikotohhh.github.io/about/","summary":"\u003cp\u003eI\u0026rsquo;m \u003cstrong\u003eLuozhong Zhou\u003c/strong\u003e, a machine learning researcher. I write here about LLMs,\ninterpretability, and the craft of running careful experiments.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eGitHub: \u003ca href=\"https://github.com/mikotohhh\"\u003emikotohhh\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eEmail: luozhong [at] sakana.ai\u003c/li\u003e\n\u003c/ul\u003e","title":"About"}]