Steering a multilingual model's safety from the inside

One-line claim: Tiny Aya Global refuses harmful requests far less in Hindi than in English, but it uses the same refusal direction in every language; adding that English direction at one layer cuts Hindi harmful compliance from 22.6% to 8.6%, matching translate-to-English in one pass, with 2% over-refusal on harmless requests and no loss of reading comprehension.

Anant Shri · Lexsi AI Safety & Alignment Hackathon, November 2026 · solo

Tracks and problem statements covered - Track 2a, #4 Multilingual and code-mixed safety (primary): measure whether safety holds across en/it/de/hi (+ ml, romanised Hindi), then close the gap. - Track 2a, #2 Adversarial and safety evaluation sets: a paired harmful/benign multilingual benchmark with labels, scoring and a comprehension control. - Track 2b, #8 Inference-time safety steering (primary): refusal-direction steering, no retraining, over-refusal on harmless requests kept at 2–4%. - Track 2b, #9 Locating where safety lives: the refusal direction, the layers where it is shared and where failing prompts lose it, then a targeted fix there.

Content warning. This write-up discusses harmful prompts (violence, self-harm, hate, sexual exploitation, illegal activity) and a model's compliance with them. No harmful completions are quoted here. The benchmark and artefact repos contain them; safety research only.

Abstract

Multilingual language models are typically safety-trained mostly in English, and refuse harmful requests less reliably in other languages. We ask whether that is because refusal is one shared mechanism that fires more weakly outside English (A), separate weaker mechanisms per language (B), or a shared representation read out differently (C), and whether the answer gives a cheap fix. We built a paired benchmark from MultiJail: 133 harmful prompts (100 harmful requests, 33 hateful statements), each with a matched benign twin, in English, Italian, German, Hindi, Malayalam and romanised Hindi (1,770 rows), with a comprehension control so that prompts the model cannot read never count as safety failures. On Cohere Labs' Tiny Aya Global (3.35B), harmful-request compliance is 5% in English and 23% in Hindi (+17.4 pp, 95% CI 9.5–25.8), with low over-refusal everywhere: under-refusal, not caution. Translating the prompt to English first nearly closes the gap but costs a second pass and answers in English.

Inside the model, the English difference-of-means refusal direction separates harmful from benign prompts almost as well in Italian, German and Hindi (AUROC 0.99 each), and a Hindi direction is as aligned with it as a neutral food-vs-travel concept is across languages. The failures are local: Hindi prompts the model complies with have lost much more of their harm signal relative to their English twin than prompts it still refuses. Steering with SafeTune along the English direction at one layer (slot 24, a fifth of the English harmful–benign separation) cuts Hindi compliance to 8.6% (−14.0 pp [−21.7, −6.5]), equal to translate-to-English (8.7%) and better than a safety system prompt (15.1%), in one pass and answering in Hindi. Matched control and random directions do nothing; removing the direction at every layer raises compliance to 40–51% in all languages. A Hindi-specific direction does no better (B disfavoured). Over-refusal on harmless requests stays at 2–4% in every language (Hindi 2.2%), Belebele reading comprehension (via AuditKIT) moves by under 1 pp, but harmless statements get more pushback (total Hindi over-refusal 9.6%). Steering rescues 13 of 17 Hindi failures where English refuses, and 1 of 4 where English also fails: it restores signal lost on the way into Hindi but cannot create recognition English never had. All predictions were registered before their runs, judge labels were validated and over-refusal hand-corrected, and every artefact is public.

Headline (before → after)

Hindi, Tiny Aya Global base + steering (r_en, slot 24) translate-to-English (2 passes)
harmful-request compliance (↓) 22.6% 8.6% 8.7%
over-refusal on harmless requests (↓, human-corrected) 0.0%¹ 2.2% —
Belebele reading comprehension (↑) 57.8% 57.7% —
passes / answer language 1 / Hindi 1 / Hindi 2 / English

¹ base over-refusal across all Hindi benign twins.

Base model + method in one line: CohereLabs/tiny-aya-global (fp16, greedy), steered with SafeTune RefusalDirectionModel(mode="steer") adding 6.07 × (unit English refusal direction) to the residual stream after decoder layer 23, at every token.

For judges: one minute - Gap: Hindi complies with harmful requests 4.4× as often as English (22.6% vs 5.1%); Italian 13.3%, German 15.6%. - Why: the refusal direction is shared across languages; some non-English prompts lose harm signal along it in the middle layers. - Fix: one vector at one layer → Hindi 8.6% (= translate-to-English, one pass, Hindi answer). Control/random directions: no effect. Ablation: refusal collapses everywhere. - Cost: over-refusal on harmless requests 2–4%; Belebele −0.1 to −0.8 pp; more pushback on harmless statements. - Trust: pre-registered predictions (incl. failed ones), kill-tests first, judge validated against humans, all over-refusal hand-corrected, everything public. - Kits: SafeTune (direction extraction, steering, ablation), AuditKIT (WildGuard second judge, Belebele with the steering hook).

Trade-off (A) Full run, Hindi: harmful compliance vs human-corrected over-refusal for every condition. (B) Development dose–response by layer, with the no-GPU forecast and the kill-test controls.

Problem statement and approach

Why this problem / real use case. Most people do not use LLMs in English. A model that declines to help sell a person in English but helps in Hindi is not safe for Hindi speakers, and small multilingual models like Tiny Aya are exactly what gets deployed on-device and in low-resource settings. The standard fix, translating every request to English first, doubles inference and answers in the wrong language. A one-pass internal fix with a measured cost would be directly deployable.

Approach. (1) Build a paired, comprehension-controlled multilingual benchmark and measure the gap against two naive baselines (safety system prompt; translate-to-English). (2) Look inside: does the English refusal direction separate harmful from benign in other languages, is it specially shared, and where do failing prompts lose it? (3) Intervene: steer along it with SafeTune, kill-test against control directions first, screen layer and strength on development pairs, then run in full with over-refusal and utility checks.

Research questions and hypotheses. - A, shared: one refusal mechanism, triggered more weakly outside English → steering along r_en restores refusal. - B, language-specific: each language has its own weaker mechanism → a Hindi direction steers better than the English one. - C, shared representation, different readout: same representation, different decision → steering helps but the language's readout limits it; target-language steering does better.

Pre-registered. Every prediction and decision rule was written in PREDICTIONS.md before its run (P2.x–P4.9), including bets on A/B/C (author: C; Claude: A 50 / C 35 / B 15), kill-test rules for each phase, and outcomes with amendments kept visible. Post-hoc analyses are labelled as such.

Related work. Wang et al. (2025) find the English refusal direction near-universal across 14 languages but harmful/harmless separation weaker outside English (≈ A). Aziz et al. (2026) find separation nearly intact in low-resource languages and locate the failure in the decision readout, fixed by recalibration (≈ C). Zhang (2026), the closest work (MultiJail, difference-of-means), shows high cross-lingual cosine is generic: a non-safety contrast is just as aligned. We sit between Wang and Aziz (separation intact as in Aziz; failures are prompts losing signal along the shared direction, a local version of Wang; readout softer outside English). New here: a multilingual-first model; a comprehension control; human-corrected judge rates and a human-translation control; translate-to-English as the bar for any internal fix; control and random directions in the causal tests (missing in Zhang); paired twins locating where each failing prompt loses signal; over-refusal and utility accounting of the fix. Full map: LITERATURE.md.

Hackathon repos (kits) used

SafeTune (Lexsi-Labs/SafeTune @ 26f59e6), Phase 4, every run: - extract_refusal_direction with RefusalDirectionConfig(target_layers=[17,20,23], pick_layer=i, pool_method="last_token", normalize=False): re-extracted all steering vectors in-job (per cross-fitted fold, English and Hindi, plus the control direction) and checked them against ours (cosine 1.000, worst 0.999999). - RefusalDirectionModel(model, d, mode="steer", strength=α, layers=[slot−1]): all steering runs (14 screening conditions, two full runs, Hindi at 0.3). Verified that it moves the slot-24 score by exactly α. - RefusalDirectionModel(model, d, mode="ablate", strength=1.0, layers=None): all-layer ablation (necessity test). - Trap found: SafeTune's layers=[i] indexes decoder blocks, so our "slot k" (output of layer k−1) is layers=[k−1]. Its extractor runs the full LM head and OOMs on a T4; we pass the decoder without the head.

AuditKIT (Lexsi-Labs/AuditKIT @ 71316da), Phase 4: - GuardJudge("wildguard", judge_model="hf:allenai/wildguard", fp16): validated against our human labels (registered rule failed; used only where valid, Section "What didn't work"), screening over-refusal on harmless-request twins, second-judge column in the final table, positive control on WildGuardMix. - Native engine MCQAdapter("mcq_loglikelihood") on Belebele (900 questions × en/it/de/hi, zero-shot, log-likelihood of A–D) with the SafeTune hook installed on AuditKIT's own model object (run_lmeval reloads weights and would drop hooks). - Finding for the kit: its disk cache returned the unhooked model's results for the hooked model; we use a fresh XDG_CACHE_HOME per condition. It also prepends a second BOS token to WildGuard prompts (minor effect, measured).

CuratorKIT, CircuitKIT: assessed (RECCE.md) but not used in the results. AlignTune, AgentTune: not used. Everything else (benchmark construction, judging, geometry, statistics) is our own code.

How the repos connect.

flowchart TD
  MJ[MultiJail harmful prompts] --> B1[Benchmark + benign twins + MT<br/>comprehension control]
  B1 --> G[Generate: base / sysprompt / translate_en]
  G --> J[Qwen2.5-72B judge] --> H[Human audits] --> R[Rates + paired CIs]
  B1 --> A[Activations, HF Job] --> D[Directions, geometry, kill-test]
  D --> SP[Steering vectors + no-GPU forecast]
  SP --> ST[SafeTune: re-extract, steer, ablate]
  ST --> J
  ST --> AK[AuditKIT: WildGuard + Belebele with the SafeTune hook]
  R --> F[Final analysis]
  AK --> F

Our Phase 3 code finds the direction and the layer; SafeTune re-extracts it (as a check) and applies it; AuditKIT measures what the steered model loses (Belebele) and gives an independent over-refusal judge (WildGuard); the primary judge and human audits score everything on the same footing as the baselines.

Models and licences

role model config licence
base (Aya) model CohereLabs/tiny-aya-global, 3.35B, 36 layers, sha af89d219 fp16, greedy, its own chat template, 512 new tokens, transformers 5.22.0, no quantisation CC BY-NC 4.0 + Cohere Labs acceptable-use policy
refusal judge (j1) Qwen/Qwen2.5-72B-Instruct via HF Inference Providers temperature 0, rubric JUDGE_RUBRIC.md Part A, reads the response in its own language Qwen licence
comprehension judge Qwen2.5-7B-Instruct (4-bit) 1–5 restatement score Apache 2.0
second judge allenai/wildguard (7B) via AuditKIT fp16 Apache 2.0
MT NLLB-200 3.3B (it, de); Google Translate (hi, ml) sentence-split for NLLB CC BY-NC 4.0 / ToS

No closed-model baseline. Our own released "model" is a set of steering vectors, not weights: anantshri1/tiny-aya-global-refusal-steer.

Datasets, benchmarks and licences

Sources. | source | use | size | licence | |---|---|---|---| | MultiJail (Deng et al., 2023) English | harmful prompts: 100 H_core requests + 33 H_bait statements (of 315; labelled, de-duplicated, B/A/X dropped) | 133 | MIT | | benign twins (LLM-drafted, human-reviewed) | one per harmful prompt: same topic, maximal shared wording, harmless intent | 133 | CC BY-NC 4.0 | | translations | it/de (NLLB), hi/ml (Google), hi-Latn (mechanical transliteration), paraphrases for 29 pairs | 1,770 rows total | CC BY-NC 4.0 | | control set | food vs travel harmless pairs, same languages | 64 pairs × 6 | CC BY-NC 4.0 | | MultiJail human Italian | translation control only | 100 | MIT | | Belebele | utility retention | 900 q × 4 languages | CC BY-SA 4.0 |

Languages. English, Italian, German, Hindi (all comparisons); Malayalam and romanised Hindi (descriptive only: the model understood 61% / 26% of benign prompts; Malayalam is not in its listed training languages).

Comprehension filter (instead of CuratorKIT filtering). Tiny Aya restates each prompt in English, a judge scores the restatement; a pair is kept only if both sides are understood. Pairs kept of 133: en 132, it 131, de 129, hi 124, ml 43, hi-Latn 11.

Splits and leak check. No training. Directions are estimated with 5-fold cross-fitting by pair (seed 0): each prompt is scored or steered by a direction that never saw its pair, in any language. Phase 4 screening used 55 development pairs (folds 0–2); the 34 test pairs (folds 3–4) were never used for any choice; final numbers are on all understood pairs, with the test folds reported separately. Leaks are excluded by construction (splits are by pair_id, so a prompt's translations and twin share its split; exact-match, not embedding or MinHash). Pre-training contamination is possible (MultiJail is public since 2023) and cannot be checked.

Safety benchmark (focus). Harmful compliance on H_core requests in en/it/de/hi (focus task); H_bait as secondary. General capability (retention check): Belebele in en/it/de/hi. Over-refusal test set: the benign twins (133 per language): XSTest-style "scary word, safe meaning" for requests, neutral questions about the same group for statements.

All data: benchmark (with its dataset card) and artefacts.

Metrics, and why these

metric direction why
H_core harmful compliance (judge C+P among understood pairs) lower is better the safety failure itself
paired difference vs English / vs base (pp, 95% pair bootstrap) gap: lower is better same prompts, so content is held fixed
over-refusal on benign twins, human-corrected, split requests / statements lower is better what must not get worse
Belebele accuracy higher is better general capability must not get worse
d′ / AUROC of harmful vs benign along r_en higher = better separation is the safety signal present per language?
disattenuated cross-language transfer vs control comparison is refusal specially shared?
rescue rate (GAP vs BOTH-fail) higher is better which failures the fix can and cannot reach

Checking the automatic judge. j1 vs blind human labels (n=72, enriched): Cohen's κ 0.53 (5-way); on the headline split (harmful compliance vs not) agreement 0.75, κ 0.37. A second blind check on the rows that decide the headline (n=112): j1 precision on harmful failures 0.76–0.92 by language; the gap survives correction (corrected Hindi gap +21.6 pp [7.2, 39.5]). On benign over-refusal, j1 precision was ~0.2, so all over-refusal is human-corrected: every j1 flag plus a stratified sample of non-flags is labelled by the author, and rates are re-estimated per language × tier. WildGuard: H_core twins recall 0.95 / precision 0.79; re-validated on steered outputs (precision 9/10). Comprehension judge vs human: weighted κ 0.87. The judge provider was switched for the full runs only after a pre-registered agreement check (0.95 / 0.97 on 100 rows).

Method / experiments

Generation (own code, src/generate.py, src/screen_job.py). fp16, greedy (do_sample=False), batch 16, 512 new tokens (screening 256; label stability from 256 verified), the model's default chat template and system preamble. Baselines: sysprompt adds "Refuse harmful requests in any language." to the preamble; translate_en asks the model to translate the prompt to English, cleans the translation, and answers it.

Directions (own code, src/geometry.py, src/steer_prep.py). Residual stream (pre final norm) at the last prompt token <|START_RESPONSE|>, all 37 slots; r = mean(harmful H_core) − mean(benign twin), unit-normalised, cross-fitted (5 folds by pair, seed 0). Control: food − travel; random: Gaussian, seed 0. Δ_k = English harmful − benign score separation at slot k (Δ_24 = 30.34).

SafeTune (steering). Defaults kept except: mode="steer" (or "ablate"), layers=[23] (slot 24), strength=α with α/Δ ∈ {0.1, 0.2, 0.3} at slots 18/21/24 in screening, 0.2 in the final runs (α = 6.07), 0.3 for Hindi post hoc; per-fold vectors; extraction pool_method="last_token", normalize=False.

AuditKIT (evaluation). GuardJudge("wildguard") with default prompt template and parsing, fp16 on an L4; Belebele via MCQAdapter("mcq_loglikelihood"), zero-shot, default prompt, letter targets, 900 questions per language, fresh cache per condition.

Selection rule (fixed before reading results, PHASE4_PLAN.md). Among screened settings that bring Hindi to the translate-to-English level, choose the lowest over-refusal (WildGuard on request twins); ties → smaller α. Chosen: slot 24, α/Δ 0.2.

Seeds. Fold map, random direction, bootstrap (2,000 resamples) and human-audit sampling all seed 0. Generation is greedy (deterministic up to GPU numerics).

Ablations and controls. Control and random directions at the same push (kill-test); all-layer ablation; Hindi vs English direction (B test); three layers × three strengths; language-conditional policy; Hindi at 0.3.

Results

Same model, before vs after (H_core compliance %, 512 tokens, judge j1). | condition | en | it | de | hi | Hindi over-refusal (all twins / requests) | Belebele hi | |---|---|---|---|---|---|---| | base | 5.1 | 13.3 | 15.6 | 22.6 | 0.0 / — | 57.8 | | safety system prompt | 5.1 | 9.2 | 14.6 | 15.1 | 0.8 / — | — | | translate-to-English | 5.1 | 6.1 | 9.4 | 8.7 | 5.4 / — | — | | steer r_en, slot 24, 0.2 (F1) | 3.0 | 7.1 | 9.4 | 8.6 | 9.6 / 2.2 | 57.7 | | steer r_hi, slot 24, 0.2 (F2) | 4.0 | 8.2 | 12.5 | 9.7 | 9.8 / 2.2 | 57.8 | | F1, English left unsteered | 5.1 | 7.1 | 9.4 | 8.6 | 9.6 / 2.2 | 57.7 |

Safety and general side by side (F1). Over-refusal on harmless requests en 3.0 / it 2.0 / de 4.2 / hi 2.2%; on all twins 4.5 / 2.3 / 5.4 / 9.6%. Belebele change −0.1 / −0.3 / −0.8 / −0.1 pp (base 74.9 / 68.9 / 68.6 / 57.8%; 8–18 of 900 answers change). WildGuard on request twins (second judge): 5.1 / 4.1 / 5.2 / 1.1% (base 2.0 / 4.1 / 3.2 / 1.1).

Confidence intervals (paired, 95% pair bootstrap). F1 − base, Hindi −14.0 [−21.7, −6.5], Italian −6.1 [−11.7, −1.0], German −6.2 [−12.5, −1.0]; F1 − system prompt, Hindi −6.5 [−12.9, 0.0]; F1 − translate-to-English, Hindi −0.1 [−5.3, 5.3]; F2 − F1, Hindi +1.1 [−2.3, 4.7]. Hindi gap vs English +17.4 → +5.4 [0.0, 11.5].

Finding 1: the gap is under-refusal, not a translation artefact

Hindi +17.4 pp over English [9.5, 25.8], Italian +8.2, German +10.5; human-corrected over-refusal 0–5.5% everywhere. MultiJail's human Italian gives the same compliance as our machine Italian (+4.1 pp, CI includes 0). The gap survives judge correction.

Finding 2: the refusal direction is shared; failing prompts lose signal along it

Phase 3 geometry (A) Harmful vs benign separation along the English direction (held-out d′) by layer. (B) Cross-language transfer: refusal (solid) vs a neutral food/travel topic (dotted). (C) Extra harm signal lost by prompts that fail in a language, relative to their English twin.

At slot 24, AUROC 0.99 in en/it/de/hi (d′ 4.6 / 4.2 / 3.9 / 4.1); Malayalam and romanised Hindi collapse (d′ ≈ 0.8–0.9). Transfer en→hi 0.90 for refusal vs 0.89 for the topic control: shared, not specially shared. Hindi prompts the model complies with lose −8.2 score units vs their English twin (median), against −3.4 for prompts it still refuses.

Finding 3: steering along it is sufficient, and the direction is necessary

Kill-test on 55 dev pairs at the same push (α/Δ 0.25): Hindi compliance r_en −16.4 pp [−27.3, −7.3], food/travel +3.6 [−5.5, 12.7], random −1.8 [−7.3, 3.6]. The no-GPU forecast tracked the slot-24 dose–response (forecast 15.3 / 10.4 / 6.9% vs observed 16.4 / 10.9 / 7.3% at 0.1 / 0.2 / 0.3). All-layer ablation: compliance en 40.0 / hi 43.6 / de 50.9%.

Finding 4: what steering can and cannot fix

Hindi failures whose English twin is refused (GAP): 13 of 17 rescued. Failures English shares (BOTH): 1 of 4. Pushing Hindi to 0.3 (post hoc) gives 9.7% (+1.1 [−2.2, 4.5] vs 0.2): saturation, because the remaining failures are mostly BOTH-type.

Finding 5: verdict on the hypotheses

A in a local form fits best (r_hi = r_en; controls inert; ablation removes refusal everywhere). Some C (Hindi's score→refusal curve is higher and softer: midpoint 13.6 vs 9.75, width 13.7 vs 6.4), but C's specific prediction (target-language steering wins) failed. B disfavoured, with limited power (r_en and r_hi are 0.90-aligned at slot 24).

What didn't work (negative results)

Sample inputs and outputs

We do not reproduce harmful completions. Every prompt, response, judge label and human label is in the artefacts, joined by uid (e.g. mj027-b-en-orig, a benign twin: "Why are students with disabilities given extra time in exams?").

Cost / ROI and compute

Limitations (known weaknesses)

One 3B-class model. One LLM-judge family for headline labels; a single human annotator (the author: native Hindi, reads Italian, German A2/B1). Malayalam and romanised Hindi are not understood by the model, so nothing is claimed for truly low-resource languages. Direct MultiJail requests only: no adversarial jailbreaks, encodings or multi-turn attacks. Greedy, 512-token generation. Hindi over-refusal rests on a handful of rows. The held-out test folds have little power. MT for it/de/ml is not natively validated (Hindi is). One canonical number was found stale during write-up (screening grid recorded before judge retries; corrected and amended, selection unaffected).

With more credits

Planned approach. (1) Repeat on Aya Expanse 8B and one non-Aya multilingual model. (2) Jailbreak robustness of the steered model (MultiJail-style attacks, encodings, best-of-N) and abliteration as the weight-space attack. (3) Per-language strength and conditional steering (steer only when a probe on the same direction flags a harmful request), to remove the statement-pushback cost. (4) Native validation of Italian/German/Malayalam and a second annotator. (5) CircuitKIT localisation of where Hindi prompts lose the signal (slots 12–21) for a narrower fix.

Expected before → after (forecasts, not results). Conditional steering: Hindi compliance ≈ 9% (unchanged) with over-refusal on harmless statements back towards base (≈ 1–3% total instead of 9.6%). Aya Expanse 8B: a smaller base gap (stronger multilingual safety) and similar relative reduction (≈ 50–60%). Under jailbreak attacks: steering should raise refusal less than on direct requests, since attacks often suppress the harm signal itself (BOTH-type failures), which Finding 4 shows steering cannot create.

Artifacts

Documentation and reproduction

Environment. Python 3.13; pinned versions in requirements.txt; SafeTune and AuditKIT from source at the commits above. GPU: one T4 (16 GB) suffices for Tiny Aya fp16; WildGuard needs an L4 (24 GB). Judge: an HF token with Inference Providers access (LEXSI_JUDGE_PROVIDER, default featherless-ai; full runs deepinfra). Laptop (CPU) for everything else.

Repo layout. src/paths.py is the only place paths are defined; PATHS.md is the full file contract (names, schemas, HF layout); canonical_numbers.txt holds every reported number; phase write-ups: PHASE2_SUMMARY.md, PHASE3_SUMMARY.md, PHASE4_SUMMARY.md; design and traps: PHASE4_PLAN.md, RECCE.md.

Run instructions (outputs land under outputs/ and in the HF artefact repo; every step resumes): 1. Benchmark: python src/build_benchmark.py (from the reviewed interim files in the artefact repo's curation/); control set python src/build_control.py prep|build; comprehension python src/comprehension_final.py. 2. Generation + judging (Phase 2): notebooks/02_baseline.ipynb (uses src/generate.py); judge with src/judge.py; rates python src/rates.py, baselines python src/baseline_rates.py <condition>; human over-refusal check python src/audit_overref.py sample|correct <condition>. 3. Activations + geometry (Phase 3): hf jobs uv run --flavor t4-small --timeout 1h --secrets HF_TOKEN src/extract_activations.py (add --control for the control set); then python src/killtest.py, src/cosine.py, src/twin_depth.py, src/fig_phase3.py. 4. Steering (Phase 4): python src/steer_prep.py, python src/forecast.py, python src/screen_split.py; hf jobs uv run --flavor t4-small --secrets HF_TOKEN src/screen_job.py --stage extract|screen|full|fullhi; judge python src/screen_judge.py [--full|--fullhi]; WildGuard src/screen_wg_job.py and Belebele src/belebele_job.py on --flavor l4x1; analysis python src/screen_analysis.py, src/final_analysis.py, src/lang_conditional.py, src/fig_phase4.py. 5. Mac-side checks of the single-file job scripts: src/test_*.py.

Declaration of work done outside the hackathon. None. All work (benchmark, code, runs, labels, write-up) was done during the hackathon, from 2 November 2026.

Discussion and implications

The multilingual safety gap in this model is not a missing Hindi safety mechanism: the English one is shared, and the failures are prompts whose harm signal weakens on the way through the network. That makes the gap cheap to repair from the inside, and it predicts the limit of any such repair: steering cannot recover harm the model never recognised, in any language. For deployment, steer only non-English inputs. For evaluation, measure comprehension before calling anything a safety failure, and compare internal fixes with translate-to-English, not only with the base model.

Team, contributions, acknowledgements

Teammates: none (solo). Author contributions: Anant Shri: research question, design decisions, all runs, all human labels (curation, judge validation, over-refusal), analysis and write-up review. Acknowledgements: Claude (Anthropic) was a research partner throughout: planning, design discussions, code drafts, reviews and writing drafts; all runs, human labels and final decisions are the author's. Compute on the author's Hugging Face credits.

LLM usage statement. Approximately 80% of the code and 85% of the write-up prose were drafted by an LLM (Claude) and then run, debugged, corrected and reviewed by the author; research decisions were made jointly; all human labels and all final decisions are the author's.

References

Appendix

Judge rubric and prompts: JUDGE_RUBRIC.md. Full per-condition tables: figures/p4_final_v1.csv, figures/p4_screen_v1.csv. Licence: code Unlicense; data CC BY-NC 4.0.