OpenMythos Documentation

File: README.md Copy Chapter

OpenMythos

Disclaimer: OpenMythos is an independent, community-driven theoretical reconstruction based solely on publicly available research and speculation. It is not affiliated with, endorsed by, or connected to Anthropic or any of their proprietary systems.

OpenMythos is an open-source, theoretical implementation of the Claude Mythos model. It implements a Recurrent-Depth Transformer (RDT) with three stages: Prelude (transformer blocks), a looped Recurrent Block (up to max_loop_iters), and a final Coda. Attention is switchable between MLA and GQA, and the feed-forward uses a sparse MoE with routed and shared experts ideal for exploring compute-adaptive, depth-variable reasoning.

Installation Copy Chapter

pip install open-mythos

#uv pip install open-mythos

To enable Flash Attention 2 in GQAttention (requires CUDA and build tools):

pip install open-mythos[flash]

Usage Copy Chapter

import torch
from open_mythos.main import OpenMythos, MythosConfig


attn_type = "mla"  # or "gqa"

base = {
    "vocab_size": 1000,
    "dim": 256,
    "n_heads": 8,
    "max_seq_len": 128,
    "max_loop_iters": 4,
    "prelude_layers": 1,
    "coda_layers": 1,
    "n_experts": 8,
    "n_shared_experts": 1,
    "n_experts_per_tok": 2,
    "expert_dim": 64,
    "lora_rank": 8,
    "attn_type": attn_type,
}

if attn_type == "gqa":
    cfg = MythosConfig(**base, n_kv_heads=2)
else:
    cfg = MythosConfig(
        **base,
        n_kv_heads=8,
        kv_lora_rank=32,
        q_lora_rank=64,
        qk_rope_head_dim=16,
        qk_nope_head_dim=16,
        v_head_dim=16,
    )

model = OpenMythos(cfg)
total = sum(p.numel() for p in model.parameters())
print(f"\n[{attn_type.upper()}] Parameters: {total:,}")

ids = torch.randint(0, cfg.vocab_size, (2, 16))
logits = model(ids, n_loops=4)
print(f"[{attn_type.upper()}] Logits shape: {logits.shape}")

out = model.generate(ids, max_new_tokens=8, n_loops=8)
print(f"[{attn_type.upper()}] Generated shape: {out.shape}")

A = model.recurrent.injection.get_A()
rho = torch.linalg.eigvals(A).abs().max().item()
print(
    f"[{attn_type.upper()}] Spectral radius ρ(A) = {rho:.4f} (must be < 1)"
)

Model Variants Copy Chapter

Pre-configured scales from 1B to 1T parameters:

from open_mythos import (
    mythos_1b,
    mythos_3b,
    mythos_10b,
    mythos_50b,
    mythos_100b,
    mythos_500b,
    mythos_1t,
    OpenMythos,
)

cfg = mythos_7b()  # returns a MythosConfig
model = OpenMythos(cfg)

total = sum(p.numel() for p in model.parameters())
print(f"Parameters: {total:,}")

Training Copy Chapter

The training script for the 3B model on FineWeb-Edu is at training/3b_fine_web_edu.py.

Single GPU:

python training/3b_fine_web_edu.py

Multi-GPU (auto-detects GPU count):

torchrun --nproc_per_node=$(python -c "import torch; print(torch.cuda.device_count())") training/3b_fine_web_edu.py

Key design choices:

Documentation Copy Chapter

The Central Hypothesis Copy Chapter

Claude Mythos is suspected to be a Recurrent-Depth Transformer (RDT) — also called a Looped Transformer (LT). Rather than stacking hundreds of unique layers, a subset of layers is recycled and run through multiple times per forward pass. Same weights. More loops. Deeper thinking.

This is not chain-of-thought. There is no intermediate token output. All of this reasoning happens silently, inside a single forward pass, in continuous latent space.

Architecture Copy Chapter

A looped transformer divides its layers into three functional blocks:

Input
  ↓
[Prelude P]        — standard transformer layers, run once
  ↓
[Recurrent Block R] — looped T times
  ↑_______↓         (hidden state h updated each loop with input injection e)
  ↓
[Coda C]           — standard transformer layers, run once
  ↓
Output

The recurrent block update rule at each loop step t:

h_{t+1} = A·h_t + B·e + Transformer(h_t, e)

Where:

• h_t is the hidden state after loop t
• e is the encoded input (from the Prelude), injected at every loop
• A and B are learned injection parameters
• The Transformer blocks apply attention and MLP as usual

The injection of e at every step is what prevents the model from drifting — it keeps the original input signal alive throughout the entire recurrence depth.

The full implementation is in open_mythos/main.py. See the OpenMythos class reference for a detailed API walkthrough, configuration options, and usage examples.

Attention Implementations Copy Section

The attention layer is switchable via cfg.attn_type:

RoPE is applied to Q and K before caching, so cached values do not need to be re-rotated on retrieval.

Why This Explains Mythos Copy Chapter

1. Systematic Generalization Copy Section

Vanilla transformers fail to combine knowledge in ways they have never seen during training. Looped transformers pass this test. The ability emerges through a three-stage grokking process:

1. Memorization — model fits training distribution
2. In-distribution generalization — model handles known compositions
3. Systematic generalization — model handles novel compositions OOD, abruptly and suddenly

This is why Mythos feels qualitatively different from other models on novel questions — the capability phase-transitions in, rather than emerging gradually.

2. Depth Extrapolation Copy Section

Train on 5-hop reasoning chains. Test on 10-hop. Vanilla transformer fails. Looped transformer succeeds — by running more inference-time loops. This maps directly to the observation that Mythos handles deeply compositional problems (multi-step math, long-horizon planning, layered arguments) without explicit chain-of-thought.

More loops at inference = deeper reasoning chains = harder problems solved.

3. Latent Thoughts as Implicit Chain-of-Thought Copy Section

Each loop iteration is the functional equivalent of one step of chain-of-thought, but operating in continuous latent space rather than token space. A looped model running T loops implicitly simulates T steps of CoT reasoning. This has been formally proven (Saunshi et al., 2025).

Furthermore, continuous latent thoughts — unlike discrete token outputs — can encode multiple alternative next steps simultaneously. This allows something closer to breadth-first search over the reasoning space, rather than a single committed reasoning path. The model is effectively exploring many possible directions inside each forward pass before converging.

4. No Parameter Explosion Copy Section

A looped model with k layers run L times achieves the quality of a kL-layer non-looped model, with only k layers worth of parameters. For Mythos-scale deployments, this matters enormously:

• Memory footprint does not grow with reasoning depth
• Inference-time compute scales with loop count, not model size
• This makes deeper reasoning "free" in terms of parameters

The Stability Problem (and How It Was Likely Solved) Copy Chapter

Training looped models is notoriously unstable. Two failure modes dominate:

• Residual explosion — the hidden state h_t grows unboundedly across loops
• Loss spikes — training diverges suddenly due to large spectral norms in injection parameters

The Dynamical Systems View Copy Section

Recast looping as a discrete linear time-invariant (LTI) dynamical system over the residual stream. Ignoring the nonlinear Transformer contribution, the recurrence becomes:

h_{t+1} = A·h_t + B·e

For this LTI system, stability is governed entirely by the spectral radius of A:

• ρ(A) < 1 → stable, convergent
• ρ(A) ≥ 1 → unstable, divergent

Empirically, every divergent training run learns ρ(A) ≥ 1. Every convergent run maintains ρ(A) < 1.

The Fix Copy Section

Constrain the injection parameters so that stability is guaranteed by construction:

1. Parameterize A as a continuous negative diagonal matrix
2. Discretize using ZOH/Euler schemes: A_discrete = exp(Δt · A_continuous)
3. Enforce negativity via A := Diag(-exp(log_A)) with a learned scalar Δt
4. This ensures ρ(A) < 1 always holds, regardless of learning rate or batch noise

The result: the looped model becomes significantly more robust to hyperparameter selection and trains cleanly even at high learning rates. This is the Parcae architecture (Prairie et al., 2026), and it represents the most likely class of solution Anthropic used to make Mythos trainable.

Scaling Laws for Looped Models Copy Chapter

Parcae establishes the first predictable scaling laws for looped training:

• Training: For a fixed FLOP budget with fixed parameters, increasing mean recurrence and reducing token count yields a lower loss than training with minimal loops on more data. Optimal recurrence and optimal token count both follow power laws with consistent exponents across scales.
• Inference: More test-time loops improves quality following a predictable, saturating exponential decay — gains are real but diminishing. This mirrors the inference-time scaling of chain-of-thought.

At 770M parameters, a looped model achieves the downstream quality of a 1.3B fixed-depth Transformer trained on the same data — roughly half the parameters for the same quality.

Applied to Mythos: if trained under these scaling laws, Mythos could be dramatically more parameter-efficient than it appears, with a large fraction of its apparent "capability" coming from loop depth rather than raw parameter count.

The Loop Index Embedding Hypothesis Copy Chapter

A key open question is whether the looped block behaves identically on every iteration, or whether it can learn to do different things at different loop depths.

Without any positional signal across loops, the same weights must handle both early-stage pattern matching and late-stage refinement — a tight constraint. A RoPE-like embedding of the loop index injected alongside the input at each step would allow the same parameters to implement functionally distinct operations across iterations, much like how RoPE allows the same attention heads to behave differently at different sequence positions.

If Mythos uses this technique, each loop is not a repetition — it is a distinct computational phase, all sharing weights but operating in different representational regimes. This would substantially increase the expressiveness of the recurrent block without increasing parameter count.

The Overthinking Problem Copy Chapter

More loops is not always better. Beyond a certain depth, excessive recurrence degrades predictions — the hidden state drifts past the solution and into noise. This is the "overthinking" failure mode.

The original Universal Transformer (Dehghani et al., 2018) addressed this with an Adaptive Computation Time (ACT) halting mechanism: a learned scalar per position that dynamically decides when to stop looping. Positions that are harder to process receive more computation; simple tokens halt early.

Mythos almost certainly has some version of this. The model cannot naively run the maximum number of loops on every input — it needs a learned signal for when the answer has converged. The ACT mechanism also makes the model Turing-complete under certain assumptions, which has theoretical implications for the class of problems it can solve.

Mixture of Experts — Suspected for Large Parameter Counts Copy Chapter

The looped transformer explains the depth of Mythos's reasoning, but not the breadth. Handling wildly different domains — code, math, literature, science, law — with the same weights requires Mixture of Experts (MoE). The suspected design replaces every FFN in the Recurrent Block with a fine-grained MoE layer: each FFN is split into many small experts (1/m the normal size), a router selects the top-mK of them per token via learned affinity scores, and a small number of shared experts are always activated regardless of routing to absorb common cross-domain knowledge — syntax, basic reasoning, general context — that would otherwise be redundantly learned by every routed expert. Routing collapse is prevented through a bias term on the router logits adjusted dynamically during training, keeping load balanced across experts without distorting the loss signal.

As the hidden state h_t evolves across loop iterations, the router may select different expert subsets at each depth, making every loop computationally distinct despite shared weights. MoE provides breadth; looping provides depth. If the activation ratio is ~5%, Mythos could hold hundreds of billions of total parameters while activating only a small fraction per token — the true parameter count, if ever disclosed, would be a storage number, not a compute number.

The Memorization-Reasoning Tradeoff Copy Chapter

Looped models exhibit an interesting dichotomy: looping improves reasoning but can hurt memorization. The recurrent structure is optimized for iterative composition — running a reasoning chain forward — but does not inherently improve the storage of rote facts.

This maps to an observable characteristic of Mythos: it reasons exceptionally well about novel problems it has never seen, but its factual recall can be inconsistent. The architecture is structurally biased toward composition over memorization.

Looping-based regularization (Saunshi et al., 2025) can be used to balance this tradeoff during training — applying stronger looping constraints for reasoning tasks while relaxing them for retrieval tasks.

Parameter Reuse via LoRA Adaptation Copy Chapter

A complementary approach from Relaxed Recursive Transformers (Bae et al., 2024): rather than requiring fully identical weights at every loop, add a small depth-wise LoRA module at each iteration. This preserves the compactness of weight sharing while allowing each loop to adapt its behavior slightly.

The result:

• Each loop shares a large common weight matrix (the recursive base)
• A small rank-r adaptation matrix shifts behavior per iteration depth
• The total parameter overhead is minimal

This bridges the gap between pure weight-tying (maximally parameter-efficient, less expressive) and fully distinct layers (maximally expressive, no parameter savings). Mythos likely sits somewhere on this spectrum.

Continuous Depth-wise Batching Copy Chapter

A downstream consequence of the recursive architecture: Continuous Depth-wise Batching. Because all tokens share the same recurrent block, the model can exit the loop at different depths for different tokens or sequences — processing easy inputs quickly and hard inputs with more iterations, all within the same batch.

Theoretical analysis suggests 2-3x improvements in inference throughput. For a deployed model like Mythos serving many users simultaneously, this would be a substantial efficiency gain.

Summary: What Mythos Probably Is Copy Chapter

References Copy Chapter

Twitter / X Copy Section

• Why Claude Mythos is so good — looped transformer theory (Sigrid Jin): https://x.com/realsigridjin/status/2044620031410266276
• LT implicit reasoning over parametric knowledge unlocks generalization (Yuekun Yao): https://x.com/yuekun_yao/status/2044229171627639004
• Looped transformer cyclic trajectories and input injection (rosinality): https://x.com/rosinality/status/2043953033428541853
• Parcae scaling laws for stable looped language models — thread (Hayden Prairie): https://x.com/hayden_prairie/status/2044453231913537927
• RoPE-like loop index embedding idea to differentiate functions across iterations (davidad): https://x.com/davidad/status/2044453231913537927
• On the Looped Transformers Controversy by ChrisHayduk: https://x.com/ChrisHayduk/status/2045947623572688943
• On the Looped Transformers Controversy Summary by @realsigridjin https://x.com/realsigridjin/status/2046012743778766875

Papers Copy Section

• Fine-grained expert segmentation and shared expert isolation in MoE: https://arxiv.org/abs/2401.06066
• Loop, Think, & Generalize — Implicit Reasoning in Recurrent Depth Transformers: https://arxiv.org/pdf/2604.07822
• Parcae — Scaling Laws for Stable Looped Language Models: https://arxiv.org/abs/2604.12946
• Parcae blog: https://sandyresearch.github.io/parcae/
• Universal Transformers: https://arxiv.org/pdf/1807.03819
• Reasoning with Latent Thoughts — On the Power of Looped Transformers: https://arxiv.org/abs/2502.17416
• Training Large Language Models to Reason in a Continuous Latent Space: https://arxiv.org/abs/2412.06769
• Relaxed Recursive Transformers — Effective Parameter Sharing with Layer-wise LoRA: https://arxiv.org/pdf/2410.20672
• Mixture-of-Depths Attention: https://arxiv.org/abs/2603.15619
• Hyperloop Transformers: https://arxiv.org/abs/2604.21254
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding: https://arxiv.org/abs/2604.21215
• LT2: Linear-Time Looped Transformers: https://arxiv.org/pdf/2605.20670

Citation Copy Chapter

If you use OpenMythos in your research or build on this work, please cite:

@software{gomez2026openmythos,
  author    = {Kye Gomez},
  title     = {OpenMythos: A Theoretical Reconstruction of the Claude Mythos Architecture},
  year      = {2026},
  url       = {https://github.com/kyegomez/OpenMythos},
  note      = {Recurrent-Depth Transformer with MoE, MLA, LTI-stable injection, and ACT halting}
}

License Copy Chapter

MIT License — Copyright (c) 2026 Kye Gomez. See LICENSE for the full text.

File: requirements.txt Copy Chapter

torch>=2.1.0
transformers>=4.40.0
datasets>=2.18.0
pytest>=7.0.0

optional — enables Flash Attention 2 in GQAttention (requires CUDA + build tools)

flash-attn>=2.8.3

File: docs/datasets.md Copy Chapter

Recommended Training Datasets

Primary Pretraining Copy Chapter

FineWeb-Edu Copy Section

• HuggingFace: HuggingFaceFW/fineweb-edu
• Size: 1.3T tokens
• License: Apache 2.0
• Why: Web text filtered for educational quality. Outperforms The Pile, C4, and RefinedWeb on downstream benchmarks. Already deduplicated and cleaned.
• Start with: sample-10BT to validate your pipeline, then sample-100BT or the full corpus for a serious run.

Supplementary Copy Chapter

OpenHermes 2.5 Copy Section

• HuggingFace: teknium/OpenHermes-2.5
• Size: ~1M instruction samples
• License: Apache 2.0
• Why: High-quality instruction-following data. Mix in ~5% by token count on top of FineWeb-Edu to improve instruction following without degrading general capability.

OpenWebMath Copy Section

• HuggingFace: open-web-math/open-web-math
• Size: ~14.7B tokens
• License: ODC-By
• Why: Math-focused web text. Add if you want stronger quantitative and symbolic reasoning. Particularly useful for the 10B+ variants where reasoning depth matters.

Token Budget Recommendations Copy Chapter

The looped architecture is more sample-efficient than a standard transformer — same validation loss is reachable with fewer tokens due to faster convergence. The "recommended (looped)" column reflects this and is based on the Tiny Shakespeare result where OpenMythos reached equivalent loss ~2.5× faster than nanoGPT.

File: docs/open_mythos.md Copy Chapter

`OpenMythos` — Class Reference

Module: open_mythos.main
Base class: torch.nn.Module

Overview Copy Chapter

OpenMythos is the top-level model class implementing the Recurrent-Depth Transformer (RDT) architecture described in the OpenMythos hypothesis. It assembles three functional stages — Prelude, Recurrent Block, and Coda — into a complete autoregressive language model.

Input token IDs  (B, T)
        ↓
   [Embedding]          token index → dim-dimensional vector
        ↓
   [Prelude]            prelude_layers × standard TransformerBlock  (run once)
        ↓
   [Recurrent Block]    one TransformerBlock looped T times
        ↑___________↓   h_{t+1} = A·h_t + B·e + Transformer(h_t, e)
        ↓
   [Coda]               coda_layers × standard TransformerBlock  (run once)
        ↓
   [RMSNorm → LM head]
        ↓
Output logits  (B, T, vocab_size)

Every architectural choice in OpenMythos can be configured through a single MythosConfig dataclass passed at construction.

`MythosConfig` Copy Chapter

@dataclass
class MythosConfig

All hyperparameters for the model are stored in this single frozen-style dataclass. Pass an instance to OpenMythos.__init__.

Core fields Copy Section

Attention fields Copy Section

attn_type selects between two complete attention implementations. All other attention fields are implementation-specific.

GQA vs MLA: GQA reduces KV cache by having fewer KV heads than Q heads (factor of n_heads / n_kv_heads). MLA achieves a much larger reduction by caching a low-rank KV latent (kv_lora_rank) and the RoPE keys (n_heads × qk_rope_head_dim), then reconstructing full K and V on the fly. At production scale MLA yields roughly 10–20× smaller KV cache than standard attention.

MoE FFN fields Copy Section

The Mixture-of-Experts FFN is used exclusively inside the Recurrent Block. Prelude and Coda use a dense SwiGLU FFN.

Approximately n_experts_per_tok / n_experts = 6.25% of routed expert parameters are activated per token, plus all shared expert parameters.

Stability and adaptation fields Copy Section

Constructor Copy Chapter

OpenMythos(cfg: MythosConfig)

Builds all sub-modules, precomputes RoPE frequency buffers, and runs weight initialization.

What happens internally:

1. nn.Embedding(vocab_size, dim) — token embedding table, weight-tied with the LM head.
2. RoPE buffers — freqs_cis (for GQA, dim = dim // n_heads) and freqs_cis_mla (for MLA, dim = qk_rope_head_dim) are precomputed once and registered as non-parameter buffers. The correct buffer is selected at forward time based on cfg.attn_type.
3. prelude — nn.ModuleList of prelude_layers TransformerBlock instances with dense SwiGLU FFN.
4. recurrent — a single RecurrentBlock containing one TransformerBlock (with MoE FFN), LTIInjection, ACTHalting, and LoRAAdapter.
5. coda — nn.ModuleList of coda_layers TransformerBlock instances with dense SwiGLU FFN.
6. RMSNorm(dim) applied before the LM head.
7. nn.Linear(dim, vocab_size, bias=False) LM head with weights tied to the embedding.
8. All nn.Linear and nn.Embedding weights initialized from N(0, 0.02).

Example:

from open_mythos.main import OpenMythos, MythosConfig

cfg = MythosConfig(
    vocab_size=32000,
    dim=2048,
    n_heads=16,
    n_kv_heads=4,
    max_loop_iters=16,
    attn_type="mla",
)
model = OpenMythos(cfg)
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

`forward` Copy Chapter

def forward(
    self,
    input_ids: torch.Tensor,
    n_loops: Optional[int] = None,
    kv_cache: Optional[dict] = None,
) -> torch.Tensor

Single forward pass through the full Prelude → Recurrent Block → Coda pipeline.

Parameters Copy Section

Returns Copy Section

Tensor (B, T, vocab_size) — raw (unnormalized) logits over the vocabulary for each position.

Behavior walkthrough Copy Section

1. Embed:     x = embedding(input_ids)              # (B, T, dim)
2. Select RoPE buffer:
     if attn_type == "mla": use freqs_cis_mla[:T]
     else:                   use freqs_cis[:T]
3. Build causal mask (upper-triangular -inf):
     if T > 1: mask = _causal_mask(T, device)
     else:     mask = None  (single-token decode step)
4. Prelude:
     for i, layer in prelude:
         x = layer(x, freqs_cis, mask, kv_cache, f"prelude_{i}")
5. Freeze encoded input:
     e = x                                          # (B, T, dim)
6. Recurrent loop:
     x = recurrent(x, e, freqs_cis, mask, n_loops, kv_cache)
7. Coda:
     for i, layer in coda:
         x = layer(x, freqs_cis, mask, kv_cache, f"coda_{i}")
8. Project:   logits = lm_head(norm(x))             # (B, T, vocab_size)

Step 5 (freeze e) is the key architectural invariant: the encoded input e is captured after the Prelude and injected at every loop iteration unchanged. This prevents the hidden state from drifting away from the original input signal regardless of loop depth.

Training example Copy Section

import torch
from open_mythos.main import OpenMythos, MythosConfig

model = OpenMythos(MythosConfig()).cuda()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

input_ids = torch.randint(0, 32000, (2, 512)).cuda()
labels    = torch.randint(0, 32000, (2, 512)).cuda()

logits = model(input_ids)                    # (2, 512, 32000)
loss   = torch.nn.functional.cross_entropy(
    logits.view(-1, 32000),
    labels.view(-1),
)
loss.backward()
optimizer.step()

Depth extrapolation at inference Copy Section

A looped transformer trained on N loops can be evaluated on N + k loops and often achieves higher quality on hard multi-hop problems. Pass n_loops at inference time:

# Trained with max_loop_iters=16 — try deeper reasoning at test time
logits_deep = model(input_ids, n_loops=32)

`generate` Copy Chapter

@torch.no_grad()
def generate(
    self,
    input_ids: torch.Tensor,
    max_new_tokens: int = 64,
    n_loops: int = 8,
    temperature: float = 1.0,
    top_k: int = 50,
) -> torch.Tensor

Autoregressive token generation with KV caching. Processes the full prompt on step 0, then decodes one token at a time using the accumulated cache.

Parameters Copy Section

Returns Copy Section

Tensor (B, T + max_new_tokens) — the original prompt concatenated with the generated token indices.

KV caching mechanism Copy Section

On step 0, the full prompt (B, T) is passed and all keys/values for every layer are populated in kv_cache. On steps 1…N only the single most recent token (B, 1) is passed; the attention layers read back all prior K/V from the cache. This makes decode cost proportional to a single token per step rather than the full growing sequence.

Each layer caches under a deterministic string key ("prelude_0", "recurrent_loop_3", "coda_1", etc.), so caches from different layers never collide.

Sampling strategy Copy Section

logits = forward(cur_ids, n_loops, kv_cache)[:, -1, :] / temperature

if top_k > 0:
    threshold = logits.topk(top_k).values[:, -1:]
    logits[logits < threshold] = -inf

probs    = softmax(logits)
next_tok = multinomial(probs, num_samples=1)

Generation example Copy Section

import torch
from open_mythos.main import OpenMythos, MythosConfig

model = OpenMythos(MythosConfig()).eval()

# Tokenized prompt (use your tokenizer of choice)
prompt = torch.tensor([[1, 450, 3118, 310, 278]])   # (1, 5)

output = model.generate(
    prompt,
    max_new_tokens=128,
    n_loops=16,        # deeper reasoning
    temperature=0.8,
    top_k=40,
)
# output.shape == (1, 133)

Internal Components Copy Chapter

The following sub-modules are assembled inside OpenMythos. They are not typically called directly but understanding them clarifies the model's behavior.

`RecurrentBlock` Copy Section

The heart of the architecture. A single TransformerBlock (with MoE FFN) is run in a loop for up to n_loops iterations, with the following per-iteration pipeline:

h_loop = loop_index_embedding(h, t, loop_dim)   # inject sinusoidal loop-index signal
combined = RMSNorm(h_loop + e)                   # add frozen encoded input
trans_out = TransformerBlock(combined, ...)       # attention + MoE FFN
trans_out = trans_out + LoRAAdapter(trans_out, t) # depth-wise LoRA delta
h = LTIInjection(h, e, trans_out)               # stable update: A·h + B·e + trans_out
p = ACTHalting(h)                                # per-position halting probability

The loop exits early for positions whose cumulative halting probability exceeds cfg.act_threshold. If all positions have halted, the loop exits before n_loops. The final output is an ACT-weighted sum of h across iterations.

`LTIInjection` Copy Section

Implements the stable recurrent update rule h_{t+1} = A·h_t + B·e + transformer_out. The diagonal matrix A is parameterized as:

A_continuous = Diag(-exp(log_A))     # always negative diagonal
A_discrete   = exp(Δt · A_continuous) # ZOH discretization, values ∈ (0, 1)

This guarantees spectral radius ρ(A) < 1 by construction, making the looped model unconditionally stable regardless of learning rate or batch noise. See Parcae (Prairie et al., 2026) for the theoretical foundation.

`ACTHalting` Copy Section

A single linear layer mapping (B, T, dim) → (B, T) followed by sigmoid. At each loop step, the scalar halting probability per position is accumulated. When the cumulative sum exceeds cfg.act_threshold, the ACT remainder trick assigns the remaining probability mass as the final weight and the position stops contributing. Implements Graves (2016) ACT.

`LoRAAdapter` Copy Section

A depth-wise low-rank adapter with three components:

• down: shared Linear(dim, rank) — down-projects the transformer output
• B: shared parameter matrix (rank, dim) — up-projects back to full dimension
• scale: Embedding(max_loops, rank) — per-loop element-wise scale

The delta per iteration is (down(x) * scale[t]) @ B. Bridges the expressiveness gap between pure weight-tying and fully distinct per-layer weights. Based on Relaxed Recursive Transformers (Bae et al., 2024).

`TransformerBlock` Copy Section

Pre-norm transformer block with swappable attention and FFN:

• Attention: MLAttention (MLA) or GQAttention (GQA), selected by cfg.attn_type
• FFN: MoEFFN (when use_moe=True, inside RecurrentBlock) or dense Expert (Prelude, Coda)
• Pre-norm via RMSNorm applied to both the attention input and FFN input

`MLAttention` Copy Section

Multi-Latent Attention (DeepSeek-V2, 2024). The cache stores only the compressed KV latent c_kv (rank kv_lora_rank) plus the RoPE-encoded keys. At each decode step, K_nope and V are cheaply reconstructed from c_kv via a shared up-projection, trading a fast linear multiply for dramatically smaller KV memory footprint.

Cache size per layer per token: kv_lora_rank + n_heads × qk_rope_head_dim vs. full GQA cache of n_kv_heads × head_dim × 2.

`GQAttention` Copy Section

Grouped Query Attention (Ainslie et al., 2023). n_kv_heads KV pairs are shared across n_heads // n_kv_heads query heads each, reducing KV cache by that factor while preserving full query expressiveness.

`MoEFFN` Copy Section

Fine-grained Mixture-of-Experts FFN (DeepSeekMoE, Dai et al., 2024):

• Routed experts: n_experts small SwiGLU FFNs. Each token's router selects the top-n_experts_per_tok via softmax over learned logits. A per-expert bias router_bias (non-gradient, updated externally) keeps load balanced.
• Shared experts: n_shared_experts always-active FFNs with width expert_dim × n_experts_per_tok, absorbing cross-domain patterns.

Total activated parameters per token: (n_experts_per_tok / n_experts) of routed capacity + all shared capacity.

`Expert` Copy Section

Single SwiGLU feed-forward unit: down(silu(gate(x)) * up(x)). Used both as individual routed experts inside MoEFFN and as the dense FFN in Prelude/Coda blocks.

`RMSNorm` Copy Section

Root Mean Square Layer Normalization (Zhang & Sennrich, 2019). Normalizes by x / rms(x) with a learned per-channel rescaling weight. No bias, no mean subtraction. Used throughout in place of standard LayerNorm.

Utility functions Copy Chapter

`precompute_rope_freqs(dim, max_len, theta)` Copy Section

Precomputes complex-valued RoPE rotation matrices as a (max_len, dim//2) complex64 tensor. Called once in __init__ and stored as a buffer.

`apply_rope(x, freqs_cis)` Copy Section

Applies precomputed RoPE frequencies to a query or key tensor by treating adjacent feature pairs as complex numbers and multiplying pointwise by the positional phasor.

`loop_index_embedding(h, loop_t, loop_dim, theta)` Copy Section

Injects a sinusoidal loop-index signal into the first loop_dim channels of the hidden state, analogous to RoPE but over recurrence depth rather than sequence position. Allows the shared recurrent block weights to behave differently at different loop iterations.

Key design properties Copy Chapter

Full configuration reference Copy Chapter

The default MythosConfig() targets a mid-scale research model. Below is a minimal configuration for quick experimentation:

from open_mythos.main import OpenMythos, MythosConfig

# Minimal config for fast iteration / unit testing
small_cfg = MythosConfig(
    vocab_size=8192,
    dim=256,
    n_heads=4,
    n_kv_heads=2,
    max_seq_len=512,
    max_loop_iters=4,
    prelude_layers=1,
    coda_layers=1,
    attn_type="gqa",
    n_experts=8,
    n_shared_experts=1,
    n_experts_per_tok=2,
    expert_dim=64,
    lora_rank=4,
)
model = OpenMythos(small_cfg)

And a production-oriented MLA configuration matching the default hyperparameters:

# Default MLA config (matches MythosConfig() defaults)
prod_cfg = MythosConfig(
    vocab_size=32000,
    dim=2048,
    n_heads=16,
    n_kv_heads=4,
    max_seq_len=4096,
    max_loop_iters=16,
    prelude_layers=2,
    coda_layers=2,
    attn_type="mla",           # Multi-Latent Attention
    kv_lora_rank=512,
    q_lora_rank=1536,
    qk_rope_head_dim=64,
    qk_nope_head_dim=128,
    v_head_dim=128,
    n_experts=64,
    n_shared_experts=2,
    n_experts_per_tok=4,
    expert_dim=512,
    act_threshold=0.99,
    rope_theta=500000.0,
    lora_rank=16,
)
model = OpenMythos(prod_cfg)

References Copy Chapter

File: training/requirements.txt Copy Chapter

torch>=2.11.0
datasets>=3.6.0
loguru>=0.7.3
open-mythos