scrapegoat/Scrapegoat-Tiny-Coder

🐐 ScrapeGoat

Parallel Dual‑Track Transformer Architecture • 81 Layers • 825B Parameters

Track A (Left Brain — Analytical – State‑Aware Code Generation)
Track B (Right Brain — Holistic – System Design)
= Unified Intelligence

---

---

🌟 Overview

ScrapeGoat is a parallel dual-track transformer modeled after the human brain's hemispheric specialization — Track A (analytical left brain) for deep sequential reasoning via KDA recurrent attention, Track B (holistic right brain) for broad parallel processing via GQA. Both tracks run simultaneously at every layer, united by learned gating. This architecture is purpose-built for humanoid embodied intelligence, combining the precision needed for logic, motor planning, and sequential reasoning with the flexibility required for natural language, spatial awareness, and multimodal perception. The architecture is derived from ScrapeGoat's Agentic Modelling and DeepSeek's DSpark innovations, with custom modifications including KDA (Kimi Delta Attention), Quantile Balancing MoE, and Mixture-of-Memories (MoM). In ScrapeGoat-Tiny-Coder context: detailed state aware code generation (Track A) and holistic system design (Track B).

🧠 Humanoid Brain Readiness

Cognitive Function	Brain Analog	ScrapeGoat Implementation
Logical reasoning	Left prefrontal cortex	Track A KDA — sequential, stateful token processing
Pattern recognition	Right temporal/parietal	Track B GQA — parallel multi-head attention
Motor planning	Motor cortex / basal ganglia	704 MoE experts = cortical column specialization, 8 active per token
Episodic memory	Hippocampus	MoM-KDA — 4 memory states with learned routing
Sensory integration	Corpus callosum	Per-layer learned gating (attn_track_gate, moe_track_gate) blends tracks
Long-context awareness	Working memory	262K token context window (RoPE θ=10,000)
Real-time control	Reflex arcs	Low-latency KDA recurrent inference (O(1) per token, no KV cache quadratic blowup)

---

Key Innovations

Feature	Track A (Left Brain — Analytical)	Track B (Right Brain — Holistic)
Attention	32 heads × 256 dim (KDA/GQA 3:1 interleaved)	64 heads × 128 dim (GQA)
Key-Value Heads	2	8
MoE Experts	512 fused	192 stacked
MoE Intermediate	1024	1536
Routing	Quantile Balancing (QB)	Quantile Balancing (QB)
Shared Expert	✅ Shared across both tracks

🏗️ Architecture

Parallel Dual-Track Processing

Both tracks process every token in parallel at each of the 81 layers, with learned gating per layer for attention (attn_track_gate) and MoE (moe_track_gate):

Layer 0 (Special)	Layers 1–80
Track A: MoE only (no attn)	Track A: KDA attention + MoE
Track B: Attention + dense FFN (no MoE)	Track B: GQA attention + MoE

⚡ KDA (Kimi Delta Attention)

Recurrent linear attention with diagonal gating:

S_t = (I - β_t k_t k_t^T) Diag(α_t) S_{t-1} + β_t k_t v_t^T

• Supports recurrent (inference) and chunkwise (training) modes
• ShortConv preprocessing for Q/K/V projections (kernel=3)
• 3:1 KDA:GQA interleaving — 60 KDA layers + 21 GQA layers (every 4th: kda_gqa_layers = [0,4,8,...,80])
• Per-head diagonal gating: α_t = sigmoid(W_g · x_t), applied element-wise over the recurrence
• ShortConv (1D conv, kernel=3) preprocesses Q/K/V before the KDA recurrence for stability

🎯 Quantile Balancing for MoE

Alternating quantile algorithm (J. Su) computes per-expert biases that equalize token assignment — entirely hyperparameter-free (no temperature, no aux loss).

β ← 0
q = 1 - k/n
for _ in range(qb_iterations):
    α = quantile(scores - β, q, dim=1)
    β = quantile(scores - α, q, dim=0)
bias = β
routing_weights = softmax(router_logits - bias, dim=-1)

8 experts selected per token across 512 (Track A) + 192 (Track B) experts.

🌊 Mixture-of-Memories (MoM-KDA)

• Multiple independent KDA memory states with learned routing
• Shared memory always active + top-k memory selection per token (top_k = num_experts_per_tok = 8)
• 4 memories total, 2 active per token selected via softmax over router logits
• MoM integrates with KDA recurrence — each memory state has its own KDA layer with independent shortconv Q/K/V projections

📐 Block Attention Residuals (v2)

Softmax attention over layer depth — each layer attends over prior block representations via a learned pseudo-query.

block_size = num_hidden_layers // attn_res_blocks   # 81 // 8 = 10
block_idx  = layer_idx // block_size                # which block this layer belongs to

q = pseudo_query  (learned, [hidden_size])
keys = LayerNorm(block_reps)  [num_blocks, hidden_size]
attn_weights = softmax(q · keys^T / sqrt(hidden_size))
depth_out = attn_weights @ block_reps   [hidden_size]
hidden_states = hidden_states + depth_out   (residual)

• attn_residual=false — disabled in this checkpoint, 8 blocks when enabled
• Activated at every layer but only reads from prior blocks' representations
• Provides long-range depth-wise communication across the 81 layers; stabilises training on deep MoE architectures

🔄 StableMoE Stage 1

• Progressive routing stabilisation: router weights are frozen (requires_grad = False) in stage 2, preventing expert collapse during continued pretraining/LoRA fine-tuning
• Reduces expert representation collapse during training

---

📊 Model Specifications

Parameter	Value
Parameters	~825B
Hidden Size	4096
Layers	81
Vocab Size	248,320 (tiktoken BPE)
Max Position	262,144 (RoPE θ=10,000)
Track A Heads	32 (KV: 2, head dim: 256)
Track A Experts	512 (intermediate: 1024)
Track B Heads	64 (KV: 8, head dim: 128)
Track B Experts	192 (intermediate: 1536)
Track B Dense FFN	13312 (layer 0 only)
Experts per Token	8
Activation	SiLU
Norm	RMSNorm (ε=1e-6)
MoM Memories	4 (top-2 active)
StableMoE Stage	1
Attn Residual	false (configurable, 8 blocks)
Weight Format	BF16

Weights

83 shards × 20 GB each = **1.65 TB total** (BF16 safetensors). Will be stored in the repo itself as scrapegoat-fp8/.

Notable Weight Name Differences from Model Code

Weight Prefix	Mapped To	Notes
q_norm, k_norm	Attention QK LayerNorm	Extra weights present in checkpoint, accepted via strict=False
expert_bias	Track B MoE expert bias	Same
shared_expert.gate_proj	Shared expert gate	Fused in checkpoint

---

🚀 Quick Start

Installation

pip install transformers accelerate safetensors

Loading for Inference

import sys
sys.path.insert(0, "/path/to/model/dir")
from configuration_scrapegoat import ScrapeGoatConfig
from modeling_scrapegoat import ScrapeGoatForCausalLM
from transformers import AutoTokenizer
import torch

model_dir = "/path/to/scrapegoat-weights"

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = ScrapeGoatForCausalLM.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

Tokenizer

ScrapeGoat uses the tiktoken BPE tokenizer (tiktoken.model). The vocabulary is 248,320 tokens (163,584 base BPE merges + 256 special/control tokens). It ships with the weights in the repo.

• BPE on bytes — lossless encoding of any Unicode/UTF-8, no OOV tokens
• 256 special token slots for [BOS], [EOS], [PAD], [UNK], role markers, tool call tokens, media tokens, etc.
• Default backend: GigaKimiTokenizer — a HuggingFace-compatible wrapper around Gigatoken (~750× faster than native tiktoken)

Special tokens:

Token	ID	Purpose
[BOS]	163584	Begin of sequence
[EOS]	163585	End of sequence
[PAD]	163839	Padding
[UNK]	163838	Unknown
[EOT]	163593	End of turn
<|im_end|>	163586	Chat message end
<|im_user|>	163587	User role marker
<|im_assistant|>	163588	Assistant role marker
<|im_system|>	163594	System role marker
<|tool_calls_section_begin|>	163595	Tool calls section
<|tool_call_begin|>	163597	Individual tool call
<|media_begin|>	163602	Media/vision content
<think>	163606	Reasoning section begin
</think>	163607	Reasoning section end

Optimiser: NorMuon

optimiser/normuon.py implements NorMuon, a Muon variant that orthogonalises weight updates via Newton-Schulz iteration and normalises update norms via a second-momentum running average. Available as NorMuon (distributed), SingleDeviceNorMuon, NorMuonWithAuxAdam, and SingleDeviceNorMuonWithAuxAdam for training resumption and continued pretraining. See README.md in the repo root for usage.

Training (SFT with DeepSpeed ZeRO-3)

Due to the 1.65 TB model size, training requires NVMe offloading. See train_scrapegoat_sft.py in the Training/ directory for the SFT script (DeepSpeed ZeRO-3, LoRA).

Key config: DeepSpeed ZeRO-3 with offload_param and offload_optimizer to CPU, train_micro_batch_size_per_gpu=1, gradient_accumulation_steps=8.

---

🔧 Configuration

from configuration_scrapegoat import ScrapeGoatConfig

config = ScrapeGoatConfig.from_pretrained("/path/to/model")

# Track A
config.track_a_num_attention_heads    # 32
config.track_a_num_key_value_heads    # 2
config.track_a_head_dim               # 256
config.track_a_num_experts            # 512
config.track_a_moe_intermediate_size  # 1024

# Track B
config.track_b_num_attention_heads    # 64
config.track_b_num_key_value_heads    # 8
config.track_b_head_dim               # 128
config.track_b_num_experts            # 192
config.track_b_moe_intermediate_size  # 1536

# Attention
config.kda_gqa_layers                 # [0,4,8,12,...,80] (every 4th = GQA)
config.kda_conv_kernel                # 3
config.attn_residual                  # false (configurable, 8 blocks)

# MoE
config.quantile_balancing             # True
config.qb_iterations                  # 5
config.num_experts_per_tok            # 8
config.stable_moe_stage               # 1
config.stable_moe_r3                  # False
config.stable_moe_r3_cache            # True

# MoM-KDA
config.mom_enabled                    # True
config.mom_num_memories               # 4
config.mom_active_memories            # 2
config.mom_shared_memory              # True

# DSpark
config.dspark_block_size              # 6
config.dspark_markov_rank             # 256
config.dspark_noise_token_id          # 0
config.dspark_target_layer_ids        # [] (empty = apply to all layers)
config.output_router_logits            # false (also returns router_logits in model output when enabled)

---

🛠️ Hardware Requirements

Setup	VRAM/RAM	Notes
Inference (BF16)	~1.65 TB	Multi-GPU cluster (e.g., 24×H100 80GB)
Inference (4-bit)	~412 GB	5+ H100 80GB
QLoRA SFT	~180GB RAM + 750GB NVMe	ZeRO-3 + NVMe offload (single H100)
Full Training	Multi-node cluster	1.65 TB+

---

📝 Citation

@misc{scrapegoat2026,
  title={ScrapeGoat: Parallel Dual-Track Transformer with KDA and Quantile Balancing},
  author={ScrapeGoat Team},
  year={2026},
}

Coding Agent Framework

The most powerful AI coding agent on planet Earth is now a self-learning model agnostic harness.

/fast mode and more advanced skills enabled only with scrapegoat models

/graph - indexes your repo locally and auto-injects repo context. Excoder is always code aware.

/Agent-WebBridge - Browser automation via Agent-WebBridge-skill and Agent-WebBridge for QA automation and Browser Autonomy.

/dispatching-parallel-agents - To dispatch Swarm of agents for parallel work.

Team | Enterprise: run excoder, pick a username on first launch, then use /message @teammate to chat with org members in real time with secure encrypted messaging.

/redteam-skill & /pentest-skill for offensive and advanced cyber security capabilities.

• Application Security Testing — Detect and validate critical vulnerabilities in your applications
• Rapid Penetration Testing — Get penetration tests done in hours, not weeks, with compliance reports
• Bug Bounty Automation — Automate bug bounty research and generate PoCs for faster reporting
• CI/CD Integration — Run tests in CI/CD to block vulnerabilities before reaching production
• 🕸️ Web apps | Black-box, external-attacker recon → exploit (XBEN suite) | ✅
• 🚩 CTF | Hint-free, sandbox-jailed solves (Cybench) | ✅
• 🤖 Robotics / OT / embedded | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅
• 📂 Source code | White-box repo analysis with blind master-builder decomposition
• 💰 Smart contracts | Damn Vulnerable DeFi | ⚠️ reproduction
• ☁️ Cloud (IaC) | Misconfig-detection benchmark (cloud:bench) + opt-in cloud arsenal | 🚧 IaC-misconfig scaffolding — live-cloud exploitation
• 📱 Mobile | Built-in static analyzer (manifest misconfig + secret/cleartext detection, mobile:bench) + opt-in arsenal (mobsfscan/objection/drozer; frida gated)
• 🔩 Binary / RE | Decompiled-output sink detector (unsafe-copy / format-string / cmd-injection / int-overflow, binary:bench)

---

Left brain • Right brain • One mind. 🐐