๐๏ธ QU-SSM-130M-MoE: Continuous Quasi-Unitary State Space Model with Sparse Mixture-of-Experts
๐ Official Research Paper (PDF) & Open Verification
Title: "Gated Quasi-Unitary Lie-Algebra Recurrent State Space Models"
Author: Prannessh K.V.A. (@prannesshkva)๐ฅ Download Full Research Paper (PDF) | ๐๏ธ Zenodo DOI: 10.5281/zenodo.22283431 | ๐ฎ Live Interactive Space
-brightgreen){kind=icon}
QU-SSM-130M-MoE is a continuous quasi-unitary state space sequence architecture co-designed with sparse SwiGLU Mixture-of-Experts (MoE). Designed and invented by Prannessh K.V.A. (Sole Architect & Inventor), QU-SSM eliminates monotonic exponential memory dissipation present in classical real-valued SSMs while delivering strictly constant O(1) step memory (0.19 MB) and 3.32x generation speedup over modern baseline Transformers.
๐งฌ Base Model Lineage & Technical Notes
- Core Architecture: Novel Continuous Quasi-Unitary Lie-Algebra Recurrent State Space Operator (SO(2) phase rotations) with 8 SwiGLU Mixture-of-Experts and Top-2 gating.
- Tokenizer Lineage: Standard GPT-2 Byte-Pair Encoding (BPE) vocabulary (50,257 tokens).
- Pre-training & Calibration: Initialized and pre-trained on
roneneldan/TinyStories(~20M+ tokens) to empirically validate non-dissipative recurrent memory and narrative coherence. - Parameter Footprint: 134.89M Total Parameters, 78.27M Active Parameters per token.
- State Space Dynamics: Strictly preserves vector norms (||R(theta)||_2 = 1.000) through dual-trigonometric rotation, eliminating contractive gradient decay across long sequence horizons.
- Official Research Contact: LinkedIn โ Prannesh K. V. A.
๐ What is QU-SSM (QU-SSM-130M-MoE)?
QU-SSM is a state-of-the-art linear-time sequence engine that formulates state transitions as continuous Lie-group rotations over SO(2) โ
U(1). Unlike classical real-decay state space models that suffer from exponential forgetting, QU-SSM preserves state norm (โR(ฮธ)โโ โก 1.00000) while dynamically decoupling memory damping (ฮณ_t).
Key Advantages of QU-SSM-130M-MoE:
- โก 3.32x Faster Inference: Generates at 5.55 tok/s (180.16 ms/tok step latency) compared to 1.67 tok/s for standard dense Transformer baselines.
- ๐พ Constant O(1) Memory: Requires strictly 0.19 MB of state RAM regardless of context length (1,894x smaller than a 360 MB KV-cache at L = 8,192).
- ๐ง 42% Lower Active Compute: Evaluates only 78.27M active parameters per token across 8 SwiGLU experts with Top-2 routing.
- ๐ Lowest Spectral Phase Distortion: Demonstrates 1.9648 rad phase error on continuous sensor and financial telemetry streams.
๐ SOTA Benchmark Leaderboard (~135M Scale)
| Model Architecture | Total Params | Active Params / Token | Generation Speed | Step Latency | RAM at L=8,192 |
|---|---|---|---|---|---|
| Standard Dense Transformer Baseline (135M) | 134.52M | 134.52M (Dense) | 1.67 tok/s | 597.86 ms | 360.00 MB |
| Classical Real-Decay SSM Baseline (130M) | 129.14M | 129.14M (Dense) | 1.98 tok/s | 506.18 ms | 0.19 MB |
| QU-SSM-130M-MoE (Ours) | 134.89M | 78.27M (Top-2 Sparse) | 5.55 tok/s (๐ฅ 3.32x) | 180.16 ms | 0.19 MB (๐ฅ Constant) |
๐ฌ Mathematical Formulation of QU-SSM
The core recurrence equation evolves according to:
h_t = ฮณ_t ยท R(ฮธ_t) ยท h_{t-1} + u_t
Where:
- R(ฮธ_t) โ SO(2) is a 2x2 rotation matrix with strict norm preservation: โR(ฮธ_t)โโ โก 1.00000.
- ฮณ_t = ฯ(W_ฮณ ยท x_t + b_ฮณ) โ (0, 1) is the decoupled memory gate controlling damping independently.
Exact Real Dual-Component Parallel Prefix Scan:
S = cumsum(log ฮณ_t).clamp(min=-12.0, max=0.0)
ฮฆ = cumsum(ฮธ_t)
h_t = exp(S) ยท [ cos(ฮฆ) ยท cumsum(u_real) - sin(ฮฆ) ยท cumsum(u_imag) ]
๐ป How to Install and Run QU-SSM
Run QU-SSM-130M-MoE directly using standard Hugging Face transformers:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Prannesshkva/QU-SSM-130M-MoE"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "Once upon a time in a world of state space models,"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50)
print(tokenizer.decode(output[0]))
๐ Universal Multimodal Support in QU-SSM
The QU-SSM backbone supports four continuous and discrete sequence interfaces:
| Modality | Python Class | Input Data Format |
|---|---|---|
| ๐ Language & Code | QUSSMForCausalLM |
BPE Token IDs (50,257 Vocab) |
| ๐๏ธ Raw 16kHz Audio | QUSSMForAudio |
Continuous 1D Waveform |
| ๐ Financial & Sensors | QUSSMForSensorTelemetry |
Continuous Time-Series Ticks |
| ๐ผ๏ธ 2D Spatial Vision | VisionQUSSM |
16x16 Pixel Patches |
โ Frequently Asked Questions (FAQ)
What makes QU-SSM different from classical real-decay SSMs?
Classical SSMs enforce negative real decay (A โ โโป), creating a dissipative low-pass filter that loses oscillatory and high-frequency phase information (e^(-ฮฑยทt) โ 0). QU-SSM uses dynamic Lie-group rotations over SO(2) with โR(ฮธ)โโ โก 1.00000, preserving state energy indefinitely while dynamically controlling damping via ฮณ_t.
Is QU-SSM-130M-MoE an MoE model?
Yes. It co-designs the quasi-unitary recurrent state space engine with an 8-expert SwiGLU Mixture-of-Experts layer using Top-2 routing, evaluating only 78.27M active parameters per step.
What is the pretraining dataset for QU-SSM-130M-MoE?
This checkpoint is an early-stage foundational prototype pre-trained on ~20 Million tokens of structured narrative reasoning (TinyStories) over 5 epochs, achieving a 70x perplexity reduction (3,358 โ 48.21 PPL).
๐ Intellectual Property & Citation
- Sole Architect & Inventor: Prannessh K.V.A.
- Permanent Research DOI:
10.5281/zenodo.22283431 - License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
- Official Contact: LinkedIn โ Prannesh K. V. A.
@software{prannesshkva_qu_ssm_2026,
author = {Prannesh K.V.A.},
title = {QU-SSM-MoE: Continuous Quasi-Unitary Lie-Group State Space Models with Sparse Mixture-of-Experts},
month = sep,
year = 2026,
publisher = {Zenodo},
doi = {10.5281/zenodo.22283431},
url = {https://doi.org/10.5281/zenodo.22283431}
}
๐ Related QU-SSM Models & Ecosystem Links
- Downloads last month
- 3,263