Updated model card

README.md

@@ -5,37 +5,125 @@ tags:
 - pytorch
 - transformer
 - custom-model
+- rope
+- rmsnorm
+- swiglu
+- from-scratch
 language:
 - en
 pipeline_tag: text-generation
+library_name: pytorch
 ---
 
-# VelocityLM - 2B Parameter Language Model
+# VelocityLM 🚀
 
-A custom transformer model with 2B parameters trained for text generation.
+A high-performance, custom transformer language model trained from scratch using modern architectural innovations. VelocityLM combines state-of-the-art techniques including RMSNorm, SwiGLU activation, and Rotary Position Embeddings (RoPE) to deliver efficient and scalable language modeling.
 
-## Model Details
+## 🎯 Quick Links
 
-- **Parameters:** ~2 billion
-- **Architecture:** Custom Transformer with RoPE, RMSNorm, SwiGLU
-- **Context Length:** 2,048 tokens
-- **Tokenizer:** GPT-2 compatible
-- **Training:** Falcon RefinedWeb dataset
+- **🚀 Try the Model**: [Interactive Demo Space](https://huggingface.co/spaces/dixisouls/VelocityLM)
+- **💻 Source Code**: [GitHub Repository](https://github.com/dixisouls/VelocityLM)
 
-## Usage
+## 🏗️ Model Architecture
+
+VelocityLM features a custom transformer architecture optimized for performance and efficiency:
+
+### Model Specifications
+- **Parameters**: ~2B parameters 
+- **Architecture**: Decoder-only transformer with causal attention
+- **Hidden Size**: 2,048
+- **Layers**: 24 transformer layers
+- **Attention Heads**: 32 heads per layer
+- **Vocabulary**: 50,257 tokens (GPT-2 tokenizer compatible)
+- **Context Length**: 2,048 tokens
+- **Intermediate Size**: 8,192 (4x hidden size)
+
+### 🔬 Key Innovations
+
+#### RMSNorm (Root Mean Square Normalization)
+- Replaces LayerNorm for improved training stability and efficiency
+- Better gradient flow compared to traditional normalization
+
+#### SwiGLU Activation Function
+- Gated Linear Unit with Swish activation
+- Superior performance compared to standard ReLU/GELU for language modeling
+- Enhanced expressivity and gradient flow
+
+#### Rotary Position Embeddings (RoPE)
+- Relative position encoding with rotational invariance
+- Better extrapolation capabilities to longer sequences
+- More efficient than learned absolute position embeddings
+
+## 🎯 Training Details
+
+- **Dataset**: [Falcon RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) - high-quality web text
+- **Training Steps**: 5,000+ completed
+- **Optimization**: AdamW with cosine annealing schedule
+- **Hardware**: Trained on 4x NVIDIA A100 (80GB) GPUs
+- **Features**: Mixed precision (FP16), gradient checkpointing, distributed training
+
+## 🚀 Usage
+
+### Basic Text Generation
 
 ```python
+# Note: This model requires custom loading code
+# See the GitHub repository for complete implementation
+
 from transformers import AutoTokenizer
 import torch
 
-# Load tokenizer
+# Load tokenizer (GPT-2 compatible)
 tokenizer = AutoTokenizer.from_pretrained("gpt2")
 
-# Load model (you'll need custom loading code)
-# See the Space implementation for details
+# For complete usage examples and model loading:
+# Visit: https://github.com/dixisouls/VelocityLM
 ```
 
-## Files
+### Interactive Demo
+Try the model immediately in our [Hugging Face Space](https://huggingface.co/spaces/dixisouls/VelocityLM) - no setup required!
+
+## 📊 Performance Features
+
+### Generation Strategies
+- Greedy decoding for deterministic output
+- Top-k and top-p (nucleus) sampling
+- Temperature control for creativity adjustment
+- Repetition penalty to reduce repetitive text
+
+### Memory Optimizations  
+- Gradient checkpointing (40% memory reduction)
+- Efficient causal attention implementation
+- Streaming data processing
+
+## 🔧 Technical Implementation
+
+This model implements several cutting-edge techniques:
+
+- **Distributed Training**: Multi-GPU support with PyTorch DDP
+- **Mixed Precision**: FP16 training with automatic loss scaling  
+- **Advanced Scheduling**: Cosine annealing with warm restarts
+- **Memory Efficiency**: Gradient checkpointing and parameter grouping
+
+
+## 🛠️ Installation & Setup
+
+For detailed installation instructions, training scripts, and advanced usage:
+
+**👉 Visit the [GitHub Repository](https://github.com/dixisouls/VelocityLM)**
+
+The repository includes:
+- Complete training pipeline
+- Inference utilities
+- Configuration management
+- Multi-GPU training support
+- Comprehensive documentation
+
+## 📈 Roadmap
 
-- config.json - Model configuration
-- pytorch_model.bin - Model weights
+Future enhancements planned:
+- Flash Attention 2.0 integration
+- Extended context length support (4K+)
+- Model quantization for efficient deployment
+- Fine-tuning capabilities for downstream tasks
+- ONNX export for production inference