YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
🚀 MiniGPT from Scratch
A decoder-only GPT implementation built completely from scratch using PyTorch.
This project was created as part of my journey to understand how modern Large Language Models work internally instead of simply using existing libraries.
Features
- ✅ Character-level Tokenizer
- ✅ Token Embeddings
- ✅ Positional Embeddings
- ✅ Multi-Head Self Attention
- ✅ FlashAttention (
scaled_dot_product_attention) - ✅ Feed Forward Network (MLP)
- ✅ Residual Connections
- ✅ Layer Normalization
- ✅ Decoder-only GPT Architecture
- ✅ Autoregressive Text Generation
- ✅ Top-K Sampling
- ✅ Top-P (Nucleus) Sampling
- ✅ Temperature Sampling
Architecture
Input Tokens
│
▼
Token Embedding
│
▼
Position Embedding
│
▼
Transformer Block × 4
│
▼
LayerNorm
│
▼
Linear Head
│
▼
Vocabulary Logits
│
▼
Next Token
Project Structure
MiniGPT-from-Scratch/
├── model.py
├── train.py
├── generate.py
├── utils.py
├── config.py
├── requirements.txt
├── notebook/
├── checkpoints/
├── outputs/
├── images/
└── data/
Dataset
Tiny Shakespeare
https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
Training
python train.py
Generate Text
python generate.py
Sample Output
FANH:
A Prusjortus:
Weath the too,
eyes glorls cobsuess fewn.
Future Work
- KV Cache
- Mixed Precision
- LoRA Fine-tuning
- Beam Search
- HuggingFace Integration
- Distributed Training
- Triton Kernels
Author
Built with ❤️ while learning PyTorch Internals and Transformer Architecture.
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support