MyKamus logo MyKamus AI Lab
Mini Malay GPT - transformer decoder neural network visualization
🇲🇾 AI Research · Bahasa Melayu · Open Source

Mini Malay GPT
Transformer Decoder from Scratch

A complete PyTorch implementation of a GPT-style transformer decoder trained on Malay language data. Build a full Bahasa Melayu language model — from BPE tokenization to chat fine-tuning — entirely from first principles.

🔥 PyTorch 🧠 Transformer Decoder 📚 Malay Wikipedia 💬 Chat Fine-tuning ✅ Open Source
📦 View on GitLab → 🚀 Quick Start ↓ 🏗 Architecture ↓
16KBPE Vocabulary
4Transformer Layers
4Attention Heads
256Embedding Dim
128Context Length
FP16Mixed Precision

Overview

What is Mini Malay GPT?

MiniGPT for Bahasa is an educational, end-to-end implementation of a GPT-style decoder-only language model built specifically for the Malay language. It covers every layer of the pipeline — corpus cleaning, BPE tokenizer training, model architecture, pretraining on Malay Wikipedia, and supervised fine-tuning for conversational responses.

🔠

BPE Tokenization

SentencePiece BPE tokenizer trained on Malay text, producing a 16,000-token subword vocabulary optimised for Bahasa Melayu.

🧲

GPT Architecture

Causal self-attention with multi-head attention, feed-forward networks (4x expansion), pre-norm layer normalisation, and residual connections.

📚

Wikipedia Pretraining

Pretrained on cleaned Malay Wikipedia with gradient accumulation (effective batch 128), AdamW optimiser, and FP16 mixed precision.

💬

Chat Fine-tuning

Supervised fine-tuning on USER: ... ASSISTANT: ... END formatted chat data with loss masked on user turns.

🖥️

Desktop GUI App

Tkinter-based desktop app (main.py) with step-by-step wizard, real-time training logs, and interactive chat interface.

🎓

Educational Focus

Every component is written from scratch in pure PyTorch — ideal for learning transformer internals without hidden abstractions.

Model Architecture

GPT-style Decoder

The model follows the decoder-only transformer architecture popularised by GPT-2. Input tokens are embedded, passed through N transformer blocks, and projected to vocabulary logits.

📝 Input Tokens
Malay text → BPE token IDs
↓
🗃 Embedding Layer
Token Embed + Positional Embed → B×T×256
↓
×4 blocks
Layer Norm (pre-norm)
↓
🎯 Masked Multi-Head Self-Attention (4 heads)
↓
+ Residual · Layer Norm
↓
⚡ Feed-Forward Network (256 → 1024 → 256)
↓
+ Residual
↓
Layer Norm (final)
↓
📊 Linear Head → Vocabulary (16K)
Cross-entropy loss / softmax sampling

Default Hyperparameters

ParameterValueDescription
block_size128Context length
embedding_dim256Model dimension
n_heads4Attention heads
n_layers4Transformer blocks
vocab_size16,000BPE vocabulary
ff_dim1,024Feed-forward (4x embed)
dropout0.1Regularisation

Quick Start

Get Up and Running

Install dependencies and train your own Malay language model in five steps.

shell — install 
pip install torch sentencepiece tqdm
1

Train the BPE Tokenizer

Train a SentencePiece BPE tokenizer on your Malay text corpus. Output: mswiki_bpe_16k.model

shell
cd tokenizers
python train_tokenizer.py
2

Prepare and Tokenize Data

Tokenize Malay Wikipedia and chat data into PyTorch tensors (.pt files).

shell
cd data
python token_mswiki.py       # Wikipedia data
python prepare_chat_data.py  # Chat data
3

Pretrain on Malay Wikipedia

Train the base GPT model. Checkpoints saved to checkpoints/latest.pt every 10,000 steps.

shell
cd training
python demo.py
4

Fine-tune for Chat

Supervised fine-tuning on conversational data. Loss is masked so only assistant responses are trained.

shell
cd training
python chat_finetune.py
5

Run Inference / Chat

Test the model interactively or use it programmatically for text generation.

shell
cd scripts
python gpt_test.py
python
import torch
import sentencepiece as spm
from scripts.gpt_test import TinyGPT

# Load tokenizer
sp = spm.SentencePieceProcessor()
sp.load("tokenizers/mswiki_bpe_16k.model")

# Load model
model = TinyGPT().to("cuda")
model.load_state_dict(torch.load("outputs/chat_finetuned.pt"))
model.eval()

# Generate a response
prompt = "USER: Apa itu kecerdasan buatan? ASSISTANT:"
ids = sp.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long).to("cuda")
output = model.generate(x, max_new_tokens=80, temperature=0.6, top_k=15)
print(sp.decode(output[0].tolist()))
shell — desktop gui
python main.py  # Opens the step-by-step training wizard

The desktop app provides a full training wizard with visual progress, real-time logs, configurable parameters, and an integrated chat interface.

Training Details

Pretraining & Fine-tuning

📚 Pretraining — demo.py

DatasetMalay Wikipedia
Batch size32Effective: 128 w/ grad accum
Learning rate3e-4AdamW optimizer
PrecisionFP16Gradient scaling
Grad clipping1.0Max norm
CheckpointsEvery 10K stepscheckpoints/latest.pt

💬 Fine-tuning — chat_finetune.py

DatasetChat conversations
FormatUSER: ... ASSISTANT: ... END
Loss maskingAssistant onlyUser turns masked
Learning rate5e-5Lower than pretrain
Batch size16
Base modelcheckpoints/latest.ptPretrained weights

Data Format

Chat Data Structure

Each conversation line follows a simple structured format that marks speaker turns and response boundaries.

text — chat_train.txt
# One conversation per line
USER: Apa itu kecerdasan buatan? ASSISTANT: Kecerdasan buatan ialah teknologi yang membolehkan mesin belajar dan membuat keputusan seperti manusia. END
USER: Terangkan bahasa pengaturcaraan. ASSISTANT: Bahasa pengaturcaraan ialah satu set arahan yang digunakan untuk berkomunikasi dengan komputer. END
USER: Apa itu transformer dalam AI? ASSISTANT: Transformer ialah seni bina model AI berasaskan mekanisme perhatian (attention) yang membolehkan model memproses data secara selari. END

📁 Preparing Your Own Data

  1. Place raw .txt files in the data/ directory
  2. Run python data/prepare_chat_data.py to tokenize chat data
  3. Run python tokenizers/token_mswiki.py for Wikipedia data
  4. Verify with python scripts/check_tokens.py

Troubleshooting

Common Issues & Solutions

Model not giving correct output
Ensure the tokenizer used for inference matches the one used for training. Debug with:

python scripts/check_tokens.py

Use chat_finetune_debug.py to overfit on a single example and verify the training loop works correctly before scaling up.
Import errors when running scripts
Always run scripts from the project root directory, not from inside a subdirectory. Python module resolution depends on the working directory being the project root.

cd /path/to/minigptforbahasa && python scripts/gpt_test.py
CUDA out of memory
Reduce the batch size in the training script, or increase gradient accumulation steps to maintain the same effective batch size. FP16 mixed precision is already enabled — ensure torch.cuda.amp.autocast() and GradScaler are active.
Poor generation quality
Adjust generation hyperparameters:

• Temperature: Lower (0.6–0.7) = more deterministic. Higher (0.9–1.0) = more creative.
• top_k: Lower (10–20) = more focused. Higher (40–50) = more diverse.

Also consider training for more steps, increasing embedding_dim and n_layers, or adding more training data.
How to check which checkpoint step was saved
ckpt = torch.load("checkpoints/latest.pt"); print(f"Step: {ckpt['step']}")

Roadmap

Next Steps & Future Work

📊 Add evaluation metrics — perplexity, BLEU score
🔭 Implement beam search for more coherent generation
🗣️ Multi-turn conversation support
📈 Scale up model size and training data volume
⚡ Model quantization (INT8/INT4) for faster inference
🌐 Web interface for browser-based chat inference

References

Academic References

📄
Attention Is All You Need — Vaswani et al. (2017)
The original transformer paper introducing multi-head self-attention. arxiv.org/abs/1706.03762
 📄
Language Models are Unsupervised Multitask Learners — Radford et al. / GPT-2
OpenAI's GPT-2 paper showing decoder-only transformers can learn diverse NLP tasks without supervision. The direct architectural inspiration for MiniGPT.
 🔠
SentencePiece — Google (Kudo & Richardson, 2018)
Language-independent subword tokenization library used to train the 16K BPE vocabulary on Malay Wikipedia. github.com/google/sentencepiece
 🇲🇾
Wikipedia Bahasa Melayu
Primary pretraining corpus. Cleaned and preprocessed using data/clean_mswiki.py. ms.wikipedia.org

🚀 Explore the Full Source Code

All model code, training scripts, tokenizers, and data preparation tools are freely available on GitLab. Clone, experiment, and build your own Malay LLM!

View on GitLab — minigptforbahasa
📘 Educational Purpose: MiniGPT for Bahasa is a minimal implementation designed for learning. For production applications, consider using established libraries like Hugging Face Transformers or training on significantly larger datasets with scaled-up model sizes. Contributions, issues, and pull requests are welcome on the GitLab repository.