Rohxn08/flame-rust-30m

FLAME Rust 30M (base)

A small (29.9M-parameter) base language model for Rust code and English, trained from scratch with FLAME, a Rust + Candle framework for training small language models.

It is a research/learning model. It is not a chatbot or a coding assistant. It continues text: given the start of some Rust, it writes more Rust-shaped text. It does not follow instructions and usually does not write correct code (see Evaluation).

Model details

Architecture	Decoder-only Transformer: RMSNorm, RoPE, causal self-attention, SwiGLU, tied embeddings
Parameters	29,893,120
Layers / width / heads	8 / 512 / 8
Context length	1024 tokens
Tokenizer	byte-level BPE, 8,192 tokens (rust-en-8k.json)
Weights	FLAME Candle safetensors, fp32 (best/model.safetensors)
Training	2 stages, 7,500 + 2,500 optimizer steps, 131,072 tokens/step, ≈1.31B tokens total
Hardware	one NVIDIA DGX Spark (GB10), BF16 mixed precision with fp32 master weights

Stage 1 data mix: 59% Rust code, 35% English, 6% Rust docs. Stage 2 (continued from stage 1): 50% Rust code, 29.5% English, 20% Rust docs, 0.5% rustc error explanations.

How to use

The weights use FLAME's own format, so they do not load with 🤗 Transformers. Use the FLAME command-line tool:

# 1. Build FLAME (needs Rust: https://rustup.rs)
git clone https://github.com/RTensorLabs/FLAME && cd FLAME
cargo build --release --locked

# 2. Download this model
pip install -U huggingface_hub
hf download Rohxn08/flame-rust-30m --local-dir ../flame-rust-30m

# 3. Generate (run from the model folder, so the tokenizer path resolves)
cd ../flame-rust-30m
../FLAME/target/release/flame generate --cpu --run . --prompt 'fn main() {' --max-tokens 120

Options: --temperature (default 0.8), --top-k (default 40), --seed, --max-tokens. It runs comfortably on a laptop CPU (a few seconds per sample).

Evaluation

Language modelling (held-out validation data)

Data	Loss	Perplexity
Overall	1.699	5.47 (2.45 bits/token)
Rust code	0.967	2.63
rustc error explanations	1.464	4.32
Rust docs	1.981	7.25
English	3.047	21.06

Task evaluation (does the output work?)

20 tasks, 3 samples each (temperature 0.5, 120 tokens), checked with rustc 1.99. Script and tasks: task_eval.py, rust_tasks.json; all samples: eval-report.json.

Task	Metric	Score
Code completion (12 small functions, e.g. add, factorial, reverse_string)	compiles	12/36 = 33%
	compiles and passes tests	1/36 = 3% (1/12 tasks for any of 3 seeds)
Error fixing (8 real rustc errors, e.g. E0382, E0384, E0433)	help: text names the known fix	0/24 = 0%

The model has learned Rust syntax and style (low perplexity on code, a third of its functions compile, and its error output has the right format), but not meaning: compiled functions usually do the wrong thing (max_of → a + b, square → x + 1), and every suggested compiler fix was wrong. The task set is small, so treat these numbers as a rough baseline.

Limitations

• Base model only: no instruction tuning, no chat format, no memory between prompts.
• Rarely produces correct programs; output must never be trusted or run without review.
• Weak English (perplexity 21); tends to drift from code into documentation-like text.
• May reproduce fragments of its training data (open-source code and web text).
• Context is limited to 1024 tokens.

Training data

Source	Contents	Licence
Rust code	.rs files of ~37 large open-source Rust projects and ~100 popular crates (with dependencies)	mostly MIT / Apache-2.0
Rust docs	The Rust Book, Rust by Example, Nomicon, Reference, Async book, Edition guide, API guidelines, RFCs, rustc dev guide, Cargo book, Rust design patterns, Clippy docs, project READMEs	mostly MIT / Apache-2.0, some CC-BY
rustc errors	rustc --explain text for 518 error codes	MIT / Apache-2.0 (Rust project)
English	FineWeb-Edu sample-10BT (≈1.5 GB)	ODC-By 1.0

We thank the Rust project, the authors of the crates and documentation above, and the FineWeb-Edu authors (Hugging Face). Training data is not redistributed here.

Files

File	What it is
best/model.safetensors	model weights (fp32)
config.toml	model shape and training settings (frozen at training time)
rust-en-8k.json	tokenizer
MODEL_CARD.json, VALIDATION.txt	training run summary and validation results
eval-report.json, task_eval.py, rust_tasks.json	task evaluation: results, script, tasks