Restore full README with training history, swarm table, and specialist status

README.md

@@ -1,54 +1,174 @@
 ---
 language:
 - en
-license: apache-2.0
+license: mit
 tags:
-- eve
+- pytorch
+- safetensors
+- eve-moe
 - moe
+- deepseek
+- nvidia-h200
 - instruction-tuning
-- deepseek-moe
+- text-generation
+- nano-lm
+- edge-ai
+- rope
+- custom_code
 base_model: anthonym21/Eve-2-MoE-272M
 datasets:
+- HuggingFaceFW/fineweb-edu
+- HuggingFaceFW/finepdfs
 - yahma/alpaca-cleaned
+doi: 10.57967/hf/7731
 ---
 
-# Eve-2-MoE-272M-IT
+<div align="center">
+<img src="eve-2-swarm.jpg" alt="Eve-2 Swarm" width="600">
+<h1>Eve-2-MoE-IT-272M</h1>
+<h3>The Foundation for Nano-Scale Swarm Intelligence</h3>
+</div>
 
-Instruction-tuned variant of [Eve-2-MoE-272M](https://huggingface.co/anthonym21/Eve-2-MoE-272M).
+**Eve-2-MoE-IT-272M** is a 272M parameter instruction-tuned model designed as the foundational base for the **Eve Swarm**—a collection of hyper-specialized, CPU-deployable adapters.
 
-## Training
+Unlike massive generalist LLMs, Eve is built for **deterministic-ish transformations**. She is designed to be "overfitted" into specialists that perform one job perfectly (e.g., SQL generation, Git commits, JSON extraction) with negligible latency and cost.
 
-- **Base model**: Eve-2-MoE-272M (272M params, DeepSeek-style MoE)
-- **Dataset**: [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned) (~52K examples)
-- **Epochs**: 3
-- **Peak LR**: 2e-05
+**Author:** [Anthony Maio](mailto:anthony@making-minds.ai) / [Public Outputs](https://making-minds.ai)
+
+---
+
+## The Eve Swarm (Specialist Ecosystem)
+
+This model serves as the parent for the following **Full Fine-Tuned (FFT)** specialists. All members were trained on an **NVIDIA H200 SXM** to ensure optimal embedding alignment.
+
+> **Note (Feb 2026):** The base model was updated with 10B tokens of continued pretraining (see Training History below). All specialists below were trained on the **v1 base weights** and need to be **retrained** on the updated base to benefit from the improved foundation. The GGUF quantizations also need to be regenerated from the new IT weights.
+
+| Specialist Model | Task                                                                              | Dataset Source                   | Size | Loss                 | Status |
+| ---------------- | --------------------------------------------------------------------------------- | -------------------------------- | ---- | -------------------- | ------ |
+| Eve-NanoFunction | Strict JSON Function Calling – produces valid JSON outputs from natural language. | glaive-function-calling-v2       | 272M | <0.4 (35k samples)   | Needs retrain |
+| Eve-NanoSummary  | Conversation Summarization – condenses dialogues into concise summaries.          | knkarthick/dialogsum             | 272M | <1.0 (12.5k samples) | Needs retrain |
+| Eve-NanoCommit   | Git Diff → Commit Message – writes conventional commits from raw code diffs.      | bigcode/commitpackft             | 272M | <1.0 (20k samples)   | Needs retrain |
+| Eve-NanoExtract  | Text → Structured Data – extracts parameters/entities into strict JSON schemas.   | Salesforce/xlam-function-calling | 272M | <0.4 (20k samples)   | Needs retrain |
+| Eve-NanoSQL      | Natural Language → SQL – converts questions to SQL using table context.           | b-mc2/sql-create-context         | 272M | <0.2 (25k samples)   | Needs retrain |
+| Eve-NanoPrompt   | Prompt Expansion – expands simple ideas into rich image gen prompts.              | Stable-Diffusion-Prompts         | 272M | <1.0 (15k samples)   | Needs retrain |
+| Eve-NanoRouter   | Intent Classification – routes user queries to the correct swarm member.          | bitext/customer-support          | 272M | <0.3 (25k samples)   | Needs retrain |
+| Eve-NanoPII      | PII Redaction – identifies and masks sensitive entities.                          | ai4privacy/pii-masking-200k      | 272M | <0.1 (35k samples)   | Needs retrain |
+
+---
+
+## Training History
+
+### Run 1: Initial Pretraining (v1)
+- **Dataset**: [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) — 10B tokens
+- **Hardware**: NVIDIA H200 SXM (141GB VRAM)
+- **Result**: Trained from scratch to a functional base model
+
+### Run 2: Continued Pretraining (v2 — current base)
+- **Dataset**: [HuggingFaceFW/finepdfs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs) (eng_Latn subset) — 10B tokens
+- **Hardware**: NVIDIA A100 80GB (Google Colab)
+- **Duration**: 14.7 hours
+- **Throughput**: 189,295 tok/s
+- **Val perplexity**: 36.8 → **32.3** (12.2% improvement)
+- **Result**: Stronger base with improved document understanding from academic/scientific PDFs
+
+### Run 3: Instruction Tuning (IT — this model)
+- **Dataset**: [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned) — ~52K instruction-response pairs, 3 epochs
+- **Hardware**: NVIDIA H200 SXM (141GB VRAM)
+- **Method**: Full Fine-Tuning with **response-only loss masking** (prompt tokens excluded from loss)
+- **Duration**: ~1 hour
 - **Batch size**: 128
-- **Context length**: 2048
-- **Loss masking**: Response-only (prompt tokens excluded from loss)
-- **Baseline val ppl**: 32.6
-- **Final val ppl**: 36.2
-- **Training time**: 1.0h
+- **Peak LR**: 2e-5 (cosine decay)
+- **WikiText-2 val ppl**: 32.6 → 36.2 (slight increase expected — model shifted toward instruction-following distribution)
+
+---
+
+## Technical Specifications
+
+### Architecture: Nano-MoE
+
+Eve uses a DeepSeek-style Mixture-of-Experts architecture scaled down to the "Nano" range.
+
+* **Total Parameters:** 272M
+* **Active Parameters:** ~80M (per token)
+* **Experts:** 8 routed + 1 shared
+* **Top-K:** 2
+* **Context Window:** 2048 tokens
+* **Vocab:** 50,304 (GPT-2 compatible)
+
+### Training Config (H200 SXM)
+
+This model was trained using **Full Fine-Tuning (FFT)**. We found that LoRA was insufficient for aligning the embeddings of such a small model; unfreezing all weights yielded significant performance gains. You don't need to use a H200, it's absurdly overkill. I love it.
+
+* **Hardware:** NVIDIA H200 SXM (141GB VRAM)
+* **Method:** Full Fine-Tuning (No PEFT/LoRA)
+* **Precision:** `bfloat16`
+* **Batch Size:** 128 (Global)
+* **Learning Rate:** 2e-5 (Cosine Schedule) — IT run; 5e-5 for specialist FFT
+* **Loss Masking:** Response-only (masked user prompts via loss_mask tensor)
+
+---
+
+## How to Tune Eve 2
+
+If you want to train your own Eve specialist, follow these rules derived from our H200 experiments:
+
+1. **Abandon LoRA:** For a 272M model, LoRA restricts the embedding space too much. You have the VRAM; use **Full Fine-Tuning**.
+2. **Mask User Prompts:** You *must* use a collator that masks the prompt (loss only on response tokens). If the model calculates loss on the instruction, it wastes capacity learning English grammar instead of the task.
+3. **Batch Size Matters:** We saturated the H200 with `batch_size=128`. High batch sizes stabilize the gradients for these volatile small architectures.
+4. **Dataset Quality > Quantity:**
+   * *Bad:* 100k rows of scraped web text.
+   * *Good:* 10k rows of "Input → Ideal Output" pairs.
+   * *Sweet Spot:* 2 Epochs. Do not over-train; these models memorize quickly.
+
+---
 
 ## Usage
 
 ```python
-from transformers import AutoModelForCausalLM
-import tiktoken
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "anthonym21/Eve-2-MoE-IT-272M"
+
+# Load with trust_remote_code=True for custom MoE architecture
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
 
-model = AutoModelForCausalLM.from_pretrained("anthonym21/Eve-2-MoE-IT-272M", trust_remote_code=True)
-enc = tiktoken.get_encoding("gpt2")
+# Standard formatting
+prompt = "User: Explain the concept of Semantic Quantization.\nAssistant:"
 
-prompt = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nExplain what a neural network is.\n\n### Response:\n"
-input_ids = torch.tensor([enc.encode(prompt)])
-output = model.generate(input_ids, max_new_tokens=200)
-print(enc.decode(output[0].tolist()))
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+out = model.generate(**inputs, max_new_tokens=150, do_sample=True, temperature=0.6)
+
+print(tokenizer.decode(out[0], skip_special_tokens=True))
 ```
 
-## Architecture
+---
+
+## GGUF Quantizations
+
+Quantized versions are available at [anthonym21/Eve-2-MoE-IT-272M-GGUF](https://huggingface.co/anthonym21/Eve-2-MoE-IT-272M-GGUF):
+
+| Quantization | Filename | Size |
+|-------------|----------|------|
+| Q8_0 | Eve-2-MoE-IT-272M-Q8_0.gguf | ~318 MB |
+| Q4_K_M | Eve-2-MoE-IT-272M-Q4_K_M.gguf | ~204 MB |
 
-- 12 transformer layers
-- 512 embedding dim, 8 attention heads
-- 8 routed experts + 1 shared expert (top-2 routing)
-- GPT-2 tokenizer (50304 vocab)
-- RoPE positional encoding
-- RMSNorm
+---
+
+## Citation
+
+```bibtex
+@misc{maio2026eve2moeit,
+  author = {Maio, Anthony D.},
+  title = {Eve-2-MoE-IT-272M: A Nano-MoE Foundation for Swarm Intelligence},
+  year = {2026},
+  publisher = {Maio, Anthony D.},
+  url = {https://huggingface.co/anthonym21/Eve-2-MoE-IT-272M}
+}
+```