Fuuga 1.0.15

Fuuga

Tired of paying tokens? Think you could train a better model? Well, now you can try.

An LLM built from scratch in F# and .NET. Fuuga implements a complete language model pipeline: tokenization, data ingestion, model training, fine-tuning, and text generation -- with no Python dependencies.

Built on TorchSharp for tensor operations and Microsoft.ML.Tokenizers for BPE, Fuuga uses idiomatic F# (discriminated unions, pipelines, immutability) throughout. Works on GPU or CPU.

Choose your path

New here? Each goal is a short sequence of commands driven by one recipe file. Scaffold a recipe, edit a few paths, then run the commands with --recipe. See docs/workflows.md for the full journeys.

Getting Fuuga

Fuuga comes in two forms, reflecting how it's used:

• F# library — NuGet. Reference it from a script (#r "nuget: Fuuga") or a project (<PackageReference Include="Fuuga" />) and call the API directly. Best for power users composing steps the CLI doesn't expose — see the examples/ catalogue (complete-pipeline.fsx, finetuning-lora-qlora.fsx, modelops-compress-and-export.fsx, and more).

  The NuGet packages come in two backend flavors, and the package choice — not a runtime flag — decides CPU vs GPU: Fuuga carries CUDA 12.8 natives (win-x64; needs an NVIDIA GPU), Fuuga.cpu carries CPU natives and runs anywhere. The same split applies to the image module: Fuuga.Image (CUDA) and Fuuga.Image.cpu (CPU). When building from source, the TorchBackend MSBuild property selects the backend: dotnet build -p:TorchBackend=cuda for CUDA, plain dotnet build for the CPU default.

• CLI and server — clone the repo. The fuuga CLI and the fuuga-serve server are not published as standalone tools: their native dependency (libtorch) is over a gigabyte — too large for a NuGet package or a dotnet tool. So clone the repo and run from source:

  • CLI — shown throughout the docs as fuuga <command>; from a clone, run it as dotnet run -- <command>.
  • Server — dotnet run --project Fuuga.Server -- --checkpoint <dir> --tokenizer <dir> [--port <N>] (this is what fuuga-serve refers to).

Improving the server's distribution story (a slimmer, CPU-only or download-on-first-run tool) is an open area for contribution.

Two ways to use Fuuga, then: the CLI for the full pipeline (config-driven via --recipe), or the F# API from scripts. Full command list below; full reference in docs/cli-reference.md.

Features

Core Pipeline:

• BPE Tokenizer -- Train a byte-pair encoding tokenizer on your own corpus with configurable vocabulary size
• Data Ingestion -- Discover and tokenize epub, markdown, Parquet, and plain text files into a binary corpus
• Parquet I/O -- Read and write HuggingFace-compatible Parquet datasets for SFT, DPO, and document data
• Corpus Compression -- Zstd compression/decompression for .fuge corpus files
• GPT-2 Transformer -- Decoder-only causal transformer with rotary position embeddings (RoPE), grouped-query attention (GQA), RMSNorm, and SwiGLU activation
• Multi-Head Latent Attention (MLA) -- DeepSeek-V2 style compressed KV cache with query/KV compression, decoupled RoPE keys, and optional weight absorption for reduced memory during inference
• Vision Encoder -- Vision model support for multimodal inputs
• Vision Bridge -- Q-Former cross-attention bridge that compresses vision patch tokens into learned query vectors for multimodal (image+text) inputs
• Paged Attention -- Paged KV-cache attention for efficient memory usage during long-context generation
• Memory Hierarchy -- Compressed memory with external retrieval for extended context
• Multi-Resolution Attention -- Chunk pooling with global tokens for efficient long-context processing
• FlashAttention Config -- SDPA backend selection and benchmarking for attention kernels
• Auto Config -- Hardware-aware auto-resolution of DU configuration cases (norm, activation, precision, offloading, communication) at startup
• Early Exit -- Adaptive depth inference for faster generation when confidence is high
• Training -- AdamW optimizer with cosine learning rate scheduling, warmup, gradient clipping, mixed precision support, and gradient accumulation
• Multi-Token Prediction (MTP) -- DeepSeek-V3 style auxiliary heads predicting multiple future tokens (configurable Depth); adds a weighted multi-depth loss during training for better sample efficiency and powers MTP-drafted speculative decoding for faster generation. Enable from the CLI with train --mtp-depth <N> [--mtp-loss-weight <f>], or set MtpConfig in the model-config JSON
• INT8 Optimizer Moments -- Optional INT8 quantization of AdamW M/V moment tensors with per-row symmetric quantization, reducing optimizer memory ~4× (--moment-quant int8). Mutually exclusive with SWA/Lookahead/SAM, differential per-group LR, gradient offload, per-param flush, and NVMe/CPU optimizer offload (combining them fails fast). INT8 moments are not persisted across checkpoint resume — model weights resume normally while momentum/variance restart fresh.
• Optimizer Variants -- Stochastic Weight Averaging (SWA) and Lookahead optimizer support with checkpointable optimizer state
• Gradient Checkpointing -- Memory-efficient training via activation recomputation
• GPU Offloading -- Layer-wise CPU/GPU offloading for reduced VRAM usage
• Optimizer Offloading -- Offload optimizer states to CPU memory
• NVMe Paging -- ZeRO-Infinity 3-tier GPU/CPU/NVMe memory management for training models larger than available VRAM
• Per-Tensor Gradient Offloading -- Bulk-copy gradients to CPU after backward pass and restore before optimizer step, freeing GPU VRAM during the optimizer phase (--grad-offload)
• Per-Parameter CUDA Flush -- Aggressive CUDA cache cleanup after optimizer step to reclaim transient VRAM spikes from M/V update temporaries (--flush-each-param)
• VRAM Guard -- In-process background thread that polls GPU memory via nvidia-smi and signals the training loop to warn, skip batches, or abort when usage exceeds a configurable threshold (--vram-guard-gb <float>)
• Memory Strategy Presets -- MemoryStrategyConfigs.none, .constrained (grad-offload + flush), and .full (all three with VRAM guard at 95%) with automatic ConfigWizard recommendations based on model-to-VRAM ratio
• Model Parallelism -- Tensor and pipeline parallelism configuration for 70B+ parameter models, with automatic DataParallel recommendation for multi-GPU setups
• Inference -- Greedy, top-k, top-p (nucleus), and temperature sampling with repetition penalty
• Fill-in-the-Middle -- FIM support with prefix/suffix tokens for code completion
• Checkpoints -- Save, load, resume training from checkpoints with full metadata; safetensors format support
• Memory-Mapped Loading -- mmap-based model loading for fast startup
• Streaming Inference -- Token-by-token generation with configurable stop conditions
• Confidence Signals -- Entropy, repetition rate, hedging detection, calibrated confidence with Platt scaling, and stop reason reporting
• Drift Detection -- Statistical drift monitoring (Kolmogorov-Smirnov, Population Stability Index) over confidence signals with ring-buffered accumulation
• Drift Alerting -- Dual-threshold alerts with adaptive sigma-based thresholds, OpenTelemetry metrics, and retraining triggers
• ONNX Export -- Export to ONNX format with fp16/int8 quantization, validation, and benchmarking
• ONNX Inference -- ONNX Runtime backend for optimized inference (--backend onnx)
• Benchmark Evaluation -- Built-in benchmark runner for MMLU, HellaSwag, ARC-Challenge, WinoGrande, PIQA, TruthfulQA (log-likelihood MCQ scoring), GSM8K (generative, numeric-answer extraction), and HumanEval (generative; candidates execute against the official assert-based test suites via a local Python interpreter, with an approximate fallback and warning when Python is absent), plus cached dataset downloads and checkpoint-attached benchmark results
• FP8 Dequantization -- FP8 format support for quantized weight loading with GPU-accelerated LUT path (256-entry cached lookup table using torch.index_select) auto-selected when CUDA is available
• Validation Pipeline -- Input validation framework with composable validators
• Scaling Heuristics -- Auto-scaling configuration from corpus and hardware stats
• Config Wizard -- Corpus analysis and hardware-aware config generation using Chinchilla scaling laws, activation memory estimates, NTK-aware RoPE, multi-GPU detection (nvidia-smi), and memory strategy recommendations
• CLI -- Subcommands for the full pipeline (tokenize, ingest, train, infer, info, experiments, sft, dpo, rl, merge, transfer, distill, merge-models, fisher, calibrate, plan-quant, distributed, export onnx, export gguf, export llama, export glm, compress, decompress, prune, eval, config, config wizard, rag, wordnet, serve, orchestrate, agent, image)
• Experiment tracking -- fuuga experiments <dir> compares training runs side by side (train/val loss, benchmark scores, approx params, fine-tuning stage, corpus hash, date) from their checkpoints, sorted and in table/CSV/JSON form. Read-only over existing checkpoint.json files — no models loaded.
• Data provenance -- ingest writes a <corpus>.manifest.json sidecar recording the source files behind the corpus (path, size, content hash) and the corpus's SHA256. That hash is the same value stamped into CheckpointMetadata.CorpusHash, so checkpoint → corpus manifest → source files is fully traceable for reproducibility.

Fine-Tuning:

• Supervised Fine-Tuning (SFT) -- LoRA-based fine-tuning on instruction/chat JSONL data with configurable rank, alpha, and target modules
• Prompt Tuning -- Soft-prompt / virtual-token fine-tuning with frozen base weights for lightweight PEFT workflows
• Direct Preference Optimization (DPO) -- Preference learning from chosen/rejected pairs with LoRA
• Reinforcement Learning (RL) -- REINFORCE++ / GRPO fine-tuning with pluggable reward functions
• Reward Functions -- Composable reward functions for RL training (correctness, formatting, safety)
• LoRA Adapter Merging -- Merge trained LoRA adapters back into the base model weights
• QLoRA -- NF4-quantized base weights with LoRA adapters for memory-efficient fine-tuning on consumer GPUs
• Data Validation -- JSONL format validation for SFT and DPO datasets with honesty pattern classification
• Data Augmentation -- Synonym replacement, rule-based paraphrasing, token-level noise injection, and SFT/DPO oversampling for training data diversity

Weight Transfer and Model Merging:

• Weight Transfer -- Transfer weights from donor models with architecture-aware mapping (Phi-3, Phi-4, LLaMA3, Gemma-3, Gemma-4, DeepSeek-V3 dense FFN, Z.AI GLM-4.5/4.6/5 MoE (experimental)) and dimension adaptation for mismatched tensors
• Donor Export -- The reverse direction: export a Fuuga model back to HuggingFace-format safetensors. fuuga export llama produces LlamaForCausalLM (near-lossless for a faithful Llama fine-tune) loadable by Transformers / vLLM / TGI / AWQ-GPTQ; fuuga export glm produces GLM (Glm4Moe) safetensors (experimental, best-effort). Round-trips (export → re-import) are verified bit-exact in the test suite.
• Knowledge Distillation -- Token-level, sequence-level, reverse-KLD, and MLA→GQA attention (distill --attention) distillation from a teacher model
• Attention Distillation -- MLA-to-GQA distillation that trains grouped-query attention layers to reproduce frozen Multi-head Latent Attention teacher outputs (DeepSeek-V3/Kimi K2) with KL-divergence + MSE loss
• N-ary Model Merging -- Merge multiple models with configurable strategies (TIES, DARE, Karcher mean, ModelSoups, ModelStock) and EWC protection
• Fisher Information -- Compute diagonal Fisher information matrices for Elastic Weight Consolidation
• N-ary Data Mixing -- Weighted multi-source mixing with static, curriculum, proxy-based DoReMi (Group DRO domain reweighting from a unigram proxy), and self-paced (difficulty-ramped) strategies

Inference Capabilities:

• Chain-of-Thought -- Thinking mode with ThinkStart/ThinkEnd token handling and dimmed thinking display
• Constrained Decoding -- Grammar-guided JSON structured output generation
• Self-Verification -- Draft/refine verification passes with learned verifier scoring for higher-confidence answers
• Tool Calling -- MCP (Model Context Protocol) client for tool discovery and invocation during generation
• Tool Policy -- Confidence-aware tool routing policy for deciding when external tools should be invoked
• Web Search -- Web search integration for grounded generation with citations
• Image Routing -- Generate or caption images via the Fuuga.Image MCP server, either with the fuuga image command or by configuring fuuga-image in mcp.json
• A2A Protocol -- Agent-to-Agent protocol client for multi-agent communication
• Tree-Structured Speculative Decoding -- Speculative decoding with tree-structured candidates for faster generation