Two-Stage Distillation Pipeline
The NorwAI model is produced through a two-stage distillation process. Each stage has a distinct purpose: Stage 1 compresses general intelligence, Stage 2 injects Norwegian domain expertise.
Stage 1: Knowledge Compression (27B → 7B)
The Qwen 2.5-27B model serves as the teacher in Stage 1. This is a 27-billion parameter model with strong multilingual capabilities and demonstrated proficiency in legal and analytical reasoning tasks. The teacher processes a curated dataset of legal documents and generates Chain-of-Thought (CoT) training pairs in JSONL format.
The teacher's role is pedagogical: it produces structured reasoning traces that expose how a large model analyses legal text — identifying relevant statutes, cross-referencing precedents, evaluating procedural compliance, and reaching conclusions. These reasoning traces become the training signal for the student.
Training pair generation example:
The 7B student model learns from the teacher's reasoning traces via LoRA fine-tuning. The result is Alpha Model v0.1 — a compact model that has internalised the 27B teacher's analytical approach but runs at approximately 4× the inference speed.
At this stage, Alpha v0.1 is a general-purpose legal reasoning model. It can analyse legal texts and produce structured reasoning, but it doesn't yet have specialised knowledge of Norwegian legislation, terminology, or procedural norms. That's what Stage 2 provides.
| Parameter | Value | Rationale |
|---|---|---|
| LoRA Rank | 48 | Balanced for legal reasoning without over-parameterisation |
| LoRA Alpha | 96 | Alpha/rank ratio 2.0 for consistent adapter scaling |
| Target Modules | q, k, v, o, gate, up, down | Full attention + MLP for comprehensive reasoning transfer |
| Learning Rate | 2 × 10-4 | Slightly higher than CO2 model due to larger domain scope |
| Context Length | 4,096 tokens | Sufficient for legal document excerpts + CoT blocks |
| Training Data | ~3,000 pairs | Mix of legal Q&A, reasoning chains, and boundary examples |
Stage 2: Norwegian Legal Fine-Tuning
Stage 2 takes the Alpha v0.1 model and applies a second round of LoRA fine-tuning with a curated Norwegian legal corpus. This corpus includes:
Primary Legislation
Barnelova (Children Act), Barnevernsloven (Child Welfare Act), Menneskerettsloven (Human Rights Act), Forvaltningsloven (Public Administration Act)
Case Law & Rulings
ECtHR rulings (Strand Lobben v. Norway, Abdi Ibrahim v. Norway), Norwegian Supreme Court (Høyesterett) family law precedents
Regulatory Guidance
Bufdir (Children, Youth and Family Affairs) guidelines, Barneombudet (Ombudsman for Children) reports, municipal Barnevern procedural manuals
Language Training
Norwegian Bokmål and Nynorsk legal terminology, professional register for government and academic audiences, bilingual EN↔NO translation pairs
The result is NorwAI Alpha v0.1-NO: a 7B model that combines the 27B teacher's reasoning capabilities with deep knowledge of Norwegian family law, child welfare procedures, and bilingual Norwegian-English proficiency. It reads and responds like a Norwegian legal professional, not a generic chatbot.
