Helix, part 1: training a European language model

Why a games studio trains a language model

NETBYTE builds games, apps and websites. Language models increasingly sit inside such products: as characters that talk back, as help that understands a question, as tools that do something. Most open models are English first; Estonian, Irish or Maltese come as an afterthought. We wanted to understand the whole chain ourselves – data, tokenizer, architecture, training – and find out where the money actually goes.

parameters

320 M

official EU languages

24

training tokens

6.5 B

for the rented GPUs

≈ $12

Helix began as a research prototype. Everything below is a decision that shaped it, each backed by a measurement – and the mistakes that cost us the most time.

Four ideas, one survivor

The prototype went after four known weaknesses of today’s text generators. It writes in blocks of 16 tokens instead of one token at a time, it can run a shared core of layers several times, it predicts a compact “plan” for each block, and it mixes cheap local attention with linear attention for long distances.

Block writing worked straight away: a small test model wrote 64 tokens in 13 passes through the network, where a conventional model needs one pass per token – about a fifth of the work. The other three ideas failed a harder test. We built tasks in which the answer sits far outside the local window, so the model has to look back across the whole text.

Only exact attention across the distance solved them. Linear attention, plan and loops stayed at chance level; the optimizer had even turned the linear memory down to a half-life of less than one token. Two exact global heads were enough to match full attention.

The final model therefore keeps eight local heads with a 256-token window in every layer and adds two exact global heads in every second layer; the layers in between keep two gated linear-attention heads. Plan and loops are switched off. They are still in the code and tested, but on real text they did not earn their cost.

Measure, don’t assume

We tested every training trick on a 94-million-parameter model with the same budget per run: 29.5 million tokens, the same seed, the same data order. Two runs of the same configuration differ by ±0.03, so anything smaller is noise.

The Muon optimizer was the only large lever, and it stacks with a warmup–stable–decay learning-rate schedule. Moving the embedding and the output layer back to AdamW, as the Muon authors recommend, gained another 0.12 and made training 5 % faster. Combinations do not simply add up: a deeper, thinner network helped under AdamW and hurt under Muon. That is why every promising combination got its own run.

24 languages, 6.5 billion tokens

Helix reads web text from FineWeb-Edu for English and FineWeb-2 for the other 23 languages – for the larger ones its quality-filtered top tenth. Code, mathematics and a small synthetic chat stream complete the mix. English gets a third. The other languages are weighted by how much text exists, smoothed so that Irish and Maltese are not drowned out, and no source is repeated more than four times – up to that point repetition is about as good as fresh data.

The chat stream is extractive: answers are copied from real paragraphs, and tool results are computed rather than invented. It teaches format, not knowledge. The tokenizer is a byte-level BPE with 64,000 entries; it packs between 3.3 and 4 bytes of text into one token for every language in Latin script.

Three bugs worth telling

• The model that read the answer. In one code path the global heads ignored the causal mask and could see future tokens. Loss fell from 6.85 to 1.71 in 75 steps, and a 16-hour run was worthless. A suspiciously good loss is a bug report; a regression test now guards exactly this.
• Characters are not bytes. Compression was counted in characters instead of UTF-8 bytes. That invented a “weakness” in Chinese and made Greek and Bulgarian look almost twice as bad as they are.
• Documents broken at every blank line. The encoder split documents at every empty line – and empty lines sit in the middle of almost every source file. Up to 35 % of the code and maths text was lost. We fixed the separator and rebuilt the whole corpus on 31 October; it took 2 hours and 12 minutes.

From the desk to a rented machine

The first 1.66 billion tokens ran on a single AMD Radeon RX 7800 XT. Two things made that usable: an attention kernel flag that is off by default – without it attention was 12 times slower – and compiling every layer with torch.compile. At a median of 11,800 tokens per second, the rest would have taken almost five more days.

So we wrote an autopilot. It rents GPUs on Vast.ai, uploads code, corpus and checkpoint, trains, brings a checkpoint home every three hours and stops before the credit runs out. On 2 November it went through three machines whose hosts never started them, then settled on two RTX 4070 Ti Super in Germany for $0.363 an hour. The 15.4 GiB upload took 40 minutes.

41,400 tokens per second is 3.5 times the desk. The rented part – 4.8 billion tokens in about 33 hours – costs roughly $12.

Where Helix stands today

As of 3 November 2026, Helix has read 5.35 of its 6.5 billion tokens. Since 5.2 billion it is in the cooldown: the learning rate falls and the mix shifts towards maths, code and chat. On held-out text from all streams it is at 3.12 nats per token.

What that means in practice shows best in an unedited sample from a checkpoint at about 4.9 billion tokens. The beginning is ours, the rest is Helix:

Grammar, word choice and topic hold; the facts do not. That is roughly what to expect from 320 million parameters and 6.5 billion tokens: a fluent model that invents things.

What comes next

Pretraining ends in a few hours. After that come a long-context stage with books from Project Gutenberg and EU legal texts from EUR-Lex, fine-tuning for conversation and tool use, and a small draft model that speeds up generation.

Back to all posts