An on-device, GPU-accelerated language model in Rust.
Noodle is a language model implemented from scratch in Rust. It's a decoder-only transformer with a complete pipeline — training, fine-tuning on instruction data, evaluation, and an interactive inference runtime — plus cloud-GPU training jobs on Modal. No PyTorch and no Python model code: the entire stack is Rust, with GPU acceleration.
> mkdir -p corpus
> curl -L https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/TinyStoriesV2-GPT4-train.txt -o corpus/tinystories-train.txtFirst, do a quick training pass with a small testing dataset to verify everything is working:
> head -10000 corpus/tinystories-train.txt > corpus/tinystories-test.txt
> cargo run --release -- train corpus/tinystories-test.txt models/noodle --max-epochs 20Then, train the model with the full dataset:
> cargo run --release -- train corpus/tinystories-train.txt models/noodle --max-epochs 20For faster training, you can use Modal to train on cloud GPUs.
Prerequisites:
> uv pip install modal
> uv run modal setupCreate Modal volume:
> uv run modal volume create noodle-dataTest with small dataset first:
> uv run modal volume put noodle-data corpus/tinystories-test.txt /corpus/
> uv run modal run jobs/modal/train.py --corpus-file /data/corpus/tinystories-test.txt --max-epochs 1Run full training on Modal GPU:
> uv run modal volume put noodle-data corpus/tinystories-train.txt /corpus/
> uv run modal run jobs/modal/train.py --corpus-file /data/corpus/tinystories-train.txt --max-epochs 20Download trained model to local machine:
> mkdir -p models/noodle
> uv run modal volume get noodle-data /models/noodle/model.json ./models/noodle/
> uv run modal volume get noodle-data /models/noodle/model.mpk ./models/noodle/You can inspect the volume contents with:
> uv run modal run jobs/modal/train.py::list_volumeWhen you have a trained model, you can use it for inference:
> cargo run --release chat models/noodle/model.mpk
Using GPU device: DefaultDevice
Loading model: 4 layers, d_model=256
initializing...
creating token embeddings (50281 x 256)...
using rotary position embeddings (RoPE)
creating 4 transformer blocks...
creating final layer norm...
creating output projection...
model ready
~(°◡°)~ Noodle
I am ready to chat! Type your message and press Enter.
> Once upon a time
, there was a little girl named Lily. She loved to design things with her crayons. One day, she wanted to design something new and pretty. So, she added some colors and shapes in it.Lily wanted to create something special for her mom. She put pretty colors on the paper and made a beautiful picture of a beautiful picture on the paper with many colors on it. When she finished drawing, they were very happy with their creative work - not just like Lily's painting!Note that the pre-trained model only performs text generation and does not follow instructions. To make the model follow instructions, you can fine-tune it on instruction data (see below).
You can fine-tune a pre-trained model on instruction data to teach it to follow instructions:
> cargo run --release -- finetune models/noodle/model.mpk corpus/instructions.txt models/noodle-finetuned --max-epochs 5The fine-tuning command takes the following arguments:
model— Path to the pre-trained model.mpkfileinput— Path to an instruction text file (training data)output— Output directory for the fine-tuned model
Optional flags:
--backend— Backend to use for training:gpu(default),cuda, orcpu--max-epochs— Maximum number of fine-tuning epochs (default:5)
The fine-tuned model is saved to the output directory and can be used with the chat command:
> cargo run --release chat models/noodle-finetuned/model.mpkThis project is licensed under the MIT license.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Noodle by you, shall be licensed as MIT, without any additional terms or conditions.