Training Llama 3.2 with GRPO and LoRA

In reward-based fine-tuning, the verifier defines what the model is encouraged to produce. I used Llama 3.2 3B Instruct on GSM8K to examine this in a setting where the final numerical answer can be checked against a reference. The reward functions score both answer correctness and the required response format.

For each question, the model samples four responses. GRPO compares their rewards within the group, and the optimiser updates LoRA adapters while the base weights remain frozen. The notebook uses Unsloth and TRL for training and vLLM for generation; its saved run covers 500 steps.

What the model sees

I used the GSM8K training split, which has 7,473 maths word problems. Each reference solution ends with #### followed by the answer. I extract that answer for the reward functions:

1
2
3
4
5
6
7
8

from datasets import load_dataset

dataset = load_dataset("openai/gsm8k", "main", split="train")

def extract_hash_answer(text):
    if "####" not in text:
        return None
    return text.split("####")[1].strip()

The model gets the question and a system instruction asking for a particular response format. It does not get the reference answer. In reinforcement learning terms, the model is the policy, which chooses the next token, and the reward is the number our checks assign to its completed response.

For a question about having 24 books and buying 12 more, the requested format looks like this:

<start_working_out>24 + 12 = 36.<end_working_out>
<SOLUTION>36</SOLUTION>

The tags make the answer easier to extract. The checks can then compare 36 with the reference answer. They do not verify the arithmetic written in the working section.

Training the adapter

I kept the base model frozen and trained LoRA (Low-Rank Adaptation) adapters on its attention and feed-forward projections. Full fine-tuning would also need gradients and optimiser state for the base weights. LoRA reduces that memory cost by learning an update through two smaller matrices, A and B:

The rank r controls the smaller dimension of those matrices. I used r=64 and lora_alpha=64, so the scaling factor alpha/r is 1. Increasing the rank would add trainable parameters and memory use.

The model setup is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3.2-3B-Instruct",
    max_seq_length=2048,
    load_in_4bit=False,
    fast_inference=True,
    max_lora_rank=64,
    gpu_memory_utilization=0.9,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=64,
    lora_alpha=64,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

The trainer reports 97,255,424 trainable parameters, or 2.94% of its reported total. I also enabled Unsloth's gradient checkpointing, which saves activation memory by recomputing some work during backpropagation.

The base model loads with load_in_4bit=False. The adamw_8bit setting used later applies to optimiser state; it does not make this a 4-bit QLoRA run.

Four answers per question

I set num_generations=4, so the model samples four responses to each question. Group Relative Policy Optimization, introduced in DeepSeekMath, scores those responses and compares them within the group.

For example, rewards of [0, 1, 2, 3] have a mean of 1.5. Subtracting that mean gives [-1.5, -0.5, 0.5, 1.5]. With group reward scaling, these differences are also divided by the group's standard deviation. The resulting value is the advantage, a measure of how well an answer did relative to the other attempts.

The training objective uses that advantage to encourage higher-scoring responses and discourage lower-scoring ones. The group provides the baseline, so GRPO does not need a separate critic network to estimate it. The optimiser applies the resulting updates to the LoRA weights.

If all four answers earn the same reward, their reward-based advantages are zero. There is no relative preference to learn from that group, even if all four answers are wrong.

What earns a reward

The notebook sums four reward functions with equal weights:

• Format pattern: match_format_exactly gives +3 when its regex matches the expected working and solution sections.
• Marker counts: match_format_approximately gives +0.5 for each marker appearing once, or -1 for a missing or repeated marker. Its maximum is +2.
• Answer comparison: check_answer gives +3 for an exact string match, +1.5 after trimming whitespace, and smaller rewards for nearby numerical answers. A parsed but incorrect answer can receive -1.5.
• Number extraction: check_numbers looks for a number after <SOLUTION>, removes commas, and gives +1.5 for numerical equality or -0.5 for a different parsed number. Failed extraction or conversion receives zero here.

The maximum combined score is 9.5, but correct formatting accounts for five points. A formatted answer of 99 to our books question would still earn 3.0: 3 + 2 - 1.5 - 0.5. This matters when reading the training curve. Reward can improve because the model learns the tags, before it gets better at answering the questions.

There are also cases the verifier handles poorly. Its number regex drops a leading minus sign and can extract an earlier number in a sentence instead of the final answer. I would test negative answers, multiple numbers, missing tags, and repeated tags before trusting it on a broader task. The reward is only as reliable as that extraction and comparison code.

The training settings

The longest measured prompt is 287 tokens. I reserved 288 for the prompt and 1,760 for the completion, keeping the total within 2,048 tokens. Short answers can finish earlier.

I used a learning rate of 5e-6 for 500 steps. The excerpt below shows the main settings and the reward callbacks. The full notebook also configures weight decay, cosine scheduling, warmup, gradient clipping, and checkpoint intervals:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from trl import GRPOConfig, GRPOTrainer

training_args = GRPOConfig(
    learning_rate=5e-6,
    optim="adamw_8bit",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    num_generations=4,
    max_prompt_length=288,
    max_completion_length=1760,
    max_steps=500,
    report_to="none",
    output_dir="outputs",
)

trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[
        match_format_exactly, match_format_approximately,
        check_answer, check_numbers,
    ],
    args=training_args,
    train_dataset=dataset,
)
trainer.train()

One detail in the log: the runtime changes per_device_train_batch_size from 1 to 4 to accommodate four generations. With gradient accumulation set to 4, it reports a total batch size of 16. Those are not 16 distinct questions, since each generation group repeats the same prompt.

The saved outputs mention TRL 0.22.2, but the installation cells and retained logs contain differing package versions. To reproduce the run, I would start in a clean runtime and record the resolved versions and full trainer configuration.

Reading the results

The saved run took approximately 6.98 hours for 500 training steps. I averaged the logged reward over the first and last 50 steps:

These numbers come from the notebook's retained outputs, collected in a small run summary. I did not rerun GPU training for this post.

The reward rises from about -0.56 to 7.49. That shows better performance against the training checks. It does not give us maths accuracy on unseen questions, and the five points available for formatting make that distinction especially relevant here.

Saving and checking the adapter

I save the adapter separately from the base weights and load it for generation. Here text is the prompt prepared with the chat template and system instruction:

1
2
3
4
5
6
7
8
9
10

from vllm import SamplingParams

model.save_lora("grpo_saved_lora")
output = model.fast_generate(
    text,
    sampling_params=SamplingParams(
        temperature=0.8, top_p=0.95, max_tokens=1024,
    ),
    lora_request=model.load_lora("grpo_saved_lora"),
)[0].outputs[0].text

For the square-root example in the notebook, the base model returns approximately 10.049875. With the adapter, it returns 10.050, rounded to three decimal places and wrapped in the requested tags. Both answers are reasonable. The adapter example also includes a system instruction that the baseline does not, so I cannot attribute the format change to training alone.

The next check would be held-out GSM8K questions, with identical prompts and decoding settings for both models. I would report answer accuracy and format compliance separately.

Saving the model in different formats

The save demonstrated above is the LoRA adapter, not a complete copy of Llama. Loading it still requires the matching base model. The notebook also has export cells for merged weights and GGUF files, but they are disabled with if False. I treat those as export options, not files produced by the saved run.

For a regular adapter checkpoint, the notebook includes these two calls. The first saves the adapter weights and configuration; the second saves the tokenizer and its chat template:

1
2

model.save_pretrained("advanced_lora")
tokenizer.save_pretrained("advanced_lora")

If I want one model without a separate adapter, I merge the learned update into the base weights. The 16-bit export keeps floating-point weights and is the starting point I would use for deployment or further conversion:

1
2
3
4
5

model.save_pretrained_merged(
    "advanced_finetune_16bit",
    tokenizer,
    save_method="merged_16bit",
)

There is also a save_method="merged_4bit" example in the notebook. This is a separate export path from GGUF quantization. Unsloth's saving guide cautions against using it as a general default, and the installed version may guard the operation. I would keep the 16-bit merged copy and check the intended runtime's requirements before choosing a lower-precision export.

For llama.cpp-style inference, the notebook uses GGUF, a model file format. GGUF can contain floating-point or quantized weights; the format itself does not mean the model is 4-bit. Quantization stores weights at lower precision to reduce their size, with a possible loss in answer quality.

These are the GGUF options included in the notebook:

I would save each variant to its own directory so the outputs are easy to identify. The loop below makes the quantization choice explicit instead of relying on the default:

1
2
3
4
5
6
7

# Optional exports after training; not executed in the saved run.
for quant in ("f16", "q8_0", "q5_k_m", "q4_k_m"):
    model.save_pretrained_gguf(
        f"advanced_finetune_{quant}",
        tokenizer,
        quantization_method=quant,
    )

The notebook also shows push_to_hub_gguf with a list of methods to export and upload several variants in one call. Saving locally and publishing to Hugging Face are separate choices. The local calls above do not upload the model. The GGUF export documentation covers both paths.

I would compare the exported models on the same questions before picking one. A smaller file does not establish faster inference or equal accuracy, and weight quantization does not remove the runtime's KV cache and other memory costs. I would also keep the same chat template, system instruction, and end-of-sequence handling when moving to another engine.

For this setup, I would work on the verifier before increasing the training budget. There is already a clear reward increase. A stricter answer parser and a held-out comparison would tell me more about what the adapter learned than another 500 steps on the same checks.

Code adapted from Unsloth's Advanced Llama 3.2 GRPO LoRA notebook, licensed under LGPL-3.0. Excerpts are shortened for explanation.