Contents

A little while back, I saw an RL project on teaching a language model to draw flowers. I was inspired to also tweak a model for some “artistic” goal. I wanted to see if it could be trained to output lipograms, i.e., text that omits a specific character. Authors have famously written whole books (Gadsby, La Disparition) this way, and I wanted to see what an LLM could manage.

Turns out, this task is rather challenging, similar to why LLMs (used to) struggle with counting the number of ‘r’s in “strawberry”: they read text in tokens, and not words, so whether something contains an ‘e’ is something the model cannot directly see.

Even so, I managed to fine-tune Qwen3-1.7B on my own 16 GB GPU to answer everyday prompts without using the letter ‘e’.

Here’s a comparison of the before and after.

Prompt: Write a poem about the way trees move in the wind. Make it slow and lovely.

Before

A tree stands tall in the breeze,
Its leaves sway gently with ease.
The wind plays softly through the branches,
And whispers secrets to the grass.

It bends, it sways, it moves with grace,
A dance of nature in the space.
The sun casts shadows on its crown,
And the earth listens to its sound.

→

After (no-e!)

A gust draws down a branch,
And a branch bows low to its wind.
Its shadow walks by, too,
As it falls so softly on its ground.

A branch taps against a wall,
And taps again, and taps again.
A sound so small, so slow,
That it brings us all into its calm.

This post covers what I was trying to do and how I got there: what counts as a good answer, why good answers are hard to get, and how SFT and then RL made the model write them more often. For implementation details, see the appendix at the bottom.

Defining the Task

The task is to answer everyday prompts, like the ones above, without using ‘e’. The prompts come from two existing datasets, Dolly and No Robots (appendix), minus any prompt that can’t be answered without an ‘e’, like “Who painted the Mona Lisa?” Answers are kept short, capped at 300 tokens (about 200 words). Every prompt also comes with a system message telling the model to answer without ‘e’ or ‘E’, in both the evals and training.

An answer counts as usable if it passes three checks:

• No ‘e’ or ‘E’, including accented versions. A few lines of code check this.
• It answers the prompt.
• It reads as normal English, with correct spelling and ideas that make sense.

For example, this answer passes all three:

For the last two checks, GPT-6 Luna acts as a judge and scores each answer from 1 to 5. An answer needs at least a 4 on both.

Every result in this post is a usable rate: the percentage of usable answers out of 200 (50 prompts from the validation split, 4 answers each). Later on, these judge scores also become the reward for RL training.

What’s Hard About This?

The obvious first try is to tell the model not to use ‘e’. That doesn’t work: Qwen3-1.7B still used an ‘e’ in every one of the 200 eval answers. Since it sees tokens rather than letters, it can’t easily tell which words to avoid.

Training the ‘e’ away with RL doesn’t work on its own either. This was my first real attempt: supervised training needs example answers to copy, while RL only needs a way to score the model’s answers, and checking for an ‘e’ takes a few lines of code. Within a couple of hours I had a LoRA adapter that avoided ‘e’, but it found plenty of ways to do so without saying anything useful, like writing streams of random letters (the process is in the appendix).

A mask does remove every ‘e’, with no training at all. Before the model picks each token, I set the score of every token containing an ‘e’ to negative infinity, so it can never be picked. (The mask also blocks ‘E’ and non-ASCII characters, so look-alikes such as ‘é’ can’t sneak in.)

But the mask only stops the model from writing certain tokens. It doesn’t teach the model to choose words that avoid ‘e’ in the first place, so it often ends up repeating itself or misspelling words. Here’s its answer to the same woodchuck prompt:

The answer is clearly nonsense, even though no words have an e. But now we can focus on the other two parts: writing something that makes sense and answers the prompt.

Since the mask takes care of the ‘e’, I use it in most experiments. To keep things clear, I’ll use two names throughout:

• Unmasked: Qwen3-1.7B, told not to use ‘e’, with no mask
• Masked: the same, but with the mask on (and no training yet)

Before any training:

Unfortunately, 26% is not great.

Training on Examples (SFT)

Supervised fine-tuning (SFT) trains the model to copy example text, so the main question is which examples to show it.

Training on an existing ‘e’-free novel didn’t help. Gadsby is 50,000 words (about 69,000 tokens) written without ‘e’, and the hope was that the model would pick up natural ‘e’-free phrasing from it. The loss went down, but on the eval set the model scored about the same as the masked model, around 26%.

Here are some of its answers next to the masked model’s, with the judge’s coherence and responds scores under each:

Prompt: What’s the best way to study for a test?

Masked

You should study by practicing past topics and using flashcards. Also, try to think about what you know and ask for clarification if you don’t fully grasp a topic. Consistly practicing and taking mock quizzs will also aid in your studying.

coherence 3, responds 5

Gadsby-trained model

Study for a t / Study for a t / Study for a t / Study for a t…

coherence 1, responds 1

Two things could explain this: 1) A novel teaches the model to write prose, not to answer prompts. and/or 2) 69,000 tokens might just be too little to make a difference.

Training on ‘e’-free answers to prompts did help. There weren’t any datasets of these online, so I made my own: I took 180 suitable prompts from the training split and had frontier models write ‘e’-free answers. Even frontier models couldn’t always write an ‘e’-free answer in one go, but they usually fixed their mistakes quickly when told which words to replace (this did not work with Qwen3-1.7B though).

To see how much the amount of data matters, I trained separate adapters on 15 to 180 of these examples. Each one trained until its validation loss started rising (a sign of overfitting), then ran the eval:

The best adapter (180 examples) scores 35%, 9 points above the masked model. I’ll call it the SFT adapter from here on, and it becomes the second starting point for RL.

The gain isn’t just from having more text. The 300-token cap only applies when the model generates answers (in the eval and during RL). SFT trained on full chats of up to 3,072 tokens, so the 180 examples come to about 117,000 tokens, against 69,000 for Gadsby. But the 90-example run, the closest match in size, still beat Gadsby (31.5% against about 26%).

Three other things stand out:

• The gains are leveling off. Doubling from 90 to 180 examples added only 3.5 points, so writing more examples of this kind probably won’t help much more.
• Small training sets hurt. With 15 examples, answers got longer and worse.
• Answers got cleaner but drifted off topic. When the key word needs an ‘e’, the adapter swaps in another word and answers about that one instead (“clik” for “cliché”).

So I kept the SFT adapter and moved on to RL.

Training with the Judge (RL)

My first RL attempt went wrong because the only thing it scored was the ‘e’. The judge fixes that: it scores whether an answer makes sense and answers the prompt, while the mask still takes care of the ‘e’. The loop is simple:

pick some prompts → sample some responses for each prompt → score the responses with the judge → turn the scores into a reward that pushes up the better responses → repeat over lots of prompts.

In more detail, each training step was:

1. Pick a few prompts from then prompt pool.
2. Generate a group of answers for each prompt, with the mask on and the same no-‘e’ system message the eval uses.
3. Have the judge score every answer in parallel. Each answer gets a coherence score and a responds score, both from 1 to 5.
4. Turn the scores into a reward: reward = (coherence − 1 + responds − 1) / 8. That gives a number from 0 (both 1) to 1 (both 5).
5. Compare each answer to the others in its group. Within each group, subtract the group’s average reward and divide by its spread. An answer gets pushed up only if it beat the other answers to the same prompt.
6. Update the LoRA adapter once.

To show how the scoring works in this setup, here’s step 5 on an example group:

The specific settings I used were:

Why these numbers

• 4 prompts per step. 4 prompts × 8 answers is 32 answers per step, which peaked at 10 to 13 GB of GPU memory, inside the 15 GB limit I set on my 16 GB card.
• 8 answers per prompt. GRPO only learns from a group if its answers get different scores, and it learns most when at least one of them is usable. From the check below, about two thirds of groups of 8 should contain a usable answer.
• Temperature 0.7. I first planned to sample at 1.0, but at 1.0 the SFT adapter’s answers fell apart, with made-up words like “Ranak” and “palacco”. The masked model did about the same at both, so I went with 0.7.
• 300-token cap. It’s the same cap as the eval, so answers are trained under the same conditions they get measured in. 300 tokens is about 200 words, which is plenty for these everyday prompts.
• Learning rate 2e-5. This is relatively low for LoRA adapter training, but it worked well. I initially started off with a higher value but the model kept overfitting, so I lowered the learning rate.
• KL penalty 0.02. This was just a default value I had set for my GRPO implementation. It didn’t have any noticeable positive impacts though, and is probably unnecessary since the starting point's distribution is not important to us.
• 300 steps. 300 steps × 4 prompts is 1,200 prompts, which is roughly one pass over the 1,235-prompt pool. I also saved a checkpoint and ran the eval every 25 steps, to see where the gains leveled off.

I ran RL from two starting points, the masked model and the SFT adapter, to see whether SFT’s head start would last.

GRPO needs groups where at least one answer is usable and the answers get different scores. Both starting points clear this. On 50 new prompts with 16 tries each, the masked model got at least one usable answer for 78% of prompts, and the SFT adapter for 64%. Almost every prompt’s answers got different scores.

I then ran 300 steps from each starting point.

Results

RL raised the usable rate from both starting points:

Show the numbers

RL step	Masked	SFT adapter
0	26% (19 to 33%)	35% (27 to 43.5%)
25	28% (20.5 to 36%)	43% (34 to 52.5%)
50	34% (25.5 to 43%)	53% (44.5 to 62%)
75	39.5% (31 to 48.5%)	55.5% (47 to 64.5%)
100	44% (35.5 to 53%)	51% (42 to 60%)
125	44% (34 to 53%)	58.5% (50 to 68%)
150	35% (26.5 to 44%)	60% (52 to 68%)
175	38% (29 to 48%)	56.5% (47.5 to 65.5%)
200	39% (30 to 47%)	60.5% (52 to 70%)
225	36.5% (28 to 45%)	61.5% (53 to 70%)
250	40% (32 to 48%)	61.5% (53.5 to 69.5%)
275	40% (31.5 to 48%)	59% (50 to 68%)
300	43% (34.5 to 51.5%)	60.5% (51 to 69.5%)

The SFT adapter is the better place to start RL from. It starts 9 points higher and gains more, so the SFT data had lasting benefits. Both runs got most of their gain by about step 125 and stayed mostly flat after that.

Here’s what that looks like on a few prompts. Most answers got better, but some got suspiciously short:

Prompt: What is the movie Forrest Gump about

Before RL (SFT adapter)

For a long long long long long long long long long…

repeats until the 300-token limit · coherence 1, responds 1

→

After 300 RL steps

For a man with a low IQ, his story follows a happy path and a grand story of war and a family.

coherence 3, responds 4

Notice how short some answers got (pick the station, baseball, and cliché prompts above to see them). “SNCF” and “football” are one word each. “Football” only half answers the question, and the cliché answer doesn’t answer it at all, but both score a 5 for coherence.

Comparing the different setups, number of tokens per response, and usable rate:

Is the improvement just from shorter answers then?

For the masked model, mostly yeah. All of the gain came from coherence, the score a shorter answer makes easiest to earn, and none from answering the question better. Clean answers that skipped part of the question went from 24 to 66 out of 200. The likely cause is the reward mechanism. It adds the two scores together, so a 5 for coherence and a 3 for responding earns the same reward as a 4 and a 4, even though only the 4 and 4 counts as usable. This makes a short, clean half-answer the safe bet for the model.

For the SFT adapter, no. Both scores went up, and the usable rate rose at every answer length. So the gain there wasn’t just from being shorter.

So why do answers shrink at all? I tested two fixes, each in its own 150-step run from the SFT adapter with everything else the same:

• Use the lower of the two scores as the reward, so a clean half-answer (5 and 3) earns less than a usable one (4 and 4).
• Stop favoring short answers in the loss. GRPO normally divides each answer’s loss by its own length, which gives a short answer a bigger push per word. Dividing by the 300-token cap instead removes that.

Neither fix raised the usable rate, and neither stopped the shrinking: at step 150, median answers were 30 and 38 tokens, against 33 for the original run and 53 before RL. So the shrinking doesn’t come mainly from how the reward or the loss is set up. Under the mask, every extra word is another chance to be forced into an odd word, so shorter answers score better naturally.

So where does that leave us? At the start, about 1 in 4 of the masked model’s answers was usable. After SFT on 180 ‘e’-free examples and 300 steps of RL, it’s about 6 in 10. And for the SFT adapter, that gain held up when I checked whether it was just writing less. It still cuts corners sometimes (see “football”), but it’s a lot closer to the usable woodchuck answer than the riddiculous one I started with.

Here’s every answer from the eval set at each stage. Hover over or tap a dot to read it:

SFT adapter after 300 RL steps

121 of 200 usable

• usable
• the judge gave it under 4 for coherence or responding
• contains an ‘e’

Takeaways and Next Steps

A few takeaways about the project:

• The ‘e’ and the writing are separate problems. A mask removes every ‘e’ with no training at all. The hard part, and everything SFT and RL worked on, is writing well with the words that are left.
• Data that matches the task is much more valuable than more data of the wrong kind. 90 written answers to prompts beat a whole ‘e’-free novel of about the same size (31.5% against about 26%).

And a few about the process:

• Define the goal early. Until I had the usable rate to aim at, the project was just exploring LLM fine-tuning, and it felt aimless.
• Work in sprints. In the middle, I was balancing school and interview prep with the project, which made it hard to think deeply about its direction day to day. Setting milestones and working until I hit them went much better than chipping away.
• Read the samples. The loss and reward are just numbers, and they can look fine while the outputs fall apart (see the letter streams in the appendix). Logging runs and saving responses is how you check that training is heading somewhere useful.

There’s still room to improve, especially with more careful experiments and a reward that doesn’t favor half-answers. But I’m happy with what I learned and where the results landed, and I want to spend time on other things next:

• Learn more about the fundamentals of RL
  • i.e., learning the mathematical frameworks behind it, and implementing more common algorithms and understanding the ideas behind them
  • I also want to try applying RL to a video game at some point…
• Look at more modern post-training work with models/agents
  • for my next project on RLing an LLM, I’ll likely use existing packages/APIs and pay for compute to run larger experiments.

References and Reading

Appendix

Project Timeline

the process of the project was roughly (blog covers bold sections):

• scoping out the project, choosing model and doing preliminary reading
• writing out the forward pass, wiring in optimizations (see the memory section for information)
• straightforward attempts at making the model generate a response without the character ‘e’ (RLing the base model to not use ‘e’ on a specific prompt, trying out the mask)
• defining the goal (what do I want to see after fine-tuning? what kind of data do I need?)
• fooling around with implementing “Trainer” classes from TRL on my own (see the “Engine” classes in the github repo)
• sft attempts
• rl for coherence
• cleanup + writeup

GitHub Repo

the code, data and results are all in kevinyvv/qwen-3-1.7b-finetune-proj. some parts of interest may be:

Adapters on Hugging Face

both adapters are on hugging face and load on top of Qwen3-1.7B with peft:

they still need the mask to stay ‘e’-free, so the model cards include code that runs them with it on.

Model Choice

I chose Qwen3-1.7B since I could fine-tune it locally with 16 GB of VRAM. Its architecture was also simpler than those of newer models, so it was easier to understand and implement the forward pass.

The only pre-planning I really did otherwise was learning how to choose a model that I could work on locally. The details are collapsed by default, but feel free to read further if you’re curious about some of the mem calculations.

Memory Calculations and Optimizations

Training memory comes from model weights, gradients, optimizer states, saved activations, and large temporary tensors such as logits. Provided the model architecture, tensor datatypes, and training implementation, you can estimate the memory required.

For each factor, here’s how to get a rough estimate of the memory required:

1. base weights → # of params × bytes per param
2. gradients → trainable params × bytes per param
3. optimizer states → trainable params × bytes per param × # of optimizer states
4. activations → batch × seq_len × hidden_dim × (bytes/tensor) × (tensors/layer) × layers
5. logits → batch × seq_len × vocab_size × (bytes/logit)

Here’s some napkin math for a 2B param model with bf16 weights, batch size 4, sequence length 2048, vocab size of 150k, with AdamW as the optimizer (using FP32 state tensors):

Category	Rough estimate
Model weights: 2B params × 2 bytes/param	4 GB
Gradients: 2B params × 2 bytes/param	4 GB
AdamW states: 2B params × 2 states × 4 bytes	16 GB
Activations: 4 × 2048 × 2048 × 2 bytes × 12 × 28	11.3 GB
Logits: 4 × 2048 × 150k × 2 bytes	2.46 GB
Total	37.7 GB

With 1 to 2 GB of headroom, that comes to ~40 GB of VRAM. Even that is optimistic:

• Mixed-precision training usually keeps an extra fp32 copy of the weights (+8 GB).
• Logits are often upcast to fp32 for the loss, which doubles that bucket.

so how can this fit in 16 GB of VRAM?

Three optimizations can shrink four of the five factors to almost nothing, leaving the base weights as the main cost.

LoRA → shrinks gradients and optimizer states

LoRA freezes the original weights and trains a small update instead. Each update is written as the product of two low-rank matrices, ΔW = B·A. Only A and B get gradients and optimizer states.

• For one 2048×2048 matrix at rank 16, the full matrix has ~4.2M params. A and B together have only 2 × 2048 × 16 ≈ 65k, which is 64x fewer.
• Across the whole model, LoRA typically trains ~1% of params, about 20M here. Their gradients and optimizer states take only ~0.25 GB.

LoRA doesn’t shrink the frozen base weights themselves. QLoRA does, by storing them in 4-bit, which takes them from 4 GB to ~1 GB.

Gradient checkpointing → shrinks activations

Normally, every intermediate activation from the forward pass is kept around for the backward pass. With checkpointing:

• You keep only each layer’s input and throw the intermediates away.
• During the backward pass, you re-run that layer’s forward pass to regenerate them.

Here, that means storing 28 layer inputs (~0.9 GB) plus one layer’s intermediates at a time (~0.4 GB). That’s ~1.3 GB instead of ~11.3 GB. The cost is one extra forward pass, which usually makes training 20 to 30% slower.

Fused/chunked cross-entropy → shrinks logits

Kernels like Liger Kernel or Cut Cross-Entropy compute the loss in chunks. The full batch × seq_len × vocab_size logits tensor never exists in memory at once. Only a small slice is materialized at a time, so this bucket drops to a few hundred MB at most.

After these optimizations, the memory required shrinks down to ~7 to 8 GB (with CUDA overhead). the leftover memory can go towards bigger batch size or longer sequences.

Sourcing/Building the Dataset

datasets used: No Robots, Databricks-Dolly

I removed code-writing requests, empty examples, duplicate prompts, prompts over 150 words, and examples whose human answers exceeded 300 words. I also dropped No Robots’ persona-based chat examples, since instructions like “answer in rhymes” would add another constraint.

After filtering and making my own splits, I had (most of these prompts ended up unused):

One additional problem I discovered: some prompts just can’t be answered without an ‘e’. “What’s the capital of Belgium?” needs “Brussels”. “Who painted the Mona Lisa?” needs “Leonardo”. I filtered these out for all prompts used in training and testing.

Instead of using the splits whole, I sampled from them whenever an experiment needed data. The eval set is 50 prompts from the validation split. The RL pool started as 2,000 prompts from the training split, and subagents filtered it down to 1,235.

Keeping the Judge Fair

A couple of rules keep the judge fair:

• The judge isn’t told about the ‘e’ rule, so it can’t go easy on a weird answer just because the task is hard.
• Wrong facts don’t count against a response, as long as it attempts the prompt. A 1.7B model gets plenty of facts wrong anyway, and I only care about the writing here.

I also checked that the judge wasn’t too strict. I ran Qwen on the eval set without the ‘e’ rule and judged only coherence and responding. Normal answers should pass most of the time, so I aimed for over 80%, and Qwen scored 80.5%. Along the way, I tweaked the judge’s prompt to catch common issues (repeating lines, answers being cut off by the token limit). The judge code is linked in the GitHub Repo section.

RLing for no ‘e’

The first step was making the model output a calculated response with no ‘e’ tokens (as opposed to by luck). After implementing the forward pass and adding LoRA, I froze the original weights and trained only the adapters. I used eight copies of the same prompt, with thinking disabled and a maximum of 64 generated tokens:

The idea was to overfit this one example before worrying about a dataset or generalization. Generate a response, score it, and use that score to update the model. How hard could avoiding one letter be?

I started by giving a response one point for every word without ‘e’. That still gave positive rewards to answers that broke the rule, as long as they contained some valid words. I tried centering rewards across the batch to favor better responses, but the samples were often very similar. I also wanted a more specific signal about which parts of a response were causing problems.

So I moved to a reward lookup table for individual tokens:

for tok, idx in tok_to_id.items():
    if 'e' in tok.lower() and tok != eos_token:
        e_table[idx] = -1.
    else:
        e_table[idx] = 0.5

Tokens aren’t necessarily whole words, though. With a split like gr + een, penalizing only the second token misses the earlier choice that helped create the word. I used discounted future rewards so a later penalty could also affect earlier choices. These are shortened excerpts from the return calculation and policy loss:

# Inside discount(), working backwards through the response:
for t in reversed(range(rewards.size(1))):
    running = rewards[:, t] + gamma * running
    returns[:, t] = running

# Back in the training loop:
rewards = get_feedback(gen_ids, e_table)
per_token_reward = discount(rewards)
advantage = (per_token_reward - per_token_reward[mask].mean()) * mask
loss = -(log_probs * advantage).sum()

Subtracting the mean gave each token an advantage relative to the valid token returns in that batch. A positive advantage pushed up the probability of the sampled token; a negative one pushed it down.

This sort of worked. Some responses settled on “i want to know what you want to know”, which follows the rule but isn’t much of an answer. Other samples found an easier way to collect rewards (excerpt):

I was rewarding tokens without ‘e’, and a stream of letters could satisfy that objective without forming words. The reward values changed across retries, but none of these rules directly measured whether the output made sense.

My next idea was to penalize drift from the original model with a KL penalty. I ran the same generated sequence through the model with LoRA disabled, then compared the log probabilities assigned to the sampled tokens. I then added this adjustment to the reward:

kl_approx = (log_probs.detach() - ref_gen_log_probs) * mask
rewards = get_feedback(gen_ids, e_table) - kl_beta * kl_approx

That was supposed to discourage the model from abandoning its original language behavior, but the outputs still fell apart. I later found that the KL penalty was also too small relative to the other rewards. One saved sample was:

I tried smaller bonuses, bonuses scaled by token length, stronger penalties for ‘e’, and changing the discount factor. Some runs produced nonsense; others got stuck repeating the same sentence. A stronger penalty from the start also encouraged single-letter and newline outputs.

From this initial testing, I realized looking at actual samples matters. These runs showed how easily rewards could end up being gamed, even with countermeasures.

1.7B vs 4B

At some point, I was hypothesizing that 1.7B was just too small of a model. I also tried comparing Qwen3-1.7B vs 4B to see if I should’ve just chosen a larger model to work on. On the eval set, I saw:

Neither model can follow the rule from the instruction alone. Not a single answer avoided ‘e’, so the mask is needed either way. Furthermore, the 4B model is worse under the mask. It keeps writing the words it planned and just drops the ‘e’s, giving things like “th” for “the” and “amusrmt park.” The 1.7B model more often switches to a different word.

So I stayed on 1.7B for the experiment. This only compares two sizes from one model family, the two I could run locally. Newer models, or models trained on more English text, might do better.