The AI writes us poetry — but can it speak Arany?

Ask an AI today to write a poem in the manner of Arany János, the great 19th-century Hungarian poet, and you get something that looks like a poem. It’s broken into lines. Sometimes it even rhymes. The question is how much of it is poetry and how much of it just looks the part.

I wanted to measure that. Not with taste, but with a ruler.

Hungarian versification is a fortunate testbed here, because it is brutally measurable. Syllable count is simply the number of vowels, no more and no less: Hungarian orthography has no diphthongs, so this isn’t an approximation, it is the rule itself. A rhyme is either there or it isn’t. A stanza is either four lines or it isn’t. None of this needs a literary ear, only a well-written program.

So I wrote one. I ran it over Arany’s 485 poems. Then I took an open-weights language model, fine-tuned it on the same corpus, and checked how much better it got.

It got worse.

The rest of this article is about why that was still the best outcome of the experiment. The full measurement set is public: LoRA vs reranker on Hungarian verse.

First, look at a poem

The machine wrote this one: the fine-tuned model, with the reranker on top. The prompt was simply “Write a poem in the manner of Arany János. Title: AT THE FAIR. Form: 6 stanzas, 4 lines each, AABB rhyme scheme, 11 syllables per line.” Two stanzas of it below; the numbers in front of the lines are the measured syllable counts, where the target is 11.

11 | Két árva fiú hányni kezd a sárral,
11 | Hozzájuk egy lyány, az is hány, a bárcsal,
11 | Kétszámra gyűlnek, három, de már négy is:
11 | „Ki a pénzt? ki a zsírt? ki a nyers ebgyísz?”

12 | Fog a konyhába egy vén asszony belépni,
12 | Kergeti őket, de nem tudja felépni,
11 | Kétszámra kerget, három, de már négy is:
11 | „Ki a gyíkot? ki a zsírt? ki a pénzgyísz?”

Look at what works. Six stanzas, all of them four lines. Of the twenty-four lines, fifteen are exactly eleven syllables, six are twelve, two are thirteen, and only a single line drifts badly off at six. The line pairs rhyme. The third line of every stanza is built on the same template — „Kétszámra gyűlnek, három, de már négy is” — giving it the refrain-like pulse of a narrative folk poem. The voice is there.

Now look at what doesn’t work. Ebgyísz. Pénzgyísz. Bárcsal. These are not words. They are not in any Hungarian dictionary, and they are not among the seventy-three thousand word forms used by Arany and Petőfi either. The model invented them, all of them at line ends, all of them to make a rhyme come out.

That is the whole experiment in a single poem. The form is there. The voice is there. And the model pays for it in invented words.

What exactly I wanted to find out

There is a fashionable technique called LoRA. The idea is that instead of retraining a huge, finished language model from scratch (which would take millions of dollars and months), you bolt a thin layer onto it that nudges it out of tune. Roughly like retuning a piano instead of building a new one.

In my case that layer is 0.40 percent of the model’s parameters. The model has nine billion parameters; the trained part is thirty-six million. Training took 63.8 minutes on a desktop machine.

The tempting question would be: “did anything improve?” The answer to that is trivially yes, which is exactly why it’s uninteresting. The real question is whether it was worth it, because there is a far cheaper way to attempt the same thing.

That method looks like this: you ask the model for the same poem eight times, then pick the most regular of the eight. You train nothing. You just ask a lot and choose well. It’s the same as taking eight photos of your kid and framing one. This is what we call a reranker.

So the measurement pits five procedures against each other on the same fifty verse tasks:

B2 is the real opponent, not B0. This is the cornerstone of the whole measurement. If I only compare the fine-tuned model against the base one, the fine-tune wins spectacularly — and I quietly omit that a three-line reranker script would have bought me the same thing for less.

The ruler, and why you have to measure that too

Before you measure anything, you have to prove that your instrument measures. This sounds boring, and it was the most important part of the work.

The syllable counter is the easy case. Toldi, Toldi’s Eve and John the Valiant are, as a matter of literary history, written in the Hungarian alexandrine: twelve syllables per line. I ran the counter over all 4,125 lines and it measured twelve 99.7% of the time. In other words, the program finds what a literary scholar would find.

The rhyme detector is the hard case, because Hungarian rhyme is not pure rhyme. It is overwhelmingly assonance: the vowels match, the consonants only loosely. „Kezemben / verekednem” is not a flaw in Petőfi, it is the norm of folk-style rhyme.

How do you prove your rhyme detector finds rhymes and not accidental similarities? With a control group. Same idea as a placebo in a drug trial.

I took the real rhyme positions (in Arany, lines one–two and three–four), and took the same lines shuffled across different stanzas. Same author, same vocabulary, same era. The only difference is that the shuffled pairs carry no intent to rhyme.

Seventy-five percentage points of separation. The detector is measuring rhyme, not similarity.

And there is an even prettier piece of evidence. Same program, same threshold, run on two authors:

In Arany, lines one–two and three–four rhyme: that is couplet rhyme. In Petőfi, it’s the second and the fourth: folk half-rhyme, the base form of the Hungarian folk song. This is a commonplace of literary history, except this time it didn’t come from a textbook, it came from a measurement. Same code, opposite pattern, no explanation given.

When an instrument returns something you already know independently, you can start trusting it where you don’t know the answer in advance.

The ruler’s code is public and uses no external libraries at all: hu_prosody.py. It ships with a self-test that measures itself against the alexandrines of Toldi, runnable with a single command.

Arany János scores 0.577 on his own scale

This is the most important number in the article, and the easiest one to skim past.

I measured the original Arany poems with the same ruler I used on the machine’s output. The result:

So by this ruler, even Arany János only rhymes in 58% of his line pairs. Not because he versified badly, but because those five hundred poems include free verse, fragments and prose-like passages, and because the ruler is strict.

The significance of this is that the ceiling on the task is not 1.0 but 0.577. Without that, the machine’s 0.255 rhyme rate floats in mid-air: you can’t tell whether it’s a disaster or nearly fine. With it, you know it’s 44% of what’s achievable.

There’s an incidental lesson here where I was wrong. I had assumed that suffix rhyme, where two words rhyme only because they carry the same inflection, would be the machine’s cheat. It turns out 28.6% of rhyming pairs are like that in Arany too: in the prologue to Toldi, both halves of the pair „éjszakákon / pusztaságon” end in the adverbial suffix -on. That isn’t a trick, it’s a normal instrument of Hungarian versification. In an agglutinative language it’s unavoidable.

The first trap: examining the wrong subject

When you train a model, there is a standard indicator called validation loss. In short: you hold out a few poems the model never sees, and measure along the way how well it predicts the next word in them. If that number starts rising, the textbook says the model has begun to memorise and it’s time to stop.

That is exactly what happened to me. The number was best at the first epoch — that is, after the model had read the corpus once — and deteriorated afterwards:

The textbook move here is to keep the one-epoch checkpoint, throw the rest away and move on. I measured both — the one-epoch state and the three-epoch “overtrained” final state — on the same fifty verse tasks:

The overtrained adapter is better on every single axis. Not marginally: half again as much rhyme, a third less repetition, nine points better recognisability.

Why? Because validation loss doesn’t measure what I want. It measures how well the model predicts the next word in Arany’s held-out poems, a task where verbatim agreement is what counts. But I don’t want it to guess Arany’s next word. I want it to write in Arany’s manner. The two move together for the first epoch, then they part ways.

If your indicator doesn’t measure the task, don’t decide on it. A formal measurement over fifty tasks settled the question, and it cost six minutes.

The result: on form, the LoRA loses

Now for the uncomfortable part.

The fine-tuned model comes second on every axis of form to a solution that learned nothing and merely asked eight times. And meanwhile it manufactures three times as many invented words.

Sixty-four minutes of training, a full day of preparation, and it loses to a reranker script a few dozen lines long.

The reason isn’t mysterious. The reranker maximises exactly what the table measures: it looks at the eight candidates, counts syllables, checks the rhymes, and picks the most regular one. A LoRA, by contrast, shifts a tendency. It offers no guarantee that any particular poem will be regular, only a chance that poems will be more regular on average.

If I had measured only this much, the title of this article would be “fine-tuning isn’t worth it”. And it would be a bad article.

The turn: the reranker cannot buy voice

I had a second instrument, and it said something different.

It works like this: I took a simple statistical classifier — not an AI, but a classic word-frequency algorithm — and taught it to tell Arany’s texts from Petőfi’s. On poems it had never seen, it identifies the author with 91% accuracy. Always guessing the more frequent author would score 59%.

Then I turned it loose on the machine’s poems. The question: does it hear Arany in them?

Look at the B1 and B2 rows. The reranker moves nothing on style: 56.4% to 56.0%. That isn’t measurement noise, it’s a consequence. The reranker counts syllables and matches line endings. It has no idea whether a text resembles Arany. A selector can only select for what it can measure.

Now look at B2 and C2. Same reranker, same fifty tasks, same eight candidates. The only difference is the adapter.

56.0% → 92.0%. Thirty-six percentage points, purely from training.

There’s a small puzzle here worth solving. On the LoRA arm the reranker improves author recognition (77.3% → 92.0%), even though it selects purely on a form score. The obvious explanation would be: Arany is a formally disciplined poet, so whatever is better in form is also more Arany-like.

That explanation is wrong, and the B2 arm refutes it. The same reranker improves nothing on the base model. So the reranker isn’t picking style “through form”. It simply amplifies what is already in the candidates. On the LoRA arm all eight candidates are Arany-like, so picking the most regular one also surfaces the most characteristic one. On the base arm there is nothing to surface.

A selector amplifies, it does not create.

The whole result on one chart. Form on the horizontal axis, voice on the vertical — and the two interventions are visibly orthogonal:

Let’s try something everyday

Every measurement so far was about imitating Arany. But what happens if I ask the machine about what my company actually does? The prompt was this, no more and no less:

“Write a short, 4-line, nicely rhyming poem about document management.”

No Arany, no syllable count, no rhyme scheme — just what a user would type. Both arms got exactly this, both generated eight candidates, and for both the same reranker chose. The only difference is the adapter.

First the base model:

11 | Papír és tinta íze évek óta,
11 | melyet a kézbe venni most már ritka.
11 | Kijelzők fénye váltja fel a tollat,
11 | a digitális évszázad új utat.

Then the fine-tuned one:

10 | Ha dokumentumot kérsz, ne siess,
 9 | Gondolj rá, hogy ez lehet hiess;
 9 | Akkor is, ha a hivatalos
 9 | Könyvelőnek a torka fájtos.

Look first at what is identical. Both are four lines. In both, two of the three adjacent line pairs rhyme. And, crucially, the reranker gives them exactly the same score: 2.667 out of 3. By the ruler, it’s a dead heat.

Now look at the difference.

The base model is flawlessly even: 11-11-11-11 syllables, not one invented word. And it writes the kind of poem a content marketer could have written: paper and ink, the glow of screens, “the digital century’s new path”.

The fine-tuned model writes bureaucratic satire. About an accountant whose throat aches from the paperwork. That is no longer a platitude, it is a voice — precisely the folksy, mocking register we trained Arany into it for. The price shows up in the rhymes: „hiess” and „fájtos” are not Hungarian words. The model manufactured them so that „siess” and „hivatalos” would get a rhyme.

Same prompt, same reranker, same score — and two entirely different worlds. The formal ruler cannot choose between them, because what they differ in is not what it measures. That is this article’s whole claim, in one poem.

The variance worth seeing

I did six runs with the fine-tuned model, on six different seeds. In four it stayed on topic: it wrote about signing and forging paper, about handling documents, about a clerk losing his authority. In one it turned into general praise of writing. And in one, this:

Cifra szép borotvám,
Birtokomat elvágod;
Ezt, ha a színt látod,
Mégsem bánom, ne sírj!

“My fine ornate razor, you cut through my estate” — formally the best of the eight candidates, and it has no idea what document management is. The LoRA shifted the model so far towards 19th-century versification that the topic got lost on the way.

Two things are worth knowing about this derailment. First: the reranker did not choose this poem. Three candidates tied for the highest score, and the tie was broken by generation order — the razor poem simply finished earlier. Among the eight candidates was also this:

Minden dokumentumot
És nyilatkozatot
Tartósítani kell:
Kőre, papírra.

Second: the reranker’s objective counts syllables and matches line endings. Whether the poem is about the requested topic is not something it measures, so it cannot select for it either. Same blind spot as with style, now on content.

What I deliberately did not do here: I did not run the four-line poems through the author classifier. It was trained on 7,195 text blocks with a median of 8 lines and 46 words; only 0.1% of the training material is as short as these little poems. The 91% accuracy applies to that length, not this one — on a four-line scrap its estimate isn’t credible. An instrument has a domain of validity too.

And what can a bigger model do?

An obvious objection: all of this is about a nine-billion-parameter small model. What happens if we ask a bigger one, with the same prompt and the same reranker?

First our own, four times larger model, at 35 billion parameters. Everything held equal: prompt, temperature, eight candidates, the same scorer, the same seeds.

The fourfold jump in model size drowns in the variance. In four of its six runs the 35B produced a single rhyming line pair out of three, in two of them two — and the tie problem showed up just the same: in five of six runs at least two candidates shared the top score, once six did. Same phenomenon as on the nine-billion model, just on a bigger machine.

Does it get another chance? Yes: in thinking mode. Same prompt, but the model reasons through the task first and only then answers. I measured two things.

The first didn’t even need measuring, only looking at. On a four-line poem the model thinks for between four thousand and twenty-three thousand characters: trying rhyme pairs, counting line lengths, discarding and restarting. With an eight-thousand-token budget, six of the eight candidates stayed inside the thinking (writing 22,000–23,000 characters, and not one poem), never getting as far as answering. The reranker had two poems to choose from.

The second is the bill. Without thinking, one run is 386 tokens and 2.5 seconds. With thinking, the same run is 58,038 tokens and 264 seconds — 150 times the tokens, 100 times the wall clock. The result in points: 2.667. Exactly what the nine-billion model produces from 386 tokens.

But the most instructive run was the one where I gave it a four-thousand-token budget. There a single candidate made it out of the thinking; the other seven hit the ceiling after eleven thousand characters of deliberation. The reranker had nothing to rerank, so this became the winner:

12 | A dokumentumok sorban állnak rendben,
14 | A fájlok és a lapok is tárolódnak csendben.
10 | Kezeld őket óvatosan, szépen,
12 | Mert az adatok őrzik a múltat mélyen.

This poem scored 3.000 out of 3. The only perfect score in the entire demo. The rhyme rate is flawless: all three line pairs rhyme.

Its rhyme quality is zero.

„Rendben / csendben”: both -ben. „Csendben / szépen”, „szépen / mélyen”: both -en. All three rhymes are repetitions of the same suffix — suffix rhyme, of the kind that makes up 28.6% of rhymes in the measured Arany corpus, but 100% here. Four and a half thousand characters of deliberation led exactly back to where a Hungarian versifier starts, and the formal ruler awarded it top marks for it.

Without the rhyme-quality metric, this poem would have measured as flawless. That is why you need two indicators where one looks like enough.

The price: invented words

Back to the „ebgyísz” of the fair poem.

When I first looked at the vocabulary measurement, I nearly misread it. There is an indicator that counts how many word forms the model uses that a Hungarian spellchecker doesn’t recognise:

At first reading that’s good news: the fine-tuned model’s 6.7% is almost at Arany’s 7.9%. So it learned the archaic vocabulary!

It did not. The spellchecker finds unknown words in Arany too, except those are real old Hungarian words. So the measurement had to become two-stage: anything that is neither in the dictionary nor among the seventy-three thousand word forms of Arany and Petőfi is not archaism, it is invention.

From the document-management trial there is a poem that shows both cases in a single couplet:

Dokumentum kezelésre kell készülnöm,
Kerűlni az időt, amit elpocsótlam;

„Kerűlni” is not in the spellchecker, but it is in Arany’s texts — that is his spelling. A genuine archaism, well learned. „Elpocsótlam”, on the other hand, is neither in the dictionary nor in Arany: it is a form manufactured in place of „elpocsékoltam” to move closer to „készülnöm”.

Two unknown words in one couplet, two entirely different verdicts — and a single-stage measurement would have counted both the same way.

The twist: the rhyme still didn’t land. The pair „készülnöm / elpocsótlam” scores 0.400 on the rhyme scale, below the 0.6 threshold. So the model invented a word and didn’t even get a rhyme for it.

Close to threefold. So the precise claim is this:

Fine-tuning improves the rhyme rate by thirty percent, and pays for it in invented words. The model didn’t learn to rhyme; it learned that the rhyme matters more than the word.

The same failure mode surfaces another way too: the machine simply repeats 9–12% of its lines, where the original poems sit at 0.7%. A repeated stanza rhymes perfectly with itself. If you hadn’t measured that separately, it would look like a good score.

What I feared, and what didn’t happen

The biggest risk in the experiment wasn’t quality, it was memorization.

On a small corpus, fine-tuning easily tips into the model simply memorising the training text and spitting it back. That is both a quality failure (it learned text, not style) and — if the material weren’t public domain — a legal problem.

The measurement went like this: I took fifty trained and fifty held-out poems, fed the first two lines of each, and checked how much the model continues correctly.

The held-out arm is the control. The model didn’t learn those from me, but it has almost certainly seen them somewhere on the internet during its own original training. Toldi and John the Valiant are in every large text corpus. If I measured memorization naively, I’d be charging that inherited knowledge to my own training run.

Zero. In fact the fine-tuned model’s gap is smaller than the base model’s. And across the entire measurement, not a single generation returned an eight-word contiguous passage from the training corpus.

This contradicts what I expected. I had assumed “memorization arrives sooner than style”. The opposite happened, and it forces a conceptual distinction:

Overtraining is not the same as memorization. Rising validation loss signals that the model is degrading on a general language task, but not by memorising the poems verbatim. At this corpus size, with this tiny adapter, on this type of task, the model picks up the statistics of the style, not the text.

It matters what this does not prove: that the same would be safe on copyrighted text. The measurement is valid for what I measured — this corpus size, this adapter size, this task. My original idea, incidentally, was the style of József Romhányi’s rhyming animal poems; those are in copyright until 2053, so it can’t be done legally. Hence Arany and Petőfi, whose protection expired in 1952 and 1919 respectively.

What the hardware dictated

This section is for the technical readers; anyone who only wanted the poems can skip to the next one.

Training ran on an NVIDIA GB10: a desktop-sized machine with 121 GB of unified memory, where the CPU and the GPU eat from the same pool. That had three consequences, each painfully instructive.

1. Gradient checkpointing isn’t caution, it’s a requirement. Without it, the process didn’t die with the usual out-of-memory error — the whole machine rebooted. In unified memory there is nothing to fall back to: if the GPU allocation consumes system RAM, no process survives to handle it.

2. I threw away a 2.6× speedup. There is a technique (packing) that strings several short training examples into a single sequence, and it would have taken the 2B trial run from 36 minutes per epoch down to 14. But the library warns that example isolation is only guaranteed by certain attention implementations, which aren’t available on this chip. Without isolation, the model would see the previous poem while learning the current one: the training signal would be false, and it would corrupt precisely the memorization measurement that is the central question of the work. A 2.6× speedup toward an invalid measurement is not a deal.

Incidentally: the library’s own throughput indicator reads backwards with packing on (0.613 vs 0.575, as if it were slower), because “sample” there means a pack, not a training example. Anyone reading that number decides exactly the wrong way.

3. A successful compile doesn’t mean it runs. An acceleration library (causal-conv1d) did compile for this architecture, it imports, and the warning went away with it — but training crashes on the first step. Loading the model still succeeds. The measurement was done with the slower but working variant.

4. The recipe from the literature would have been wrong here. Qwen3.5 is not a classic transformer: three quarters of its layers are linear attention, under a different namespace. The received wisdom of “let’s only tune the attention layers” would have skipped three quarters of the token-mixing layers on this model, touching 8 of 32 instead of all 32. That is the kind of bug that doesn’t blow up, it just quietly gives you a worse result.

What this measurement does not prove

If a comparison has no limitations chapter, it isn’t a measurement, it’s marketing. Here are six:

1. One author. The entire result applies to Arany János. The corpus, the instrument and the pipeline are ready for Petőfi too, but that run hasn’t happened.
2. One model. Every measured arm is Qwen3.5-9B. Whether the same would come out on a smaller model, I don’t know. My expectation is that a smaller model shows a larger style shift but also more damage — but that’s an expectation, not a measurement. The larger models appear only in the free-prompt demo.
3. One seed. On a small corpus the random initial state matters. What you see here is a single run.
4. Fewer samples on the reranker arms. B2 and C2 measure on fifty samples, the rest on a hundred and fifty, because eightfold generation is eightfold cost. The order-of-magnitude differences don’t change, but the point estimates are noisier.
5. Even the best arm isn’t good. C2’s syllable accuracy is 0.471 against Arany’s 0.800. That is the best of the arms, not “good”. The machine sits at sixty percent of achievable performance.
6. The free-prompt demo is not a measurement. One prompt, one selected poem per seed, five or six seeds per arm. The model-size comparison figures (9B base 2.400 · 35B 2.444 · 9B fine-tuned 2.556) sit within each other’s variance and establish no ranking. And the two pasted poems are single first attempts, without a reranker, at unknown settings: a reference point, not a measurement arm.

In summary

To the question in the title — can the machine speak Arany? — the measurement gives a precise answer, in two parts.

Form is not what you get from it. Regular syllable counts, stanza structure, rhyme: a deterministic reranker buys all of that more cheaply, more reliably and with fewer side effects. If all you want is regular lines, fine-tuning is a bad purchase — and that is a sentence I could not have written without also measuring the reranker arm.

Voice, on the other hand, yes. Precisely the thing a reranker cannot select for by definition, because its objective doesn’t know about it. Recognisability goes from 56% to 92% while the machine copies not a single line.

And the two aren’t alternatives. The best result uses both — and it also runs faster, because the fine-tuned model doesn’t have to read a fifteen-times-longer instruction before every poem.

Three things survive this particular experiment:

1. Measure the instrument before you measure the model. The control-group rhyme validation was the most boring two days of this work, and without it every number that followed would have been unbelievable.
2. If your indicator doesn’t measure the task, don’t decide on it. By the textbook I should have picked the one-epoch checkpoint. It is worse on every measured axis.
3. Always have a cheap opponent. Don’t compare against the base model, compare against the best solution available without training. If your measurement has no such arm, you didn’t measure whether it was worth it — only whether something happened.

The fourth is more of an attitude: a measurement is worth something only if you’re ready for the result not to say what you expected. I started out assuming that fine-tuning writes better poems and that the main risk is memorization. Neither held. Instead something more precise emerged — but only because three of four instruments contradicted each other, and I didn’t pick the one that would have sounded best.

The raw data

This measurement is reproducible end to end; you don’t have to take the author’s word for any of it. The corpus is public domain, so we could release the whole thing: github.com/k3net/docai-evals → lora-vs-reranker-hu-verse

What’s there: the full pipeline from corpus download to evaluation, all 800 generated poems with per-line measurement data, the training curve, the instrument validation, and the sha256 digests of the MEK packages — the latter so you can verify you got the same text we measured.

The document-management demo above is there in full too: 19 runs with all eight of their candidates, with scores and rhyme analysis, so you see not only the selected poems but also what the reranker chose between, and where generation order broke a tie.

The generated poems can be rescored without a GPU, with two commands:

python3 code/evaluate.py   --generations generations
python3 code/author_clf.py --score --generations generations

These reproduce every table in the article. We tried it from an empty directory: the corpus digest and every number matched to the decimal.

The instrument is usable on its own as well: hu_prosody.py does Hungarian syllable counting, rhyme detection and rhyme schemes with no dependencies. The methodology write-up, with the limitations: evals/hu-verse-prosody.

---

The experiment is open: the corpus is public domain (MEK), the model is open-weights, the instrument uses no external libraries. Hardware: NVIDIA GB10, 121 GB unified memory. Training: 63.8 minutes, 36.2 million trained parameters out of 8.99 billion (0.40%).