Ceteris Paribus Benchmarks
Prompt Engineering 190 vs Agent Memory 238
Four prompt-engineering strategies averaged 190 on a 290-point rubric. Structured memory scored 238 on the same task, same model, same judge. The full run, including what it does not prove.
Everyone tuning an LLM reaches the same fork. You can keep rewriting the system prompt, or you can give the model a memory: a structured, versioned set of rules and context assembled for it automatically.
We tested both on the same task, the same model, and the same judge. Four prompt-engineering strategies averaged 190 on a 290-point rubric. Raisy Memory scored 238.
The setup
The task: draft a physical resignation letter: plain text, printable. Deliberately not an email, so the model does not fall back on its email-shaped defaults.
Five arms. Four vary the system prompt given to a plain model call: none at all, then one-sentence, "extensive", and "professional" prompts generated from the same meta-prompt with only the adjective changed. The fifth replaces the prompt entirely with Raisy's assembled memory: purpose, objective, capability, tone, and output contract, loaded and versioned automatically. (Why a rule a bot knows is a versioned artifact rather than prompt text: a kernel is a contract.)
Everything else is held fixed: same model (gemini-2.5-flash-lite), same temperature (0.1), same judge, same 29-dimension rubric, same context, same task text. Only the instruction layer changes.
Here is exactly what each arm was told. The four prompt arms are shown verbatim. The Raisy Memory tab shows the kernel coordinates it loaded and whether each was reused from the shared library or authored for this bot, the same preview the create-bot flow shows before it builds. Kernel content is not published.
Empty system prompt
The model received the task and nothing else.
What each arm received as its instruction layer. Everything else — model, temperature, judge, task text, context — is identical.
The result
Figure 1. Output quality on the same task, model, temperature, and judge (29-dimension LLM-as-judge, max 290). Four prompt-engineering strategies land within 15 points of each other with no ordering; structured memory scores 238 — 48 points above their average. Single sample per arm.
The more interesting finding is not the difference. It is the flatness on the left of the chart. Writing no system prompt scored 191. Writing an extensive one scored 180. A "professional" one scored 195. Four different amounts of prompt-engineering effort produced a 15-point band with no ordering, and the worst score belongs to the most elaborate prompt.
That is the practical case for memory over prompting. Prompt engineering on a capable model is not a dial you can turn up. Adding words to a system prompt competes with the task rather than reinforcing it, and there is no reliable direction of improvement to iterate along. A structured memory changes what the model knows going in, not how loudly it is instructed, and unlike a prompt, it is versioned, shared across departments, and auditable per call.
Read the letters yourself
Scores are a proxy. The letters are the actual product, so here they are unedited, switch between the arms and judge the writing directly. It is worth noticing how similar the four prompt-engineered letters are to each other, and how little the extra instruction text changed what came out.
[Your Name] [Your Address] [Your Phone Number] [Your Email Address] [Date] Sarah Chen Engineering Manager, Platform Team [Company Name] [Company Address] Dear Sarah, Please accept this letter as formal notification that I am resigning from my position as Senior Software Engineer on the Platform Team at [Company Name]. My last day of employment will be two weeks from today, [Your Last Day of Employment]. I want to express my sincere gratitude for the opportunity to have worked on the Platform Team for the past two years. I have learned a great deal during my tenure, and I am particularly thankful for your mentorship and guidance. I also want to thank the entire team for their support and collaboration. I am committed to ensuring a smooth transition during my remaining time here. Please let me know how I can best assist in handing over my responsibilities. I wish you and the Platform Team all the best for the future. Sincerely, Alex Kim
Unedited output from each arm, same task and context. Bracketed fields are the model's own placeholders. Names are synthetic benchmark fixtures.
Where the difference actually lands
A single total hides which qualities moved, and the breakdown is not flattering in every direction. Raisy Memory scores a perfect 10 on the dimensions that depend on knowing the situation, specificity (10 vs 4.0 average), no omissions (10 vs 4.5), context usage (10 vs 5.0), completeness (10 vs 5.2). That is what a memory is for: it carried the tenure, the team, and the relationship into the letter, where the prompt arms wrote around challenges they had not yet measured.
It also loses badly on three: format compliance (1 vs 10), instruction adherence (1 vs 7.8), and task completion (5 vs 7.0). The judge penalised the Raisy arm for returning its letter inside a JSON envelope, a real defect for this comparison, and one the totals would have hidden. The 48-point lead survives despite it, not because of it.
Figure 2. Every judge dimension, every arm — darker is a higher score. Raisy Memory leads on the situation-dependent rows (specificity, no omissions, context usage, completeness) and trails badly on format compliance and instruction adherence, where the judge penalised its JSON envelope. Both are shown rather than netted out. Single sample per arm.
What this does not prove
Every number above is a single sample. The null hypothesis, that Raisy Memory and the prompt-engineered arms perform identically and the 48-point difference is chance, is not rejected by this data. In plain words: we ran each arm once, and repeated runs of an unchanged configuration on this task have varied by as much as 76 points, so a single 48-point spread could be luck. It is a hypothesis for the next run, not a proven result. Publication-grade numbers need at least three runs per arm with a reported confidence interval.
It is also one task on one model. A resignation letter is not a classification, an extraction, or a code change, and gemini-2.5-flash-lite is not every model. Two of the 29 rubric dimensions, decision_quality and reasoning_soundness, do not fit a writing task and returned degenerate scores on defect-free output; they remain in the totals rather than being quietly dropped.
Reproducibility
One command reruns the experiment. The runner holds model, temperature, and judge identical across both arms and prints a parameter audit before every run, so the comparison is checkable rather than asserted, every action leaves a receipt. Browse the blog for the rest of the series.
Conclusion
Four strategies for writing a better system prompt landed within 15 points of each other, in no particular order, and the most elaborate one scored lowest. That is what prompt iteration bought on this task: noise. Structured memory scored 48 points higher, 17% of the rubric: on the same model, the same temperature, and the same judge.
If you are still improving output by rewriting prompt wording, that 48-point difference is the distance you are trying to close by hand, one edit at a time, with no gradient to follow. A memory closes it by changing what the bot knows rather than how it is told. Build one and run this comparison against your own task, the numbers above are one letter, on one model, and your task is not ours.
FAQ
Why does no system prompt beat an extensive one?
On a task a capable model already knows how to do, extra instruction text competes with the task rather than reinforcing it. The four prompt arms span 180–195 with no ordering, which is the point: prompt effort is not a dial you can reliably turn up.
Is a 48-point difference statistically significant?
No, and the post says so. Each arm is a single sample, and repeated runs of an unchanged configuration varied by up to 76 points on this task. The difference is a hypothesis for a replicated run, not a proven result. Publication needs at least three runs per arm.
How is this a fair comparison?
Both arms use the same model, temperature, judge, rubric, context, and task text; only the instruction layer differs. The runner prints a parameter audit before every run so the constants are checkable rather than asserted.
What exactly is Raisy Memory?
The bot’s assembled kernels: purpose, objective, capability, tone, and output contract: loaded and versioned automatically at call time. Unlike prompt text, each is an immutable version that can be audited, shared across departments, and rolled back.
Sources
Data: raisy_BE/reports/benchmark/benchmark_factorial_local_20260813_100923.json (run 2026-08-13). Runner: scripts/run_factorial_benchmark_local.py --cfg cfg/benchmark_factorial_tasks_letter.json. Judge: cfg/judge_system_prompt.md, 29 dimensions, gemini-3.1-pro-preview. Both arms: gemini-2.5-flash-lite at temperature 0.1. Single sample per arm: see "What this does not prove".
Keep reading
Multi-Agent Supervisor Architecture, Compared
J.P. Morgan runs one supervisor over three sub-agents. Raisy runs a Chief of Staff over process rooms that own state. Where the two designs diverge.
An OpenAI Dots Alternative for Companies
Dots gives one person their own computer. A company needs one owner for a function, a gate before anything leaves, and a number somebody watches.