JournalLLM Developments
Prompts Are Production Code and Should Be Versioned Like It
A string that determines system behaviour, can be edited by anyone, and has no history is not configuration. It is an outage waiting for a calendar slot.
A prompt determines the behaviour of a system, can be changed by anyone, and frequently lives in a string literal, an admin panel, or a spreadsheet cell with no history. By any reasonable definition that is production code being managed as a preference.
The regression you find from a support ticket
The failure mode is familiar to anyone who has run an LLM feature for six months. A wording change fixes one customer complaint and silently regresses three behaviours nobody wrote tests for. Because there is no diff, the regression is discovered by users.
Treating prompts as versioned artefacts removes most of this:
- Keep them in the repository, in files, reviewed like any other change.
- Attach a small evaluation suite that runs on every edit — twenty examples covering the behaviours you care about will catch the majority of regressions.
- Record the model version each prompt was validated against.
The model is part of the prompt
That last point is the one teams skip. A prompt tuned for one model is not guaranteed to transfer to its successor. Upgrading the model is a change to the system even when no prompt text moved — and it is a change that ships without a diff unless you pin and record it.
prompts/
triage.v4.md # the text
triage.v4.eval.jsonl # 20 cases with expected behaviours
triage.v4.lock # model id + params it was validated againstIf you cannot answer 'which prompt and which model produced this output last Tuesday', you cannot debug your own product.