A language model can produce an impressive demo in an afternoon. The same feature, put in front of customers or staff, will meet badly worded questions, outdated documents, edge cases nobody listed, and a monthly invoice.
The features that last share a pattern. They give the model the right information instead of trusting its memory, put a person in the loop where mistakes are expensive, measure quality with tests rather than impressions, and know what each request costs. None of this is exotic. It is engineering discipline applied to a probabilistic component.
Pick a job with a narrow shape
The best first AI features are bounded: summarise a support ticket, draft a reply from approved material, classify incoming enquiries, extract fields from a supplier invoice, answer questions about a specific policy library. They have a clear input, a clear output and an obvious way to tell whether the result was useful.
Open-ended assistants that are supposed to “help with anything” are harder to test, harder to trust and harder to explain when they get something wrong. If you are not sure where to start, choose the task a staff member does repeatedly and dislikes, where a first draft saves time even if it needs correcting.
Retrieval beats memory
A model does not know your prices, your contract terms or your returns policy, and it will sometimes sound confident anyway. The standard fix is retrieval: when a question arrives, the system searches your own documents or database, hands the relevant passages to the model, and asks it to answer from those passages only.
That puts the quality of the answer in your hands. If the policy page is outdated, the answer will be outdated, so retrieval makes content ownership a real job. Decide who maintains the source material, how changes are picked up, and what the system does when nothing relevant is found. “I could not find that, here is who to ask” is a feature, not a failure.
- Show which source passages an answer used so users can check them
- Restrict retrieval by permission, so staff see only what they are allowed to see
- Handle no-match explicitly instead of letting the model improvise
- Keep source documents versioned and dated
Decide where a human reviews
Review should be proportional to the cost of being wrong. A suggested subject line needs none. A drafted reply to an unhappy customer deserves a quick look. Anything that moves money, changes records, sends legal language or touches health or safety needs explicit approval before it happens.
Design the review step to be fast, or it will be skipped. Show the draft, the sources, and a one-click way to approve, edit or reject. Log what was changed. Those edits are your best signal for where the system is weak.
A review step nobody has time for is not a safeguard. Make it quick, or reduce what needs it.
Evaluate with a test set, not a feeling
Without tests, every prompt tweak is a guess, and an improvement for one kind of question can quietly break another. Build a set of real examples, anonymised where needed: typical questions, awkward ones, and ones the system should refuse. Write down what a good answer looks like for each.
Run that set whenever you change a prompt, a model or the retrieval setup, and compare against the last run. Have a person grade a sample as well, because automated graders have their own blind spots. Then keep adding to the set from real failures. After a few months it becomes the most valuable asset in the project.
Watch cost and latency from day one
Cost scales with how much text you send and receive, and with the model you choose. Long documents passed in on every request add up quickly. Track cost per request and per completed task, set sensible limits, and cache results for repeated questions where answers do not change.
Not every step needs the largest model. Classification and extraction often work well with a smaller, cheaper one, with the bigger model kept for drafting or reasoning. Check speed as well: a useful answer that takes too long to arrive will not get used.
Plan for failure. Providers have outages and rate limits, so build timeouts, retries and a plain fallback, such as routing to a person or to the old form. And decide up front who is accountable when the feature is wrong, because someone will be asked.
The question is not whether the model is clever. It is whether the whole system is reliable, checkable and affordable.
Takeaways
- Start with a narrow, repeatable task that has a clear definition of good.
- Ground answers in your own documents and show the sources.
- Match human review to the cost of an error, and make review quick.
- Keep a test set of real cases and re-run it on every change.
- Track cost and latency per task, and build a fallback for outages.
Related services
