An AI assistant on a property developer's website has one job: tell the customer which flats are available, how much they cost and how they differ. If it says a reserved flat is available, or quotes a price tens of thousands below the price list, the sales team has a problem, not a time saving.
This article describes what we did with such an assistant for a residential developer with three developments and about 210 flats on offer. The most important conclusion is an unglamorous one: most of the invented answers did not come from a weak model, but from the data that reached it.
What the errors looked like
The previous version of the assistant worked and often answered well. A comparison run, however, turned up errors that are unacceptable in sales:
- a reserved flat described as available,
- a list for a question about three rooms in which two flats had two rooms,
- an inflated number of available flats, because the system counted knowledge base entries instead of flats,
- completion dates from brochures that had already passed, given as current,
- for a question with a budget, a guessed "from around" price instead of the cheapest flat on offer.
When we traced each case, most of them had the same mechanism behind them. The model did not have complete, current information in its context, so it filled the gaps with something plausible. The knowledge base contained counts of available flats and price ranges that were months old. Semantic search returned a few similar fragments instead of all the matching flats. In one place, a database function substituted the description of the oldest flat as the description of the whole development, so the model received a flat that had been sold long ago.
Principle one: numbers only from live data
We split the work between two mechanisms. Hard criteria, meaning the number of rooms, the budget and the development, are recognised by deterministic code, and the matching flats are fetched by a plain SQL query against the current offer. The model receives the full list of matches together with their total count, in a separate data section. If nothing matches, the data says so explicitly and suggests the nearest options, so the model does not have to guess.
Vector search stayed for what it is good at: descriptive questions about the neighbourhood, the finishing standard or the purchase process. It does not decide numbers, prices or statuses.
We rewrote the knowledge base in a factual style. The counts of available flats, the price ranges and the marketing exclamation marks were removed. The facts stayed, and completion dates from brochures gained the note "according to the brochure". We import the offer every hour from the developer's source, and sold flats do not stay in the data with an old price.
Principle two: data in a form the model cannot garble
How data is written into the context matters more than it seems. Each flat is one line written in plain language: number, rooms, floor area with a Polish decimal comma, floor and price. A flat that is no longer on offer appears without a price but with its status, so a question about a specific number gets a true answer even if the flat is sold.
Two observations from testing surprised us most. The first: examples in the instructions with realistic-looking numbers do harm. The model could assemble a flat that does not exist out of them, so we replaced the examples with templates containing fields to fill in. The second: when a development had only a few available flats and the data held just a summary without numbers, the model added the number and floor of a flat it knew from elsewhere. A prohibition in the instructions did not help. What helped was supplying those few flats as full lines.
Principle three: a defence that does not depend on the model
An assistant on a public website also receives questions meant to pull it out of its role. Instead of counting on the model to always refuse, we added layers that work independently of it:
- offer and knowledge base data go into a separate block, from which text in the data cannot "escape" and pose as instructions,
- a narrow filter in front of the model recognises unambiguous attempts to extract the instructions and answers with fixed text, without calling the model and at no cost,
- an output guard cuts off a response mid-stream if it starts repeating the instructions, and the widget replaces the bubble with a refusal,
- the widget turns every link to a domain outside the allowed list into plain text.
The input filter is deliberately narrow. It is better to let a suspicious question through to a model that has its own safeguards than to block a customer's ordinary question about reservation rules. In fairness, modern small models fend off typical attacks even without these layers. The layers provide something else: predictability that does not depend on the model, so the model can be swapped without rebuilding the safeguards.
How we measure it
Without measurement, each of these changes would be only an opinion. So we built a comparison test: 106 scenarios in 16 categories, from general questions and criteria, through specific flats, comparisons, parking and the purchase process, to multi-turn conversations, typos, questions in English and abuse attempts.
Both versions of the assistant, old and new, answer the same questions on the same copy of the offer and on the same model, with the prompt cache switched off. Scoring then runs on two tracks:
- Facts are checked by a program, not a model. Every price, amount and floor area in an answer must exist in the database, and a flat named in a sentence must be in the offer. A separate test checks whether the number, rooms, floor area, floor and price in one sentence belong to the same flat. Each of those figures can be true on its own while their combination is invented.
- Style and helpfulness are scored blind by a judge model. It sees a summary of the offer data and both conversations as A and B in random order, scores them on five dimensions and picks the better one.
The judge makes mistakes too. Re-scoring the same conversations can shift the result by a few wins, and in an extreme case by about ten. That is why we base decisions on what the program counts and treat the judge's verdict as a second signal.
Results
In the first round, the number of answers containing invented facts fell from 5 to 0. In the third round, the new version won 81 times, drew 18 and lost 6, and after a repeated blind scoring the figures were 89, 11 and 6. The judge's average scores on a 1-5 scale rose from 3.38 to 4.16 for accuracy and from 3.04 to 4.65 for conciseness. The median answer length fell from 403 to 178 characters, because a customer in a chat does not want to read an essay.
We optimised cost separately. We moved the fixed part of the instructions to the beginning and the variable data after it, so the provider's cache covers a longer shared prefix. The cost of one answer fell from 0.268 to 0.109 cents, and the median time to first word from 1.6 to 0.85 s. A unit test now ensures nobody changes the order by accident.
What we learned
A prohibition in the instructions turned out to be the weakest tool. In conversation summaries for the sales team, the model ended 37 of 54 texts with the same faulty phrase despite an explicit ban. The ban reduced the problem but did not remove it. Examples of good summaries helped, and code cuts out the rest. Across the next 71 summaries the phrase did not appear once.
A cache can distort a model comparison. In the first test of six models, the prompt cache returned to each of them an answer saved earlier for a different model. It has to be switched off for the duration of an evaluation.
The cost of evaluation also has to be designed. On the first day, tests with a judge through an external API cost about 18 USD. After moving scoring to parallel agents and splitting conversations into batches, a full run costs about 0.57 USD, and 106 conversations are scored in about a minute.
Limitations
Zero invented answers applies to this set of 106 scenarios and is not a guarantee in production. The model still has its habits, for example ending some answers with the question "Would you like me to…?". In a few conversations the judge preferred the old version because it was warmer and described the neighbourhood at greater length. We will measure the assistant's effect on the number of sales enquiries after a full season, not after a few weeks.
If you are considering an AI assistant for sales or customer service, start with the question of where the model will get its numbers from, and with a test set, before the first version of the instructions exists. The whole deployment is described in the case study.
