What it is
We build models that score, classify, extract or predict: risk scores, document reading, request routing, anomaly detection. Before training anything we measure what a simple method already achieves; a model is built only if it clearly beats it on data it has never seen.
The studio below is a demonstration build of the kind of workspace we set up for a modelling team: every run logged, compared and explained.
How we build a model you can rely on
The bar is set before training
Before any model is trained we agree on the exact question, the figure that will judge the answer, and the simplest method that could answer it: a rule your team already uses, an average, a linear model. That simple method is the bar. A model is worth its cost only if it clears that bar on data it has never seen, by a margin that matters to your business.
Leakage, the most common reason a model fails in use
A model that scores well in testing and poorly in use has usually seen information during training that it will not have in real life. A field filled in after the outcome, a customer present in both the training and the test data, a random split of data that should be split by time. We check for each of these before trusting any score.
The best of many runs looks better than it is
A team that tries two hundred settings and keeps the best one has partly selected for luck. We log every run, keep a sample of data locked away during development, and score the chosen model on it once. That single reading is the result we report, with its interval, whether it is good or bad. Our research note on detecting overfitting explains the method in detail.
Language models, measured like any other model
For reading documents, sorting requests or extracting fields, language models are often the right tool. We build an evaluation set from a sample of your own documents, labelled by your staff, and measure the error rate of each approach on it, along with the cost per document. Where errors are costly, uncertain cases go to a person.
After launch
The world changes and models drift. We compare live predictions with what actually happened on a fixed schedule, alert when accuracy falls below the agreed level, and write down when and how the model is retrained. A model card states what the model is for, what data it learned from and where it is known to fail.
Example build
Demonstration builds made for our portfolio, running on generated data. No client data is shown.
What you receive
What a project of this kind typically hands over. The exact list is set in the written scope.
A comparison with the simple method
You see what the model adds over what you have now, with its margin of error.
A written leakage check
Every field and every split examined before any score is trusted.
Code you can rerun
Training and evaluation scripts, the log of every run and a model card.
The model where you need it
Delivered as an API or inside your software, with the reasons behind each score.
Language models measured
Error rate and cost per document on your own documents.
Checks after launch
So you know when the world has changed under the model.
How a project runs
Typical timings. Your scope sets the real ones.
Week 1
Question, metric, data
The exact question, the figure that judges the answer, the data allowed, and a sample locked away for the final test.
Weeks 2 to 3
Baseline and leakage checks
What a simple method achieves, and a search for fields or splits that leak the answer.
Following weeks
Models on development data
Models tried on development data only, every run logged with its settings.
Final test
One reading
The chosen model scored once on the locked sample, reported with its interval.
Delivery
Into your tools
Report, code, model card, and the model running where it is used, with monitoring.
Have a project like this one? Tell us what you need in a few lines.
Send us the briefOther services
- Custom software and internal toolsSoftware built around the way your team already works, not the other way round.
- Platforms, dashboards and client portalsMulti-user platforms where your clients, partners or staff log in and get their own view.
- Websites and web appsFast websites and web apps that work on every screen and in several languages.
- Data pipelines and CRMClean, connected data and a CRM your sales team actually keeps up to date.
- Forecasting and statistical studiesForecasts of demand, load, prices or volumes, and studies that answer a precise question.