The Enrollment Model We Refuse to Let Make Decisions (Yet)
By Theodora Christou · Co-Founder, Dabster Labs · Client engagement, anonymised at the client’s preference
The situation
A training provider’s outreach team opens every course with the same problem: a list of hundreds of contacts and no principled way to decide who to call first. Experience helps, but experience does not scale, and the team was spending its best hours on names that were never going to convert while better prospects sat untouched at the bottom of the list.
What we built
A machine-learning model that scores every contact’s probability of enrolling in a given course, so the team can work the most promising leads first. Each contact gets a calibrated percentage, learned from who actually registered for past courses, sitting quietly next to their name in the tool the team already uses.
Why it is not making decisions yet
This is the part most people skip, and it is the part that matters. We did not ship a score and call it done. We tested the model the honest way, training it on the past and measuring it on a future it had never seen. That immediately caught an early version cheating: it scored 0.90 on data it had memorised and only 0.73 on data it had not, the classic sign of a model that looks brilliant in the lab and useless in the field. We fixed it by making it forget more, which lifted the honest, held-out score to roughly 0.78.
Then we did the harder thing: we left it in shadow mode. The model scores and logs, but it does not decide who gets called. On the first real course its predictions were no better than a coin toss, because only a handful of people had enrolled, far too little to conclude anything. So the model watches. Its accuracy is checked every week as more courses resolve, and it will only start influencing a single call once it has earned that trust on real numbers. A model you cannot yet trust should not be allowed to make decisions.
What changed
- The team has a signal, not a slogan. A calibrated probability next to each contact, honest about its own uncertainty, instead of a vendor’s promise of “AI-powered lead scoring.”
- The evaluation is built in. The model grades itself every week against what actually happened, so its real accuracy is never a mystery.
- Go-live is a human decision. Nothing switches from shadow to live automatically. A person looks at the evidence and decides, which is exactly where that call belongs.
- We say so when the ceiling is the data. We tested a dozen ways to make the model smarter and reported honestly that the limit is the amount of history, not clever features. Knowing that saves everyone from chasing a better model that does not exist yet.
Where it stands
Running in shadow, scoring every contact for every course, re-graded weekly, with a deliberate human checkpoint standing between it and any real decision. That is not a limitation we are apologising for. It is the whole point.
Want AI that earns its place before it touches a decision, not after? Book a free audit and we will show you what that looks like.
Related service: Lead Qualification
