Live events and ticketing
The problem was economic, so the answer was too
A ticketing platform wanted an ML bot-detector. Every control that replaced it is standard anti-bot practice and none of it is proprietary; what the engagement decided was to refuse the classifier, and to start measuring whether any of it works.
The controls were off-the-shelf. The decision was refusing the model and finally measuring the thing everyone argued about.
Problem
High-demand on-sales were being drained in seconds by automated buyers, then resold. Fans complained, promoters threatened to leave, and the engineering team proposed a machine-learning classifier over request telemetry.
Context
A platform running scheduled on-sales with extreme, short traffic peaks and a long quiet tail.
Current architecture
A web front end, a queue at on-sale, a monolithic booking service, and rate limits by IP address.
Constraints
- Adversarial: the other side is funded, motivated, and observes every change you make.
- Accessibility: friction cannot exclude legitimate buyers using assistive technology or shared connections.
- Latency: anything in the buy path must add negligible time at peak.
- Legal: resale rules differ by territory and by promoter contract.
Evidence
Each statement placed on the ladder before it was used.
Most inventory in the first minute goes to automated buyers. Believed from complaint volume; never measured against actual fulfilment data.
Rate limiting is applied per IP address only. Residential proxy pools defeat it, and the platform had already seen this.
Any classifier must decide inside the buy path, where the added latency is itself a competitive disadvantage at peak.
A model over request telemetry could separate automated from human buyers well enough to act on.
A false positive is a fan who cannot buy a ticket. That is the most expensive error in the business.
Questions that changed the answer
- What fraction of first-minute inventory actually ends up on resale? This is knowable from resale listings, and it had not been checked.
- How quickly would an adversary notice a new detector and adapt?
- Who retrains the model when they do, and how fast can they ship?
- What is the acceptable false-positive rate when the error is a genuine fan turned away?
Options
Including the one nobody wanted to discuss.
ML bot classifier in the buy path
Train on labelled traffic, score each request, block or challenge.
What it costs: Labels are a guess, the adversary adapts within days, latency is added in the worst place, and false positives hit fans.
Change the economics
Randomised queue admission, per-identity purchase limits verified at delivery, delayed ticket issuance, and price caps on resale where contracts allow.
What it costs: Does not identify bots at all. It makes owning a thousand of them worth less.
Buy a bot-mitigation vendor
A commercial edge product with its own detection.
What it costs: Moves the arms race to a supplier with more data than the platform. Kept as a complement rather than the strategy.
Economics
Four horizons, not one estimate.
- Build
- Classifier: model plus labelling plus serving. Economic controls: queue changes and identity checks in the existing booking service.
- Run
- The decisive difference. A classifier needs continuous labelling, retraining and monitoring forever; a purchase limit needs none.
- Change
- Every adversary adaptation is a model change on one path and a parameter change on the other.
- Exit
- Low for economic controls. Medium for a vendor, whose challenge pages become part of the fan experience.
Decision
Do not build a classifier. Change the payoff: randomised queue admission, identity-bound limits enforced at delivery rather than checkout, delayed issuance, and resale price caps where the promoter contract allows. Keep a commercial edge product for volumetric abuse only.
Why
Two things, and only the first is arguable. Detection problems where the adversary profits and adapts are not stable learning problems, because the distribution moves in response to your model, on the adversary’s schedule, and the retraining bill never ends. Controls that reduce the value of a successful purchase do not need to identify anyone, which is why they do not decay. The controls themselves, from queue randomisation and identity-bound limits to delayed issuance and resale caps, are published advice from every vendor in this market, and were presented as such, not as a design of ours. The genuinely missing piece was measurement. Nobody had matched resale listings back to fulfilment records, so every claim about how much inventory bots were taking, and every claim about whether a control helped, was an opinion.
Why not the alternatives
- ML bot classifier in the buy path: It commits the team to a permanent retraining race, adds latency where it is least affordable, and pays for errors in turned-away fans.
- Buy a bot-mitigation vendor: Useful against volumetric abuse and retained for that, but it outsources the same arms race rather than ending it.
Trade-offs accepted
- Randomised admission removes the ‘first click wins’ story some fans believe in, and that has to be explained honestly.
- Identity-bound limits add friction at delivery, and the accessibility path has to be designed in, not bolted on.
Reversibility
Every control is a parameter. Limits, queue randomisation and issuance delay can be tuned or withdrawn per on-sale, which is exactly what an adversarial setting demands.
Non-functional requirements
- No added latency in the buy path at peak
- No control may depend on JavaScript execution alone
- Every rejection must be explainable to the fan it affected
Decision gate
GO, with the measurement the argument had been missing: resale listings matched against fulfilment, so the effect of each control is observable.
Implementation
Queue randomisation and purchase limits in the existing booking service. Delivery-time identity binding. Resale caps configured per promoter contract. No new platform.
What the decision was expected to achieve
A smaller share of inventory reaching resale, and fewer fan complaints per on-sale. It would be proven by matching resale listings to fulfilment records. That measurement should have existed before anyone proposed a model.
No outcome is claimed. This is an illustrative example, so there is nothing measured to report, and a real engagement would state what happened and how it was verified.
Lessons
- Against an adaptive adversary, a model is a subscription to an arms race. Price the retraining before you price the build.
- You do not have to identify the attacker to beat them. Make the attack worth less.
- When the remedy is commodity advice, say so, and spend the engagement on the part nobody has done, which here was measuring the problem at all.
- When the expensive error is a false positive on your own customer, that fact sets the design and the accuracy metric does not.