All projects

Case study · Cabify Spain · Jan to Jun 2026

Hybrid ML + LLM Vision Engine

The first LLM version of AQM asked the model everything: which side of the car it was looking at, which vinyl design, which plate, and then the hard questions about condition. That meant up to 18 model calls per vehicle, most of them for things a small model can answer for free. So I split the work by its nature and trained my own models for the repetitive half.

~80% AI cost cutlocal ML
PythonPythonPyTorchPyTorchKerasKerasGoogle ColabGoogle ColabOpenCVOpenCVYOLOv8 · UltralyticsYOLOv8 · UltralyticsClaudeClauden8nn8n
My roleBuilt alone: dataset, training, the cascade, the worker and the hand-off to n8n.
Whenv1 Jan to Feb 2026 · hybrid cascade May to Jun 2026
Built withPython, PyTorch (ResNet18 transfer learning), Keras for v1, Google Colab GPUs, OpenCV, YOLOv8, EasyOCR, n8n, Claude
StatusRuns under AQM. The vinyl model is retrained in Colab when a new design appears.
~80%
lower vision inference cost per vehicle
16,000+
labelled vehicle photos used for training
13
vinyl designs recognised by the local model
0.6
confidence gate: below it, the LLM decides

Split the work by its nature

The rule is simple: what is deterministic and repetitive runs on a local model and costs nothing in tokens; what is subjective or comparative goes to the LLM in n8n, where it is worth paying for.

CheckEngineWhy
Side of the vehicleLocal CNNA fixed visual pattern with plenty of examples; nothing to reason about
Vinyl typeLocal CNNA closed classification, cheap to repeat thousands of times
PlateLocal OCRPure OCR plus a format rule
Vinyl state against the referenceLLMNeeds to compare the real photo with the reference design
Dirt and damageLLMSubjective perception, and the answer has to be described in words
Rear defectsLLMWhether a logo or a label is present and legible, with nuance
INLOCALLOCALGATELLMOUTVehicle photofrom the AQM formsModel 1 · sideCNN: right, left or rearModel 2 · vinyl typeCNN, 13 designs, forside photosPlate OCREasyOCR + Spanish formatcheck, for rear photosConfidence gatebelow 0.6 goes to theLLMLLM in n8nstate against reference,dirt, damage, reardefectsFleet databaseone verdict per field
Deterministic and repetitive goes local and costs no tokens. Subjective or comparative goes to the LLM, where it adds real value.
# Cascade: each model learns one narrow task, which makes it easier to train and debug.
def analyse(photo):
    side, p_side = side_model(photo)                      # right / left / rear
    if side == "rear":
        return {"side": side, "plate": read_plate(photo)}   # OCR + ^[0-9]{4}[B-DF-HJ-NP-TV-Z]{3}$
    vinyl, p = vinyl_model(photo)                         # 13 designs
    if p < 0.6:                                           # rare or new design
        return {"side": side, "vinyl": None, "ask_llm": True}
    return {"side": side, "vinyl": vinyl}
All six checks on the LLMAll six checks on the LLM: 100100Hybrid: three local, three LLMHybrid: three local, three LLM: 2020
Cost of the vision checks per vehicle, indexed to 100 for the all-LLM version. Moving three of the six checks to local models removes about nine model calls per vehicle.

The models

  • Two classifiers in cascade, not one. Model 1 decides the side (right, left, rear); model 2 classifies the vinyl design on side photos. Each is a ResNet18 fine-tuned from ImageNet weights, which trains well on a free Colab GPU.
  • Plates are read with EasyOCR and checked against the Spanish format. The OCR is not trained at all.
  • Data. 16,000+ photos labelled by the column they came from in the capture sheet, which already says which side each photo shows. Augmentation never flips images horizontally, so logos and text are never mirrored.
  • Confidence gate. Rare designs have few examples, so when the vinyl model is less than 60% sure, the row is flagged and the LLM decides, using a text description of each design.
  • A new design from marketing only means retraining the vinyl model once there are enough photos of it; the side model and the OCR stay as they are.
Workerpolls every ~30 s,outgoing onlyn8n: pending rowsreturns the image asbase64Local modelsside, vinyl, platen8n: result webhookwrites the prediction tothe tableLLM auditpicks up rows with avinyl type already set
The n8n server cannot reach the machine running the models, but the machine can reach n8n. So the worker starts every exchange, nothing is exposed and the plates never leave company infrastructure.

Version 1: from scratch

Before the cascade I built everything from zero in Colab: a dataset of 2,156 photos in 10 classes and a Keras CNN with four convolution blocks at 256 px, YOLOv8 to find the car and EasyOCR with several contrast and threshold variants for plates, and a SIFT and RANSAC homography that aligned each photo to the reference design to estimate how much of it was covered. It ran in batch over real submissions, but it confused similar designs and I did not keep a proper evaluation, so there was no number to defend. The LLM version of AQM replaced it, and the hybrid cascade brought the local models back where they are strong.

What I learned

  • An LLM is a great generalist and an expensive one. Knowing which questions do not need it is most of the saving.
  • Keep the evaluation from the first day. The v1 models may well have been good; I just could not prove it.

No vehicle photos are shown on this page: they show real plates and partner branding.