Smart Trust Meter: DistilBERT, Roman Urdu, and why the model lives on Hugging Face

Smart Trust Meter was my BS thesis: score trust for Lahore Honda and Suzuki dealerships from real customer reviews, then ship a small dashboard on top.

This log is the NLP path — live vs backup models, the dataset, Roman Urdu vs English, and the deploy split that actually worked on free tiers.

Live model vs backup

DistilBERT multilingual was the live model for the dashboard — every real scoring path that could reach the ML service went through it.

The dual-input LSTM (padded text + TextBlob features: polarity, subjectivity, length, punctuation) stayed as the backup. If the live transformer was offline — HF Space asleep, cold start timeout, deploy blip — the dashboard could still score reviews instead of going dark. English test numbers for that backup sat around 82% accuracy / 0.86 F1. Usable. Not the primary path.

Reviews in Lahore are not clean English. They look like "service bohot acha tha but pricing zyada hai" — Latin-script Urdu mixed with English. The LSTM preprocessor stripped non-alpha characters and quietly hurt Roman Urdu. That is exactly why DistilBERT owned live traffic and the LSTM only covered downtime.

Why DistilBERT multilingual for live

We compared what we could actually fine-tune and host as the primary path:

Candidate	Role
Dual-input LSTM	Backup when live ML is offline; weaker on Roman Urdu
DistilBERT English-only	Fast, misses code-mixed Urdu
distilbert-base-multilingual-cased	Live model — cased multilingual base, fits Colab/free GPU, small enough to serve
Heavier XLM-R / large BERT	Better ceiling, worse memory story for free hosting

Live DistilBERT was fine-tuned for binary sentiment (negative / positive), max length 128. English test set: ~90% accuracy, ~0.92 F1. Held-out Roman Urdu went from ~46% before augmentation to ~79% after — roughly +33 points from the RU data alone.

# illustrative — base + binary head
model_name = "distilbert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2,
)

Inference keeps casing and only light-cleans (URLs, whitespace). Softmax on the positive class becomes the continuous trust signal; 0.5 is the label threshold.

The dataset

Source material: Google Maps-style review exports for 10 Lahore dealerships (five Honda, five Suzuki). Raw text was noisy, bilingual, and full of dealership-specific slang.

Workflow was boring on purpose:

1. Prepare CSVs for labeling
2. Manual positive / negative labels (and aspect tags offline)
3. Split into training_data.csv / testing_data.csv
4. Add roman_urdu_augment.csv so the fine-tune saw enough RU examples
5. Keep a small held-out roman_urdu_eval.csv so RU gains were measurable, not vibes

Labels are binary for trust scoring: positive vs negative. Dealership trust is then positive / (positive + negative). Aspect scores (pricing, service, parts, etc.) were computed offline from the labeled set — not reinvented on every live submit.

Deploy: Render for the API, Hugging Face for the model

The product is three services: Vite frontend (Vercel), Express API (Render), Flask ML (/predict).

First attempt put the live DistilBERT service on Render next to the API. On a 512 MB free instance it OOMs. Cold starts already hurt; when the live model is unreachable the dashboard falls back to the LSTM backup — better than nothing, worse on Roman Urdu.

Fix: host DistilBERT on Hugging Face Spaces (Docker Space, enough RAM), keep the Node API on Render, point ML_SERVICE_URL at the Space. Weights (~500 MB+) stay on the Hub and download at Space boot via a repo id secret — not baked into the Render image. LSTM stays wired as the offline backup for the dashboard.

Browser → Vercel (SPA)
       → Render (Express + Mongo)
       → HF Space (live DistilBERT /predict)
         ↳ LSTM backup if live ML is offline

Both free tiers sleep when idle. First request after sleep is slow. That is the tradeoff for a thesis demo that stays online without a GPU bill.

Takeaways

1. If your users write Roman Urdu, English-only tokenizers and alpha-only cleaners are a silent failure mode — keep that model off the live path.
2. Ship a live model and a backup: DistilBERT for accuracy when up, LSTM so the dashboard still scores when HF is asleep.
3. Measure Roman Urdu on a held-out set. Augmentation without an eval file is storytelling.
4. Split hosting by memory: API on Render, live transformer on HF Spaces, when free-tier RAM is the blocker.

More STM notes if I revisit the aspect pipeline. The private write-ups stay in the project notes — this log is the lessons.