Enterprise feedback ingestion, escalation triage, and a GenAI resolution copilot — built with the overfitting fix made visible, live, instead of just claimed.
Same pipeline. Same code. The only difference is the training data — and the app lets you flip between them and watch every downstream metric change, live.
Support tickets, app reviews, tweets, call notes — thousands of records a day, across channels that don't talk to each other. A billing outage or crash spike can sit unnoticed for hours.
Detect urgency spikes in real time instead of discovering an outage from angry tweets.
Cluster unstructured feedback into operational themes without hand-labeling.
Cut manual ticket-handling time with a structured, reviewable AI-drafted reply.
Regex cleaning strips HTML, URLs, mentions, and repeated characters before anything touches the database. Every batch insert runs inside one transaction:
Naively maximizing silhouette picks whichever k is largest on templated text — it never plateaus. Replaced argmax(silhouette) with the Kneedle max-distance-from-chord elbow method.
| Logistic Regression | 0.914 |
| Random Forest | 0.912 |
| Metric | V1 Naive | V2 Hardened |
|---|---|---|
| Unique training sentences | 9 | 212 |
| K-Means silhouette (best k) | 1.000 | 0.328 |
| Baseline classifier macro-F1 | 1.000 | 0.914 |
| BiLSTM test macro-F1 | 1.000 | 0.821 |
| BiLSTM test MSE | 0.0137 | 0.0162 |
V2 scored an obviously-critical complaint ("crashes every time I check out, I've switched to a competitor") almost identically to an obviously-positive one. Tests check code correctness — not whether a model actually generalizes to language it never saw in training.
V2's BiLSTM learns its own embedding table from scratch — its vocabulary is exactly 196 words, because the synthetic training templates are narrow. Tokenizing genuinely natural sentences showed 33–63% of words mapping to <UNK>. With over half a sentence invisible, the model has almost nothing to work with — output collapses toward the training mean regardless of true severity.
Same lesson from an unrelated image-classification project: fine-tuning a slice of a pretrained model beat training a CNN from scratch by a wide margin. Applied here: fine-tune a pretrained DistilBERT instead of training a BiLSTM's vocabulary from 4,000 rows.
The standard test-set metrics look nearly identical — that's expected, since that test set is drawn from the same narrow templates both models trained on. The real difference only shows up on text neither model saw verbatim, which is exactly the point.
~79% of that 265MB was the frozen, unmodified DistilBERT backbone — identical to what AutoModel.from_pretrained() already fetches fresh. Rather than reach for Git LFS, saved only the parameters that actually changed and reconstructed the rest at load time.
Tested ~20 real sentences directly against V3 to find what actually crosses the 0.8 CRITICAL threshold. Two patterns reliably do:
| 0.860 | "deleted all data... fix immediately... unacceptable" |
| 0.808 | "unauthorized access... password changed... financial risk" |
"The iPhone 15 crashed after the iOS 17.2 update, and Apple support in California hasn't responded."
5 distinct crisis reports (crash, billing, shipping, outage, latency) condensed to one brief in ~5s (cached weights).
Few-shot examples + Chain-of-Thought system prompt. Backend is pluggable:
0.3-pt gain from tuning means 0.5 was already close to optimal — no hidden accuracy left on the table.
Almost entirely the deliberately-ambiguous "neutral" template class (urgency 0.35–0.6) — the model lands on the wrong side of a boundary the dataset was built to contain, not a comprehension failure.
Channel vs. urgency independence: χ²=4.18, p=0.840. The synthetic generator assigns channel independent of content — a real gap, documented and fixed in the roadmap, not hidden.
| Finding | Severity | Status |
|---|---|---|
| Naive generator made the pipeline overfit-prone | High | Fixed |
| Docker healthcheck silently broken (missing curl) | Medium | Fixed |
| Cloud deploy Python-version mismatch blocked launch | High | Fixed |
| Channel/aspect carry no signal in synthetic data | Medium | Open |
| Playground & Copilot don't share pasted text | Low | Open |
Correlate channel/aspect with template category; validate on real Yelp/Sentiment140 data; share state across pages.
OpenAI embeddings for semantic clustering; Pinecone for similar-ticket RAG; Claude API already wired in.
Fine-tuned DistilBERT vs. BiLSTM comparison; active learning on boundary cases; drift monitoring.
| Incident | Root cause | Fix |
|---|---|---|
| "Gone over its resource limits" (crash) | 4 transformer models loaded at once (bert-large-NER 1.33GB + bart-large-cnn 1.63GB + DistilBERT V3) ≈ 2.96GB, measured via the HuggingFace file-metadata API, not guessed | Fixed — swapped to bert-base-NER + distilbart-cnn-12-6, cutting the combined footprint to 1.66GB, verified output quality held at both toy and production batch scale |
| "Your app has been throttled" (CPU) | Streamlit Community Cloud's free-tier CPU cap under sustained transformer inference load, with no GPU | Documented, not code-fixed — expected behavior at this hosting tier; the real fix is a paid inference endpoint (see Enhancement Roadmap) |
31 passing tests · V1→V2→V3 model evolution, live and toggleable · live Streamlit deployment · Dockerfile verified · full build log in PROGRESS.md · data insights, product review, and roadmap in /docs.
Next: blog series, research paper, Twitter/LinkedIn threads.