We present OmniFeedback AI, a feedback-triage system combining a transactional warehouse, K-Means clustering, an urgency regressor, transformer NER/summarization, and a GenAI copilot. We document two sequential post-deployment corrections. V1→V2: an initial corpus yielded suspiciously perfect metrics (F1=1.000, silhouette=1.000) — an artifact of 9 unique training sentences, not competence — fixed with a hardened generator and a Kneedle elbow method replacing naive silhouette-maximizing K-selection. V2→V3: a user of the deployed app found the resulting BiLSTM scoring natural critical and positive feedback almost identically; we trace this to a 196-word from-scratch vocabulary (42–55% OOV on natural text) and correct it via partial transfer learning — a fine-tuned DistilBERT, 21.4% of parameters trainable — yielding a ~13× larger critical-vs-low score separation on hand-written text with no additional data. We further validate this correction on 150 examples/class of real, independently-authored Yelp and Sentiment140 text: V3's separation remains statistically significant on both (p<0.0001) at roughly half the hand-written magnitude, and its correlation with Yelp's full 5-class rating (Spearman ρ=0.682) is markedly stronger than V2's (ρ=0.073, not significant). We report full before/after metrics for both fixes, an error analysis, a null channel–urgency result (p=0.84) traced to generator design, a checkpoint-size engineering note, and an explicit characterization of what the corrected model's CRITICAL threshold actually detects. All three configurations remain live and independently reproducible.
Enterprises triage feedback across uncoordinated channels — reviews, tickets, social posts — too slowly to catch emerging incidents. Automated triage combines clustering, regression-based urgency scoring, and LLM-based drafting, each typically validated in isolation on whatever data is on hand. When that data is synthetic, near-perfect offline metrics can reflect trivial separability rather than generalizable competence — a failure mode this paper documents concretely.
We trace that failure as it occurred in OmniFeedback AI: the diagnosis, two applied fixes, and the resulting metrics. Both the naive (V1) and hardened (V2) configurations remain live and re-runnable, so Table 1 is reproducible rather than merely asserted.
Bidirectional LSTMs [1,2] remain a strong, low-resource baseline for sequence regression relative to transformer fine-tuning. K-Means selection commonly relies on the silhouette coefficient [3]; the Kneedle algorithm [4] formalizes elbow detection as maximum distance from the chord joining a curve's endpoints — applied here to a "higher is better" metric rather than its original inertia framing. NER and summarization use BERT [5] and BART [6]; Section 6's transfer-learning correction fine-tunes DistilBERT [7], a distilled encoder retaining most of BERT's language understanding at a fraction of its parameter count. The copilot draws on chain-of-thought [8] and few-shot prompting [9], optimized with Adam [10].
Feedback is cleaned by a regex ingestor and loaded into a SQLite star schema (Fact_Feedback + three Dim_* tables) under transaction and constraint enforcement. Three parallel paths consume it: TF-IDF/K-Means with LogReg/RF baselines, a BiLSTM urgency regressor, and on-demand BERT-NER/BART pipelines. A GenAI copilot drafts a structured JSON resolution from the BiLSTM's output, falling back to a deterministic generator absent an LLM credential.
V1 (naive): 9 unique phrases (5 negative, 4 positive) sampled at a flat 40% negative rate across 4,000 records. V2 (hardened): templated, randomized phrasing (212 unique surface forms), an ambiguous class straddling the 0.5 threshold, and ~3% label noise. Both share identical downstream code, isolating any metric difference to the data alone.
Naive argmax(silhouette) over k∈[2,12] scores 1.000 for every k≥9 on V1 (Fig. 2) — clusters of exact duplicates, not signal. We replace it with Kneedle: normalize both axes, select k at maximum perpendicular distance from the endpoint chord.
A single fixed-seed 70/15/15 split precedes training. The BiLSTM (embed 128 → BiLSTM 64/dir → dropout 0.3 → sigmoid) trains under Adam (decay 1e-5) with MSE loss and early stopping on validation loss (patience=4); all metrics are test-split only.
| Metric | V1 | V2 |
|---|---|---|
| Unique training sentences | 9 | 212 |
| Silhouette @ selected k (k=9) | 1.000 | 0.328 |
| Baseline macro-F1 (LogReg) | 1.000 | 0.914 |
| BiLSTM test MSE | 0.0137 | 0.0162 |
| BiLSTM test macro-F1 | 1.000 | 0.821 |
A post-hoc sweep (Fig. 5) finds the F1 optimum at t=0.48 (0.8285) vs. t=0.5 (0.8260) — the small gap indicates the shipped default was already near-optimal. Predicted/true urgency correlation: r=0.869. Channel–urgency independence: χ²=4.18, p=0.840 (8 d.f.) — not significant.
Post-deployment, a person testing the live application (not any automated test) found that V2 scored natural-language critical and positive feedback almost identically whenever phrasing diverged from the training templates. This section documents the diagnosis and fix as a second, independent case study in the same spirit as Section 4.2's elbow-selection correction: a suspicious result, investigated to a measurable root cause, fixed, and verified — this time a generalization failure rather than a data-construction artifact.
V2's BiLSTM (§4.3) learns its embedding table from scratch; its vocabulary — words occurring ≥2 times in the training split — is 196 tokens. Tokenizing natural stress-test sentences against it showed 42–55% out-of-vocabulary. With roughly half the input mapped to a single <UNK> token, the model retains little signal and its output regresses toward the training mean irrespective of true severity.
We replace the from-scratch embedding with a fine-tuned distilbert-base-uncased encoder, freezing the embeddings and first 4 of 6 transformer layers and training only the last 2 layers plus a linear regression head — 21.4% of parameters trainable (14.2M / 66.4M), deliberately mirroring a partial-fine-tuning ratio observed to outperform from-scratch training in an unrelated image-classification task. Subword tokenization eliminates the closed-vocabulary failure mode entirely: any English string decomposes into known subword units regardless of training-corpus coverage.
On the held-out synthetic test split, V2 and V3 are comparable (MSE 0.0162 vs. 0.0160; macro-F1 0.821 vs. 0.825) — expected, since that split is drawn from the same template family both models trained on and cannot itself reveal an out-of-distribution generalization gap. On 8 natural-language sentences constructed to fall outside those templates, V2's critical-minus-low mean-score separation is 0.022; V3's is 0.298 — a difference this large is the finding, not the in-distribution parity above.
The naively-saved full model checkpoint is 265MB, exceeding GitHub's 100MB per-file limit. Since ~79% of that size is the frozen, unmodified pretrained backbone — recoverable at load time via the same from_pretrained() call already used for this project's NER/summarization components — we persist only the fine-tuned parameter subset (56.7MB) and reconstruct the frozen weights on load, verified to reproduce bit-identical predictions to the full checkpoint before adopting the change.
Testing ~20 additional natural-language sentences against V3 found its ≥0.8 CRITICAL threshold is not a general severity scale: sentences describing severe operational impact alone (e.g., extended production downtime with revenue loss) reliably score 0.69–0.79 (HIGH) without crossing 0.8. Two patterns reliably do: data-deletion-plus-immediate-fix-demand phrasing (0.86) and unauthorized-account-access-plus-financial-risk phrasing (0.81) — both closely matching specific template families in V2's training data that V3 inherited via its training labels. This is expected behavior for a threshold calibrated to a training distribution, not a defect, but it is a genuine limit on what "CRITICAL" means in this system that we state explicitly rather than leave for a user to discover unassisted.
§6.1–6.5 used 8 hand-written sentences — sufficient to demonstrate the failure mode but vulnerable to the objection that curated examples inflate the apparent fix. We re-ran the identical separation metric on 150 real examples per class from Yelp Review Full (proxy label: star rating) and Sentiment140 (proxy label: binary sentiment), neither authored by us. V3's separation remains significant on both (Yelp 0.127, Sentiment140 0.084; both p<0.0001 by Mann-Whitney U), confirming the fix generalizes beyond curated text — but at roughly half the magnitude of the hand-written result (0.30), indicating real text is measurably harder to separate than examples written to be unambiguous. A secondary result: V2's separation is not uniformly null — it is small but statistically significant on Yelp's longer reviews (p=0.0048) yet indistinguishable from noise on Sentiment140's short tweets (p=0.39), consistent with §6.1's mechanism (shorter text leaves proportionally less in-vocabulary signal). A Spearman correlation against Yelp's full 5-class rating gives the most granular comparison available: V3 ρ=0.682 (p<0.0001) versus V2 ρ=0.073 (p=0.14, not significant) — V2 shows no reliable monotonic relationship with real sentiment across the full ordinal scale, while V3 does.
V2 false negatives cluster almost entirely in the deliberately-ambiguous class (target 0.35–0.6) — a boundary miss, not a comprehension failure. One false positive traces directly to the label-noise injection on a canonically critical sentence, meaning the model was semantically correct against a deliberately corrupted target.
An initial threshold sweep optimized precision_recall_curve's positive-class F1, not the macro-F1 used elsewhere — yielding a "best" threshold that scored worse than the shipped default, a logical contradiction that only became visible because the numbers didn't add up.
The null channel result reflects the generator's independent metadata assignment, not evidence that channel is generally uninformative — a distinction worth stating plainly rather than treating a synthetic design choice as a finding about the world.
Two incidents surfaced only under real hosted deployment, not local testing. First, a memory-ceiling crash ("gone over its resource limits") traced — via HuggingFace's file-metadata API rather than inference — to three transformer models resident simultaneously (bert-large-cased-conll03, 1.33GB; bart-large-cnn, 1.63GB; the V3 DistilBERT backbone), ≈2.96GB combined with no GPU. Resolved by substituting smaller same-task models (bert-base-NER, 433MB; distilbart-cnn-12-6, 1.22GB), reducing the combined NER+summarization footprint to ≈1.66GB, with output quality re-verified at both single-record and full production-batch scale before the smaller models were accepted as a replacement, not merely assumed equivalent from parameter count alone. Second, a CPU-throttling event followed under sustained inference load — an expected consequence of running four transformer-scale models on a free CPU-only tier rather than a defect in the code; documented as a hosting-tier constraint rather than papered over with a workaround that does not exist at this service tier.
Batch executive summaries produced by the BART/DistilBART pipeline (§3, §6.6's summarizer) were reported as reading close to extractive on this corpus — reusing ticket phrasing nearly verbatim rather than synthesizing cross-ticket themes. This is consistent with BART's pretraining regime (span corruption and reconstruction over long-form CNN/DailyMail news prose): on short, already-terse, keyword-dense complaint sentences, there is comparatively little redundant prose to compress, so the model's salient-span selection surfaces directly as near-verbatim reuse. No parameter-count variant within the BART family addresses this, since the behavior follows from the pretraining objective and input register, not model capacity. We extended the existing pluggable-backend pattern used for the resolution copilot (§3): an instruction-following LLM (Claude), given an explicit synthesis-oriented system prompt directing it to name recurring themes rather than reuse ticket sentences, drafts the brief when an API key is configured; the BART pipeline remains the zero-configuration fallback, labeled in the interface as more extractive on this register rather than presented as equivalent.
§6.6 validates V3's separation on real Yelp/Sentiment140 text, but only for the binary critical-vs-low comparison; the two narrow CRITICAL-triggering patterns identified in §6.5 were not re-tested against real data, and real data would plausibly broaden them. The copilot's local fallback is rule-based, not learned, and untested here against its optional Claude-backed path. V3 corrects the urgency regressor only; the TF-IDF-based clustering and baseline classifiers retain the same closed-vocabulary property that motivated Section 6 and have not yet received an equivalent correction or external validation. The NER component (§3, §7.4) is trained on the CoNLL-2003 tag scheme, which defines exactly four entity types (PER/ORG/LOC/MISC) with no dedicated product or software-version category; product mentions ("iPhone 15", "iOS 17.2") are consequently tagged MISC by construction rather than by model error, a limitation shared by essentially all CoNLL-trained NER models regardless of parameter count. An OntoNotes-trained alternative would add a PRODUCT tag, but the small candidates checked were not reliably available and the one confirmed-available option is a ~1.7GB Flair model — larger than this system's entire current NER+summarization footprint (§7.4) and not adopted for that reason; this is reported as a considered trade-off rather than an oversight. The Claude-backed summarization path in §7.5 was verified only for control flow and output shape (backend selection, schema conformance) via a substituted response, not for the actual synthesis quality of live model output, since no API credential was available in this evaluation environment — a claim of improved semantic quality for that path specifically therefore awaits end-to-end verification with a live key.
We documented several sequential diagnostic cycles — suspicious metrics traced to a data-construction artifact, a live-deployment failure traced to a closed-vocabulary generalization gap, resource-constrained hosting failures traced to measured model memory footprints rather than assumed ones, and a summarization-quality complaint traced to a mismatch between pretraining objective and input register — each resolved (or, where a measured trade-off argued against a fix, explicitly documented as such) with a verifiable, measured basis inside a system where all three model configurations stay live specifically so every table and figure here is reproducible, not asserted.