QA & Product Review
End-to-End Flow, QA Findings & Business Justification
A QA/PM-style walkthrough of OmniFeedback AI as it exists today — written against the
actual shipped product, not an aspirational spec. Live at
the deployed app.
1. End-to-end user journeys
Support Manager
- Pastes a ticket into Inference Playground → urgency score in under a second
- If HIGH/CRITICAL, moves to GenAI Copilot for root cause + draft reply
- Sends the reviewed draft, logs the action
Friction: text doesn't carry over between the two pages yet — queued in the roadmap.
Product Manager
- Checks Executive Overview for critical rate, channel trends, aspect volume
- Opens Clustering Explorer — elbow chart justifies K, not just asserts it
- Checks Statistical Tests before trusting any channel-based intuition
Gap: no exportable report snapshot — screen-share only, for now.
Technical Reviewer
- Opens Model Comparison first — it's page 0
- Sees V1 hit F1=1.0, sees why that's a red flag, not a win
- Scrolls to the V2-vs-V3 section, sees the real 13× separation gap on real-world sentences
This journey is the strongest single feature for a skeptical audience — it now covers two independent, verifiable fixes, not one.
2. QA findings
| Finding | Severity | Status |
|---|
| Naive generator made the whole pipeline overfit-prone | High | Fixed |
| Naive argmax(silhouette) picks the range boundary on templated text | Medium | Fixed |
| Docker HEALTHCHECK silently broken (missing curl) | Medium | Fixed |
| Streamlit Cloud defaults to a Python version incompatible with pinned torch | High | Fixed |
| Channel/aspect carry no real signal — assigned independent of text | Medium | Open |
| Playground and Copilot don't share pasted text | Low | Open |
| No exportable report snapshot for Executive Overview | Low | Open |
| NER/BART cold-start ~95s combined on first use | Low | Open (UX, not correctness) |
| GenAI local backend is keyword-pattern-based, not learned | Medium | By design |
| V2's from-scratch 196-word vocabulary scored real critical/positive feedback almost identically — found by a person testing the live app, not by any test | High | Fixed (V3) |
| V3's initial checkpoint (265MB) exceeded GitHub's 100MB per-file limit | Medium | Fixed |
| V3's CRITICAL threshold (≥0.8) only reliably triggers on two narrow patterns (data loss, account compromise) — a severe outage complaint alone tops out at HIGH | Medium | Open (by design, stated plainly) |
| V3's fix was only validated on 8 hand-written sentences — could plausibly be an artifact of curated text | High | Validated on real data |
Real-data validation (150 examples/class, Yelp reviews + Sentiment140 tweets, proxy labels): V3's
separation is statistically significant on both (p<0.0001), at roughly half the hand-written magnitude
(0.08–0.13 vs. 0.30) — the fix replicates, honestly reported at its real, smaller effect size. See
Data Insights §7 for the full breakdown.
3. Business value, honestly framed
Crisis detection speed
Urgency scored in well under 350ms; catches ~80% of true critical tickets at the deployed 0.5 threshold (real confusion-matrix number, not a marketing claim of near-100%).
Generalization to real customer language (V3)
V2's from-scratch BiLSTM scored real-world critical and positive feedback almost identically (0.02 separation) whenever phrasing left its narrow training templates — found live, not in a test. V3 (transfer-learned DistilBERT, ~21% of parameters retrained) raised that separation to 0.30, ~13× larger, with zero additional training data. This is the difference between a demo that only works on cherry-picked examples and one that holds up under real scrutiny.
Clustering for roadmap prioritization
Works, but currently capped by the channel/aspect-independence finding above — clusters describe how people phrase feedback, not yet what product area it's about. Worth stating to a prospective client rather than overselling.
Executive summarization
BART live-tested and confirmed coherent — genuinely condenses a batch of tickets to a short brief, verified rather than assumed.
GenAI resolution copilot
Zero API cost via the local backend, transparent upgrade to Claude — directly answers the common enterprise objection: "we don't want per-token costs during a pilot."
The platform is credible to demo today, with three caveats a presenter should state
proactively: (1) clustering/channel findings are currently constrained by synthetic-data design, not
model capability; (2) the GenAI local backend is intentionally simple by design, not a limitation to
hide — its value is the zero-cost default and clean upgrade path; and (3) V3's CRITICAL threshold is
calibrated to two specific training patterns (data loss, account compromise), not a general "how bad
does this sound" scale — a severe complaint outside those patterns will land HIGH, which is expected
behavior, not a defect, but worth saying before a reviewer discovers the boundary themselves.