OmniFeedback AI

Theory sheet

Every concept used in OmniFeedback AI, with its full form, how it works, where it appears in the code (GitHub links with line numbers) and how it makes the project more efficient or more honest. 72 concepts · 7 groups · 222 code references.

← PresentationCode links are pinned to commit d27a741, so the line numbers stay correct even if main changes.

Data & text 8 concepts

How the training data is made, kept honest, and cleaned.

Synthetic data & random seeds PRNG = pseudo-random number generator #

Computer-generated records that stand in for real data. A seed makes the “random” sequence repeat exactly, so everyone who runs the script gets the same dataset.

Where it is used in this project

generate_naive (V1) uses a private random.Random(seed). generate_hardened (V2) seeds the global generator but saves and restores its state with getstate()/setstate(), so calling it does not change randomness elsewhere. Both build 4,000 rows. Timestamps use datetime.now(), so the dates are not reproducible even though everything else is.

scripts/generate_data.py · L142–188 global seed + save/restore

How it helps

There is no real customer data, so the project can manufacture a deliberately easy dataset (V1) and a hard one (V2) on demand, with no privacy issues, and re-create either exactly.

Shortcomings & limits

Synthetic text is only as rich as its templates: V2 has just 212 distinct sentences in 4,000 rows. Models learn the generator, not real customers, which is why V3 and the real-data validation exist.

Theory & history

A PRNG is a deterministic algorithm that produces numbers that look random. Starting it from the same seed always yields the same sequence. That is what makes an experiment reproducible.

Python offers a global generator (random.seed, random.random) and private generators (random.Random(seed)). A private one cannot be disturbed by other code.

See also →Template-based text generationLabel noiseOverfitting & memorisation

Template-based text generation slot filling #

Writing a sentence once with {placeholders} and filling each slot with a random choice, so one template yields many different sentences.

Where it is used in this project

CRITICAL_TEMPLATES holds sentences such as "App keeps crashing during {step} step…"; CRITICAL_FILLS maps step to ["checkout", "payment", …]. _fill swaps each placeholder for random.choice(options). The ambiguous templates deliberately mix positive and negative words.

How it helps

V1 picked from 9 fixed sentences (9 unique texts in its database). Templates lift that to 212 in V2, so keyword memorisation no longer gives a perfect score.

See also →Synthetic data & random seedsOverfitting & memorisation

Label noise simulated annotator disagreement #

Deliberately wrong labels, mimicking real people who disagree or make mistakes when labelling data.

Where it is used in this project

About 3% of rows (label_noise_rate=0.03) get label = 1 - label and urgency_score = 1.0 - urgency_score. Only V2 has this; V1 has none.

How it helps

With perfectly clean labels any model can reach a flawless score. Noise puts a ceiling below 1.0 so a perfect metric becomes a warning sign instead of a trophy.

Shortcomings & limits

The noise here is random and symmetric. Real label noise is usually systematic (ambiguous cases flip more).

See also →Overfitting & memorisationSynthetic data & random seeds

Overfitting & memorisation #

A model that fits its training data so closely that it fails on new data. Perfect scores on a tiny, repetitive dataset are the classic symptom.

Where it is used in this project

V1 is the textbook case. Its 4,000 rows contain only 9 distinct sentences, so the baseline classifiers score macro-F1 = 1.0 and K-Means reaches silhouette = 1.0 with inertia ≈ 0. The model memorised the sentences. V2 changes the data (not the code) and the numbers become realistic: F1 ≈ 0.91, silhouette ≈ 0.33.

src/ml_models.py · L36–70 Kneedle avoids rewarding the artefact

How it helps

Recognising the V1 result as memorisation rather than success is the central lesson of the project.

Theory & history
  • Detect it with a held-out test set and by comparing train vs validation error.
  • Fight it with more varied data, regularisation (dropout, weight decay), early stopping, or simpler models.
See also →Synthetic data & random seedsEarly stoppingDropoutSilhouette score

Data leakage & the train / validation / test split #

Leakage happens when information from the test data sneaks into training, making results look better than they are. The cure is to split first and fit everything on the training part only.

Where it is used in this project

The deep models shuffle indices with np.random.RandomState(42).permutation(n) and slice 15% test, 15% validation, 70% train. The vocabulary is built from train_texts only. V3 reuses the same seed and fractions, so V2 and V3 are scored on the identical held-out rows. The classical models use train_test_split(test_size=0.2, stratify=…).

src/dl_lstm.py · L97 train-only vocab

How it helps

Makes the V2-versus-V3 comparison fair and keeps test numbers honest.

Shortcomings & limits

data_insights.py scores all 4,000 rows, including training rows, so its confusion matrix and “best threshold” are in-sample and optimistic.
Theory & history

Typical split: train (learn weights), validation (tune and early-stop), test (touch once at the end).

See also →Class imbalance & stratified samplingEarly stopping

Class imbalance & stratified sampling #

When one class is rare (critical feedback is 15% of V2's generated rows), a model can score high accuracy by always predicting the common class. Stratified splitting and class weights keep the rare class visible.

Where it is used in this project

stratify=df[label_col] keeps the same critical/non-critical ratio in train and test. Both baselines use class_weight="balanced", and the headline metric is macro-F1 so each class counts equally.

src/ml_models.py · L83 logistic regression

How it helps

Accuracy alone would hide a model that ignores the minority class.

See also →Precision, recall, F1 & macro-F1Data leakage & the train / validation / test split

Regular expressions (regex) a pattern mini-language for text #

A compact notation for describing text patterns: \s+ is “one or more whitespace”, #(\w+) is “a hash followed by word characters, and remember the word”.

Where it is used in this project

TextIngestor defines six compiled patterns as class attributes, including the back-reference (.)\1{2,} that shrinks “sooooo” to “soo”. The copilot also uses regex: _ROOT_CAUSE_KEYWORDS (case-insensitive keyword rules) and _parse_json_response (strip markdown fences, grab the first {…}).

How it helps

Cleaning 4,000 texts costs about 0.024 ms per record end to end (read + clean + insert, from training_summary.json).

Shortcomings & limits

Regex matches characters, not meaning. _ROOT_CAUSE_KEYWORDS treats “down” inside “download” as an outage, and ignores sentiment entirely.

Theory & history
  • Capture group ( ) remembers a match; \1 in a replacement or pattern refers back to it.
  • Compiling (re.compile) pre-builds the pattern so it is not re-parsed on every call.

History. Formalised by Stephen Kleene in the 1950s as “regular events”; Ken Thompson brought them into text editors in the late 1960s.

See also →Text normalisationClasses, class attributes & @classmethod

Text normalisation (cleaning) #

Turning messy raw text into a consistent form (lower-case, no HTML or URLs, no stray punctuation) so models see the same word the same way every time.

Where it is used in this project

clean_text runs a fixed pipeline of substitutions, in an order that matters: lower-case, remove HTML, URLs and @mentions, keep the word of a #hashtag, shrink repeated letters, drop other punctuation, collapse spaces. Both raw_text and cleaned_text are stored in the warehouse. It is called from 7 places outside the tests. The BiLSTM and TF-IDF get cleaned text; V3 deliberately gets raw text because its tokenizer uses casing and punctuation (“!!!”) as signal.

src/urgency.py · L17–23 raw for V3, cleaned for BiLSTM

How it helps

One shared function guarantees training and live scoring clean text identically, which prevents “train/serve skew”.

See also →Regular expressionsClasses, class attributes & @classmethodRegistry & dispatcher pattern

Database 10 concepts

The SQLite star-schema warehouse and why writes are transactional.

SQL & SQLite Structured Query Language #

SQL is the language for defining and querying relational tables. SQLite is a database that lives in a single file and needs no server.

Where it is used in this project

Each variant has its own file (omnifeedback_v1_naive.db, omnifeedback_v2_hardened.db). WarehouseManager opens it with sqlite3.connect, runs the schema script, and exposes query(), which returns a pandas DataFrame.

How it helps

Zero setup: the repository ships the warehouse files, so the deployed app works without installing a database server.

Shortcomings & limits

One writer at a time and no network access, which is fine for a demo but not for many concurrent writers.

Theory & history

History. SQL grew out of IBM's SEQUEL (Chamberlin & Boyce, 1974). SQLite was created by D. Richard Hipp in 2000 and is the most widely deployed database engine.

See also →Star schemaACIDParameterised queries & SQL injection

Primary & foreign keys #

A primary key uniquely identifies a row; a foreign key is a column that must match an existing primary key in another table.

Where it is used in this project

Dim_User.user_id is a primary key; Fact_Feedback carries foreign keys to all three dimension tables. SQLite ignores foreign keys unless you switch them on, so every connection runs PRAGMA foreign_keys = ON.

src/database.py · L36–38 three foreign keys
src/database.py · L55 must be enabled per connection

How it helps

A feedback row can never point at a channel or user that does not exist.

See also →Star schemaConstraints

Star schema (fact & dimension tables) a data-warehouse layout #

One central fact table of events (here, feedback) surrounded by small dimension tables of descriptive labels (user, channel, aspect). Drawn out, it looks like a star.

Where it is used in this project

Fact_Feedback stores integer ids plus raw_text, cleaned_text, urgency_score, label and timestamp. In the V2 warehouse the dimensions hold just 5 channels, 6 aspects and 886 users for 4,000 feedback rows. _get_or_create_dim looks a name up or inserts it; fetch_all_feedback joins everything back together.

src/database.py · L66–72 dimension look-up/insert
tests/test_database.py · L62–65 proof of de-duplication

How it helps

Each channel name is stored once instead of 4,000 times, spelling cannot drift (UNIQUE), and counting by channel or aspect is a simple GROUP BY on a small integer column. tests/test_database.py checks five rows with one channel create exactly one dimension row.

Shortcomings & limits

A pragmatic mini star: each dimension has a single column and there is no date dimension. predicted_cluster exists in the fact table but the pipeline never fills it (0 of 4,000 rows).

Theory & history
  • Fact table: one row per event, holding measurements (urgency score) and foreign keys.
  • Dimension table: one row per distinct thing; each name is stored once.
  • Queries JOIN the fact table to dimensions to turn ids back into names.

History. Popularised by Ralph Kimball in his data-warehouse methodology (The Data Warehouse Toolkit, 1996).

See also →JOINPrimary & foreign keysACIDIdempotent writes

JOIN #

Combines rows from two tables where a column matches. An INNER JOIN keeps only rows that match on both sides.

Where it is used in this project

fetch_all_feedback joins Fact_Feedback f to Dim_Channel c ON f.channel_id = c.channel_id and to Dim_Product p ON f.aspect_id = p.aspect_id, aliases channel_name AS channel, and sorts by f.timestamp. Every dashboard page reads data through this one method.

src/database.py · L125–136 two JOINs + ORDER BY

How it helps

Storage stays compact (ids) while the UI still gets a flat, human-readable table.

Theory & history

Other kinds: LEFT JOIN (keep every left row even without a match), RIGHT, FULL. AS renames a column or table.

See also →Star schemaDatabase index

ACID Atomicity, Consistency, Isolation, Durability #

The four guarantees that make a database transaction trustworthy.

Where it is used in this project

A – insert_feedback_batch wraps the loop in BEGIN … COMMIT and calls conn.rollback() on any exception. C – the CHECK, NOT NULL and FOREIGN KEY rules reject bad rows. D – conn.commit() writes to the file. I – provided by SQLite's own locking; the project writes no isolation code. test_rollback_on_bad_urgency_score inserts one good row and one with score 5.0 and asserts zero rows remain.

How it helps

A half-loaded batch can never reach the dashboard.

Theory & history
  • Atomicity: all of a transaction's changes happen, or none do.
  • Consistency: the data always satisfies the rules (constraints) before and after.
  • Isolation: concurrent transactions do not see each other's half-finished work.
  • Durability: once committed, data survives a crash.

History. Transaction ideas come from Jim Gray's work in the late 1970s and early 1980s; the acronym ACID was coined by Theo Härder and Andreas Reuter in 1983.

See also →Transaction, commit & rollbackConstraintsStar schema

Transaction, commit & rollback #

A group of database operations treated as one unit. COMMIT makes them permanent; ROLLBACK undoes them all.

Where it is used in this project

The pattern is try: BEGIN … loop … commit() / except Exception: rollback(); raise. The bare raise re-throws the original error so callers still see it. A single transaction for all 4,000 rows is also faster than committing each row, because every commit forces a write to disk.

scripts/run_pipeline.py · L38 called by the pipeline

How it helps

Correctness (all-or-nothing) and speed (one disk sync per batch).

See also →ACIDContext managersIdempotent writes

Constraints (CHECK, UNIQUE, NOT NULL) #

Rules the database enforces on every write, so bad data is rejected at the source.

Where it is used in this project

CHECK (urgency_score >= 0.0 AND urgency_score <= 1.0) stops impossible scores; channel_name TEXT NOT NULL UNIQUE stops duplicates and blanks; AUTOINCREMENT numbers dimension rows.

tests/test_database.py · L47–53 constraint triggers the rollback

How it helps

The rule lives in one place (the schema), not scattered across every script that writes data.

See also →ACIDPrimary & foreign keys

Database index #

An auxiliary lookup structure (usually a B-tree) that lets the database find or sort rows without scanning the whole table.

Where it is used in this project

idx_fact_timestamp and idx_fact_channel are created on the fact table. fetch_all_feedback orders by timestamp, and charts group by channel.

How it helps

Speeds reads that filter or sort on those columns. Honest note: at 4,000 rows the gain is negligible; the indexes are there so the design still scales.

Shortcomings & limits

Indexes use extra space and make writes slightly slower.

See also →JOINStar schema

Idempotent writes (INSERT OR IGNORE / OR REPLACE) #

An operation is idempotent if running it twice leaves the same result as running it once.

Where it is used in this project

Users go in with INSERT OR IGNORE (skip if present); fact rows with INSERT OR REPLACE (overwrite the same feedback_id); dimensions through _get_or_create_dim. Re-running run_pipeline.py therefore refreshes the warehouse instead of duplicating it.

How it helps

Safe to re-run after a failure or a model change.

Shortcomings & limits

OR REPLACE silently overwrites an existing row, so a wrong re-run would also overwrite quietly.
See also →Transaction, commit & rollbackStar schema

Parameterised queries & SQL injection #

Passing values separately from the SQL text (with ? placeholders) so user data can never be interpreted as SQL.

Where it is used in this project

Every value is bound with ?. Table and column names cannot be parameters, so _get_or_create_dim inserts them with an f-string; that is safe here only because the callers pass hard-coded names.

src/database.py · L67–68 values bound, identifiers formatted

How it helps

Removes the classic injection risk, where a value such as x'; DROP TABLE … becomes a command.

See also →SQL & SQLite

Classical machine learning 9 concepts

TF-IDF, K-Means clustering, baseline classifiers and the metrics that judge them.

TF-IDF Term Frequency–Inverse Document Frequency #

Turns each text into a vector of numbers where a word scores high if it is frequent in this text but rare across all texts.

Where it is used in this project

fit_tfidf uses max_features=2000, ngram_range=(1, 2), min_df=2 (ignore words that appear in fewer than 2 texts). The matrix feeds K-Means, both baseline classifiers and the PCA plot, and the fitted vectoriser is saved with joblib so the app transforms new text identically.

How it helps

Fast, needs no training data beyond the text itself, and the 2,000-feature cap keeps matrices small.

Shortcomings & limits

Bag-of-words: ignores word order and meaning (“not good” looks like “good”).

Theory & history

tfidf(t, d) = tf(t, d) × idf(t). tf counts the word in the document; idf is roughly log(N / df), so common words such as “the” get a low weight and distinctive words such as “outage” a high one. N-grams are word sequences: with ngram_range=(1,2), “app crash” is its own feature. The output is a sparse matrix (mostly zeros), stored compactly.

History. The idf idea is credited to Karen Spärck Jones (1972).

See also →K-Means clusteringLogistic regressionModel serialisation

K-Means clustering (and inertia) #

An unsupervised algorithm that groups similar points into k clusters without being given any labels.

Where it is used in this project

select_k_via_silhouette fits K-Means for k = 2…12 and records inertia and silhouette; fit_kmeans fits the final model at the chosen k (k = 9 for both V1 and V2). Only the text is used: the dataset's aspect_category column is random noise, so the clusters are discovered topics.

How it helps

Finds themes in feedback with no labelling effort.

Shortcomings & limits

You must choose k; it assumes roughly round clusters; high-dimensional sparse TF-IDF vectors make Euclidean distance a weak signal.

Theory & history

Lloyd's algorithm: place k centres, assign every point to its nearest centre, move each centre to the mean of its points, repeat until stable. Inertia is the total squared distance from points to their centres (lower means tighter clusters). n_init=10 repeats the whole thing from 10 random starts and keeps the best, because the result depends on the start.

History. Stuart Lloyd devised the algorithm in 1957 (published 1982); James MacQueen coined the name “k-means” in 1967.

See also →TF-IDFSilhouette scoreElbow method & KneedlePCA

Silhouette score #

Scores how well each point fits its own cluster compared with the next-nearest cluster, from −1 (wrong cluster) to +1 (perfect).

Where it is used in this project

V1 reaches silhouette = 1.0 at k = 9 (inertia ≈ 0): nine identical sentences form nine perfect clusters. V2 reaches 0.328. That contrast is shown on the Model Comparison page.

How it helps

A model-free way to compare different values of k.

Shortcomings & limits

Costs O(n²) distance computations, and on templated text it keeps rising with k, which is why a plain maximum fails.

Theory & history

For a point, a = mean distance to others in its cluster, b = mean distance to the nearest other cluster; s = (b − a) / max(a, b). The score is the average over all points.

History. Peter Rousseeuw, 1987.

See also →Elbow method & KneedleK-Means clusteringOverfitting & memorisation

Elbow method & Kneedle choosing k at the knee of a curve #

Pick k where adding more clusters stops paying off: the “knee” of the curve.

Where it is used in this project

select_best_k replaces argmax(silhouette). On templated text silhouette climbs almost steadily with k, so argmax would always pick the largest k tested. Two tests lock this in: the edge case (must not return the boundary k) and the clear elbow (must return 3). The chosen k is drawn as a dashed line on the Clustering Explorer page.

How it helps

Avoids rewarding a metric artefact; a robust, parameter-free rule.

Shortcomings & limits

Still a heuristic; for V2 the silhouette keeps rising past the chosen k = 9.

Theory & history

Kneedle normalises both axes to 0–1, draws a straight line from the first to the last point, and returns the point farthest (perpendicular distance) from that line. The distance uses a 2-D cross product divided by the line length.

Worked example (the project's own test): silhouettes {2: .10, 3: .30, 4: .32, 5: .33} → k = 3, because the jump from 2 to 3 is the knee.

History. Satopää et al., “Finding a ‘Kneedle’ in a Haystack”, 2011 (cited in the code's docstring).

See also →Silhouette scoreK-Means clustering

PCA Principal Component Analysis #

Squashes many dimensions into a few while keeping as much variation as possible. Here: 2,000 TF-IDF dimensions down to 2 for a scatter plot.

Where it is used in this project

The Clustering Explorer calls PCA(n_components=2, random_state=42).fit_transform(X.toarray()) and plots each feedback item coloured by cluster. PCA is for the picture only; clustering itself is done in the full space.

How it helps

Lets a human see the clusters.

Shortcomings & limits

Two components keep only part of the variance, so clusters that overlap in the plot may be well separated in the full space. The page recomputes PCA on every interaction (not cached).

Theory & history

History. Karl Pearson, 1901; Harold Hotelling, 1933.

See also →K-Means clusteringTF-IDF

Logistic regression a linear classifier #

Combines features linearly and squashes the result with a sigmoid into a probability, then thresholds it into a class.

Where it is used in this project

One of two baselines on TF-IDF features, predicting the critical/non-critical label (LogisticRegression(max_iter=1000, class_weight="balanced")). Macro-F1: V1 1.00, V2 0.914.

How it helps

A cheap yardstick: if a deep model cannot beat this, it is not earning its complexity.

See also →Random forestPrecision, recall, F1 & macro-F1TF-IDF

Random forest #

Many decision trees trained on random subsets of the data and features; the final answer is their majority vote.

Where it is used in this project

RandomForestClassifier(n_estimators=200, max_depth=12, class_weight="balanced"). Macro-F1: V1 1.00, V2 0.912. run_ml_pipeline saves whichever baseline scores higher.

src/ml_models.py · L120 saves the better one

How it helps

Handles non-linear patterns without much tuning; limiting max_depth curbs overfitting.

Theory & history

History. Leo Breiman, 2001.

See also →Logistic regressionPrecision, recall, F1 & macro-F1

Precision, recall, F1 & macro-F1 (confusion matrix) #

Metrics for classification that are more informative than accuracy when classes are unbalanced.

Where it is used in this project

The BiLSTM and DistilBERT predict a score; the code turns it into a class with 0 if p >= 0.5 else 1 (0 = critical) and calls f1_score(..., average="macro"). Test macro-F1: V1 1.00, V2 0.821, V3 0.825. Worked example (V2, all 4,000 rows, in-sample): TP=1070, FP=355, FN=275, TN=2300 gives F1(critical) ≈ 0.773 and F1(non-critical) ≈ 0.880, so macro-F1 ≈ 0.826.

How it helps

Prevents a model that ignores the minority class from looking good.

Shortcomings & limits

Depends on the 0.5 cut-off; data_insights.py sweeps 181 thresholds and finds 0.48 is barely better (F1 0.8285), in-sample.

Theory & history
  • Precision = TP / (TP + FP): of what I flagged critical, how much was right?
  • Recall = TP / (TP + FN): of all truly critical, how much did I catch?
  • F1 = 2PR / (P + R), the harmonic mean of the two.
  • Macro-F1: compute F1 for each class, then average, so a rare class counts as much as a common one.

History. The F-measure comes from C. J. van Rijsbergen's Information Retrieval (1979).

See also →Class imbalance & stratified samplingMSE loss & regression

Model serialisation (joblib, torch.save) #

Saving a trained object to disk so the app can load it later instead of retraining.

Where it is used in this project

joblib.dump stores the TF-IDF vectoriser, K-Means model and best baseline as .pkl; torch.save(state_dict) stores the weights of the neural nets. Loading uses torch.load(..., weights_only=True).

src/ml_models.py · L118 classical models
src/dl_lstm.py · L172 BiLSTM weights

How it helps

Training happens once; the deployed app just loads files.

Shortcomings & limits

Pickle files can run code when loaded, so only load files you trust; weights_only=True restricts what torch will unpickle.

See also →state_dict & lightweight checkpoints

Deep learning 17 concepts

The from-scratch BiLSTM (V1/V2) and the transfer-learned DistilBERT (V3).

Tensors, batches, epochs & compute devices #

A tensor is PyTorch's n-dimensional array (like numpy, but GPU-capable and differentiable). Data is fed in batches; one full pass over the training set is an epoch.

Where it is used in this project

Batch size 32 (BiLSTM) and 16 (DistilBERT); up to 30 and 8 epochs. The device is chosen at run time: BiLSTM tries CUDA then CPU, DistilBERT tries Apple's MPS, then CUDA, then CPU. Tensors and models are moved with .to(device).

src/dl_lstm.py · L86 BiLSTM device

How it helps

Mini-batches give smoother, faster learning than one example at a time and fit in memory.

See also →Dataset & DataLoaderBackpropagation & the training loop

Embedding, vocabulary & out-of-vocabulary words OOV = out-of-vocabulary #

A vocabulary maps each known word to an integer id; an embedding maps each id to a learned vector of numbers (here 128).

Where it is used in this project

TextVocabulary builds the vocabulary from the training text only (min_freq=2) – V2's contains just 196 entries. encode truncates or pads every sentence to 128 ids. This closed vocabulary is V2's weakness: real customer wording falls to <UNK>, so very different sentences look alike.

How it helps

Cheap and fast to train.

Shortcomings & limits

Any word not seen in training is lost. This is what the V3 transformer fixes with sub-word tokens.

Theory & history

Special tokens: <PAD> (id 0) pads short sentences to a fixed length; <UNK> (id 1) stands in for any unseen word. padding_idx=0 tells the embedding to keep the PAD vector at zero.

History. Learned word embeddings were popularised by word2vec (Mikolov et al., 2013).

See also →Sub-word tokenisation & the attention maskBiLSTMData leakage & the train / validation / test split

LSTM Long Short-Term Memory #

A recurrent neural network that reads a sequence one step at a time while carrying a memory, with gates deciding what to keep, forget and output.

Where it is used in this project

nn.LSTM(embed_dim=128, hidden_dim=64, batch_first=True, bidirectional=True) inside BiLSTMUrgencyRegressor.

How it helps

Captures word order, which TF-IDF cannot.

Shortcomings & limits

Processes words sequentially (slow, hard to parallelise) and, here, starts with no language knowledge. Padding is not masked, so a forward pass also steps through the padding at the end of short sentences.

Theory & history

Plain recurrent nets forget early words because gradients shrink over many steps (the vanishing gradient). An LSTM adds a separate cell state and three gates (forget, input, output) so useful information can travel across a sentence.

History. Sepp Hochreiter and Jürgen Schmidhuber, 1997.

See also →BiLSTMTransformer & self-attention

BiLSTM Bidirectional Long Short-Term Memory #

Two LSTMs read the sentence, one left-to-right and one right-to-left, and their final states are joined.

Where it is used in this project

forward() embeds the ids, runs the LSTM, then torch.cat((h_n[-2], h_n[-1]), dim=1) joins the last forward and backward states (64 + 64 = 128 numbers). Dropout, then nn.Linear(128, 1), then a sigmoid gives the urgency score between 0 and 1. It is trained from scratch for V1 and V2.

src/dl_lstm.py · L59–73 inherits nn.Module

How it helps

Each word is seen with context from both sides (“not bad”), which a one-way LSTM gets only partly.

Shortcomings & limits

It matches V2's template language, not English in general: critical and positive real sentences scored only 0.022 apart on the hand-written test. V3 is better because DistilBERT arrives already knowing English: separation 0.298 with no extra training data.

Theory & history

History. Bidirectional recurrent networks: Schuster & Paliwal, 1997.

See also →LSTMInheritance & super()Embedding, vocabulary & out-of-vocabulary wordsTransfer learning, fine-tuning & freezing

Dropout #

During training, randomly switch off a fraction of neurons each step so the network cannot lean on any single one.

Where it is used in this project

nn.Dropout(0.3) before the BiLSTM's output layer; nn.Dropout(0.2) in the DistilBERT head. It is active under model.train() and disabled by model.eval().

How it helps

A built-in guard against overfitting.

Theory & history

History. Srivastava, Hinton et al., 2014.

See also →Overfitting & memorisationBackpropagation & the training loop

Sigmoid #

A smooth S-shaped function that squashes any number into the range 0–1: 1 / (1 + e^−x).

Where it is used in this project

The last step of both urgency models, so output is a valid 0–1 urgency score. .squeeze(-1) then drops the trailing size-1 dimension.

How it helps

Matches the target range exactly.

Shortcomings & limits

Saturates at the extremes, and with MSE it nudges uncertain predictions toward the middle, which is one reason real-text scores cluster around 0.4–0.55.

See also →MSE loss & regressionBiLSTM

MSE loss & regression Mean Squared Error #

Regression predicts a number rather than a class. MSE averages the squared gap between prediction and target.

Where it is used in this project

nn.MSELoss() trains both models to predict urgency_score. Predicting a number preserves severity (0.92 vs 0.55), and the score is only thresholded at 0.5 afterwards for F1. Test MSE: V1 0.0137, V2 0.0162, V3 0.0160.

How it helps

Keeps a ranking of urgency, which a yes/no classifier would throw away.

Shortcomings & limits

Squaring punishes large errors heavily and encourages hedging toward the mean.

See also →Precision, recall, F1 & macro-F1SigmoidBackpropagation & the training loop

Optimisers: Adam, AdamW & weight decay #

The rule that updates the weights from the gradients. Weight decay (L2) shrinks weights slightly each step to discourage over-complex fits.

Where it is used in this project

BiLSTM: Adam(lr=1e-3, weight_decay=1e-5). DistilBERT: AdamW with a much smaller lr=2e-5, given only the parameters that are not frozen.

How it helps

A small learning rate for fine-tuning avoids wrecking the pretrained weights; passing only trainable parameters saves compute.

Theory & history

History. Adam: Kingma & Ba, 2014. AdamW (decoupled weight decay): Loshchilov & Hutter, 2017.

See also →Transfer learning, fine-tuning & freezingBackpropagation & the training loopOverfitting & memorisation

Backpropagation & the training loop #

Backpropagation computes how much each weight contributed to the error; the optimiser then nudges each weight to reduce it.

Where it is used in this project

The same five-line recipe appears in train_bilstm_model and train_transformer_regressor.

src/dl_lstm.py · L132–136 validation, no gradients

How it helps

This loop is how every neural network here learns.

Theory & history

Each batch: zero_grad() → forward pass → loss → loss.backward() → optimizer.step(). Validation runs under model.eval() and torch.no_grad() (no dropout, no gradient bookkeeping).

See also →Optimisers: Adam, AdamW & weight decayEarly stoppingMSE loss & regression

Early stopping #

Stop training when validation error stops improving, and keep the best version seen.

Where it is used in this project

If the validation loss does not beat the best by more than 1e-4 for patience epochs (4 for BiLSTM, 3 for DistilBERT), training breaks. A .clone() snapshot of the best weights is restored afterwards with load_state_dict. Stopped at epoch 6 (V1), 6 (V2), 7 (V3), from the saved training summaries.

How it helps

Saves compute and prevents the model from over-training.

See also →Overfitting & memorisationBackpropagation & the training loop

Dataset & DataLoader #

Dataset says how to fetch one example; DataLoader batches and shuffles examples for you.

Where it is used in this project

FeedbackDataset returns (token ids, score); UrgencyTextDataset returns a dict with input_ids, attention_mask and score. Both inherit from torch.utils.data.Dataset and implement __len__ and __getitem__.

How it helps

Any dataset that honours that contract plugs straight into the same loader and loop.

See also →Inheritance & super()Tensors, batches, epochs & compute devices

Transformer & self-attention #

A neural architecture that lets every word look at every other word at once (attention), instead of reading one word at a time.

Where it is used in this project

DistilBERT is a transformer encoder. AutoModel.from_pretrained("distilbert-base-uncased") provides it; the project adds a small regression head.

src/transformer_nlp.py · L11 memory trade-off note

How it helps

Parallel, and captures long-range context; pretrained versions already understand language.

Shortcomings & limits

Attention cost grows with sentence length, so max_len=64 truncates long text (Yelp reviews are cut). Models are large: the project swapped its NER and summariser models to smaller ones after hitting Streamlit Cloud's memory limit (about 2.96 GB down to about 1.66 GB, per the code comments).

Theory & history

History. “Attention Is All You Need”, Vaswani et al., 2017.

See also →BERT & DistilBERTSub-word tokenisation & the attention maskLSTM

BERT & DistilBERT Bidirectional Encoder Representations from Transformers #

BERT is a transformer pretrained on huge text by guessing masked words. DistilBERT is a smaller student trained to imitate it (knowledge distillation).

Where it is used in this project

BASE_MODEL_NAME = "distilbert-base-uncased": 6 transformer layers, about 66M parameters. The NER pipeline uses a BERT-base model (dslim/bert-base-NER).

How it helps

Strong language knowledge at a size that fits a free cloud app.

Theory & history

History. BERT: Devlin et al. (Google), 2018. DistilBERT: Sanh et al. (Hugging Face), 2019: about 40% smaller and 60% faster, keeping about 97% of BERT's language-understanding performance. “uncased” means the tokenizer lower-cases text.

See also →Transfer learning, fine-tuning & freezingNamed Entity RecognitionTransformer & self-attention

Transfer learning, fine-tuning & freezing #

Reuse a model pretrained on a huge task and adapt it to yours. Freezing layers (requires_grad = False) stops them changing; fine-tuning trains the rest.

Where it is used in this project

The embeddings and the first N_FROZEN_LAYERS = 4 of DistilBERT's 6 layers are frozen; only the last 2 layers and the new linear head train: 21.4% of parameters. trainable_param_fraction() reports the figure. It is the same idea as freezing an ImageNet CNN backbone.

How it helps

Much less compute and less overfitting on 4,000 rows. Result: critical-versus-low separation of 0.298 against V2's 0.022 (about 13×).

Shortcomings & limits

It was still fine-tuned on synthetic labels, so on real Yelp text V3's average scores for 1-star (0.55) and 5-star (0.42) reviews are both mid-range: it ranks well but is not calibrated.

See also →BERT & DistilBERTOptimisers: Adam, AdamW & weight decayBiLSTM

Sub-word tokenisation & the attention mask #

Splitting text into word pieces from a fixed list, so an unseen word breaks into known fragments instead of becoming <UNK>. The attention mask marks real tokens (1) versus padding (0).

Where it is used in this project

UrgencyTextDataset tokenises every text up front with truncation=True, padding="max_length", max_length=64 and returns input_ids plus attention_mask. The model's forward needs both.

How it helps

Removes the closed-vocabulary problem that broke V2.

See also →Embedding, vocabulary & out-of-vocabulary wordsMean pooling

Mean pooling #

Averaging the model's per-token vectors into one vector per sentence, ignoring padding.

Where it is used in this project

mask = attention_mask.unsqueeze(-1).float(), then (hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9): multiply by the mask to zero padding, sum, divide by the real-token count (clamp avoids division by zero).

How it helps

More robust than using only the first (“[CLS]”) token on short, varied feedback text, and it ignores padding properly.

See also →Sub-word tokenisation & the attention maskBiLSTM

state_dict & lightweight checkpoints #

A state_dict is a dictionary of a model's learned tensors. Saving it is how a trained network is stored.

Where it is used in this project

trainable_state_dict() keeps only the fine-tuned parameters. The docstring explains the saving: about 55 MB instead of about 265 MB, which also stays under GitHub's 100 MB file limit. load_transformer_regressor rebuilds the model, downloads or reads the frozen backbone, and loads the rest with strict=False.

How it helps

Smaller repository and faster deploys.

Shortcomings & limits

The first V3 start needs the Hugging Face backbone, so it needs internet access or a warm cache.

See also →Model serialisationTransfer learning, fine-tuning & freezing

NLP & generative AI 8 concepts

Entity extraction, summarisation and the LLM resolution copilot.

Named Entity Recognition (NER) & CoNLL-2003 #

Finding and labelling names of people, organisations, places and similar items in text.

Where it is used in this project

get_ner_pipeline loads dslim/bert-base-NER with aggregation_strategy="simple" (merge word pieces into whole entities). extract_entities returns entity_group, word and score. CoNLL-2003 has four tags (PER, ORG, LOC, MISC), so product names such as “iPhone 15” come back as MISC; the page explains why.

How it helps

Pulls out who and what a ticket mentions, for routing and reporting.

Shortcomings & limits

No PRODUCT tag in this scheme. An OntoNotes model would add one, but the smallest option was too large for the free cloud tier.

Theory & history

History. The task name comes from the MUC-6 evaluation (1995); CoNLL-2003 is the standard English benchmark.

See also →BERT & DistilBERTHugging Face pipelines & the lru_cache singleton

Summarisation: extractive vs abstractive (BART / DistilBART) #

Extractive summaries copy key sentences; abstractive ones write new text. BART is a sequence-to-sequence transformer; DistilBART is its distilled version.

Where it is used in this project

summarize_batch joins texts, cuts them into 3,000-character chunks (BART reads about 1,024 tokens), summarises each, and if there were several chunks summarises the summaries again (map-reduce). _bounded_lengths keeps min and max lengths sensible for short inputs.

src/genai_copilot.py · L198–219 why the LLM is preferred

How it helps

Condenses many tickets into a brief.

Shortcomings & limits

Trained on long news, it behaves almost extractively on short, keyword-dense complaints. That is why generate_executive_summary prefers Claude when a key is present and falls back to this.

Theory & history

History. BART: Lewis et al. (Facebook AI), 2019. The model used here, sshleifer/distilbart-cnn-12-6, was trained on CNN/DailyMail news.

See also →LLMPluggable backend & graceful degradationHugging Face pipelines & the lru_cache singleton

Hugging Face pipelines & the lru_cache singleton #

pipeline(task, model=…) bundles a tokenizer, a model and post-processing into one callable. functools.lru_cache memoises a function's result.

Where it is used in this project

@lru_cache(maxsize=1) on get_ner_pipeline() and get_summarizer_pipeline() means the model loads once on the first call and the same object is returned afterwards (a singleton). device=-1 pins them to the CPU. from transformers import pipeline happens inside the function (a lazy import).

How it helps

Loading takes tens of seconds and hundreds of MB; doing it once keeps the app responsive and memory bounded.

See also →DecoratorsLazy importsNamed Entity Recognition

LLM Large Language Model #

A very large neural network (almost always a transformer) trained to predict the next token over vast text, then tuned to follow instructions.

Where it is used in this project

The copilot uses an LLM for two jobs, drafting a structured resolution and writing an executive brief, behind a switch that falls back to rules or BART when no key is set.

How it helps

Handles open-ended language (root cause, tone of a reply) that rule-based code cannot.

Shortcomings & limits

  • Hallucination: can state fluent falsehoods, so the prompt says never to invent account details.
  • Unreliable format: may wrap JSON in markdown or omit keys, hence the parser and validation.
  • Prompt injection: customer text is pasted into the prompt, so a ticket could try to steer it. A human reads the output; sanitising inputs would be a sensible next step.
  • Cost, latency, privacy: each call is billed, takes time and sends ticket text to an external service.
  • Knowledge cut-off and non-determinism.
Theory & history

Training stages: (1) pre-training on a huge corpus; (2) instruction tuning on example tasks; (3) preference tuning (for example RLHF, learning from human feedback) so replies are helpful and safe.

History. Transformer (2017) → GPT-1 and BERT (2018) → GPT-2 (2019) → GPT-3 (2020, 175 billion parameters) → ChatGPT (November 2022), which brought LLMs to the mainstream; many labs and models have followed.

See also →ClaudePrompt engineering: system prompt, few-shot & chain-of-thoughtPluggable backend & graceful degradation

Claude (Anthropic) #

Claude is Anthropic's family of LLMs, accessed here through the anthropic Python SDK.

Where it is used in this project

_claude_backend and _claude_executive_summary call client.messages.create(model="claude-sonnet-4-5", max_tokens=…, system=…, messages=[…]) and read message.content[0].text. The key comes from the ANTHROPIC_API_KEY environment variable or Streamlit secrets; with none, the app runs entirely on the local backend.

src/genai_copilot.py · L190 executive-summary call

How it helps

Good at following a strict JSON schema and writing professional replies.

Shortcomings & limits

  • The model name is hard-coded in two places; if that model were retired, the broad except would hide it and the app would quietly use the local backend. One constant, plus logging, would fix this.
  • All general LLM limits apply (hallucination, cost, privacy).
Theory & history

History. Anthropic was founded in 2021. Claude was first released in March 2023, followed by Claude 2 (July 2023), the Claude 3 family of Haiku, Sonnet and Opus (March 2024), Claude 3.5 Sonnet (June 2024), the Claude 4 generation (2025) and Claude Sonnet 4.5 (September 2025); newer generations have appeared since. Its training uses RLHF plus Constitutional AI, where the model critiques its own replies against written principles (Anthropic, 2022).

See also →LLMPrompt engineering: system prompt, few-shot & chain-of-thoughtPluggable backend & graceful degradation

Prompt engineering: system prompt, few-shot & chain-of-thought #

Steering an LLM with carefully written instructions. A system prompt sets the role and rules; few-shot examples show the desired output; chain-of-thought (CoT) asks the model to reason before answering.

Where it is used in this project

SYSTEM_PROMPT defines the role, the exact JSON schema and the constraints (“ONLY a single valid JSON object”). FEW_SHOT_EXAMPLES holds two complete worked cases (one critical, one praise). thought_process is the first key in the schema, so the model writes its reasoning before the conclusions that depend on it. EXEC_SUMMARY_SYSTEM_PROMPT demands synthesis, not sentence copying.

src/genai_copilot.py · L12–19 role, schema, constraints
src/genai_copilot.py · L120 examples injected into the user message

How it helps

Better, more consistent output without any model training.

Theory & history

History. Few-shot prompting was highlighted by the GPT-3 paper (Brown et al., 2020); chain-of-thought prompting by Wei et al. (2022).

See also →LLMStructured JSON output & validation

Structured JSON output & validation #

Asking an LLM for machine-readable output and checking it before trusting it.

Where it is used in this project

_parse_json_response strips markdown fences with a regex, takes the span from the first { to the last }, runs json.loads, and checks that the five required keys are present using set operations (required_keys.issubset(data.keys()); the missing ones are required_keys - data.keys()). Two tests cover fence-stripping and the missing-key error.

How it helps

The UI can rely on the fields existing.

Shortcomings & limits

It checks that keys exist, not that risk_level is one of the four allowed values.

See also →Regular expressionsPrompt engineering: system prompt, few-shot & chain-of-thoughtPluggable backend & graceful degradation

Pluggable backend & graceful degradation #

Design the system so that if the best component is unavailable, a simpler one takes over and the app still works.

Where it is used in this project

generate_resolution tries Claude inside try/except; on any failure (no key, network, bad JSON) it uses _local_backend, which maps the urgency score to a risk band and matches keyword rules to a root cause. Both return the same JSON shape plus a _backend tag that the page shows. generate_executive_summary uses the same pattern with BART as the fallback. A test runs with the key removed.

pages/3_GenAI_Copilot.py · L55 UI shows which backend ran

How it helps

The public demo works with zero configuration and never shows an error to the viewer.

Shortcomings & limits

except Exception: pass hides why a fallback happened. Logging the exception would make it debuggable.
See also →LLMStructured JSON output & validationLazy imports

Statistics 6 concepts

The tests used to check that differences are real, and how big they are.

Hypothesis test & p-value #

A procedure for asking “could this pattern be chance?”. The null hypothesis says there is no effect; the p-value is the probability of results at least this extreme if the null were true.

Where it is used in this project

The Statistical Tests page sets alpha = 0.05 and branches on p_value < alpha to either reject or fail to reject the null.

How it helps

Separates real differences from noise.

Shortcomings & limits

A small p-value does not mean a big effect, and “fail to reject” does not prove the groups are equal. That is why effect sizes are reported too.

See also →Chi-squared test of independenceCramér's V & effect sizeMann–Whitney U test

Chi-squared test of independence #

Tests whether two categorical variables are related by comparing observed counts in a table with the counts you would expect if they were independent.

Where it is used in this project

pd.crosstab(channel, urgency_bucket) builds the contingency table; chi2_contingency returns chi2, p_value, dof, expected. With 5 channels and 3 buckets, dof = (5−1)(3−1) = 8. Running the page's logic on the stored data gives p = 0.94 (V1) and 0.84 (V2): no association, which is expected because the generator assigns channels at random.

scripts/data_insights.py · L71 also in data_insights

How it helps

Doubles as a sanity check: a significant result here would signal a bug in the data.

Theory & history

History. Karl Pearson, 1900.

See also →Hypothesis test & p-valueCramér's V & effect size

Cramér's V & effect size #

A 0-to-1 measure of how strongly two categorical variables are associated, built from chi-squared.

Where it is used in this project

cramers_v = (chi2 / (n * min_dim)) ** 0.5 where min_dim = min(rows, cols) − 1. On the generated data it is about 0.02, essentially nothing.

How it helps

Significance says whether; effect size says how much.

Theory & history

History. Harald Cramér, 1946.

See also →Chi-squared test of independenceHypothesis test & p-value

Mann–Whitney U test #

A non-parametric test of whether values in one group tend to be larger than in another. It compares ranks, so it assumes no particular distribution.

Where it is used in this project

mannwhitneyu(crit, low, alternative="greater") is a one-sided test used on real data. Yelp: V2 p = 0.0048 (separation only 0.025: real but tiny), V3 p < 0.0001. Sentiment140: V2 p = 0.39 (indistinguishable from noise), V3 p < 0.0001.

How it helps

Valid for scores that are not normally distributed.

Theory & history

History. Mann & Whitney, 1947.

See also →Spearman rank correlationCritical-minus-low separationHypothesis test & p-value

Spearman rank correlation #

Measures how consistently two things rise and fall together, using ranks, from −1 to +1.

Where it is used in this project

Urgency score against an inverted Yelp star rating (4 - label), on 400 reviews (80 per star). V2: ρ = 0.07 (not significant). V3: ρ = 0.68. This is the strongest evidence in the project that V3 ranks real reviews sensibly.

scripts/validate_real_data.py · L61 invert so higher = worse

How it helps

Tests ranking quality across all five ratings, not only the two extremes.

Theory & history

History. Charles Spearman, 1904.

See also →Mann–Whitney U testCritical-minus-low separation

Critical-minus-low separation #

The project's own simple metric: the mean urgency score of critical examples minus the mean of low-urgency examples.

Where it is used in this project

Hand-written stress test (8 sentences): V2 0.0224 vs V3 0.2984. Real Yelp: 0.0249 vs 0.1272. Real tweets: 0.0050 vs 0.0837.

How it helps

Easy to explain and directly measures the V2 failure (critical and positive text scored almost alike).

Shortcomings & limits

  • The 8-sentence test has only 3 critical and 2 low sentences, and the “ambiguous” ones are ignored.
  • Yelp and tweet labels are sentiment proxies, not true urgency.
  • It measures ranking, not whether a score lands in the right risk tier.
See also →Mann–Whitney U testSpearman rank correlationTransfer learning, fine-tuning & freezing

Python & software engineering 14 concepts

Inheritance, shared functions, patterns and tooling used across the code.

Inheritance & super() #

A class can inherit from a parent class to reuse its behaviour. class Child(Parent): gets everything the parent defines; super().__init__() runs the parent's constructor.

Where it is used in this project

Four classes inherit: FeedbackDataset(Dataset), UrgencyTextDataset(Dataset), BiLSTMUrgencyRegressor(nn.Module), TransformerUrgencyRegressor(nn.Module). From nn.Module they get: model(x) calling forward(x), .parameters(), .train()/.eval(), .to(device) and .state_dict(). super().__init__() is mandatory: without it PyTorch would not register the layers. Not inheriting: TextIngestor, WarehouseManager, TextVocabulary. The urgency dispatcher uses a tagged dictionary instead of subclasses.

src/dl_lstm.py · L61 parent constructor

How it helps

PyTorch's training, saving and device logic comes for free.

Theory & history

Overriding a method (such as forward) customises behaviour while the parent's machinery still works. Inheritance also gives polymorphism: code written for the parent works with any child.

See also →Dataset & DataLoaderBiLSTMRegistry & dispatcher pattern

Classes, class attributes & @classmethod #

Class attributes are shared by every use of the class. A @classmethod receives the class itself (cls) instead of an instance (self).

Where it is used in this project

TextIngestor keeps its compiled regexes as class attributes and exposes clean_text, extract_hashtags and extract_mentions as classmethods, so you call TextIngestor.clean_text(…) with no object. TextVocabulary.from_dict is a classmethod that uses cls.__new__(cls) to rebuild an object from saved data without re-running __init__.

How it helps

A stateless utility needs no instance; the patterns are compiled once.

See also →Regular expressionsDecorators

Decorators #

A decorator (@name above a function) wraps the function to add behaviour.

Where it is used in this project

@classmethod (ingestor), @contextmanager (get_connection), @lru_cache (NER and summariser loaders), @st.cache_resource (database handle, model bundles) and @pytest.fixture (tests).

How it helps

Cross-cutting behaviour (caching, resource handling) without cluttering the function body.

See also →Context managersHugging Face pipelines & the lru_cache singletonStreamlit

Closures & nested functions #

A function defined inside another can use the outer function's variables even after the outer one has moved on.

Where it is used in this project

make_loader inside train_bilstm_model uses the surrounding texts, scores and vocab; sep inside stress_test_compare.main uses results; _bounded_lengths uses max_length/min_length. In the Streamlit pages, get_bundle(model_dir, model_type) ignores its arguments and reads the outer variant; they only act as the cache key.

pages/1_Inference_Playground.py · L31–33 arguments are only the cache key

How it helps

Small helpers stay next to the code that needs them.

See also →DecoratorsStreamlit

Comprehensions & generator expressions #

A compact way to build a list, dict or set (or a lazy stream) from a loop with an optional filter.

Where it is used in this project

List with a filter: [w for w, c in counter.most_common(...) if c >= min_freq]. Dict: {w: i for i, w in enumerate(self.itos)} and {k: v.clone() for k, v in state_dict().items()}. Set: the trainable-parameter names in trainable_state_dict. Generator: " ".join(t for t in texts if t and t.strip()).

How it helps

Shorter and clearer than the equivalent loops; generators avoid building a temporary list.

See also →Closures & nested functions

Context managers (with) #

with guarantees set-up and clean-up around a block, even if an error occurs.

Where it is used in this project

get_connection is a generator turned into a context manager with @contextmanager: connect, yield conn, then finally: conn.close(). Other uses: with torch.no_grad():, with open(…), with st.sidebar:, with tempfile.TemporaryDirectory() in tests.

How it helps

Connections cannot leak.

See also →DecoratorsTransaction, commit & rollback

Type hints & default arguments #

def f(n: int = 4000) -> pd.DataFrame documents expected types and supplies a default value. Python does not enforce hints at run time; editors and linters use them.

Where it is used in this project

Used throughout, including the modern union syntax str | Path and str | None. Defaults let callers write generate_naive() or override one setting.

How it helps

Self-documenting signatures with convenient defaults.

See also →Guard clauses & early return

Registry & dispatcher pattern (shared functions) #

Keep configuration in one dictionary and route work through one function, so the rest of the code never needs to know the details.

Where it is used in this project

VARIANTS maps v1_naive, v2_hardened, v3_transformer to their paths, labels and model_type; it is referenced from 12 files. select_variant() is called by 7 pages and returns the chosen entry. load_urgency_model returns a “bundle” dict tagged with "type"; score_urgency branches on that tag. V3 shares V2's db_path and raw_csv but has its own model_dir, which is how “same data, better model” is expressed in code.

How it helps

Adding or changing a variant means editing one dictionary; pages do not change.

Shortcomings & limits

If a variant had an unknown model_type, load_urgency_model would silently treat it as a BiLSTM. Real subclasses would make a third backend easier to add.

See also →Inheritance & super()StreamlitText normalisation

Lazy imports #

Importing a module inside a function so it loads only when needed.

Where it is used in this project

transformers in the NER and summariser loaders and in the regressor constructor; anthropic in _claude_backend; streamlit in _get_anthropic_key; datasets in the fetch script.

How it helps

Fast start-up, and optional dependencies (an API key, the dataset library) are only required if used.

Shortcomings & limits

A missing library shows up late, at the first call.

See also →Hugging Face pipelines & the lru_cache singletonPluggable backend & graceful degradation

Guard clauses & early return #

Handle the bad or trivial case first and return immediately, so the main logic stays un-nested.

Where it is used in this project

clean_text returns "" for non-strings or blanks; extract_entities returns [] for empty text; summarize_batch returns "" for no input; select_best_k returns the first k if the line has zero length.

How it helps

Prevents crashes on missing values such as NaN from pandas.

See also →Type hints & default arguments

Streamlit (rerun model, session state, caching) #

A Python framework that turns a script into a web app. Every interaction re-runs the script top to bottom; widgets return their current value.

Where it is used in this project

st.session_state remembers the chosen variant across pages; @st.cache_resource keeps one database handle or model bundle alive across reruns; st.stop() ends a page early; files in pages/ become sidebar pages in number order. app.py and six pages form the dashboard.

How it helps

A full interactive dashboard in pure Python.

Shortcomings & limits

Uncached work repeats on every interaction (for example PCA on the Clustering page).

See also →DecoratorsRegistry & dispatcher patternClosures & nested functions

pytest: fixtures, monkeypatch & mocks #

A testing framework. A fixture prepares a fresh resource; monkeypatch temporarily replaces things; a mock is a fake that records how it was called.

Where it is used in this project

31 tests across seven files. db fixture builds a throw-away database in a temp folder; monkeypatch.setattr points MODELS_DIR at a temp folder and delenv removes the API key; MagicMock with assert_called_once_with proves the dispatcher sends cleaned text to the BiLSTM and raw text to V3. pytest.raises checks failure paths. Rule used there: patch where the name is looked up (src.urgency), not where it was defined.

How it helps

Regression protection: for example the Streamlit colour test exists because “yellow” once silently broke rendering.

See also →DecoratorsACID

Docker (containerised deployment) #

Packages the app and its exact environment into an image that runs the same everywhere.

Where it is used in this project

The Dockerfile starts from python:3.11-slim, copies requirements.txt before the code (so the slow dependency layer is cached), installs with --no-cache-dir, exposes 8501, adds a HEALTHCHECK and starts streamlit run app.py. requirements.txt pins version ranges (for example torch>=2.2,<2.6).

Dockerfile · L9 dependency layer first

How it helps

Reproducible builds; layer caching makes rebuilds quick.

Theory & history

History. Docker was released in 2013.

See also →Streamlit

pandas: DataFrame, masks, apply, groupby #

The standard Python table library.

Where it is used in this project

df["raw_text"].apply(TextIngestor.clean_text) (function passed without parentheses); a boolean mask df[df["cleaned_text"].str.len() > 0] to drop empties; value_counts, resample("6h") and drop_duplicates in the dashboard; pd.crosstab for the chi-squared table; groupby(…).apply(lambda g: g.sample(…)) for stratified samples; pd.read_sql_query to load the warehouse.

How it helps

Vectorised operations replace explicit loops.

See also →Comprehensions & generator expressionsSQL & SQLite
No concept matches your search.