Classify the whole queue with Jev

One ticket and a confident label make a pleasant demo. A queue makes the tradeoffs visible.

Imagine opening a batch of support requests. Categories appear with counts. One category contains straightforward refund requests; another mixes failed transfers with transfers that are merely pending. A review queue contains the uncertain cases. Clicking any count reveals the records behind it.

Now move the automation threshold. Some records leave the automatic queue. The interface updates coverage and the observed accuracy on labeled records. You can see what the system gives up to become more selective.

This is the categorization experience we would build around Jev.

Start with a dataset that can contradict you

For a public first version, BANKING77 provides 13,083 queries across 77 banking intents, with separate training and test sets and a CC BY 4.0 license. Keep attribution and pin the downloaded revision.

Its labels give the demo an external reference. They do not supply urgency, frustration, fraud, or customer-value labels. If those columns appear in your product, they need separate evidence and evaluation.

Keep model fitting, calibration, policy selection, and final evaluation separate. A dataset used repeatedly to improve the demo is development data, even if its original filename says test.

Declare the taxonomy before inference

Version the category IDs and their descriptions. Distinguish a pending transfer from a failed transfer using definitions an annotator can apply. Avoid categories that overlap unless the task explicitly allows multiple labels.

For exclusive categories, inspect a Choice distribution. For independent tags, ask separate questions and evaluate each tag separately. Their probabilities do not need to add to one.

Large taxonomies deserve an explicit experiment. Compare asking over every category with retrieving a short candidate list and scoring those candidates. The second design introduces a new failure: the correct label might never reach the classifier. Report candidate recall before reporting reranker accuracy.

Keep the unit of evidence visible

Every record should carry its input ID, taxonomy version, responding model, classification result, and any truncation or failure. Add the reference label only in the evaluation view; do not accidentally include it in model state.

The interface needs four connected views:

• Category totals, including a review count.
• A record list filtered by category or review reason.
• A record inspector with the original text and leading alternatives.
• A policy view showing coverage and observed errors at each threshold.

Use horizontal bars when counts are the point. A cluster cloud can obscure the distinction between nearby text embeddings and actual category membership.

Make the threshold honest

Let the denominator remain visible. “97% accurate” over 30 automatically routed records tells a different story from the same percentage over 3,000.

For a selected threshold, calculate coverage over all submitted records, including failures. Calculate accuracy only where reference labels exist. If there are no automatically routed labeled records, show no estimate rather than 100%.

The demo slider explores a recorded run. It does not validate a new production threshold. Select the shipping policy on separate data and evaluate it under expected traffic conditions.

A new topic can receive a confident wrong label. Treat unsupported inputs as a separate evaluation problem. Holding out entire intents and collecting genuinely unrelated requests can reveal failure modes that random train/test splits miss.

Let a correction become useful data

When a reviewer changes a category, retain both the prediction and the correction. Record the taxonomy version and the reviewer action. A visible membership update is immediate; a model update is not.

Later, collect corrections into a new training revision and test the resulting model against an unchanged holdout. Sample some automatically accepted records for review too. Otherwise the feedback dataset only describes the cases your current policy already finds difficult.

The first release can serve a recorded, reproducible inference run. Label it recorded. When live inference is added, edited input must receive a fresh result or an explicit failure—not a recycled example distribution.

That turns a model demonstration into something an operations team can evaluate: a queue, a policy, and the records that show whether either is working.

Proposed workflow. No Jev accuracy or latency measurement is claimed here. See the implementation plan for the local build sequence.