# Categorization lab: implementation and compute plan Status: build specification. No new model has been trained or deployed by writing this document. ## First deliverable: a reproducible measured batch Start with BANKING77: 10,003 training examples, 3,080 test examples, 77 intents, CC BY 4.0. Preserve attribution and source revision/checksums. Do not present the data as live bank traffic. 1. Pin dataset revision; validate rows, labels, duplicate texts and split overlap. Keep the official test labels out of model selection. Document any exclusions. 2. Split the official training set reproducibly into fitting, model-selection, probability-calibration, and policy-selection subsets. Keep identical texts together; preserve per-class counts where possible. Avoid using the same examples to choose the model, calibrate scores, select a policy, and report its success. 3. Run a simple text-classification baseline, then a fine-tuned ModernBERT classifier. A lexical baseline establishes whether a complex model adds value. Our existing gate heads cannot substitute for an intent head. 4. Compare zero-shot Laya separately. Audit option truncation, label-order sensitivity, and actual encoded option lengths. Fine-tuned Laya belongs in a separate matched-training row. Jev comparison requires actual API access and the same examples and taxonomy. 5. Fit temperature scaling on calibration logits only. Select a review policy on its own split; report risk/coverage and uncertainty. A threshold on maximum probability is a policy input, not a universal correctness guarantee. 6. Freeze the artifact, model identity, preprocessing, taxonomy, calibration, and policy. Run the official test once for the release report. If repeatedly used to tune the demo, relabel it development data and obtain a new final holdout. 7. Export every prediction and the evaluation manifest. The public experience starts with this recorded run, so it stays fast and usable even while a Spark is offline. Acceptance: per-class metrics, macro-F1, accuracy, NLL, multiclass Brier score, documented ECE binning, selective accuracy and coverage with numerator/denominator, and error examples are reproducible from exported rows. No target accuracy or millisecond claim before measurement. ## Then support live input Use the same frozen backend for pasted text or an authorized batch. Return actual inference status; never substitute recorded scores for edited text. Report network-inclusive latency separately from warm model latency. Test malformed inputs, empty text, oversized records, unsupported languages, rate limiting, timeouts, and backend unavailability. For an input unsupported by the trained taxonomy, return a review recommendation. Before marketing unknown detection, measure it on a distinct out-of-domain set and on withheld intents. Low softmax confidence is only one weak signal. ## Model design Reuse the serving/validation structure, not the gate's learned heads: normalized record -> shared encoder -> intent logits -> optional independent tag logits -> optional ordinal rubric logits For 77 stable intents, a supervised classification head is a practical first model. The taxonomy is bound to the artifact: adding a new intent requires a new fitted head or an explicitly different open-vocabulary backend. Renaming labels must not silently change semantics. Later, test reusable state encoding with option-conditioned scoring for dynamic taxonomies. Compare it with the fixed-head baseline using the same backbone, precision, hardware, length distributions, and batch sizes. Learned pooling for event sequences is a separate task requiring sequence labels and temporal split discipline. Use cross-entropy/BCE for the corresponding supervised tasks before experimenting with RL. Neither RL nor a proper scoring rule automatically gives calibrated out-of-domain probabilities. ## The length budget Reusable state encoding is cheap in the number of questions and not in the length of the state, and the second half of that decides how event segmentation is built. Measured on one GB10 (`research/architecture-2026-09-20/`): one encoding answering thirty-two questions costs six per cent more than answering one, so a long question set is close to free; but one encoding of a 8,192-token state costs 297 ms on ModernBERT-base and 535 ms on ModernBERT-large, against 8.9 ms and 18.8 ms for a 256-token one. Attention is superlinear, and a single long sequence pays for it. A support query is a sentence and never meets this. A session of events, a transcript or a contract does. So: - Chunk a long input and answer per chunk, rather than encoding it whole. Each call then sits in the region where the shape pays, and throughput is a batch of short sequences rather than one long one. - A chunk boundary is a modelling decision, not a buffer size: it decides what evidence a question can see. Record the chunking with the result, and evaluate it, because two chunkings of one document are two different tasks. - Where a decision genuinely needs the whole document at once, expect hundreds of milliseconds per call and size the batch for throughput rather than latency. Do not quote a short-input latency for a long-input workload. - The same measurement bounds the comparison rows: a per-question shape re-reads the state per question, so at 8,192 tokens it feeds 65,536 and costs seconds. That gap is real but it is not an argument for encoding documents whole. ## FOLD's role FOLD can serialize repeated structured events compactly and cache stable input representations. Keep canonical JSON as the interchange/reference representation initially. Benchmark JSON and FOLD with the actual tokenizer, task quality, truncation, and serialization overhead. Fewer bytes does not guarantee fewer tokens. FOLD supplies neither semantic labels nor calibrated confidence. ## Cloudflare architecture browser -> static lab + recorded public results -> Worker API -> authenticated inference gateway -> Spark model -> D1 metadata / labels / jobs -> R2 large inputs and result artifacts - Keep model weights and PyTorch inference on a Spark. The existing Worker plan is not a GPU hosting plan. - Static recorded results need no inference calls. Cache immutable assets by artifact hash. - D1 stores job status, taxonomy versions, review labels, ownership, and result pointers; R2 stores bulk records and distributions. - A queue can handle batch job dispatch. Start with one serial worker and bounded batches; add orchestration only when failure recovery and volume require it. - Do not expose a Spark or an unrestricted model endpoint publicly. Authenticate the gateway and enforce request limits at the Worker. Avoid accepting arbitrary fetch URLs as batch input. - Uploaded records need tenant-scoped access and deletion/retention rules. An anonymous public demo should use approved public data and short-lived pasted input, with analytics excluded from text fields. - Keep the waitlist database/schema separate from experimental classification data. Do not store per-record classification payloads in analytics. Cloudflare billing depends on usage and enabled services. Model training/inference uses local power and hardware capacity; “self-hosted” is not zero cost. Verify current Cloudflare limits before provisioning additional services. ## Compute decision Use existing Sparks first, after checking current workloads. One hosts a warmed model; the second can train/evaluate when available. This is a proposed allocation, not a claim that both are idle or newly configured. Measure warm/cold p50/p95, batch throughput, tokenization, device transfer, peak memory, and end-to-end API latency. If training or serving misses the required turnaround, rent a GPU for a bounded experiment with a cost cap. More compute does not fix missing labels, poor task definition, or bad evaluation splits. ## Rollout order 1. Reproducible batch benchmark and recorded results explorer. 2. Actual live inference for the same taxonomy, with explicit offline/error states. 3. Transparent cohort predicates and review correction workflow. 4. Drift views and discovery suggestions, evaluated on suitable time-series data. 5. Customer-specific adaptation, calibrated policy, shadow run, then opt-in operational actions. Each phase should ship evidence of the specific capability. The first phase does not establish real customer segmentation or production business impact.