Model changelog
What each version of the OpenTextShield mBERT classifier brought, and the numbers behind it. The model version is what /health reports; the API has its own version.
Model 2.9
CurrentSeptember 2026Learned the labeling rules from a strong teacher model instead of from a small hand-picked subset, and stopped blocking legitimate branded notices. On a 100-message head-to-head it matches the teacher at 96 to 98% while staying self-hosted and under 100 ms.
Highlights
- Real notices get through. Bank fraud checks, one-time codes, delivery updates, bills and account alerts from real senders are no longer rejected. On a set of 40 legitimate notices that look like scams, 2.7 blocked 13; 2.9 blocks none.
- Social engineering without a link is caught. "Hi Mum, new number", "send me the 6-digit code", fake order confirmations with a call-back number, and account-block texts with the link stripped out.
- Prize, job and delivery lures are labelled phishing, as the labeling guide defines them, instead of spam. Both are still rejected; reports keyed on category will show phishing rising and spam falling.
- Confidence means something. 2.7's mistakes were made at 0.99 confidence. 2.9's cluster at low confidence, so a gateway threshold can act on them.
- 14 languages of branded notices in training, with real domains and routine calls to action, each paired with its look-alike lure.
Numbers
| Set | 2.7 | 2.9 |
|---|---|---|
| Hand-written bench, accuracy (100) | 83.0% | 96.7% |
| Hand-written bench, legitimate messages blocked | 17.6% | 1.0% |
| Hard legitimate notices blocked (40) | 32.5% | 1.7% |
| Teacher-labelled held-out corpus, accuracy (5,660) | 80.1% | 98.8% |
| Mishra & Soni, accuracy (5,971) | 89.4% | 94.7% |
| IMC25 smishing, block rate (8,005) | 72.4% | 71.5% |
| UCI SMS Spam, 3-class accuracy (5,574) | 99.5% | 92.9% |
Three training seeds, mean shown. The UCI drop is by design: 308 of its 747 spam rows are prize lures the guide calls phishing; block accuracy on UCI stays at 98%. Full write-up in evals/results/DISTILL_2.9.md.
How it was built
- TypeSafe (jev-1.13.0) read the labeling guide and answered for 154k unique texts: the whole 137k corpus plus generated sets. Training used its label and its class probabilities (knowledge distillation) on 134,710 rows.
- 12,000 generated legitimate notices and 10,000 lure twins, kept only where the teacher agreed (96%).
- Fine-tuned from the 2.5 weights, 2 epochs, three seeds; the shipped checkpoint is seed 13.
Still open
- Wrong-number and "is this still your number?" openers with no ask in the first message. Neither 2.9 nor the teacher catches those from one message.
- Some 2004-era premium-rate receipts labelled spam in old public sets now pass.
Model 2.7
July 2026Taught the model modern smishing. 2.5 was excellent at classic 2010-era spam but let most current attacks through; 2.7 fine-tuned it on a targeted synthetic set of today's scam archetypes and their benign twins, and beat 2.5 on every benchmark without losing the classic-spam strength.
Highlights
- Modern attack archetypes covered: toll and delivery-fee scams, bank-lock, government refunds, family impersonation, vishing call-backs, gift-card requests, crypto-wallet scares, OTP-forwarding theft, sextortion, and four obfuscation styles.
- 15 languages with locale-correct brands, currencies and courier names.
- Hard negatives by construction: every attack shipped with a benign twin (real OTP vs OTP theft, real fraud alert vs fake "verify now") so the model learned the decision boundary rather than new keywords.
- A gentle recipe on purpose: plain loss, low learning rate, 2 epochs, with a rehearsal sample of the old corpus. Heavier runs scored better on validation but regressed on real-world smishing.
Numbers
| Set | 2.5 | 2.7 |
|---|---|---|
| Adversarial suite, block rate (127) | 75.6% | 96.9% |
| Adversarial suite, phishing recall | 11.9% | 80.6% |
| IMC25 smishing, block rate (8,007) | 69.9% | 72.4% |
| IMC25 smishing, phishing recall | 17.1% | 45.7% |
| UCI SMS Spam, accuracy (5,574) | 99.3% | 99.5% |
| Mishra & Soni, block accuracy (5,971) | 99.2% | 99.3% |
Method and caveats in evals/REPORT.md. A 2.8 candidate followed with a written labeling guide and a cleaned training subset; it was not released, and its work became the foundation for 2.9.
Model 2.5
October 2025The long-running production model and the base every later version fine-tunes from. Trained on the combined v2.4 corpus of about 137,000 messages across the supported languages, with a parameter-efficient (LoRA) incremental pass.
Highlights
- Classic spam essentially solved: 99.3% on the UCI SMS Spam collection, 99.2% block accuracy on Mishra & Soni.
- Three-class output (ham, spam, phishing) on a multilingual BERT base with a 96-token window, the shape the API still serves today.
- Known limit that motivated 2.7: phishing collapsed into spam. Only 2.7% of Mishra & Soni's smishing was labelled phishing, and modern attacks in IMC25 were blocked 69.9% of the time.
Earlier
2024The versions that took the model from English-only to multilingual.
- 2.1 · May 2024
- Tamil and Sinhala for Sri Lanka; better handling of non-Latin scripts and fewer false positives in multilingual traffic.
- 1.9 · Apr 2024
- German and Italian datasets; five languages in training, stronger cross-lingual transfer.
- 1.8 · Apr 2024
- Initial German support, compound-word tokenization, better non-ASCII handling.
- 1.7.6 · Apr 2024
- URL-based phishing detection, including shortened and look-alike links.
- 1.7.5 · Apr 2024
- Full Indonesian dataset.
- 1.7 · Apr 2024
- Arabic and initial Indonesian; right-to-left script support.
- 1.6 · Apr 2024
- Multilingual framework on the cross-lingual BERT base.
- 1.5 · Apr 2024
- First release: English SMS spam and phishing detection, ham/spam/phishing output, Apple Silicon support.