
Part 3 turned the agent’s patterns into sensitive information types. Some findings had no pattern at all: a patent template, an R&D expense report. They are defined by what they are, not by a string of characters. That is the job of a trainable classifier, which learns to recognise a category of content from examples.
- Part 1 - Let the data tell you: what Purview already sees, and why the Posture Agent changes the starting point
- Part 2 - Getting the Posture Agent running: prerequisites, gotchas and what it actually cost
- Part 3 - From discovery to definition: using the agent to find the sensitive information types you did not know you needed
- Part 4 - Teaching Purview what matters: trainable classifiers (this post)
- Part 5 - From classification to protection: labels, DLP and the evidence trail
Start with the built-in classifiers
Microsoft provides pre-trained classifiers that need no training. The Contoso tenant has 79, and they had already classified content without anyone configuring anything: Finance 134 items, HR 52, Budget 20, Agreements 7, IT and Tax 5 each, Statement of Work 3 and IP 2.

This is the quickest win in Purview. Before writing a single custom rule, an organisation can see roughly how much finance, HR and contract content it holds and where it lives.
Now look at IP: 2 items. Contoso is an engineering company with a product portfolio. Either it holds almost no intellectual property, or the generic IP classifier does not recognise what Contoso’s IP looks like. It is the second. Generic classifiers are trained on generic examples.
Choosing the right classifier type

- Sensitive info type - patterns and keywords, for data with a recognisable shape (Part 3).
- Exact data match - exact values from a structured source, such as a customer list.
- Document fingerprinting - documents based on a standard form or template.
- Trainable classifier - categories of content learned from examples.
Contoso’s engineering design specifications fit the last category. Each one is different, written by different engineers for different products, but a person recognises one instantly.
Building Contoso Engineering Design Specifications
Prepare the seed content

- At least 50 and up to 500 positive samples, and at least 150 and up to 1,500 negative samples.
- Text-based files only. Images and image-only PDFs cannot be used.
- A SharePoint site such as a communication site, with positive and negative content in separate folders used only for seed content.
- At least an hour for a new site or folder to be indexed before you create the classifier.
- English only for custom classifiers, and only items that are not encrypted.
The lab uses 60 fictional design specifications and 160 negatives, uploaded to a communication site called Classifier Seed Content. The generator (make_seed.py) and an upload script are in the GitHub repository for this series, so you can recreate the same seed content in your own test tenant.

The negative set matters as much as the positive set. Alongside meeting notes, HR policies and newsletters, I added hard negatives: quick start guides and marketing pages for the same products named in the specifications. Without them, the classifier can learn the shortcut “mentions a Contoso product” instead of “is a design specification”.
In a real organisation, ask the engineering lead for genuine specifications and take negatives from the same team’s non-confidential content. Negatives from the same team teach far more than random documents from elsewhere.
Gotcha: the communication site never appeared in the Posture Agent’s data source picker. The picker searches people and groups, and a communication site has no Microsoft 365 group.
Create and train

The wizard has four steps: name, positive sample folder, negative sample folder, and review. The wizard estimated 5 to 7 hours of training, and Microsoft says 24 hours or less.
Reality was slower. Seventeen hours after creation, the classifier was still showing Training in the unified Classifiers page.

Plan for at least a day between creating a classifier and reviewing it. Do not book the business owner’s review session for the same afternoon.
Review before publishing
When training finishes, Purview tests the classifier and asks you to review its predictions as correct, incorrect or uncertain. Only publish when you are satisfied. Retraining a published custom classifier is not supported; if it needs improving, delete it and start again with more samples.
By default only the creator can review the predictions. Decide who owns that review before you start: someone who knows the content, which usually means the business rather than IT.
The results: 96% accuracy
Training finished about 23 hours after creation, for just 220 short text files: 60 positive and 160 negative. The accuracy score was 96%: Purview tested the classifier against items from the seed folders and got all but four right.
Microsoft now quotes far shorter training times than it used to: the portal’s create tile estimates 5 to 7 hours. Our lab took 23 hours, even with a small data set. That is not a problem in itself. Training a classifier is rarely time-sensitive for an organisation, and real seed sets are usually much larger. Just plan for a day rather than an afternoon.

Gotcha: two pages, two different answers
The new unified Classifiers page (preview) still showed the classifier as Training after it had finished. The classic Trainable classifiers page showed the truth: 95.7%, Ready to publish.


If a classifier looks stuck in Training, check the classic Trainable classifiers page before you wait any longer.
Read the misses, not just the score
Review test results lists every item the classifier got wrong. There were no false positives: no negative document, including the hard negatives, was mistaken for a specification. There were four false negatives, all genuine specifications the classifier missed.

Three of the four were specifications for the same product, the Northwind heat exchanger. A score hides that kind of pattern; the list shows it. In a real organisation, that is the moment to ask whether one product line writes its specifications differently and needs more examples, before anyone relies on the classifier for it.
No false positives and a few false negatives is the right way round to start. A classifier that misses some specifications is safer to put into a simulation policy than one that flags newsletters.
Publish
Publish for use makes the classifier available to Information Protection, Data Lifecycle Management and Communication Compliance, and to DLP conditions.

What comes next
A classifier on its own protects nothing; the Affected policies column from Part 1 made that clear. Its value comes from what you connect it to: DLP, sensitivity label auto-labelling and retention. Part 5 adds it to the DLP policy alongside the SITs from Part 3.
References
- Learn about trainable classifiers - learn.microsoft.com/purview/trainable-classifiers-learn-about
- Get started with trainable classifiers - learn.microsoft.com/purview/trainable-classifiers-get-started-with
- Trainable classifier definitions - learn.microsoft.com/purview/trainable-classifiers-definitions
Pingback: Where do we start with data classification? Part 5: From classification to protection | Cloud-Secure.ai
Pingback: Where do we start with data classification? Part 1: Let the data tell you | Cloud-Secure.ai
Pingback: Where do we start with data classification? Part 2: Getting the Posture Agent running | Cloud-Secure.ai
Pingback: Where do we start with data classification? Part 3: From discovery to definition | Cloud-Secure.ai