Where do we start with data classification? Part 4: Teaching Purview what matters with trainable classifiers

Part 4 - Teaching Purview what matters with trainable classifiers

Part 3 turned the agent’s patterns into sensitive information types. Some findings had no pattern at all: a patent template, an R&D expense report. They are defined by what they are, not by a string of characters. That is the job of a trainable classifier, which learns to recognise a category of content from examples.

Start with the built-in classifiers

Microsoft provides pre-trained classifiers that need no training. The Contoso tenant has 79, and they had already classified content without anyone configuring anything: Finance 134 items, HR 52, Budget 20, Agreements 7, IT and Tax 5 each, Statement of Work 3 and IP 2.

Built-in trainable classifiers already sorting Contoso content
Built-in trainable classifiers already sorting Contoso content

This is the quickest win in Purview. Before writing a single custom rule, an organisation can see roughly how much finance, HR and contract content it holds and where it lives.

Now look at IP: 2 items. Contoso is an engineering company with a product portfolio. Either it holds almost no intellectual property, or the generic IP classifier does not recognise what Contoso’s IP looks like. It is the second. Generic classifiers are trained on generic examples.

Choosing the right classifier type

The four classifier types side by side
The four classifier types side by side
  • Sensitive info type - patterns and keywords, for data with a recognisable shape (Part 3).
  • Exact data match - exact values from a structured source, such as a customer list.
  • Document fingerprinting - documents based on a standard form or template.
  • Trainable classifier - categories of content learned from examples.

Contoso’s engineering design specifications fit the last category. Each one is different, written by different engineers for different products, but a person recognises one instantly.

Building Contoso Engineering Design Specifications

Prepare the seed content

What to prepare before creating a trainable classifier
What to prepare before creating a trainable classifier
  • At least 50 and up to 500 positive samples, and at least 150 and up to 1,500 negative samples.
  • Text-based files only. Images and image-only PDFs cannot be used.
  • A SharePoint site such as a communication site, with positive and negative content in separate folders used only for seed content.
  • At least an hour for a new site or folder to be indexed before you create the classifier.
  • English only for custom classifiers, and only items that are not encrypted.

The lab uses 60 fictional design specifications and 160 negatives, uploaded to a communication site called Classifier Seed Content. The generator (make_seed.py) and an upload script are in the GitHub repository for this series, so you can recreate the same seed content in your own test tenant.

Positive seed content in a dedicated SharePoint folder
Positive seed content in a dedicated SharePoint folder

The negative set matters as much as the positive set. Alongside meeting notes, HR policies and newsletters, I added hard negatives: quick start guides and marketing pages for the same products named in the specifications. Without them, the classifier can learn the shortcut “mentions a Contoso product” instead of “is a design specification”.

In a real organisation, ask the engineering lead for genuine specifications and take negatives from the same team’s non-confidential content. Negatives from the same team teach far more than random documents from elsewhere.

Gotcha: the communication site never appeared in the Posture Agent’s data source picker. The picker searches people and groups, and a communication site has no Microsoft 365 group.

Create and train

Naming the new trainable classifier
Naming the new trainable classifier

The wizard has four steps: name, positive sample folder, negative sample folder, and review. The wizard estimated 5 to 7 hours of training, and Microsoft says 24 hours or less.

Reality was slower. Seventeen hours after creation, the classifier was still showing Training in the unified Classifiers page.

Still Training 17 hours later, alongside the new SITs from Part 3
Still Training 17 hours later, alongside the new SITs from Part 3

Plan for at least a day between creating a classifier and reviewing it. Do not book the business owner’s review session for the same afternoon.

Review before publishing

When training finishes, Purview tests the classifier and asks you to review its predictions as correct, incorrect or uncertain. Only publish when you are satisfied. Retraining a published custom classifier is not supported; if it needs improving, delete it and start again with more samples.

By default only the creator can review the predictions. Decide who owns that review before you start: someone who knows the content, which usually means the business rather than IT.

The results: 96% accuracy

Training finished about 23 hours after creation, for just 220 short text files: 60 positive and 160 negative. The accuracy score was 96%: Purview tested the classifier against items from the seed folders and got all but four right.

Microsoft now quotes far shorter training times than it used to: the portal’s create tile estimates 5 to 7 hours. Our lab took 23 hours, even with a small data set. That is not a problem in itself. Training a classifier is rarely time-sensitive for an organisation, and real seed sets are usually much larger. Just plan for a day rather than an afternoon.

Training complete: 96% accuracy and Ready to publish
Training complete: 96% accuracy and Ready to publish

Gotcha: two pages, two different answers

The new unified Classifiers page (preview) still showed the classifier as Training after it had finished. The classic Trainable classifiers page showed the truth: 95.7%, Ready to publish.

The unified Classifiers page still says Training
The unified Classifiers page still says Training
The classic page: 95.7% and Ready to publish
The classic page: 95.7% and Ready to publish

If a classifier looks stuck in Training, check the classic Trainable classifiers page before you wait any longer.

Read the misses, not just the score

Review test results lists every item the classifier got wrong. There were no false positives: no negative document, including the hard negatives, was mistaken for a specification. There were four false negatives, all genuine specifications the classifier missed.

Four false negatives, three of them for the same product
Four false negatives, three of them for the same product

Three of the four were specifications for the same product, the Northwind heat exchanger. A score hides that kind of pattern; the list shows it. In a real organisation, that is the moment to ask whether one product line writes its specifications differently and needs more examples, before anyone relies on the classifier for it.

No false positives and a few false negatives is the right way round to start. A classifier that misses some specifications is safer to put into a simulation policy than one that flags newsletters.

Publish

Publish for use makes the classifier available to Information Protection, Data Lifecycle Management and Communication Compliance, and to DLP conditions.

Published and ready to use in policies
Published and ready to use in policies

What comes next

A classifier on its own protects nothing; the Affected policies column from Part 1 made that clear. Its value comes from what you connect it to: DLP, sensitivity label auto-labelling and retention. Part 5 adds it to the DLP policy alongside the SITs from Part 3.

References

4 thoughts on “Where do we start with data classification? Part 4: Teaching Purview what matters with trainable classifiers”

  1. Pingback: Where do we start with data classification? Part 5: From classification to protection | Cloud-Secure.ai

  2. Pingback: Where do we start with data classification? Part 1: Let the data tell you | Cloud-Secure.ai

  3. Pingback: Where do we start with data classification? Part 2: Getting the Posture Agent running | Cloud-Secure.ai

  4. Pingback: Where do we start with data classification? Part 3: From discovery to definition | Cloud-Secure.ai

Leave a Reply