Where do we start with data classification? Part 1: Let the data tell you

Part 1 - Let the data tell you

“Where do we start?” It is the question I get asked in nearly every Microsoft Purview engagement. Purview has a huge toolset - sensitivity labels, data loss prevention, insider risk, retention, Data Security Posture Management - but none of it works until the organisation has told Purview what it needs to protect. Purview does not know that a design specification is Contoso’s crown jewels. Contoso has to tell it.

That is classification, and it is where most projects stall. Workshops ask the business to list its sensitive data, the business lists what it can remember, and the result is a handful of sensitive information types (SITs) that miss most of what matters.

This series takes a different route. Instead of asking people what they think is sensitive, we ask the data. I use a Microsoft CDX demo tenant, Microsoft Security Copilot and the Data Security Posture Agent (preview) to discover what Contoso actually holds, and then turn those findings into classifiers. The series runs in five short parts:

All data in this series is fictional and lives in a Microsoft CDX demo tenant. Every script and template is published on GitHub at https://github.com/cloud-secure-ai/Purview-data-classification

A word you will see a lot in this series: gotcha, short for “got you”. A gotcha is something to watch out for that is not documented on learn.microsoft.com or anywhere else, and that you usually only find by hitting it.

Start with what Purview already sees

Before building anything, look at what Purview has already found. Out of the box, Purview scans Microsoft 365 content against hundreds of built-in SITs and Microsoft’s pre-trained classifiers. In the Contoso tenant the Purview home page reported 1,866 items containing sensitive information. The top three types were All Full Names (301 items), a custom SIT called Proseware Merger (152 items) and EU Tax Identification Number (130 items).

The second most common sensitive type in this tenant is not a regulatory identifier. It is a custom type someone built to find a confidential merger. That is intellectual property, and built-in rules will never find it on their own.

The unified Classifiers page

Microsoft is rolling out a unified Classifiers page (preview) under Information Protection > Classifiers. It brings SITs, exact data match, trainable classifiers and document fingerprinting into one list, and runs alongside the classic pages.

The unified Classifiers page in Microsoft Purview (preview)
The unified Classifiers page in Microsoft Purview (preview)

The most useful change is two columns side by side: Classified items and Affected policies. Filter to trainable classifiers and custom SITs and the gap is obvious. Finance matched 134 items, HR 52. The custom Proseware Merger and Mark 8 Project types matched 152 and 101 items. Every one shows 0 affected policies.

Sensitive content found, no policy using it
Sensitive content found, no policy using it

That single view is a ready-made gap analysis: Purview has found the data and nothing is protecting it. It is also the limit of what you get for free. These columns only count what an existing classifier already recognises. Anything Contoso has never defined does not appear at all.

Why starting from SITs is backwards

Most guidance on the Posture Agent shows it searching for a SIT you already have: “find files containing credit card numbers that are shared externally”. That is useful for remediation, but it does not answer “where do we start?”. If you already knew your SITs, you would not be asking.

I learned this the hard way in the lab. My first prompt asked the agent to find unlabelled Mark 8 confidential data. It ran successfully and found 0 items. The prompt was turned into a literal keyword query, limited to the default 30-day window, over content that did not use those words. Searching for what you think exists finds nothing new.

The real value is the other way round. Describe the kind of content that would hurt the organisation if it leaked, let the agent find it by meaning, and then look for the patterns that repeat. Those patterns are your candidate SITs.

The Data Security Posture Agent (preview)

The Posture Agent is a Security Copilot agent in Microsoft Purview. It has two capabilities in two places:

  • Data discovery, in DSPM > Discover > Asset explorer > Posture Agent tab. You describe what you are looking for in natural language, choose the people, groups or sites to search, and the agent analyses files, emails and messages for that content.
  • Credential scanning, in Data Security Investigations > Posture agent. It scans selected locations for exposed passwords, keys and tokens and tracks each scan on a task board.
Data discovery lives in the new unified DSPM, under Asset explorer
Data discovery lives in the new unified DSPM, under Asset explorer

Note the banner: DSPM (classic) and DSPM for AI (classic) retire on 30 November 2026, and the agent only lives in the new unified DSPM. If you have been waiting to move, this is the push.

Things worth knowing before you plan around it:

  • It finds and reports. It does not act on its own. You can label selected files from the results, export a report or schedule the prompt to run daily, weekly or monthly.
  • Only two prompts can run at once, and a run may be capped at 5 GB or 10,000 items, returning the top 1,000 matches. It is a fast, targeted discovery tool, not a full-tenant inventory.
  • The default time range is the last 30 days. Ask for a wider range in the prompt if your content is older.
  • It needs Security Copilot capacity, and every run consumes it. Part 2 covers what that took and cost.

The agent is a non-human identity

Expand Permissions and roles in the setup dialogue and you will see that starting the agent creates its own identity, with permission to access, read and see metadata for documents and content.

The Posture Agent runs under its own identity with read access to content
The Posture Agent runs under its own identity with read access to content

That is a non-human identity with broad read access to your most sensitive data. Treat it like one: know who deployed it, review it with your other agent identities and make sure its activity is in your audit process. In the lab, the credential scanning task card even reads “Started by Data security posture agent” - the agent, not the admin, is the actor.

Role design matters too. Compliance Administrator and Security Reader can run the agent and see summaries. Viewing matched snippets or opening matched files needs content-level roles such as Data Classification Content Viewer or Purview Content Analyst. Our audit log showed exactly that split: the admin was granted the compliance roles and blocked on the content roles.

What we found

Once the prerequisites were in place (Part 2), one discovery-style prompt over a single user’s mailbox and OneDrive returned 20 items, none of them labelled:

  • A document named CR-227 Product Overview - a reference number format that no SIT covered.
  • Project Tailspin and Customer Purchasing Data for Project 9 - internal codenames that no SIT covered.
  • Contoso Patent Template and an R&D expense report - content defined by what it is rather than a pattern, which is a job for a trainable classifier.
  • An annual financial report draft, already covered by the built-in Finance classifier.
  • Noise: Power BI visual definitions, Outlook task reminders and meeting titles.

None of the useful findings were on anyone’s list. That is the point. Part 3 shows how each one becomes a classifier, and how to deal with the noise.

So where do we start?

  1. Look at what Purview already sees. The unified Classifiers page shows classified items against affected policies. That is your first gap list.
  2. Do not start by asking the agent about the SITs you already have. Ask it about the kind of content that would hurt you if it leaked.
  3. Treat the agent as a non-human identity from day one.
  4. Get the prerequisites right before you pay for capacity. That is Part 2.

References

Where do we start with data classification? Part 2: Getting the Posture Agent running

Part 2 - Getting the Posture Agent running

In Part 1 I argued that the Data Security Posture Agent changes where classification starts. This post is the honest version of getting it running. It took two evenings in a CDX tenant, a Pay-As-You-Go Azure subscription and a PowerShell session, and almost every step had a catch. Here they are, in the order you will hit them, with what it cost at the end. Where you see “gotcha”, it means something to watch out for that is not documented on learn.microsoft.com or anywhere else.

The checklist

If you only read one section, read this one. Do these in order, and do the first one days before you need it:

  1. Turn on Purview Audit and confirm records are flowing.
  2. Get Security Copilot capacity, either through the Microsoft 365 E5 inclusion or by provisioning it in Azure.
  3. Allow Security Copilot to access Microsoft 365 data.
  4. Complete the unified DSPM setup, reviewing the pay-as-you-go collection policies before you click Start setup.
  5. Assign roles, including content-level roles for whoever needs to see matched content.
  6. Write your prompts and pick your data sources before you create any hourly capacity.
  7. Decide which Purview pay-as-you-go features you need, such as Data Security Investigations, and link an Azure subscription for Purview billing. These are billed separately from Security Copilot and E5 does not cover them.

1. Audit first, and days ahead

Purview Audit was off in the CDX tenant. Security Copilot setup could not enable audit logging of Copilot activity, and the unified DSPM setup will not complete without it. Turning it on was the longest delay of the whole lab.

The portal offered “Complete organisational setup”, then failed with InvalidOperationInDehydratedContextException. The Exchange Online organisation was dehydrated, which is common in demo and small tenants. The fix is in Exchange Online PowerShell:

Connect-ExchangeOnline
Get-OrganizationConfig | Format-List IsDehydrated
Enable-OrganizationCustomization
Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true
Get-AdminAuditLogConfig | Format-List UnifiedAuditLogIngestionEnabled

What to expect:

  • IsDehydrated reported False well before audit could actually be enabled. Reads and writes hit different replicas, so it is replication lag, not throttling. Disconnect, reconnect and retry every 15 minutes.
  • Audit was finally accepted about 73 minutes after the first attempt.
  • The first audit record appeared about seven hours later.
  • Audit is not retroactive. Nothing from the first evening - Security Copilot setup, the agent’s first runs or the classifier creation - was captured.

Gotcha: if you need audit evidence of an agent rollout, enable audit days before you start, not on the day.

2. Security Copilot capacity

Selecting Get started on the Posture Agent produced: “You do not have Security Copilot set up in your tenant.” Start agent stays greyed out.

Start agent is unavailable until Security Copilot is set up
Start agent is unavailable until Security Copilot is set up

The Microsoft 365 E5 route

Microsoft now includes Security Copilot with Microsoft 365 E5 and E7. Eligible tenants get 400 security compute units (SCUs) per month for every 1,000 paid user licences, up to 10,000 SCUs a month. It is provisioned automatically with 30 days’ notice, the allocation resets monthly and unused SCUs do not roll over.

Two words matter: eligible and paid. Demo and developer tenants do not qualify, and the rollout has been phased. I have yet to see it switched on in a customer tenant. Smaller customers should also do the maths: a 300-seat E5 organisation gets 120 SCUs a month.

The Azure route

Without the inclusion, you provision capacity yourself from an Azure subscription. In a CDX tenant that brought its own set of problems:

  • CDX tenants cannot start an Azure free trial. I created a Pay-As-You-Go subscription on a separate account and moved it into the CDX directory with Change directory.
  • After the move, the CDX admin had no role on the subscription. I used elevated access to assign one - remember to remove elevated access afterwards.
  • Assign Contributor rather than Owner. Owner triggers the role assignment condition screen, and Contributor is enough to create the capacity.
  • The role assignment defaults to Eligible through PIM. Switch it to Active, or it has to be activated before every use.

Then the capacity itself. The minimum is 1 SCU at $4 USD per hour, billed from the moment it is created. The form shows an estimated $2,920 a month, which is what happens if you forget to delete it.

Overage is on by default with no limit
Overage is on by default with no limit
  • Use overage units when needed is on by default at $6 USD per unit, with No limit selected. Turn it off for a lab, or set a max limit.
  • Both “Help improve Copilot” data sharing options are on by default and allow Microsoft human review of prompts and responses. Decide deliberately.
  • Form fields silently reset. Resource group, location and units did not stick on the first pass. Check every field before Create.
  • Deleting the capacity does not delete the Security Copilot workspace. When you provision again, you must Apply the new capacity to the existing workspace.

3. Let Security Copilot see Microsoft 365 data

The Posture Agent needs Security Copilot to read Microsoft 365 data. This is under Security Copilot > Owner > Plugin settings > Accessing data from Microsoft 365 services. Opting out means the agent cannot be used with Purview.

Security Copilot must be allowed to access Microsoft 365 data
Security Copilot must be allowed to access Microsoft 365 data

4. The unified DSPM gate

The first time you open the Posture Agent tab, Purview asks you to “Complete setup to unlock the unified DSPM experience”. It turns on auditing and analytics and creates three collection policies for AI, tagged Pay-as-you-go.

Review the pay-as-you-go collection policies before Start setup
Review the pay-as-you-go collection policies before Start setup
  • Select Edit and review the collection policies before Start setup. They capture Copilot and enterprise AI interactions and may generate charges.
  • Until this completes, and therefore until audit works, the gate keeps coming back. A first job may run, but Add data sources and View insights on finished jobs reopen the pop-up. You cannot even read your results.

5. First runs

With everything in place, the first prompt about five minutes after the capacity was created failed with “Could not start analysis job”. A retry a few minutes later worked.

The first prompt after creating capacity failed; a retry worked
The first prompt after creating capacity failed; a retry worked
  • Data sources must be added for every prompt. They do not carry over.
  • Searching by person works. Group search hung on Loading, and communication sites without a Microsoft 365 group never appeared in the picker.
  • Job estimation alone took 13 to 20 minutes, then reported 40 to 64 minutes remaining for a single person or group.
  • If you delete the capacity while a job is running, the job reports “found 0 items”. That looks exactly like a genuine empty result.
  • Every run, including failed ones, creates a session in Security Copilot history. That is useful evidence.

What it actually cost

Cost is the question every customer asks and few posts answer, so here are the real numbers from Azure Cost analysis. The lab used one Security Copilot capacity of 1 SCU on two occasions: about an hour and a half on the first evening, and 09:24 to 13:13 the next day.

Azure Cost analysis by resource: the full cost of the lab
Azure Cost analysis by resource: the full cost of the lab
ItemBilledCost (EUR)Status
Capacity, evening one (18 to 19 Nov)3 SCU-hours€10.77Actual
Capacity, day two (09:24 to 13:13)5 SCU-hours€17.95Actual
DSI and Purview pay-as-you-goNo investigations created€0Actual
Total8 SCU-hours€28.73 (about $32)Actual

The surprise was how the hours add up. The capacity existed for about 5.3 hours in total but was billed for 8 SCU-hours. Billing works per clock hour started: the first evening’s capacity crossed midnight and was billed for 3 hours, and day two touched five clock hours (09:00 to 13:00) and was billed for 5.

For context, €3.59 per SCU-hour is the $4 USD list price shown in the capacity form. Usage takes time to reach Cost analysis: the day two charges had still not appeared an hour after the capacity was deleted, and showed up later the same evening.

What this means for planning:

  • A focused lab or proof of concept costs tens of euros, not hundreds, provided you delete the capacity the moment you finish.
  • Most of my SCU-hours were spent waiting on job estimation, replication and retries, not running prompts. Doing the prerequisites first is the biggest saving available.
  • With the Microsoft 365 E5 inclusion, the same runs would come out of the monthly SCU allocation rather than an hourly bill. The trade-off is that a shared allocation can be used up by other teams’ agents and prompts.

How to keep it down:

  • Do the audit, Microsoft 365 data access and unified DSPM steps before you create capacity. Anything that waits on replication should not be waiting on an hourly meter.
  • Create the capacity at the top of the hour and delete it just before the next one.
  • Write your prompts and choose data sources in advance.
  • Turn overage off, set an Azure budget alert and delete the capacity as soon as you are finished. Budget alerts warn you; they do not stop spending.

Pay-as-you-go: the bill E5 does not cover

Security Copilot is only one meter. Several Purview features are billed pay-as-you-go through an Azure subscription linked to Purview, and the Microsoft 365 E5 inclusion does not cover them. You link the subscription once, Purview creates a billing resource in a resource group you choose, and you pay for what you use. The ones this series touched:

  • Data Security Investigations - storage per GB per month, plus AI compute units for investigations and AI analysis. Both have their own setup tasks: pay-as-you-go billing and AI capacity.
  • Collection policies for AI - created by the unified DSPM setup in step 4 to capture Copilot and enterprise AI interactions.
  • Some DLP policy features - the DLP Policies page shows “Set up pay-as-you-go billing” until a subscription is linked.
Linking an Azure subscription for Purview pay-as-you-go billing
Linking an Azure subscription for Purview pay-as-you-go billing

Gotchas from linking it in the lab:

  • The wizard suggests “Contoso” as the resource name, then rejects it because default names are not allowed. Enter your own.
  • Put the billing resource in the same resource group as your Security Copilot capacity, so teardown is one delete.
  • Linking took about three minutes, and the confirmation warns it can take a few hours to show in the account.
  • The billing resource was created in West US, the tenant’s geography, whatever AI processing location you choose later.
  • Deleting the billing resource unlinks pay-as-you-go billing in Purview. The DLP banner came straight back.

Here is the catch I hit: the credential scanning half of the Posture Agent lives in Data Security Investigations, so I set up DSI billing and AI capacity expecting them to unblock a stuck credential scan. They did not, and Microsoft Learn confirms the Posture Agent does not need them - only investigations and AI analysis do. If you are only using the agent, you can skip DSI, but budget for pay-as-you-go as soon as you move from discovery into investigations.

If you do set up DSI AI capacity, watch the data residency choice. The processing location (Australia, Europe, United Kingdom or United States) is where analysis runs, not where your data lives, so pick your tenant’s home geography. Leave “allow the AI to process data anywhere in the world” unticked if sovereignty matters - it is labelled “recommended for optimal performance”, which makes it an easy click.

Pick the tenant's home geography and leave the "anywhere in the world" option unticked
Pick the tenant’s home geography and leave the “anywhere in the world” option unticked

Next

With the agent running, Part 3 gets to the point of the series: using its findings to define the sensitive information types Contoso did not know it needed.

References

Where do we start with data classification? Part 3: From discovery to definition

Part 3 - From discovery to definition

This is the part of the series I most wanted to write. Most guidance on the Data Security Posture Agent shows it checking for sensitive information types (SITs) you already have. Here I use it the other way round: to find the SITs Contoso did not know it needed, then build and test them.

How you ask decides what you get

The agent turns your prompt into one of two kinds of search, and the wording decides which.

A narrow prompt becomes a keyword query

My first prompt asked for “files and emails containing confidential Mark 8 product design, roadmap or project details that do not have a sensitivity label”. The agent could not express the OR and AND combination, or “no sensitivity label” without a label name, as one query. It said so, rewrote the prompt and showed the KQL it would run.

The agent rewrites a prompt it cannot express as a single KQL query
The agent rewrites a prompt it cannot express as a single KQL query

That job finished with 0 items. The query was a literal phrase, limited to the default 30-day window, over content that did not use those words. Searching for what you already think exists is the wrong way round.

A discovery prompt becomes a semantic query

The prompt that worked described intent rather than words:

Find confidential business documents created in the last 3 years that contain
internal project codenames, product specifications or recurring internal
reference numbers

The agent accepted it as written and ran it as a semantic query over the requested date range. Note the three things the prompt asks for. I was not looking for a specific document; I was looking for the shapes that sensitive content tends to have.

A discovery-style prompt runs as a semantic query over three years
A discovery-style prompt runs as a semantic query over three years

Tip: ask for a time range in the prompt. The default is the last 30 days, which excludes most of an organisation’s long-lived intellectual property.

Reading the results

Over one user’s mailbox and OneDrive, the job found 20 items, none labelled. The report bands them by how well they match the intent: 5 High (over 90%), 15 Medium (50 to 90%) and 0 Low.

The Posture Agent report: 20 items, 0 labelled
The Posture Agent report: 20 items, 0 labelled
The unlabelled items behind the summary
The unlabelled items behind the summary

The list is where the work is. I sorted it into four piles:

FindingWhat it tells usBecomes
CR-227 Product Overview.docxA recurring reference format: CR followed by three digitsA pattern SIT (regex)
Project Tailspin, Customer Purchasing Data for Project 9Internal codenames nobody had listedA keyword SIT
Contoso Patent Template, RD Expense ReportContent defined by what it is, not by a patternA trainable classifier (Part 4)
pbiviz.json files, DataMashup, task reminders, meeting titlesNoise: system files and calendar itemsNothing - exclude

Two things to know about the report. It counts duplicates (one Power BI visual appears three times), so 20 items is fewer distinct files. And it gives a band per item but no reason, so use Asset explorer to look at each file before you trust a pattern.

The findings that matter were not on anyone’s list. Nobody in a workshop would have said “we need to protect anything with a CR number”. The agent found the pattern because it appears in content that looks confidential.

Building the SITs

Creating a SIT costs nothing and needs no Security Copilot capacity. In the unified Classifiers page, select Create > Sensitive info type.

Contoso Change Request Reference

Name and describe the SIT so policy authors know what it detects
Name and describe the SIT so policy authors know what it detects

I built it with two patterns, so the confidence reflects the evidence:

  • High confidence: the regex CR-\d{3} plus at least one supporting keyword within 300 characters (change request, product overview, specification, confidential, internal only).
  • Low confidence: the regex on its own.

The first gotcha came straight away. Selecting Word match replaces whatever you typed with a template, and your regex has to go inside its capturing group:

(?:^|[\s,;\:\(\)\[\]"'])(CR-\d{3})(?:$|[\s,;\:\(\)\[\]"']|\.\s|\.$)
Word match wraps your pattern in a template; only one capturing group is allowed
Word match wraps your pattern in a template; only one capturing group is allowed
High confidence: the regex plus supporting keywords
High confidence: the regex plus supporting keywords
Two patterns, two confidence levels
Two patterns, two confidence levels
  • Choose from existing regular expressions does not list a regex from the SIT you are still creating. For the second pattern, enter it again.
  • The recommended confidence level only offers levels your patterns use. I chose High, so policies default to the stricter match.

Contoso Project Codenames

The codenames are a simple keyword list at Medium confidence: Project Tailspin, Tailspin and Project 9.

A keyword SIT for the codenames the agent found
A keyword SIT for the codenames the agent found

Be careful with short or generic keywords. “Project 9” is fine in Contoso’s demo data, but in a real organisation check how often a codename appears in ordinary content before you build a policy on it. A codename that is also a common word needs supporting keywords and a higher confidence pattern.

Test before you trust

The SIT’s Test option needs a file; there is no box to paste text into. I tested with a short file containing three real references (CR-227, CR-228, CR-231) and two decoys (CR-12 and CR-1234).

Three real references matched, with supporting keywords; the decoys did not
Three real references matched, with supporting keywords; the decoys did not

All three real references matched, with the supporting keywords listed for each. Neither decoy matched, which proves the word boundaries are doing their job. This is the check that stops a SIT from flooding a DLP policy with false positives.

The new SITs in the unified Classifiers page, ready for policies
The new SITs in the unified Classifiers page, ready for policies

The loop

That is the process this series is about. It repeats for every team and every data source:

  1. Ask the agent a discovery-style question about a team’s content.
  2. Sort the results into patterns, categories and noise.
  3. Build a SIT for each pattern, and test it with real and decoy examples.
  4. Send each category to a trainable classifier.
  5. Schedule the prompt to run monthly, so new patterns surface as the business changes.

Part 4 picks up the categories: teaching Purview to recognise documents such as Contoso’s design specifications with a trainable classifier.

References

Where do we start with data classification? Part 4: Teaching Purview what matters with trainable classifiers

Part 4 - Teaching Purview what matters with trainable classifiers

Part 3 turned the agent’s patterns into sensitive information types. Some findings had no pattern at all: a patent template, an R&D expense report. They are defined by what they are, not by a string of characters. That is the job of a trainable classifier, which learns to recognise a category of content from examples.

Start with the built-in classifiers

Microsoft provides pre-trained classifiers that need no training. The Contoso tenant has 79, and they had already classified content without anyone configuring anything: Finance 134 items, HR 52, Budget 20, Agreements 7, IT and Tax 5 each, Statement of Work 3 and IP 2.

Built-in trainable classifiers already sorting Contoso content
Built-in trainable classifiers already sorting Contoso content

This is the quickest win in Purview. Before writing a single custom rule, an organisation can see roughly how much finance, HR and contract content it holds and where it lives.

Now look at IP: 2 items. Contoso is an engineering company with a product portfolio. Either it holds almost no intellectual property, or the generic IP classifier does not recognise what Contoso’s IP looks like. It is the second. Generic classifiers are trained on generic examples.

Choosing the right classifier type

The four classifier types side by side
The four classifier types side by side
  • Sensitive info type - patterns and keywords, for data with a recognisable shape (Part 3).
  • Exact data match - exact values from a structured source, such as a customer list.
  • Document fingerprinting - documents based on a standard form or template.
  • Trainable classifier - categories of content learned from examples.

Contoso’s engineering design specifications fit the last category. Each one is different, written by different engineers for different products, but a person recognises one instantly.

Building Contoso Engineering Design Specifications

Prepare the seed content

What to prepare before creating a trainable classifier
What to prepare before creating a trainable classifier
  • At least 50 and up to 500 positive samples, and at least 150 and up to 1,500 negative samples.
  • Text-based files only. Images and image-only PDFs cannot be used.
  • A SharePoint site such as a communication site, with positive and negative content in separate folders used only for seed content.
  • At least an hour for a new site or folder to be indexed before you create the classifier.
  • English only for custom classifiers, and only items that are not encrypted.

The lab uses 60 fictional design specifications and 160 negatives, uploaded to a communication site called Classifier Seed Content. The generator (make_seed.py) and an upload script are in the GitHub repository for this series, so you can recreate the same seed content in your own test tenant.

Positive seed content in a dedicated SharePoint folder
Positive seed content in a dedicated SharePoint folder

The negative set matters as much as the positive set. Alongside meeting notes, HR policies and newsletters, I added hard negatives: quick start guides and marketing pages for the same products named in the specifications. Without them, the classifier can learn the shortcut “mentions a Contoso product” instead of “is a design specification”.

In a real organisation, ask the engineering lead for genuine specifications and take negatives from the same team’s non-confidential content. Negatives from the same team teach far more than random documents from elsewhere.

Gotcha: the communication site never appeared in the Posture Agent’s data source picker. The picker searches people and groups, and a communication site has no Microsoft 365 group.

Create and train

Naming the new trainable classifier
Naming the new trainable classifier

The wizard has four steps: name, positive sample folder, negative sample folder, and review. The wizard estimated 5 to 7 hours of training, and Microsoft says 24 hours or less.

Reality was slower. Seventeen hours after creation, the classifier was still showing Training in the unified Classifiers page.

Still Training 17 hours later, alongside the new SITs from Part 3
Still Training 17 hours later, alongside the new SITs from Part 3

Plan for at least a day between creating a classifier and reviewing it. Do not book the business owner’s review session for the same afternoon.

Review before publishing

When training finishes, Purview tests the classifier and asks you to review its predictions as correct, incorrect or uncertain. Only publish when you are satisfied. Retraining a published custom classifier is not supported; if it needs improving, delete it and start again with more samples.

By default only the creator can review the predictions. Decide who owns that review before you start: someone who knows the content, which usually means the business rather than IT.

The results: 96% accuracy

Training finished about 23 hours after creation, for just 220 short text files: 60 positive and 160 negative. The accuracy score was 96%: Purview tested the classifier against items from the seed folders and got all but four right.

Microsoft now quotes far shorter training times than it used to: the portal’s create tile estimates 5 to 7 hours. Our lab took 23 hours, even with a small data set. That is not a problem in itself. Training a classifier is rarely time-sensitive for an organisation, and real seed sets are usually much larger. Just plan for a day rather than an afternoon.

Training complete: 96% accuracy and Ready to publish
Training complete: 96% accuracy and Ready to publish

Gotcha: two pages, two different answers

The new unified Classifiers page (preview) still showed the classifier as Training after it had finished. The classic Trainable classifiers page showed the truth: 95.7%, Ready to publish.

The unified Classifiers page still says Training
The unified Classifiers page still says Training
The classic page: 95.7% and Ready to publish
The classic page: 95.7% and Ready to publish

If a classifier looks stuck in Training, check the classic Trainable classifiers page before you wait any longer.

Read the misses, not just the score

Review test results lists every item the classifier got wrong. There were no false positives: no negative document, including the hard negatives, was mistaken for a specification. There were four false negatives, all genuine specifications the classifier missed.

Four false negatives, three of them for the same product
Four false negatives, three of them for the same product

Three of the four were specifications for the same product, the Northwind heat exchanger. A score hides that kind of pattern; the list shows it. In a real organisation, that is the moment to ask whether one product line writes its specifications differently and needs more examples, before anyone relies on the classifier for it.

No false positives and a few false negatives is the right way round to start. A classifier that misses some specifications is safer to put into a simulation policy than one that flags newsletters.

Publish

Publish for use makes the classifier available to Information Protection, Data Lifecycle Management and Communication Compliance, and to DLP conditions.

Published and ready to use in policies
Published and ready to use in policies

What comes next

A classifier on its own protects nothing; the Affected policies column from Part 1 made that clear. Its value comes from what you connect it to: DLP, sensitivity label auto-labelling and retention. Part 5 adds it to the DLP policy alongside the SITs from Part 3.

References

Where do we start with data classification? Part 5: From classification to protection

Part 5 - From classification to protection

Four posts ago Contoso had 1,866 sensitive items and no policy using any of them. It now has two SITs that came from its own data, and a classifier in training. This final part connects them to protection, looks honestly at what did not work, and shows the evidence trail.

A DLP policy, in simulation

The policy is deliberately simple: if content containing a Contoso change request reference or project codename is shared with people outside Contoso, restrict access for those external users. It runs in simulation mode, so it reports what it would have done without blocking anyone.

  • Template: Custom policy, with Create or customise advanced DLP rules.
  • Locations: Exchange, SharePoint, OneDrive and Teams. The wizard ticks everything by default, including Devices, Instances and On-premises repositories. Untick what you are not ready to cover.
Scope the policy to the Microsoft 365 workloads you are ready for
Scope the policy to the Microsoft 365 workloads you are ready for
  • Conditions: Content contains either SIT (each at its recommended confidence) AND content is shared with people outside my organisation.
The SITs from Part 3, combined with external sharing
The SITs from Part 3, combined with external sharing
  • Action: Restrict access, blocking only people outside the organisation.
The finished rule
The finished rule

The gotcha

Submitting the policy failed:

"Block only people outside your organisation" needs user notifications
“Block only people outside your organisation” needs user notifications

Blocking only external people requires user notifications. The fix is to turn on User notifications in the rule and tick “Notify users in Office 365 service with a policy tip or email notifications”. The portal does not warn you until you submit, and the error is written in PowerShell parameter names.

Simulation first

Simulation mode, with automatic activation left unticked
Simulation mode, with automatic activation left unticked

Simulation mode is the default. Leave “Turn the policy on if it’s not edited within fifteen days of simulation” unticked. A policy built on newly discovered SITs needs a human to review the matches before it blocks anything.

Adding the classifier

Once the classifier from Part 4 was published, I added it to the same rule. Edit the rule, and in the Content contains group choose Add, then Trainable classifiers.

The custom classifier appears alongside the built-in ones
The custom classifier appears alongside the built-in ones

The group operator stays at Any of these, so the rule now matches a change request reference, a project codename or a design specification, shared outside Contoso. The SITs catch the patterns; the classifier catches the documents.

Two SITs and a trainable classifier in one Content contains group
Two SITs and a trainable classifier in one Content contains group

The policy stays in simulation. One small oddity: the custom classifier shows no supported language in the picker, although custom classifiers are English only.

A word on people

DLP is a safety net, not a substitute for awareness. Most data leaks are not malicious: someone attaches the wrong file, shares a link with “Anyone”, or pastes a specification into the wrong chat. Every organisation should run regular security awareness training so staff know what sensitive data looks like and how to handle it.

DLP then protects people as much as it protects the organisation. A policy tip at the moment of sharing catches the honest mistake before it becomes an incident, and teaches the policy in context far better than a slide deck. Policy tips are also why this policy shows them during simulation: users start learning before anything is blocked.

The same SITs and classifier can drive sensitivity label auto-labelling, so Contoso’s design IP is labelled and encrypted wherever it lives, not only blocked when shared. That is the natural next step after a clean simulation.

A preview reality check: credential scanning

The Posture Agent’s other capability, credential scanning in Data Security Investigations, never completed in the lab. Both scans, one tenant-wide and one scoped to a single user, sat at “Waiting for agent” for hours. The first could not be stopped (“Failed to cancel investigation as the job is still in queue”), and the cards offer no detail.

Both credential scans stuck at "Waiting for agent"
Both credential scans stuck at “Waiting for agent”

I checked again the following night. The tenant-wide scan had been waiting for about 24 hours and the user-scoped scan for 11, with no change, and the Security Copilot capacity had been deleted since early afternoon.

Still waiting, a day later
Still waiting, a day later

Microsoft Learn says the Posture agent does not need Data Security Investigations billing or AI capacity, only Security Copilot capacity, which we had. One third-party roadmap tracker reports the credential scanning roadmap item as cancelled; I could not confirm that from Microsoft. Treat credential scanning as preview: test it in your own tenant before you promise it to anyone.

The evidence trail

The agent reads your most sensitive content under its own identity, so its activity needs to be auditable. Three lessons from the lab:

  • Audit is not retroactive. Ingestion was enabled at midnight; the first record appeared seven hours later. None of the first evening’s agent runs were captured.
  • The audit log showed the role checks behind the agent: the admin was granted the compliance roles and blocked on content-level roles such as Purview Content Analyst. That matches what Learn says about who sees summaries and who sees content.
  • Every agent run creates a session in Security Copilot history, and credential scans show “Started by Data security posture agent”. Both are useful evidence alongside the audit log.

Exporting the audit log with PowerShell was faster and more reliable than the portal. Search-UnifiedAuditLog is in the Exchange Online PowerShell module (ExchangeOnlineManagement), and the account needs the View-Only Audit Logs or Audit Logs role. Install the module once, then connect:

Install-Module ExchangeOnlineManagement -Scope CurrentUser
Connect-ExchangeOnline -UserPrincipalName admin@contoso.onmicrosoft.com

Then export the last two days of records:

Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-2) -EndDate (Get-Date) -ResultSize 5000 |
  Select-Object CreationDate, UserIds, RecordType, Operations, AuditData |
  Export-Csv .\audit-posture-agent.csv -NoTypeInformation -Encoding UTF8

Gotcha: -SessionCommand ReturnLargeSet without -SessionId fails with BadRequest. Leave it out for up to 5,000 records.

Teardown

Anything that costs money or grants access should be removed when you finish:

  1. Delete the Security Copilot capacity. The workspace survives; that is expected.
  2. Delete any Purview pay-as-you-go resource you linked. This also unlinks pay-as-you-go billing in Purview, and the DLP Policies page shows the “Set up pay-as-you-go billing” banner again. That is expected.
  3. Remove elevated access (Entra ID > Properties > Access management for Azure resources) for every account you elevated.
  4. Check Cost analysis after 24 to 48 hours, then cancel or move the subscription.
  5. Leave the DLP policy in simulation, or delete it, and decide what happens to the stuck credential scans.

So where do we start?

Five posts later, my answer is this:

  1. Look at what Purview already sees, and at the gap between classified items and affected policies.
  2. Ask the Posture Agent discovery questions about intent, not about the SITs you already have.
  3. Turn repeated patterns into SITs, test them with real and decoy examples, and send categories of content to trainable classifiers.
  4. Protect in simulation first, with a human reviewing the matches.
  5. Audit from day one, and treat the agent as the non-human identity it is.

The agent did not do the classification for Contoso. It found what nobody had thought to list, in a few hours of compute for under 30 euros. That is a better starting point than a workshop.

Coming next: a new blog post (or short series) on the DLP triage agent, and how to use it effectively on the alerts a policy like this produces.

GitHub repository

Everything used in this series is published at https://github.com/cloud-secure-ai/Purview-data-classification:

  • make_seed.py - the parameterised generator for positive and negative classifier seed content.
  • SharePoint upload script for seed content.
  • The audit log commands and the CR test file used to test the SIT.
  • A prerequisites and teardown checklist.

References