Leaked Credentials and Sign-in Metrics Make for an Interesting Day
Last Saturday morning was interesting. After barely being able to ingest my first coffee, I noticed that Teams was complaining that it couldn’t sign in. A quick check against the Entra ID sign-in log revealed that the problem was that Teams had blocked the user (me) because of risk detected by the home tenant (Figure 1). In other words, something had happened to make Entra ID consider my account to be at risk.
When unexpected things happen, my go-to source is the audit log. The root of the problem occured at 05:01 UTC when a component called MACE Credential Revocation changed the StsRefreshTokensValidFrom property for my account to set a new date and time from which refresh tokens can be used (thus forcing reauthentication). The action was captured in a Update StsRefreshTokenValidFrom Timestamp event.
MACE Credential Revocation is not something I have encountered before. The MACE part of the name means Microsoft Account Compromise Exchange and it’s part of Entra ID used to distribute leaked credential notifications. It is an Entra ID enterprise app with an application identifier of 7d636ec3-f39c-44f5-8b73-fa28a0e0c5bc. Interestingly, the service principal for the app was created in my tenant at at 19 April 2025 05:00;42 by a process called Microsoft Azure AD Internal – Jit Provisioning. It looks like Microsoft flipped a switch to enable some new Entra ID processing that immediately caused accounts to be flagged with leaked credentials. As I later discovered, it seems like many tenants had the same problem.
Despite my tenant being affected, nothing relating to a problem with Entra ID appeared in the service health dashboard, but I have heard from others who received the following statement from Microsoft support:
On Friday 4/18/25, Microsoft identified that it was internally logging a subset of short-lived user refresh tokens for a small percentage of users, whereas our standard logging process is to only log metadata about such tokens. The internal logging issue was immediately corrected, and the team performed a procedure to invalidate these tokens to protect customers. As part of the invalidation process, we inadvertently generated alerts in Entra ID Protection indicating the user’s credentials may have been compromised. These alerts were sent between 4/20/25 4AM UTC and 4/20/25 9AM UTC. We have no indication of unauthorized access to these tokens – and if we determine there were any unauthorized access, we will invoke our standard security incident response and communication processes.
The odd thing is that Microsoft refers to alerts being sent on Sunday (April 20) rather than Saturday (April 19), which is when the problem occurred. The mystery deepens.
Leaked Credentials
Managing risky users with Entra ID Protection is an Entra ID P2 feature. Being able to detect problems by monitoring for different signs of potential compromise is one of the reasons why tenants might consider the Microsoft E5 Security add-on. In my case, risky users is covered by the Microsoft 365 E5 license. Checking the Risky Users section of the Entra admin center, I duly found that my account was in the bad accounts list because of “leaked credentials” (Figure 2).
Leaked credentials aren’t a huge problem for accounts protected by strong multi-factor authentication and are considered a “nonpremium” issue in the list of risk detections. I wanted to find out more and use the link to Microsoft Defender to see if it could throw more light onto the subject (Figure 3).
Remediation is a Password Change
Despite feeling that the warning was a false positive, you shouldn’t take chances with passwords. The easiest way to remediate the risk and unblock the account is to reset the password, so I used the self-service password reset feature to set a new account password (secure reset of a password is always accepted as a remediation). Reporting the detection of risky users and their remediation can be done through PowerShell.
Just one account was affected in my tenant. Others tenants reported that access for multiple accounts was blocked. The evidence to date indicates that Microsoft noticed a problem, took action to fix the problem, and made the problem worse by causing alerts to fire about perfectly good user accounts. In one way, it’s good that this happened over a holiday period when administrators have time and space to fix accounts, but you have to ask how a remediation process can suddenly make Entra ID believe that user accounts suffer from leaked credentials. It’s all very odd.
Success Metrics for Sign-ins
All of which brings me to some interesting metrics that are available through the Graph beta endpoint. All require the Reports.Read.All permission. For instance, the MFA sign in success metrics report “the number of times users successfully completed interactive MFA sign-ins using the Microsoft Entra MFA cloud service during a specified time period.” The metrics can be reported for up to 30 days using intervals of 5, 10, 15, or 30-minute intervals. Each item reported therefore represents the number of activities that occurred in an interval.
For example, here’s the code to extract the MFA sign-in success metrics at 30-minute intervals for the last 29 days.
$StartDate = $StartDate = ((Get-Date).AddDays(-29).ToString("yyyy-MM-ddTHH:mm:ssZ"))
$EndDate = ((Get-Date).ToString("yyyy-MM-ddTHH:mm:ssZ"))
$Uri = ("https://graph.microsoft.com/beta/reports/serviceActivity/getMetricsForMfaSignInSuccess(inclusiveIntervalStartDateTime={0},exclusiveIntervalEndDateTime={1},aggregationIntervalInMinutes=30)" -f $StartDate, $EndDate)
$Data = Invoke-MgGraphRequest -Uri $Uri -Method Get | Select-Object -ExpandProperty Value
$Data
Name Value
---- -----
value 0
intervalStartDateTime 17/04/2025 11:30:00
value 0
intervalStartDateTime 17/04/2025 12:00:00
value 0
intervalStartDateTime 17/04/2025 12:30:00
value 0
intervalStartDateTime 17/04/2025 13:00:00
value 0
…
1,390 objects were duly written into the output array. That’s too many to check manually, so I created a quick loop to report the intervals when successful MFA operations occurred:
ForEach ($Item in $Data) {
If ($Item.Value -as [int] -ne 0) {
Write-Host ("{0} successful MFA sign-ins during interval starting {1}" -f $Item.Value, $Item.intervalStartDateTime) -ForegroundColor Cyan
}
}
Remember that these represent operations where a user goes through the full MFA process. If someone authenticates because their credentials haven’t expired, that operation doesn’t count.
Possibly of more interest is the API to get conditional access blocked sign in metrics, or “the number of user sign-in attempts that were blocked by a Conditional Access policy during a specific period.” The command is similar:
$Uri = ("https://graph.microsoft.com/beta/reports/serviceActivity/getMetricsForConditionalAccessBlockedSignIn(inclusiveIntervalStartDateTime={0},exclusiveIntervalEndDateTime={1},aggregationIntervalInMinutes=30)" -f $StartDate, $EndDate)
$Data = Invoke-MgGraphRequest -Uri $Uri -Method Get | Select-Object -ExpandProperty Value
Once again, we need a little help to extract the intervals when something interesting might have happened to guide us where to check in the Entra sign-in log:
If ($Item.Value -as [int] -ne 0) {
Write-Host ("{0} Blocked Conditional access sign-ins at {1}" -f $Item.Value, $Item.intervalStartDateTime) -ForegroundColor Red
}
}
Interestingly, no metrics for blocked conditional access sign-ins are detected in my tenant. I guess the folks who secured a copy of my leaked credentials haven’t tried to use them yet only to find that MFA will block their efforts.
Learn how to exploit the data available to Microsoft 365 tenant administrators through the Office 365 for IT Pros eBook. We love figuring out how things work.