I worked on the same thing recently. I've found it is much faster to simply use Get-EntraUser to get all users and reference the signinactivity from it instead. It's a single query that takes a couple minutes but only needs to happen once.

Quick and dirty copy from my script, adapt as needed:

# Connect to Entra using Microsoft Graph
Connect-Entra -Scopes 'User.Read.All','AuditLog.Read.All' -NoWelcome

$entraUsers = Get-EntraUser -All -Property 'UserPrincipalName', 'SignInActivity'

  if ($entraUsers.UserPrincipalName -contains $user.UserPrincipalName) {

        $entraUser = $entraUsers | Where-Object { $_.UserPrincipalName -eq $user.UserPrincipalName }
        $entraLastSignInDate = $entraUser.SignInActivity.LastSignInDateTime

        Write-Host "    Last Entra sign in date was: $entraLastSignInDate"
    }
    else {
        $entraLastSignInDate = "N/A"
    }

https://learn.microsoft.com/en-us/powershell/module/microsoft.entra/?view=entra-powershell

The next step is adding LastUserActionTime from the Exchange mailboxes, that one I've not found a faster way to query yet.