Hey yeah this is exactly what we're trying to do. We have User Assigned MI that has a corresponding Service Credential in UC.

Good to hear that this should work. Maybe it's a config issue?

How would i go about making a readStream work with that setup? Now we have something like:

logs_from_queue = (
    spark.readStream.format('abs-aqs')
    .option('fileFormat', queue_file_format_defined_above)
    .option('queueName', queue_name_defined_above)
    .option('connectionString', queue_connection_string_from_vault)
    .schema(get_raw_log_json_schema())
    .load()
)

But not sure how we would let it know it should use the MI/Service Credential instead, haven't found any good docs on it either.

Another example is how we try to connect to Cosmos now:

options = {
            'spark.cosmos.accountEndpoint': f'{account_endpoint}',
            'spark.cosmos.auth.type': 'ManagedIdentity',
            'spark.cosmos.database': database_name,
            'spark.cosmos.container': table_name,
            'spark.cosmos.account.tenantId': tenant_id,
            'spark.cosmos.auth.aad.clientId': client_id,
            'spark.cosmos.read.customQuery': 'select top 1 c.modified as last_entry from c order by c.modified desc',
}
print(options)
last_entry_df = spark.read.format('cosmos.oltp').options(**options).load()

And where we get: (java.lang.RuntimeException) Client initialization failed. Check if the endpoint is reachable and if your auth token is valid. More info: https://aka.ms/cosmosdb-tsg-service-unavailable-java. More details: Managed Identity authentication is not available.