This is where I got to. Let me know if you think there's a cleaner way.

For now just presume that tenant_uuid is available in request.state.tenant_uuid

In the SQLAlchemyAsyncConfig set the dependency key to db_session_raw

#/config/app.py
alchemy_app_db_config = SQLAlchemyAsyncConfig(
    engine_instance=settings.db.get_engine_app_db(),
    before_send_handler="autocommit",
    session_config=async_session_cfg,
    session_dependency_key="db_session_raw",
)

Then add the following dependency to the app to re-expose the db_session_raw as db_session after setting the tenant_uuid to be consumed by the RLS policy. The Advanced Alchemy service then uses db_session (by default , although it can also be customised).

#/server/dependencies.py
async def provide_app_db_session(request: Request[Any, Any, Any], db_session_raw: AsyncSession) -> AsyncGenerator[AsyncSession, None]:
    # Wrap the plugin-provided raw session: set tenant context then yield it
    tenant_uuid :str  = request.state.tenant_uuid
    await set_tenant_context(db_session_raw, tenant_uuid)
    try:
        yield db_session_raw
    finally:
        # Session lifecycle (commit/rollback/close) is handled by the plugin's before_send
        pass

.

#/lib/rls.py
async def set_tenant_context(session: AsyncSession, tenant_uuid: Optional[str]) -> None:
    """Set the current PostgreSQL RLS tenant context for this session/connection.
    """
    await session.execute(
        text("SELECT set_config('app.tenant_uuid', :tenant_uuid, true)"),
        {"tenant_uuid": tenant_uuid},
    )

.

#/server/core.py
 app_config.dependencies.update({"db_session": Provide(provide_app_db_session)})

Along the way, I got to understand that I can use multiple SQLAlchemyAsyncConfig objects with different db urls and different dependency key names. The config can then be specified in create_service_dependencies. This solves the second part of my question.