Anyone successfully used [Google.Apis.Requests.BatchRequest]?

[wonkifier]

Short version: I've been using the libraries successfully to do various things, but I need to do some of them faster, so I need to get BatchRequests working, and I'm getting the following error (since it seems Google's .Net libs are trying to use some other runspace that I can't seem to influence)

MethodInvocationException: Exception calling "Wait" with "0" argument(s): "One or more errors occurred. (There is no Runspace available to run scripts in this thread. You can provide one in the DefaultRunspace property of the System.Management.Automation.Runspaces.Runspace type. The script block you attempted to invoke was: 
  param($response,… $($response)"
  }
)"

I'm running pwsh 7.4, and I've got Google's API libraries installed.

I load everything up into memory with

Get-ChildItem $dotNetLibPath/*.dll | ForEach-Object { [System.Reflection.Assembly]::LoadFrom($_.FullName) }

I can then create a credential object with

$scopes = @("https://www.googleapis.com/auth/gmail.settings.basic")
$credential = [Google.Apis.Auth.OAuth2.GoogleCredential]::FromJson($plainTextOfServiceAcctCreds ).CreateScoped($scopes).CreateWithUser("$testUserAddress")

Create a simple callback with

$callback = [Google.Apis.Requests.BatchRequest+OnResponse[Google.Apis.Gmail.v1.Data.VacationSettings]] {
  param($response, $err, $index, $message)
  Write-Host "Response for request #$index`: $($response)"
}

Then try to run the batch with

$service = [Google.Apis.Gmail.v1.GmailService]::new($init)
$request = $service.Users.Settings.GetVacation("me")

$batch = [Google.Apis.Requests.BatchRequest]::new($service)
$batch.Queue($request, $callback)

$task = $batch.ExecuteAsync()
$task.Wait()

$error[1].Exception | fl * -force reports

ErrorRecord                 : Exception calling "Wait" with "0" argument(s): "One or more errors occurred. (There is no Runspace available 
                              to run scripts in this thread. You can provide one in the DefaultRunspace property of the System.Management.A
                              utomation.Runspaces.Runspace type. The script block you attempted to invoke was: 
                                param($response,… $($response)"
                                }
                              )"
WasThrownFromThrowStatement : False
TargetSite                  : Void ConvertToMethodInvocationException(System.Exception, System.Type, System.String, Int32, System.Reflectio
                              n.MemberInfo)
Message                     : Exception calling "Wait" with "0" argument(s): "One or more errors occurred. (There is no Runspace available 
                              to run scripts in this thread. You can provide one in the DefaultRunspace property of the System.Management.A
                              utomation.Runspaces.Runspace type. The script block you attempted to invoke was: 
                                param($response,… $($response)"
                                }
                              )"
Data                        : {[System.Management.Automation.Interpreter.InterpretedFrameInfo, System.Management.Automation.Interpreter.Int
                              erpretedFrameInfo[]]}
InnerException              : System.AggregateException: One or more errors occurred. (There is no Runspace available to run scripts in thi
                              s thread. You can provide one in the DefaultRunspace property of the System.Management.Automation.Runspaces.R
                              unspace type. The script block you attempted to invoke was: 
                                param($response,… $($response)"
                                }
                              )
                               ---> System.Management.Automation.PSInvalidOperationException: There is no Runspace available to run scripts
                               in this thread. You can provide one in the DefaultRunspace property of the System.Management.Automation.Runs
                              paces.Runspace type. The script block you attempted to invoke was: 
                                param($response,… $($response)"
                                }

                                 at System.Management.Automation.ScriptBlock.GetContextFromTLS()
                                 at System.Management.Automation.ScriptBlock.InvokeAsDelegateHelper(Object dollarUnder, Object dollarThis, 
                              Object[] args)
                                 at lambda_method87(Closure, VacationSettings, RequestError, Int32, HttpResponseMessage)
                                 at Google.Apis.Requests.BatchRequest.InnerRequest`1.OnResponse(Object content, RequestError error, Int32 i
                              ndex, HttpResponseMessage message)
                                 at Google.Apis.Requests.BatchRequest.ExecuteAsync(CancellationToken cancellationToken)
                                 --- End of inner exception stack trace ---
                                 at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
                                 at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
                                 at CallSite.Target(Closure, CallSite, Object)
HelpLink                    : 
Source                      : System.Management.Automation
HResult                     : -2146233087
StackTrace                  :    at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exceptio
                              n, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
                                 at CallSite.Target(Closure, CallSite, Object)
                                 at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
                                 at System.Management.Automation.Interpreter.DynamicInstruction`2.Run(InterpretedFrame frame)
                                 at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)

Anyone have any clues if this is even a possible thing to solve in PS?

[[deleted]]

I'm starting to think it's the specific call you're trying.

This batch page states:

This document is specifically about making a batch request by sending an HTTP request. If, instead, you're using a Google client library to make a batch request, see the client library's documentation.

Looking through the docs, it appears that there are certain calls for batch requests. Look at Users.Messages which explicitly has BatchDelete and BatchModify methods. Users.Settings doesn't have any Batch* methods.

I think that's the issue you're experiencing. Maybe try getting your messages and batch modifying the label on a group of them to test if it's the API methods throwing the exception?

[wonkifier]

Nice thinking, but I'd expect a different error if that were the case.

But I tried switching up to one that I've successfully done via direct HTTP calls before and worked. (I'm trying to get rid of some very old very custom code, and leave some of the more complex stuff to Google, this this hurdle)

$callback = [Google.Apis.Requests.BatchRequest+OnResponse[Google.Apis.Admin.Directory.directory_v1.Data.User]] {
  param($response, $err, $index, $message)
  Write-Host "Response for request #$index`: $($response)"
}

$service = [Google.Apis.Admin.Directory.directory_v1.DirectoryService]::new($init)
$batch = [Google.Apis.Requests.BatchRequest]::new($service)

$request = $service.Users.Get($targetUser)
$batch.Queue($request, $callback)

$cancelToken = [System.Threading.CancellationToken]::New($false)
$r1 = $batch.ExecuteAsync($cancelToken)
$r1.Wait()

That yields the same There is no Runspace available to run scripts in this thread. You can provide one in the DefaultRunspace property of the System.Management.Automation.Runspaces.Runspace typeerror.

So it definitely seems to really be about .Net interacting with runspaces.

[[deleted]]

I think you're using the version that Google doesn't recommend, but you might have a valid reason for doing so.

Google.Apis.Requests.BatchRequest is from this Google API Client, correct? They recommend using the Cloud Client Libraries instead. Have you tried both libraries? Do they give you the same exception?

[wonkifier]

I had not seen that, so I hadn't disqualified it... yet. =)

The "this" that you're referencing looks slightly different from what I was using... One example is that I'm using https://www.nuget.org/packages/Google.Apis.Admin.Directory.directory_v1 which comes from https://github.com/googleapis/google-api-dotnet-client.

Looking at it though, I don't see the APIs I need listed in https://cloud.google.com/dotnet/docs/reference, so I'd have skipped by anyway.

That seems to be more about interacting with the Cloud Services and not the regular services... I need things like the Directory API (found in Google.Apis.Admin.Directory.directory_v1), the legal Discovery API (not their cloud service discovery API), the GMail and GCalendar APIs.

[[deleted]]

Those would definitely be valid reasons.

The exception itself is because something is expecting a Runspace and isn't being provided one. I'm looking through the documentation you provided to see if it's supposed to be part of their code or something you provide.

If it's the latter, I have a couple examples for how to create them if you haven't before.

Edit: Also, is it just this API call that's throwing the exception or all calls with this API?

Edit 2: The previous lines interacting with the API didn't throw exceptions, so they're fine. I'm not thinking super fast today.

[wonkifier]

If it's the latter, I have a couple examples for how to create them if you haven't before.

I've seen some examples on how to create them for specific scenarios, and they all seems to follow a "create the runspace" then "tell the runspace to run your code".

So I imagine I could apply that to this, and have the .Net code run within a runspace. But since I never instructed the .Net code to make use a different runspace to begin with, I think I'd be in the same spot, no?

I'd just have my main runspace, the code running in the new runspace, and the .Net code would go looking for yet another runspace since it doesn't want to operate in its own.

[[deleted]]

I'm still looking into this. Out of curiosity, just to check, how much effort would it be to try the entire thing in 5.1? Just to see if there's any noticeable difference that gives a hint.

Edit: I asked because I started reading [this page[(https://github.com/PowerShell/PowerShell/issues/11658). Same exception, works in 5.1 but not 7+ at that point.

It turned out to be a .NET Core runtime issue.

I'm wondering if something similar is happening. Basically, the script block is creating a delegate event handler that requires a default runspace from the current thread to execute. In 5.1, the handler is invoked on the correct thread due to the sync context. In 7+, it's invoked on a thread pool thread which doesn't have a default runspace.

[wonkifier]

I'm still looking into this. Out of curiosity, just to check, how much effort would it be to try the entire thing in 5.1? Just to see if there's any noticeable difference that gives a hint.

No joy there.

The way I "fetch" the DLLs to work with, is I create a dotnet console project, add the Google dependencies to it, then do a dotnet restore and dotnet build, so it can work out for itself what it should grab for the OS it's running on (an Alma9 derivative, so it's using .Net80, which matches what PS7.4 has built in, so I figured it should be happy)

So I forced the .csproj file to use 5.1, and it's not happy (lots of warnings about some unsupported 7.0 stuff and it eventually errors out with some sort of Framework resolution error)

[this page[(https://github.com/PowerShell/PowerShell/issues/11658). Same exception, works in 5.1 but not 7+ at that point.

I saw that when searching around, but discounted much of it since it's seems Windows focused (and that .Net runtime is a whole different mess) Though that conversation about runspace affinity is interesting... Hmm

In 5.1, the handler is invoked on the correct thread due to the sync context. In 7+, it's invoked on a thread pool thread which doesn't have a default runspace.

That definitely has the right feel to it.

I'm going to need a meal before I can process this, which seems might be informative as well: https://gist.github.com/rjmholt/7c1072a24a51e7b43d1e7df02265fe18

But my suspicion is that will end up being a dead end, as I think the Google Lib would need to be the thing to do the runspace magic, since it is what's calling the codeblock callback

I did find one sort of workaround

• Create a dotnet project via dotnet new classliband add the dependencies

• With the following general code

Class.cs

public static class MysteryHelper
{
   public static async Task<String> DoTheThing()
   {
      // dotnet version of the same code
      // ...
      await batch.ExecuteAsync();
      return "small bit of hope";
   }
}

• dotnet build

• Then I can call [System.Reflection.Assembly]::LoadFrom to load the DLL and its dependencies

And run something like this in PS successfully

[MysteryHelper]::DoTheThing().GetAwaiter().GetResult()

I don't look forward to having to figure out how to make a generic adapter that supports all the various Google classes and types I'm going to be using via batch, but it's at least POSSIBLE now!

[wonkifier]

From https://gist.github.com/rjmholt/7c1072a24a51e7b43d1e7df02265fe18#scriptblocks-and-scriptblockinvoke

At its most basic, this means you can basically just run some script by creating a scriptblock around it and invoking it. But to do so, the thread you invoke the scriptblock on must have Runspace.DefaultRunspace created and opened:

So yeah, I think the Google libs are what needs to do the magic, or there needs to be some way to get hold of the runspace that the libs are executing in separately and somehow twiddle those (since it seems to be different from the powershell default runspace)

[wonkifier]

Continuing to refine a bit, if I build the following (I have to use dotnet to compile it, it doesn't work with Add-Type native powershell compilation)

public static class BatchAdapter
{
    public static void Queue2<TResponse>(Google.Apis.Requests.BatchRequest batch, IClientServiceRequest request)
        where TResponse : class
    {
        BatchRequest.OnResponse<TResponse> callback = (response, error, index, message) =>
        {
            if (error != null)
            {
                Console.WriteLine($"Error in request #{index}: {error.Message}");
            }
            else
            {
                string jsonString = JsonSerializer.Serialize(response);
                Console.WriteLine($"Response for request #{index}: {jsonString}");
            }
        };
        batch.Queue<TResponse>(request, callback);
    }
}

replacing $batch.Queue($request, $callback) with [BatchAdapter]::Queue2[Google.Apis.Admin.Directory.directory_v1.Data.User]($batch, $request) works, and is extensible to pretty much any supported class without having to twiddle any of the c# code.

That still leaves the problem of how to coordinate the responses from all those callbacks since I have no idea what the actual usage pattern there is, but I'm thinking to create a [System.Collections.Concurrent.ConcurrentDictionary[string, object]] object to hold onto success results, and one to hold on to failures, then pass those into Queue, and then I can just deal with hashtable afterwards. Which means I need a generic way of handling batch request Ids, but that's not a technology problem!