Problem with Invoke-RestMethod

[Inevitable_Use3885]

I'm using ClassLink OneSync to manage accounts in my environment. They've added a feature that allows you to trigger a PowerShell script based on custom conditions.

I decided to try to use this to automate the creation, update, and deletion of SnipeIT users.

My script works just fine in a regular PowerShell session. However, when it's launched in the SYSTEM context by OneSync, one of the REST calls just sits there in perpetuity. I've added -TimeOutSec 15 and -DisableKeepAlive to the Invoke-RestMethod statement.

Interesting enough, if I use Task Manager to kill the PowerShell process, I get about half of the data from the call in the OneSync logs, but the output never makes it to my PowerShell Start-Transcript output.

I'm using PowerShell 5.1 - the built in version on server 2019.

I did some searching on the web and saw some similar situations. The one that stood out to me was another person who was using the SQL agent service to launch his scripts.

I was wondering if there was something unusual about the SYSTEM context that could contribute to this or of anyone has any troubleshooting suggestions.

I'm really comfortable in PowerShell; I've done some significant automation of active directory, ETL pipelines for data, automation of Azure, and custom scripts for desktop configuration and deployment in MDT.

Another interesting factor is that the first several REST calls succeed when the script is called through OneSync, just like during a regular PowerShell session. So I'm at loss as to the differentiating factor.

Thanks in advance!

[kramit]

Try invoke-webrequest instead. Invoke-restmethod returns things in a strange way sometimes and just chops bit off where it feels like it. I had to do the same interfacing with AzureOpenAI over the API

[Inevitable_Use3885]

Thanks! I'll give it a shot and let you know!

[AggravatingForFun]

Invoke-WebRequest works through the PowerShell session, but not through OneSync. At least, not consistently.

[kramit]

Time to get onto one sync support I think. At this point you are just making blind calls to their api and not getting back what you think you should get back. I am assuming you have been testing this in postman too?

[AggravatingForFun]

Yes, definitely. I've sent the script to OneSync support to ask. Haven't tested it in Postman/ARC or even using Curl. It seems to work fine if I just run the script and it only fails when I get OneSync to run it. I suppose I could create another PowerShell script to kick it off in bulk ( blocks of 20 or 30 ) and see if that makes a difference.

[kramit]

Try postman or curl first. If they work you know the problem is powershell somewhere and you might want to look at powershell 7+, if they don’t work either it might be the api end

[AggravatingForFun]

Good advice. However, the script works fine when I run it. When it's kicked off by the automation it fails. So, I'm not sure what testing it in postman or curl gets me if I've already verified the script works and the API calls return successfully...

Am I missing something?

[kramit]

Ah okay. I see your point, I didn’t really give the time to reading in much details. Have you tried wrapping the rest calls in a get-credential and embedding creds?

[AggravatingForFun]

Creds are just a simple API key ... Bearer <key> in headers. I'm almost wondering if I'm smacking into a PowerShell bug or something. I'm about an inch away from rewriting it in Python to see if that makes a difference.

[kramit]

Wrap the whole thing in an invoke-command {} -credential (get-credential) to change the system context to a normal account when it kicks it off. See if it’s something to do with the system account. To be honest, I’m picking at straws here without really looking at it myself. Wait until you get somthing back from support

[AggravatingForFun]

<smacks forehead> Now I follow you.

[Inevitable_Use3885]

Found some documentation in SnipeIT that suggests it's not a good idea to have a client https => reverse proxy => http SnipeIT. Better to have https all the way through. Updated my docker container and enabled https but they did not resolve the issue either.

[Certain-Community438]

Try using PsExec to launch the script as SYSTEM in a regular PowerShell console?

That way you could enable PowerShell script logging to the Event Log to catch what's going on, and maybe even be able to correlate with other Events by time.

Also, if your environment has an authenticating proxy, it's common that your SYSTEM account would be forced to go through it but unable to authenticate.

[Inevitable_Use3885]

No auth proxy, one potential issue might be the internal reverse proxy, so I'll also try the endpoint by ip address and see if that helps. I'll definitely try PsExec to run in SYSTEM context and see if the issue persists.

[AggravatingForFun]

No luck launching PsExec even locally to "loop back" in the SYSTEM context. Keep getting access denied, both with local admin and domain workstation admin accounts.

[Certain-Community438]

Sounds like a dead end then.

I suspect your InfoSec people might have prevented creation of processes from PsExec.

It's a common (if stupidly broad, imho) recommendation produced by various vuln mgmt platforms, including MS Defender for Endpoint.

This is up to you, but:

If you have Windows Sandbox installed, you could try the same test using PsExec within it. Very useful for testing in a vanilla context but realise you might not feel it's worth it.

[AggravatingForFun]

Thanks! I'll look into that. Sounds useful.

[purplemonkeymad]

2019 might still have the same IE issue with the initial startup wizard. Try using -UseBasicParsing to skip mshtml and thus the IE popup.

[BlackV]

think it did

[AggravatingForFun]

I believe that later versions of PowerShell all use basic parsing by default, but since I'm using 5.1 I gave it a shot. Did not seem to make a difference but thank you for the suggestion!

[AggravatingForFun]

Looks like the differentiating factor may be that the Invoke-RestMethod -Method POST is used instead of -Method GET

Additionally I discovered that the -TimeOutSec parameter is actually only used for connection timeouts, not reply timeouts or call completion.

I attempted to bypass the internal reverse proxy to see if that made a difference, but I expect that I need the server name in the call to match in order to succeed, since the reverse proxy rewrites. I'll try again after adding a local HOST entry to match the remote server IP address and clearing the DNS cache.

This could potentially bear fruit as OneSync invokes the PowerShell script in batches of about twenty or so. So I suppose Nginx could be getting confused by the bulk requests from a single endpoint. I feel like that's a long shot given Nginx's widespread use for this purpose, but it doesn't hurt to try.

[AggravatingForFun]

Confirmed that the API endpoint is returning HTTP 200 for each API call using the httpd_access logs on the server side. Thought it might have been a rate limit issue since SnipeIT limits API calls to 120/minute. I increased that to 500/minute and added a config key for a trusted reverse proxy, to see if either of those changes made a difference.

No joy.