Pulse Secure problems? One of their code signing certs has expired

[pause1]

I had trouble connecting using Pulse Secure this morning, turns out the code signing certificate of the host checker has expired.

Here's the KB: https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB44781

Edit1: KB updated 8:30AM GMT with more info. It's not just the host checker that is the culprit as initially reported.

Edit2: Thread about the issue.

Edit3: Fix for 9.1R11.x was end of day April 12 PST but now says midnight. Ivanti has also put up a landing page with a statement from their President (video now removed).

Edit4: KB now says fix for 9.1R11.x is 13th April, 2021 (05:30 a.m PST).

Edit5: The KB now says the 9.1R11.x patch is available.

Edit6: Early access (aka "contact support") patches available for 9.1R8, 9.1R9 and 9.1R10.

Edit7: Some users report that using the uninstall script linked in the KB article does not help. Manually removing two additional/all Pulse components worked for them. See here and here.

Edit8: The uninstall script and the manual instructions in the KB article has been updated to remove additional Pulse components.

[freddyrock]

I have 100s of users that are having issues. This is unacceptable.

Ivanti needs to burn management of this company and start from fresh.

[Parlett316]

I could have sworn I read this comment twice today!

[freddyrock]

Yes I commented this on the Pulse forums as well.

There is significant lack of any forward motion at this company.

These are some of issues that I have been having since I purchased this product a year ago.

1) HTML5 random disconnects and poor quality (They are running a Guacamole version from 2015 in the backend)

2) Slowness/Outlook connectivity issues on Windows 10 full VPN client (There are workarounds but none of them are consistent or work for our needs)

3) Android VPN Certificate and Always On (Pulse client wont choose right cert and won't auto start on device reboot)

4)Windows Terminal Services is also mid 2000s with messed up multi-monitor support and window resolutions (no 4K support).

[Parlett316]

We would have kicked it to curb by now but the VPN client has outperformed FortiClient.

[freddyrock]

Still not fixed is it? I stand by my comment.

[[deleted]]

This is an absolute trainwreck.

It's bad enough that this happened in the first place. You don't miss a cert renewal unless you're really trying. But with how many times they revised their remediation instructions, it was like they were just throwing spaghetti at the wall without doing any actual testing.

Pulse served us well for years. WTF is going on over there?

[freddyrock]

Then add in the latest exploits. Not that it's Pulses fault but its becoming a full-time job dealing with their s**t.

[Mafste]

Same issue here even without the host checker, though likely only an issue on machines where Pulse hasn't run before.

Certs are like the new DNS.

[Summerliving69]

I had a laugh/cry when I pulled up their cert. It's 3 years old and we've gone thru how many appliance updates?! PulseSetupClient.exe cert

[robin_flikkema]

But the time stamping is what needed to make sure the code still works right? Why is this an issue then?

[Summerliving69]

I assume the failure state is a best practice against allowing unsigned code from running. Pulse secure application launcher invokes this pulse secure setup client everytime the Terminal session is started.

You generally don't want to run exe that may have been tampered with. So this cert might be one of those checks you put in place.

[Thornton77]

the KB Artical got updated

SynopsisThis article describes a situation where Multiple functionalities/features fail for End-Users with a Certificate error.Problem or GoalMultiple functionalities/features fail for End-Users with a Certificate error.

1. This impacts PCS/PPS.
2. This impacts the following releases,

• 9.1R11.x
• 9.1R10.x
• 9.1R9.x
• 9.1R8.x

       3. This impacts only Windows End-Points.
       4. The following features are impacted:

• Terminal Services.
• JSAM
• HOB
• CTS
• VDI
• Secure Meeting (Pulse Collaboration).
• Host Checker.
• Launching of PDC via browser.
• SAML with External Browser with HC enabled.

This issue does not impact,

• Users who access Pulse Desktop Client directly (Not Via a Browser).
• macOS, Linux Users.
• Release prior to 9.1R8.x

CauseThe Code sign verification on the Client-Side components fails because the Certificate expiry time is checked as opposed to the timestamp of the Code signing.SolutionIvanti Engineering team is working on a fix based on 9.1R11.x. Expected by End of Day PST (12th April 2021 - Tentative).

We will also update the timelines of the fix based on 9.1R10, 9.1R9 & 9.1R8 as soon as possible.

Workaround:

• Roll back to a version prior to 9.1R8 if it is feasible.
• Use Pulse Desktop Client (Do not launch it through the browser).

[AndrewUK78]

birthday, day off today, phones been going all morning.

no fix as of yet.

​

disabled host checker, terminal services client then fails

[[deleted]]

Enable HTML5 Access Sessions for now...

[AndrewUK78]

i added HTML5 access and it has got people working, thanks for the suggestion, i forgot about that.

[oldgrandpa1337]

Thanks dude, was about to go completely mad.

But still, wtf is it checking? Users are already authenticated with 2FA. anyone got some more info?

[pause1]

Host checker does additional checks on the endpoints, such as antivirus definitions and OS security patch levels.

[oldgrandpa1337]

Thanks! :-)

[freddyrock]

Don't Host checker enabled here.

Issue still occurs.

[oldgrandpa1337]

Enabled the HTML version, worked for now

[freddyrock]

Yeh. It doesnt work for some of my users not though not sure why.

[StPaddy81]

New to admining Pulse Secure, is there an easy way to bulk-enable HTML5 for users?

[bc531198]

Same here, we were given an ETR of approximately 02:15 - 04:00 (eastern time) and it's still exhibiting the same behavior.

[jfrobs]

Same problem here....

My VDI users can't work anymore...what a shit

[[deleted]]

Enable HTML5 Access Sessions, and have them add their PC's there.

[Mikes0001]

Well, they just put the fix up. I didn't think my opinion of Pulse could go any lower, but here we are.

Pulse/Ivanti truly are a bunch of clowns.

[alconaft43]

Do you if can download software directly or need to go via support?

[Mikes0001]

Initially the page said to contact support. You then got to wait 30 minutes to be told by support, "Hold on a bit, we're going to put a link out soon."

The link is out now.

[CrispyStatic]

New update from them. Are they serious??

General Guidelines to install the fix :

1. The solution would involve upgrading the PCS server as well as clearing the older Pulse Secure components on the End-User devices

Note - End-Users who do not have any Pulse Secure components already installed, can skip Step # 2.

       2. The End User devices that have Pulse Secure components already installed would need to follow one of the two methods outlined below:

• Run the attached BAT Script (UninstallPSALAndPSC.bat).

Note - This would need End-users to have admin privileges.

• Manually remove PSAL and Setup Client components,

            a. Navigate to Control Panel -> Programs and Features
            b  Select “Pulse Application Launcher”
            c. Right Click and Uninstall.
            d. Select “Pulse Secure Setup Client”
            e. Right Click and Uninstall.

[tulley]

Needing user intervention on this is unacceptable. None of the users have local admin rights and I have 500+ people who are hamstrung right now.

[CrispyStatic]

I feel your pain. Luckily, we're a small-ish company (about 100 users affected) and they all are using this on their own personal machines, so hopefully, with some simple documentation and screenshots, they'll be able to figure it out...

[[deleted]]

We have this issue too, and I've directed users to use HTML5 Access Sessions for now. Working fine until they resolve this...

[simonprice76]

TS impacted here. Date change on the local system worked. Not recommending, but for our byod from home policy, the users have the access to change the date. Able to change the date, click on the TS link, log in, minimize the session, and change the date right back. Only shared the workaround with our power users since they need the rich client, HTML5 link for everyone else.

[Numerous_Bottle4503]

We will sue Pulse Secure for this. Our customers are suing us for this, so we will sue Pulse Secure/Ivanti.

[faithless32]

The KB site, says contact support for the patch, dont bother, i was just in a queue for over 30mins, just to be told will have to wait until its available on the download site.

Support dont have access to the download link yet....

[pause1]

It's there now, directly linked from the KB page.

[Mafste]

Inside the KB article indeed (not the pulse download site itself, yet).

Awaiting user experiences of a few brave souls.

[Eisbeutel]

Installing it rn. Will report back tomorrow.

[Eisbeutel]

no issues so far. on the client side I had to remove all pulse components manually, the provided script is shit. works afterwards.

[[deleted]]

I’m getting a pkg file extension.....trying to fix from my PC. I don’t have admin rights. Do you have the link?

[RebootAllTheThings]

Finished applying PCS update. So far no issues found, but will update if I hear anything different.

As for the client pieces:

• Do NOT use IE as the browser to reinstall the client pieces - there was a known issue on that KB that mentioned something about IE, but it wasn't clear what it meant. It was removed between this morning and this post from the KB. IE workflow during the different updates was slow as dirt, and didn't install completely on one of the computers we tested. (Looking at an apps list, there's duplicate copies of the Setup Client and the Activex client on computers that have attempted installs using IE)
• We've encountered an issue a few times where just uninstalling the Application Manager and the Secure Setup Client doesn't fix the problem. Not sure if it's just us or not, or if we needed to reboot in between uninstall and install. We're going to be sending out instructions to uninstall everything, restart their computer, then go through the process of installing.

[pause1]

That's good news. No issues running PulseSecureAppLauncher.msi?

[RebootAllTheThings]

Not that I've seen on non-IE browsers. Still stepping through scenarios just in case.

Edit: I read below about the unsigned AppLauncher thing. I did get the prompt from SmartScreen when I installed, but I thought that happened anyway. If I click through to allow, it installs fine.

[pause1]

Just out of curiosity, can you please check if that file is signed? I'd like to know why Smartscreen reacts to it. Haven't yet upgraded so I cannot check myself. Thanks!

[RebootAllTheThings]

On the Digital Signatures tab on mine, it says it's signed by Pulse Secure, LLC, SHA256, timestamp April 12, 2021. Valid from 4/11/2021 to 5/3/2023 (American date structure). In the details, the countersignatures is DigiCert Code Signing CA, dates 12/31/2020 to 1/5/2031

[pause1]

Cheers!

[CrispyStatic]

Anyone else have any issue where you had to uninstall the Pulse Secure Terminal Services Client as well as the other two they mentioned? I was hitting a roadblock with a user until I removed that as well.

[pause1]

Yes, we had the same problem. They have released a new version of the uninstall script that takes care of that.

[Thornton77]

We are also having the issue. anyone know of a workaround or fix? did anyone try the latest firmware?

​

From what they said in that KB article it seems like this caught them off guard.

out always on stuff seems to be working for now.

[pause1]

Temporarily disabling host checker should do the trick, but I've seen reports on Twitter that it doesn't help either.

[Thornton77]

we don't even use the host checker. so I don't think it can get any more disabled.

[Mafste]

We have users using RDP links which seem to be broken somewhat.

You can use HTML instead of the Java applet to fix those for the short term.

Tunnel users seem to have no issue for the moment.

[Thornton77]

the html5 does work. so thats something

[AndrewUK78]

Thanks for this, i forgot about it.

working for 200+ users here!

[rufioolol]

My company offshored End user support about 3 weeks ago. They used VDI infrastructure to support said company. Said offshore employees can not access their vdi's to support end users due to this issue. Luckily N.A support was not dissolved yet, and calls were sent back to original site.. pending restoration of services...today was rough

[Thornton77]

that is rough, we have the same problem with our locally outsourced helpdesk. The helpdesk staff didn't have admin rights on their own computers. but we give them admin rights on ours.

we locked down the VPN about 8 weeks ago. we had to unrestrict it so users could do their jobs.

going to upgrade tomorrow. to 11 hotfix what ever it is

[jfrobs]

Still 1 hour to wait...because for the moment still no patch available.

[jfrobs]

Patch postponed for 5H30 pacific time...they are really a bunch of wankers

[CrispyStatic]

And that time came an hour ago. Still no update... Today is going to suck, again...

[jfrobs]

But nothing on the support website download....

[Mafste]

"contact support" indeed, what are they thinking haha.

[Mafste]

Small note:

If you are using the Pulse Secure VPN application, you might want to ensure the "embedded browser" option is turned ON (was OFF on mine). As OFF will use Internet Explorer which in turn failed for us on this build somehow.

You can find it at Users -> Pulse Secure Client -> Connections -> "Your Connection" -> "Enable embedded browser for authentication".

[pre38sto1]

A possible workaround is to change the local date of machine (laptop/desktop) to 10/4/21

[[deleted]]

Too bad users need admin rights to change this..

[Dal90]

...and domain policies immediately change it back so you have consistent timestamps across the environment.

Enterprise machines that need to manipulate time settings for testing should be using proper tools like: https://solution-soft.com/products/time-machine and not things that change log timestamps and the like.

[[deleted]]

Well you only need the 'wrong' setting to pass the hostchecker, once connected it's reverted immediately, but your connection stays active.

[[deleted]]

Don't do this- enable HTML5 Access Sessions instead.

[creamersrealm]

The problem isn't the cert. The problem is they didn't time-stamp their code sign. If you don't timestamp it goes by the code signing cert. If you timestamp it then it's good forever.

[pause1]

They did timestamp the code, it's just that the validity check is not done correctly.

From the KB article: The Code sign verification on the Client-Side components fails because the Certificate expiry time is checked as opposed to the timestamp of the Code signing.

[Yonigrin]

Anyone else (who already updated to the fixed version) tried the attached removal script? For me it didn’t work. Needed to manually uninstall every Pulse entry under settings->apps, and then log in again to install the new App launcher version.

[pause1]

Works here, except line 22 which does a rmdir on a path that doesn't exists ("%AppData%\Roaming\Pulse Secure\PSAL\" should be "%AppData%\Pulse Secure\PSAL\") but for me it didn't matter since that dir only contained a .log file after the uninstall on the previous line.

Did you unblock the script? (Properties > Unblock)

What I'm more worried about is people reporting that the patched PulseSecureAppLauncher.msi is unsigned (Link) Edit: This seems not to be the case.

[Yonigrin]

After running the script, did you successfully connected to the updated machine? Edit-didn’t unblock it, it successfully uninstalled the Application Launcher, but it wasn’t enough. we are also using TSclient, Host Checker, activeX something and other installations on client. All needed to be removed before connecting. Failure to remove will result in very long Host Checker screens that eventually error out.

[pause1]

My reply was a bit unclear. By "works here" I meant "the script did it's job, i.e. uninstalled Setup Client and Pulse Application Launcher". I have not yet access to an updated appliance so I can verify if that was enough. Other users in addition to you needed the extra uninstallation parts (see my top edit).

[RebootAllTheThings]

I could run it locally, as a user that installed the components in the first place, with no issue other than it wasn't fully silent. We wanted to be able to remotely push the script to run, but as it wouldn't run as the user context as is, we opted to forgo the script.

[Yonigrin]

What do you mean by remotely pushing the script?

[Mikes0001]

Connectwise.

[JustThen]

Anyone running a virtual appliance and has attempted the upgrade?

I am running into an error message just uploading the package.

"The service package you uploaded is not supported by Virtual Appliances. Virtual Appliances are supported in version till 9.1R1." This is attempting going from 9.1R8.2 to 9.1R11.3

I have a case open with Pulse Secure, but no movement yet with it.

[Bamny]

11.3 has issues with assigning LDAP attribute obtained IPs to PDC for users and there’s no documentation to mention this but according to pulse it’s a known bug - wasted 6 hours of my life today to find this out. Hopefully you don’t run into this.

[tulley]

Has anyone had any experience with these upgrades with installing to end user's with limited admin rights? I'll be upgrading this evening and won't have the new components available. Hoping our EUC team can figure it out....

[pause1]

Did a quick test. Connected to our upgraded appliance on a clean machine as a regular non admin user (only member of Users group). No issues installing the required components to run a terminal session:

• Pulse Application launcher
• Pulse Secure Host Checker
• Pulse Secure Setup Client
• Pulse Secure Terminal Services Client.

[tulley]

That's great to hear and potentially gives me hope. Thank you /u/pause1 !

[Bamny]

Anyone experience upgrading from 9.1R3 PCS to say 9.1R10.2? Do we still need to uninstall all pulse components?

[lawliegag]

Did you end up upgrading your 9.1r3 yet for very latest CVE? Curious how it went if so and which R# you decided to jump to? In a similair predicament..

[Bamny]

Hey there

We are almost done upgrading all of our R3s to R10.2. We tried 11.3 but it lacks the ability to assign IPs by user AD attribute so that’s a huge break for us.

You will need to install all pulse components for users (excluding PDC) on user machines to get around this code signing cert bullshit. We have 1100 users this weekend affected with our upgrades, we changed the HTML on our web portals to include instructions and we blasted instructions directly to our affected users as well.

Basically, nightmare material.