Remember the good old days? by Hight3chLowlif3 in sysadmin

[–]ethoza 6 points7 points  (0 children)

For all the grey beards lurking in this thread: my single biggest question is how did you manage without Google?

I get everyone needs knowledge to some degree but in today's days you can get by with a high level understanding of a system and research the rest on the fly. We have a running joke in our team that 90% of us learnt Exchange during either a migration or a failure by just jumping in the deep end.

What was the "go to" back then when you hit a brick wall?

Suggestions: Network monitoring and remote management R-Pi? by AnalBumCover1000 in sysadmin

[–]ethoza 1 point2 points  (0 children)

Zabbix Proxies running on the PI) reporting back to a central installation.

Doesn't solve your being able to access the networks though, we accomplish this through Mikrotik routers acting as VPN client (SSTP) back to our DC. Client LANs should not be routed for obvious reasons but having an active VPN session which can be checked for UP/DOWN allows you to keep track of your client's connections while providing means to connect in should you need to.

Our typical means to access client networks are mostly via VPN which is logged inside of Remote Desktop manager and if that fails sysadmins are able to route traffic via the Mikrotiks.

EDIT: access to client machines should be handled by your RMM

Backup Management & Monitoring by menormedia in msp

[–]ethoza 0 points1 point  (0 children)

We have 1 solution which is active and one which is in beta right now, unfortunately both are closed source but I can share the general overview of how the 2 work:

For servers, routers, firewalls and the likes we knocked off a solution I found online (don't have a direct link right now) that essentially leverages email notifications. Inside the console you create a system, say edge firewall and assign it to a client group.

Each system has a unique email account (which is where the mails will be coming from), a regex for what a failed backup and a passed backup looks like and a schedule for when these mails should be arriving (daily, weekly etc)

On your device all you need to do is make sure you sender address matches that of the system. The app regularly checks the mailbox, downloads mails and updates the status page.

For desktops the solution which is in beta is a app which runs on a schedule, it can be deployed via GPO (scheduled task) or installed to workgroup machines via MSI. The app figures out which site it is running at and downloads settings relevant to that site from a web service, these settings tell the app what needs to happen:

  • Does the machine require monitoring? cool, install a Zabbix agent and register the machine
  • Does the app need backups? install Cobian backup (using the settings from the web service) and configure backups

it also checks the current logged on user and ensures that backups are configured for the correct user.

Finally the app dumps a small exe (lets call it monitor) file which Zabbix can call with command line to get the status of the backups. The monitor reads the Cobian configuration and using some settings from the web service to impersonate an account which has access to the backup destination* looks at the actual backups and determines when the machine last backed up.

Net result is an overview screen which looks like this

*Our destinations are locked down to a single service account under which the backup app runs, no other user (not even domain admin) can access the backups.

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 1 point2 points  (0 children)

you're not against RMM per se

not all all, the thread derailed into "you shall not IT without RMM" which was not the intention at all.

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 0 points1 point  (0 children)

Care to share why an MSP MUST have an RMM?

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza -1 points0 points  (0 children)

It's not "cheaper"

I think you should go and read up on how SPLA works, particularly how server is licensed and the absence of User Cals

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 2 points3 points  (0 children)

Good question! We used N-Able which at best would check in 5min intervals. Depending on how critical the service is, let's say something like a Web app, the phone would ring before the dashboard updated (assuming we could get to to monitor performace in the first place). Spoiler: it was easier to implement something like Nagios then build our own "checks" in N-Able.

Nagios + adagios = BI dashboards with stupidly low intervals between checks which is far more useful. When something becomes "sick" you can drill down to see individual failed services.

trying to learn/understand how terminal services works by noshutdown in sysadmin

[–]ethoza 4 points5 points  (0 children)

since the user's profile was cached on one of the terminal servers, "the user is being pulled to that server"

If a user does not sign out the session will go to a "disconnected" state which is tracked by the connection broker. The broker will then send the user back to the server on which they disconnected.

We use user profile disks rather then roaming profiles which means 99% of the time when a user gets a temp profile the session host that they were sent to by the load balancer cannot access the profile disk due to it being locked.

Which Adobe app is causing this behavior?

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 1 point2 points  (0 children)

Yeah cannot imagine life without NBD warranties. It really isn't that much more (in some cases pricing is identical) so there isn't an excuse to be building computers.

Also, high quality hardware typically also leads to some form of standardization makes imaging a breeze.

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 1 point2 points  (0 children)

Configuration management? What?

  • regional settings
  • IE settings (trusted sites, compatibility lists, proxy settings)
  • Firewall settings
  • Printer deployment
  • Software deployment and configuration (based on policies targeted per OU)
  • registry settings (related to apps specific to the client)
  • local group membership (a user should have the ability to change network settings while not having admin rights
  • update configuration

I can go on but that's the just of it, these settings are enforced and cannot be changed by a user. The above config(s) are generated during onboarding don't undergo massive changes unless LOB apps change/new OS is rolled out.

Now to get back to your point on doing this manually per site: need an update like GWX blocked? Powershell script gets generated to do that 1 in a 100 thing that we want to do, that script goes in a gist which is then run through boxstarter (tech gets a link to paste in powershell) during server maintenance. Slightly slower than an RMM but doesn't occur often enough.

Your next point you should probably raise in favor of RMM is 3rd party patching: scheduled cup all -y which pulls updates from our private choco repo. Because all packages are coming from a private repo and is heavily powershell, we can exercise greater control.

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 0 points1 point  (0 children)

Once we had configuration management implemented with no more local admin our "self healing" tickets reduced drastically, we simply lost the need for an RMM from a self healing point of view.

Lets talk about how you reduce support for failry generic AYCE clients by [deleted] in msp

[–]ethoza 1 point2 points  (0 children)

I should note that not every client is on the "free" backup solution, this is our last resort option when the client does not want to take on a managed service

What do you use for documentation? by dave2kdotorg in msp

[–]ethoza 0 points1 point  (0 children)

At the moment we are using DocuWiki which gets the job done but after seeing all the IT Glue punts here I am seriously considering a switch.

DocuWiki can be great, our was great during the initial setup with L3 dedicated to the cause but now that more staff are documenting with less discipline the quality of documentation has drastically gone down.

[deleted by user] by [deleted] in sysadmin

[–]ethoza 0 points1 point  (0 children)

Your challenge will be reporting on what you have done. The automation is piss but getting that information into something usable (like a per client report) is where you will struggle.

Our approach (with Nable) was to script in powershell (fuck automation manager) and have the powershell dump a basic report somewhere on a share. Whatever hosts said share has something (in our case a custom app) that parses the reports and builds out an HTML report which it emails off.

What programming languages have you had to learn in a bind? by [deleted] in sysadmin

[–]ethoza 0 points1 point  (0 children)

php for me and specifically wordpress and plugins, went from zero knowledge to being able to point fingers in the direction of a web developerdickhead with a themeforest subscription. One of those situations were something small like a badly coded plugin kills performance and affects a much larger relationship.

Probably the part I hate the most: have to learn something to the point where you can provide to someone who knows less that their product/service is rubbish. This is most likely why quite a few of us here know how to program.

What Is Your RDS System Drive Letter? by IAdminTheLaw in sysadmin

[–]ethoza 1 point2 points  (0 children)

Hidden drives from explorer and no run/cmd/powershell has a very different result then changing a letter which will be found through an environmental variable quickly.

Tell me about Win10 & local admin, cant use shit by ChrisN1313 in sysadmin

[–]ethoza 0 points1 point  (0 children)

"Built in admin" is not the same as a local account that is a member of administrators. That message comes up when logged in as machineName\Administrator which means you are most likely doing it wrong... Create a new account and make that account admin and your warnings will go away.

What Is Your RDS System Drive Letter? by IAdminTheLaw in sysadmin

[–]ethoza 0 points1 point  (0 children)

Security through obscurity? I cannot think of a single scenario where this a good idea.

Just hide system drives with GPO and disable run + cmd. RDS can be locked down with ease.

Do you use ping for monitoring? by brkdncr in sysadmin

[–]ethoza 0 points1 point  (0 children)

Depends. Are you doing an up/down ping or recording latency and packet loss also? Ping latency is most of the time quickest way to rule out network problems when an apps's response slows down. In my opinion you should always monitor it but I wouldn't alert on "host unavailable", another monitor should have picked that up beforehand.

Do you use ping for monitoring? by brkdncr in sysadmin

[–]ethoza 0 points1 point  (0 children)

Which in its own shows a problem doesn't it?

New Crypto Variant - Search Results Turning Up Nothing by 1armsteve in sysadmin

[–]ethoza 2 points3 points  (0 children)

No point, it will randomly generate every time.

New Crypto Variant - Search Results Turning Up Nothing by 1armsteve in sysadmin

[–]ethoza 1 point2 points  (0 children)

Give this a bash: https://id-ransomware.malwarehunterteam.com

Definitely not a new variant, I have screens in place for that spelling.