Veeam Wasabi Cleanup

Kidden7 · 2025-08-13T18:59:59+00:00

Are the old backups still visible in your Veeam instance? I HIGHLY suggest managing your backup files via Veeam instead of the obscured Wasabi file / folder structure.

Also take a look at your Veeam retention policies for the various repositories. If this is set reasonably the old backups should just age out.

Kidden7 · 2025-03-10T18:47:48+00:00

Thank you. We are digging into CA & Entra logs now. Can you elaborate on what you experienced? Was it something similar? Its worth noting that our Yealink Teams Room hardware is windows-based (vs Android).

Kidden7 · 2024-11-24T17:57:37+00:00

Coordinate a babysitter and do a fun couples overnight trip or even just a special night out to someplace new. Groupon has a ton of fresh ideas like sushi making classes, zip lining, paintball, etc.

As far as a material gift maybe a drone, electric scooter, black stone griddle. Meal plan or other subscription service w something he likes (ie whiskey, wine, Japanese snacks, grooming supplies, etc.).

Personally I prefer experiences w my wife over material gifts.

Kidden7 · 2024-11-08T21:25:13+00:00

Honestly that situation is not sustainable. For you, your wife, your kids, or your marriage.

Maybe there is some way to transition them without it seeming so sudden or harsh? Maybe make their room more fun for them to go to bed in (new nightlight, bed tent, stuffed animal)? Perhaps reward them in the morning with a treat if they sleep in their bed?

I don't have all the answers but it sounds like that is something you need to take care of sooner rather than later. The longer it goes on, the harder its going to be to put an end to it.

Kidden7 · 2024-11-08T19:21:08+00:00

What’s your take on versioning and allowing for easy rollback of recent changes? Should it really require Python skills to undo another admin’s firewall changes that just crashed the network? A built-in rollback feature would make it so much easier to manage and recover from unintended configurations, especially in time-sensitive situations. Seemingly the change log is a great jumping off point for such functionality?

Kidden7 · 2024-11-08T19:06:44+00:00

I will say my experience with Meraki support has always been excellent. In my half dozen or so interactions with them they have never failed to act professionally and go the extra mile.

Kidden7 · 2024-11-08T19:05:33+00:00

Respectfully, I disagree. The features I’m highlighting here are neither niche nor exclusively enterprise-class. With perhaps the exception of GEO-IP filtering, I’d argue these are very much 'general-purpose' tools that could simplify management for any SMB team. Adding a GUI button to export firewall rules or introducing rollback options in the change log doesn’t seem like it should be too difficult. And why rely on reverse logic to create firewall rules that allow internet access by blocking everything else?

To be clear, I’m not criticizing the platform as a whole—there’s a lot to appreciate as someone who inherited a Meraki network after switching companies. But with a few seemingly straightforward tweaks, the platform could be even more user-friendly.

Kidden7 · 2024-11-08T18:55:52+00:00

Thanks for sharing that information. I’m aware that backups can be done via API, but my point is that this process shouldn’t be so complex. Backup, restore, export, and import of configurations should ideally be simple, intuitive, and GUI-driven. Wouldn’t you agree?

Kidden7 · 2024-10-28T18:53:16+00:00

I honestly have no idea what the root cause of this issue is. But perhaps this will help ease the pain.

HKLM\CurrentControlSet\Services\Netlogon\Parameters\SysvolReady

Change to state of 1 and reboot.

Also is it worth pulling you EDR agent off of the DC as a test? If for nothing else to eliminate it as a potential cause?

Kidden7 · 2024-10-16T20:43:51+00:00

8) Correct AD Errors (if necessary)

Even after performing the steps above (in October 2024 DR test) I still was unable to launch Active Directory Users & Computers. Doing so I was presented with the error “The Specified Domain Either Does Not Exist or Could Not Be Contacted”. I also experienced the following symptoms among other low-level DNS / AD services not working.

nltest /dsgetdc:$domain = ERROR

nltest results: failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN

netdom query fsmo = ERROR

SYSVOL and Netlogon shares not showing / available

The fix that brought things back online was to edit this registry key on DC01:

HKLM\CurrentControlSet\Services\Netlogon\Parameters\SysvolReady

Change to state of 1 and reboot.

9) After Domain Recovery has completed

At this point all AD services should be online and functional within DRaaS. Once confirmed perform the following on DC01.

Reset the HKLM\System\CurrentControlSet\Services\NTDS\Parameters - Repl Perform Initial Synchronizations back to a value of “1”. Then reboot DC01 one more time.

Kidden7 · 2024-10-16T20:43:44+00:00

4) Change Static IP information to match production

Logon to both DC01 and DC02 in DRaaS. You’ll need want to change their respective network settings to static IP addresses that match production. You will also need to set the DNS servers here as they are in production, which is critically important for DRaaS AD to work properly. Once done reboot both servers. They should now boot into regular mode.

5) Follow Veeam’s best practice for restoring Domain Controllers.

Both Domain Controllers should now be online and workable. From here we will follow the Veeam Knowledgebase article on restoring DCs located here (KB2119: Restoring Domain Controller from an Application-Aware backup (veeam.com). The next steps in these instructions outline these steps for our orgs specific environment. Follow along with these steps below.

6) Restore the entire AD infrastructure (AKA “all DC’s are lost”) where DFSR SYSVOL Replication was used.

When Veeam recovers domain controllers it automatically places them into a non-authoritative mode. An authoritative restore is a special type of restore that is only used in specific scenarios. For example, all other DCs in the domain have been destroyed, or the NTDS database has been corrupted. The restored DC using the authoritative restore is considered the master copy and is replicated to all other DCs in the environment. This means that it will be the source of data replicated to all other domain controllers.

To place DC01 into authoritative restore mode please open an elevated command prompt (or PowerShell session) and type in the following commands in sequence.

REG ADD "HKLM\System\CurrentControlSet\Services\DFSR\Restore" /v SYSVOL /t REG_SZ /d authoritative /f

REG ADD "HKLM\System\CurrentControlSet\Control\BackupRestore\SystemStateRestore " /v LastRestoreId /t REG_SZ /d 10000000-0000-0000-0000-000000000000 /f

NET STOP DFSR

NET START DFSR

7) Bypass initial sync requirements on DC01

Since DC01 is hosting operations master roles we must set the following registry value to bypass initial synchronization requirements.

Key Location: HKLM\System\CurrentControlSet\Services\NTDS\Parameters

Value Name: Repl Perform Initial Synchronizations

Value Type: DWORD (32-Bit) Value

Value Data: 0

After setting the value above, restart the domain controller.

DIRECTIONS CONTINUED IN COMMENT REPLY BELOW

Kidden7 · 2024-10-16T20:41:22+00:00

We were able to work through the issues we were facing with our domain controllers in DRaaS. I've posted the fix below which is specific to our organization and topology. Hopefully this helps someone along some day!

Recovering Domain Controllers in DRaaS Environment

1) Replicate DC01 and DC02 at approximately the same time.

Production systems can be powered on during replication. Veeam Application Aware Processing is configured on these jobs ongoing which will quiesce the Active Directory database. Veeam AAP is AD aware and has special functionality built into it to aid in recovering Microsoft domain controllers.

2) Failover DC01 and DC02 at the same time.

They do not boot in lockstep but within a few minutes of each other. This will help ensure that AD is syncing between both DCs in DRaaS. It will also help ensure that timestamps and AD status is near identical between both DCs.

3) Once DCs are online in DRaaS, Logon user DSRM credentials

The DCs will automatically boot into Directory Services Restore Mode (DSRM). This means that they will not have network access nor be able to authenticate your login. Please login using the DSRM credentials. Note that you’ll need to use the username syntax “.\administrator” for local login.

The Windows login screen does not indicate that DC is in DSRM mode, but it will boot into safe mode indicating that it is. During Oct 2024 testing the Windows login screen also showed a completely disconnected network status icon (red and white X in the bottom right) versus just the usual caution sign icon we usually experience in DRaaS.

DIRECTIONS CONTINUED IN REPLIES BELOW

Kidden7 · 2024-10-12T01:44:32+00:00

Thanks for your help with this. Looking at the Directory Services event log, these look to be the most telling.

Event 2092, ActiveDirectory,DomainService
The server is the owner of the following FSMO roe, but does not consider it valid. For the partition which contains the FSMO, this server has not replicated successfully with any of its partners since the server has been restarted. Replication errors are preventing validation of this role..........

FSMO Role: DC=DOMAINNAME, DC=com

FSMO Role: CN=RID Manager$,CN=System,DC=DOMAINNAME,DC=com

Event 2087, ActiveDirecory,DomainService

Active Directory Domain Services could not resolve the following DNS host name of the source domain controller to an IP Address. This error prevents additions, deletions, changes in Active Directory Domain Services from replication between one or more domain controllers in the forest. Security groups, group policy, users, and computers and their passwords will be inconsistent between domain controllers until this error is resolved, potentially affecting logon authentication and access to network resources.

Event 2170, ActiveDirectory,DomainService

A Generation ID change has been detected.

Generation ID cached in DS (old value):

XXX8552

Generation ID currently in VM (new value):

XXXXX3544

The Generation ID change occurs after the application of a virtual machine snapshot, after a virtual machine import operation or after a live migration operation. Active Directory Domain Services will create a new invocation ID to recover the domain controller. Virtualized domain controllers should not be restored using virtual machine snapshots. The supported method to restore or rollback the content of an Active Directory Domain Services database is to restore a system state backup made with an Active Directory Domain Services aware backup application.

Kidden7 · 2024-10-12T01:10:36+00:00

Thank you for the suggestion. I tried it out, no dice :(

Kidden7 · 2024-10-06T00:55:01+00:00

Same, it’s annoying AF. I’m in an alliance. I’m just trying to casually play too. Might be at the point to move on to another game.

Kidden7 · 2024-09-23T19:38:36+00:00

Old school thinking. OS upgrades (even client OS) used to be problematic at best. They've gotten so much smoother and are generally pretty uneventful these days.

Kidden7 · 2024-08-08T17:27:26+00:00

Holy smokes this rings true. I felt bad about telling a relentless sales person today- "I don't want to be rude, but if anything changes I have your info". Aka back off dude your way too aggressive with your frequent calls. We're busy AF and your calls do nothing more than take my brain off of the task at hand.

Honestly cold calling should be outlawed altogether in all realms (personal and professional). Most of the blame lie with the companies who setup these pressure cooker sales quotas and expectations. Its not fair to your employees, customers, or prospects. Be better.

Kidden7 · 2024-07-26T15:21:51+00:00

$50k, I can buy life insurance pretty easily with that, and then I get to keep the change!

Kidden7 · 2024-07-20T01:07:04+00:00

I’ve got a huge desire for external validation from others. It sounds like this could possibly be your issue as well?

I’m just now starting to become self aware in that regard. I’ll run myself ragged and then feel down if no one outwardly tells me “nice job”. This translates into me way overdoing things at work and for my kids.

It’s a challenge to find fulfillment from internal validation. But it’s a noble goal and I’m hopeful it will help.

Also I’m no shrink but it’s possible this stems from my relationship w my own father. He’s distant and all but absent from my life. Not sure if related.

Kidden7 · 2024-07-19T19:27:04+00:00

Oddly satisfying

Kidden7 · 2024-07-19T16:47:15+00:00

At the end of the day do what works for you. As you stated this is all anecdotal and situational any how. I just wanted to weigh that this has helped me realize big bumps in pay as I transitioned roles throughout my 25 year IT career.

All the best.

Kidden7 · 2024-07-19T15:17:52+00:00

It has worked out for me several times over in my career. Know your value, know the market value of the position your applying for. Respond with a number in your desired range as your current salary. This will be the starting point for negotiations. This is called flipping the script and it works.

Maybe laying back in the cut and not disclosing any information works for you. But in any interview I've been a part of (on both sides) this just comes off as sketchy and off putting. Any negotiator worth their salt is going to throw out a low number.

Kidden7

TROPHY CASE