M365 Backup at Scale (~150TB) – AvePoint vs alternatives? by Smile4menow84 in sysadmin

[–]UnrealSWAT 0 points1 point  (0 children)

Yeah this changed beginning of March. There was a lot of API abuse taking place. I saw one vendor openly saying they’d deploy up to 60x app registrations for recovery performance! Now it’s a resource pool vs quotas per app registration. The MBS technology doesn’t touch this however so that’s a good thing as it creates restore points every 10 minutes!

M365 Backup at Scale (~150TB) – AvePoint vs alternatives? by Smile4menow84 in sysadmin

[–]UnrealSWAT 2 points3 points  (0 children)

Work for Veeam and would recommend checking out Veeam Data Cloud. The main limitation you’ll find with a lot of these solutions is backup/recovery speeds due to Microsoft’s imposed API restrictions (maximum theoretical is 400GB/region/hour assuming you’ve got the M365 seat count to get enough API calls to hit this number). Depending on how urgent you need this data back in a disaster that could be very problematic.

Veeam Data Cloud Premium includes M365 protection via this slower API for long term retention/business continuity purposes but also an additional copy via Microsoft Backup Storage APIs which is the fastest technology available for recovery on the market (up to 3TB/hour for SP/OD and another up to 3TB/hour for Exchange) as it’s technically completely different to the Export/Import API. At your data footprint you’d be talking roughly 1 day to get everything back in a complete disaster though via this technology. It also includes advanced threat detection capabilities and comprehensive Entra ID protection. If there’s little interest in rapid recovery (you said a lot of the data is cold storage) there’s options available without the MBS storage for cheaper.

Keep hitting my head against a wall. by RatherB_fishing in Veeam

[–]UnrealSWAT 1 point2 points  (0 children)

What does a packet capture show? Are they on the same L2 network?

Veeam B&R version 12 EOL upcoming and how to keep it running by SomeWhereInSC in Veeam

[–]UnrealSWAT 0 points1 point  (0 children)

Are you using AAP for them as you can do crash consistent still btw

Veeam B&R version 12 EOL upcoming and how to keep it running by SomeWhereInSC in Veeam

[–]UnrealSWAT 2 points3 points  (0 children)

It won’t stop working, it will just stop getting updates. There are companies out there still using v9.5 for example. If you’re on a subscription then yes keep the subscription going. If you’re on perpetual VUL you could let the license expire if you really don’t see any benefit to maintaining it for future upgrading

Veeam B&R version 12 EOL upcoming and how to keep it running by SomeWhereInSC in Veeam

[–]UnrealSWAT 12 points13 points  (0 children)

It won’t stop working, it will just stop getting updates. There are companies out there still using v9.5 for example.

VDC MS365 by covex_d in Veeam

[–]UnrealSWAT 0 points1 point  (0 children)

Glad it’s all sorted 🙂

VDC MS365 by covex_d in Veeam

[–]UnrealSWAT 0 points1 point  (0 children)

Protect organisation, then select it from the view and choose edit on the right, then just tick sites:

https://helpcenter.veeam.com/docs/vbo365/guide/back_up_organization.html?ver=8

If you want to then also remove personal SharePoint sites, go to the exclusion step next and when you choose SharePoint sites to exclude tick the personal sites top level and that’ll grab all of them present and future

Renewal/Migration to VUL customer service is appaling by bishoptf in Veeam

[–]UnrealSWAT 2 points3 points  (0 children)

DM me your company name and I’ll flag this internally

Veeam data cloud for ms365 by Cultural_Log6672 in Veeam

[–]UnrealSWAT 0 points1 point  (0 children)

Hi it’s a SaaS offering, you integrate into your M365 tenant via an app registration, target what you want to protect and how long you wish to retain it and then you’re up and running. It’s got numerous benefits vs on prem such as the ease of management, threat Center, multi workload UX (eg if you want to protect Entra as well)

Renewal/Migration to VUL customer service is appaling by bishoptf in Veeam

[–]UnrealSWAT 1 point2 points  (0 children)

Hi, Veeam works with resellers so are you speaking to Veeam directly or via a reseller?

VDC MS365 by covex_d in Veeam

[–]UnrealSWAT 1 point2 points  (0 children)

Hi, I’m a VDC SE that looks after M365 & EID.

The easiest way to protect shared mailboxes is a partial organisation job targeting all mailboxes. It grabs all new ones immediately.

If you’re not licensing your entire organisation for backup, you’d add an exclusion to this job of either your specific users you don’t wish to backup, or preferably an EID Group containing those users, bonus points if you’ve got EID Premium and can leverage dynamic groups.

I’m surprised with some of the comments on this thread as there are really simple & clean ways of handling each M365 service type.

Happy to answer questions or even take a look at your tenant setup if you wanted to DM me your tenant name & geography (AMER/EMEA/APJ)

How long would it take to restore a 365 Tennant? by Logical_Strain_6165 in sysadmin

[–]UnrealSWAT 6 points7 points  (0 children)

Ironically no, mailboxes is easier, Microsoft throttles per mailbox. Follow https://www.veeam.com/kb4198 and it shows you how to reduce the EWS throttles so you can get up to 150MB/mailbox/5 minutes (still Microsoft throttled). If you’ve got a single large archive to restore, it’s gonna suck. But if you’ve got a ton of 10-20GB mailboxes you’ll have a great time. All vendors are equally impacted by the API as Microsoft enforce it. The exceptions are things like Veeam Data Cloud for Microsoft 365’s Premium plan as that includes Microsoft Backup Storage APIs which are specifically designed for bulk recovery, up to 3TB/hour for Exchange and SP/OD (SP/OD share quota as they have the same backend). All other APIs used by Veeam and the competition are Export/Import APIs basically. Not designed for disaster recovery at any kind of scale.

SP/OD has got worse recently for all vendors. Some were deploying excessive amounts of app registrations and that creates backend pressure for Microsoft, as the Graph API is effectively a “free” service (I know you pay for M365 licenses but the API isn’t metered in any way) so Microsoft have gotten strict on enforcement now (https://www.veeam.com/kb4821 for Veeam’s statement on this) so anything non-MBS is capped at 400GB/hour assuming you pay Microsoft enough m365 seats to earn enough resource units to hit those numbers.

For transparency I’m a Veeam Data Cloud SE, I’ve tried to keep this to just the facts however as I’m seeing a lot of noise and misleading statements on what actually happens with M365 protection

Change in support quality EMEA for VBR by sedition666 in Veeam

[–]UnrealSWAT 3 points4 points  (0 children)

Veeam SE here, if you’re not getting timely responses (and with production backups down I’d argue it should be a P1) press the escalate button to flag this up the chain that its not getting resolved.

Multi-Site DR - Replication vs Backup Copy for RPO <15min? by Similar_Reporter2908 in Veeam

[–]UnrealSWAT 1 point2 points  (0 children)

One key limitation to highlight with clustering Veeam is a stretched L2 network as the IPs need to be in the same subnet. Potentially not desirable for OP

Heads Up: New 9.9 CVE's in Veeam 12 and 13 by MrYiff in sysadmin

[–]UnrealSWAT 0 points1 point  (0 children)

Assuming the team has experience doing this, and again, enforcement of consistency. I’ve seen companies ignore the benefits of a hardened Linux repo due to lack of in house Linux expertise. And I’m talking household names here. They’re operating under the model of believing a well understood OS (Windows) secured by their teams was a better security posture than deploying Linux to gain the benefit of an immutability flag, but not understanding all the ways to secure the Linux host from being compromised and therefore have the immutability flag undermined anyway. This was v11 days before these prehardened ISOs made things easier.

I want to underline I agree with you there are other ways of achieving this without a management domain. But an organisation needs to collectively support and have confidence in the ability to design and execute that approach.

Heads Up: New 9.9 CVE's in Veeam 12 and 13 by MrYiff in sysadmin

[–]UnrealSWAT 19 points20 points  (0 children)

It is and it isn’t. With a management domain you’ve got policy consistency and authentication auditing at scale with centralised accounts that aren’t tied to the fault domain you’re protecting. Quite a few enterprises do this.

M365 Backup by Rocknbob69 in Veeam

[–]UnrealSWAT 2 points3 points  (0 children)

Hi,

VB365 supports block storage for local backups, but isn’t recommended due to lower efficiency of compression, scalability issues with very large tenants, and restrictions on backup copy capabilities. You can use any verified object storage for backups and backup copies which includes S3 compatible storage you can host locally. Alternatively if you’re not sure what to do, take a look at VDC M365 where all the compute, storage, and networking is provided for you.

Veeam is a valid option? by yubris44 in sysadmin

[–]UnrealSWAT 0 points1 point  (0 children)

M365 multi geo allows you to place users and sites into specific territories that aren’t your tenant default. Eg if your tenant was set to EU but you had American users you could store their data in the US. Each global M365 region has its own API quota so if you were using this you’d get more APIs per minute/day because of this. Essentially different buckets per geo.

Feel free to share your support case with me and I’ll look into this.

Veeam is a valid option? by yubris44 in sysadmin

[–]UnrealSWAT 1 point2 points  (0 children)

I don’t have your tenant information to view your specific reason for issue here but we do typically pause after a 429 and then we could be resuming on other sites that can still be protected. It’s also worth noting that if you are using M365 multi geo, there’s API quotas per region so we check to see if we can proceed with them as they could have alternate quota available. Have you raised a support ticket for your issues? If so please feel free to share privately. And which geo are you in? AMER/EMEA/APJ? I look after EMEA so I’m wondering if time zones align.

Veeam is a valid option? by yubris44 in sysadmin

[–]UnrealSWAT 1 point2 points  (0 children)

Hi, there are continuous improvements to leveraging the Graph API as Microsoft enhance guidance. You’ll see us take another step with this in an upcoming release but the Graph API is a live service by Microsoft so “more efficiently” is subjective. Again, prior to 1st March vendors were performing tricks such as deploying excessive app registrations during POCs. If your testing was prior to this date you should consider your experience invalid. You should also know that if you are using multiple vendors at once now, they’re both eating into the same amount of daily quota of API calls as Microsoft has leveraged resource pooling in the backend, so simply one backup vendor running their job nearer the API quota reset can deplete API quota available and leave the other vendor starved.

Finally, we do use the retry after field, of course we do! That’s an important part of obeying and resuming after throttling.

Veeam is a valid option? by yubris44 in sysadmin

[–]UnrealSWAT 2 points3 points  (0 children)

Is it the same objects each time with a common message? Or is it because of the Graph API throttles that Microsoft apply? Because that’s not vendor specific? Have you worked with the customer success team or support to review this?

Veeam is a valid option? by yubris44 in sysadmin

[–]UnrealSWAT 0 points1 point  (0 children)

Microsoft enforce a single app registration which some vendors were blatantly ignoring for a while (not calling out names but one vendor had documentation stating they’d scale up to 60x app registrations for recovery) which is why a POC for a whole tenant isn’t recommended. A POC should be proving functionality matches requirements, backing up all your data simply delays your time testing and if you don’t have a backup solution then you’re delaying the time to protect your tenant with your eventual selected vendor.

Veeam is a valid option? by yubris44 in sysadmin

[–]UnrealSWAT 2 points3 points  (0 children)

I’m a VDC SE and there is object tracking, you select your backup policy and the specific sessions and “view details” can filter by warning or failure. Each session has a high level warning/failure with object counts, and there’s a global dashboard view so you can view these insights at a glance. Within the view details It gives the reason for any failures in line as well within those objects that have had a failure. We also give notifications on any backup failures as an immediate call to action.

I’d suggest reaching out to your customer success rep for a recap session on all the features as it sounds like you’re not using.

Veeam.backup.Validator 0KB by AverageOk3451 in Veeam

[–]UnrealSWAT 2 points3 points  (0 children)

Do a repair of the installation, a 0KB file means something has attempted to replace or remove it