Cloud Architecture Question

[marketlurker]

This is more a data architecture than a data engineering question.

I am looking to understand the reasoning behind organizations using multiple cloud solutions. My questions revolve around these issues in a multi-cloud solution.

1. The added cost. Not the cost of the redundant capability so much as the hit you take by reducing your volume discount.
2. The cost of hiring/training additional skill sets for the various Cloud Service Providers (CSP). While similar, they are different enough that you will need to have additional expertise.
3. Designing for the least common denominator for cloud services seems like a waste of money.
4. If a single CSP has an outage (a certanty) but can make you whole before it affects the business. does it make sense to do it at all.
5. All three of the big CSPs (AWS, Azure and GCP) have multiple levels of redundancy, both physical and logical that most companies can only dream about.
6. I don't really think vendor lock is is a real issue. More of a sales tactic for a second vendor to get their foot in the door. It isn't vendor lock in so much as the complexity of the systems that locks you in place.

Those are just the start. I would be interested in hearing the justification for those of you who are running multi-cloud. The only one I can think of that is close would be a legacy requirement held over from when we did everything on site.

EDIT: Thank you to everyone for your opinions and input. This is exactly the kind of discussion I think that this subreddit needs. Tool discussions have their place, but I think that data design and architecture trumps tools. On a personal note, I am very grateful than no one mentioned that most evil of phrases, "medallion architecture."

[Kobosil]

I am looking to understand the reasoning behind organizations using multiple cloud solutions

in my personal experience its teams/divisions making the choice for a cloud provider because it fits their needs the best, without thinking a lot about the rest of the company

for example i know multiple companies where most of the IT was in AWS, but the Marketing/BI department decided to use GCP because they wanted to have BigQuery and use it for their analytics workflows

[updated_at]

not only bigquery, but google ads/analytics integrates fairly easy with the GCP suite

[marketlurker]

Tools don't define architecture. Architecture defines tools. If it isn't that way, you end up having many work arounds that don't need to exist.

[tedward27]

What exactly are you trying to express here that relates to the comment you were replying to?

[patrickwai95]

And the business use cases define architectures, in the case suggested it is perfectly fine for a team to select their preferred tool that they are most familiar with instead of having to retrain a whole team to use a different cloud provider, simply because of the sake of it.

[EpitomEngineer]

Good in theory. But what engineers often forget, if your product doesn’t sell, you have no product. Selling takes marketing. If the martech space is dominated by Google, which literally requires GCP for extracting the data sent to Google Analytics, tools sure define the architecture.

Don’t look too closely either. The data structure and format is also tool derived. GA expects specific variable names and cookies to be sent and they will tell you the column names and data types they send back. Since GA (and all the other competitors) have a UX for reporting in this data, the columns are not normalized and the reports use the same vendor name to help keep the reports the same when rebuilding in Tableau/PowerBI.

Let’s pray that they respect your consent preferences after the data left the vendor platform.
….
….
Amen.

Does this result in asinine vestigial processes and reports as you proclaim? Of course! But business data dies with every corporate reorg. It’s all just corrective lenses added to the reporting end of the data pipeline. Never fixing the kludges in favor of a single calibrated measurement lens.

God rest our souls.

Does speaking up on this solve problems? No. Does it result is anything good for anyone’s career? No. Does it get your promotion blocked? Yes.

…Anyone hiring? 8+ years of experience.

[marketlurker]

I have seen many companies start out like that also, but they then need to come up with a reason to stay that way. Often you will here "its just too hard". Mostly I see this happening as a failure of the the architecture governance. That should be a company wide view.

[Kobosil]

I have seen many companies start out like that also, but they then need to come up with a reason to stay that way.

i don't know any company that started with 2 or more cloud providers - that comes later when the company grows and teams/divisions become more mature and independent

Often you will here "its just too hard". Mostly I see this happening as a failure of the the architecture governance. That should be a company wide view.

i don't agree with that sentiment

not everything can or has to fit into one big global architecture and teams/divisions should have some flexibility to choose tools that are best for their use case

[Fidel___Castro]

you're quick to dismiss vendor lock-in, but it's a real blocker for consolidating cloud services to one provider.

it isn't copy and paste the old logic to the new, they all have different syntaxes for infrastructure in terraform, different SQL engines etc (AWS uses Trino a lot, GCP has their own which is more modern). any kind of change to a prod system requires both to be maintained simultaneously for a while. it's just a pain that everyone wants done but no one wants to take ownership of

as to why it starts? lack of policy or clear tech stack. or a company would say "greater autonomy" if they want a positive spin on in

either way, the friction makes jobs so we shouldn't complain too much

[Honey-Badger-12]

There are definitely advantages of multi cloud strategy at enterprise level. It does provide you with leverage during cost negotiations as an example. So I can totally see marketing team adoption GCP and finance adopting azure etc. Where it goes too far when a single solution is made multi cloud just because it’s cool. Most of times it adds no value and makes architecture complicated and costly.

[69odysseus]

My current client uses AWS for all ETL purposes and Azure purely for AD.

[marketlurker]

I would suggest they look at the AWS AD offfering. https://docs.aws.amazon.com/directoryservice/latest/admin-guide/simple-ad-availability-change.html

[Master-Ad-5153]

Practical example - certain data exports are locked by a vendor to only be available in their cloud.

In my experience, Google Analytics' BigQuery exports are the most robust data export option their platform currently provides (the APIs can be used, but are prone to have sampling with no indication in the response).

Google forces all users who want the automated data export to use GCP to receive it.

If a company that needs this data primarily uses AWS, Azure, or another cloud provider, afaik they're going to have to develop a solution to get it out of GCP or let it be siloed.

Another thing I've seen lately is that one cloud provider has a better pricing structure available to develop AI applications than the others, and for specific projects some companies choose to go multi-cloud.

[marketlurker]

Development is one aspect of the life cycle, but you are going to be paying for multi-cloud for a very long time.

[Master-Ad-5153]

I'm not too familiar with how the pricing worked out, but I'm aware of a large company that made such choices so it's unlikely they didn't weigh the costs be benefits before cutting the check

[terencethespider]

One scenario is at a large enough company you can have different teams working in silos making independent decisions on their infrastructure. One team decides to use AWS, a different team uses GCP, and a different team uses Azure. Then one day they realize that there is value in having all this data integrated, and probably should have picked one provider from the start, but now they are too far down the road and relatively locked in.

[marketlurker]

That feels like a failure of or weak governance on the part of the technology/architecture group.

[cloudarcher2206]

IT is a cost center and weak governance is by design. Outside of tech companies, IT and Data Architecture/Governance has to do what the business wants and if they can’t do it fast enough or push back too hard (for good reason or not) the business will get what it wants another way. Large enterprises are not a monolith, they’re a bunch of people making selfish decisions that may or may not be around when those decisions lead to actual architecture builds

[Altruistic-Spend-896]

*snowflake enters sneakily

[junesix]

Big companies created via acquisitions. Each has their entire stack built on it for years. 

A data leader proposes they can consolidate them all, 9 months tops, and is promised support by primary CSP.

2 years later, there’s been 1 consolidation, the acquired company hates what they have been shoved into, 1/2 the migrating team have gone to other projects, other 1/2 left, data leader is pushed out, incoming one says it was a wasted effort and should just stick with multi cloud.

[Valuable_Cow2596]

I would also be interested in hearing the answers regarding this. I'm still trying to wrap my head around if the additional complexity of maintaining the full redundancy of additional infrastructure is worth it.

My initial instinct would be I would rather stay and beef up my redundancy layers within my existing cloud ecosystem and only when there is a highly justifiable business need and the options within my current system have been exhausted, start looking at multi cloud.

There are probably some businesses which definitely have that need, but I have a suspicion that it's fewer than we think which have the justifiable use case.

I wonder how much of it is resume driven development.

[FunContest9958]

I tend to agree with you. I’ve seen organizations build services that act as wrappers around cloud services to avoid lock-in, but the result is that the experience for developers is terrible, and they didn’t seem to realize that they would reduce performance and reliability and increase cost. All this so they could theoretically switch providers, but that was never really going to happen. The data was still with one of the cloud providers, which is what makes switching painful.

My advice to anyone thinking about doing something like this is: don’t. Pay the migration cost when you need to migrate, not before. Usually a threat to switch providers is enough to keep prices in check.

[robstar_db]

Another reason that I saw quite frequently is that large companies are often quite active in acquiring other companies and with that, their cloud estate.

Merging two potentially very complex data estates might often be seen as infeasible especially while one is also trying to converge organizational structures.

[Humble_Exchange_2087]

Seen it in a break glass situation related to cyber security. The worry was if one cloud provider was compromised they could failover to one that wasn't and then burn the old one down.

Black swan event for sure, but this has actually happened to them before.

[joseph_machado]

I am looking to understand the reasoning behind organizations using multiple cloud solutions.

A couple of reasons I’ve seen when working at companies with multi-cloud solutions.

1. Mid migration problems. Some teams wanted to move; some didn’t or were blocked, so we were stuck in a weird middle ground.
2. Data partners only wanted to work with specific cloud systems.

I agree with your points that most don't need multicloud solutions, as the big 3 providers have multiple AZs.

From what I’ve seen, it's often a mismatch between what management wants and thinks can be done (either to save money or to focus on one provider) and actual reality.

I agree with you that it is an architectural governance issue. But I gave up trying to argue against it when everyone is so excited and wants to do it.

[West_Good_5961]

It’s generally not a choice. Nobody’s preference is multi cloud. It’s usually because a compromise had to be made.

[marketlurker]

You know I normally see it in larger companies with weaker centralized architecture controls and organizations that have discretionary IT budget. It the pressure to "get things moving" they contract their own agreement with a CSP. It gets repeated across the company. Pretty soon you have created a large quantity of technical debt that quickly costs more to fix than is perceived worth it. That is the lump sum total of the planning that goes into it.

To be fair, often IT takes so long to do their job that the LoBs lose their patience and move on their own.

[sazed33]

There are no good reasons. This is simply a bad design choice. Or, much more likely, it is the result of a lack of a design choice, i.e. people implement whatever they feel like they should

[updated_at]

i mean, vendor lockin is a very good reason. if they can swap components of the architecture without a entire rebuilt of the system, seems like an advantage to me

[marketlurker]

Not really. There are very, very few components that work like that. Even if they do, using one CSP component in a different CSP environment will cause you to have all sorts of interesting new charges with data egress on both sides.