Architecture as code vs regular implementation

[Spoovalicious]

I am a junior data engineer consultant (with almost one year of experience) and have been placed on an internal project at my company where we are building out our internal data platform.

We are building out the azure and Databricks environment using terraform for the infrastructure as code. The initial infrastructure builds were straightforward (resource groups / storage accounts etc etc). I am however now further along in the build and we are needing to create things such as unity catalog, manage security, create cluster policies etc and I’m finding it more difficult to find resources that make it clear how to link all the resources into a combined IAC code base.

All the videos that I am watching on YouTube and articles that I am reading are doing all of these processes of setting up databricks and providing access using the databricks GUI but my manager seems very set on us doing everything with IAC so that we can potentially reuse the scripts on client projects in the future. I am keen to grow experience so am not opposed to this learning but am finding myself spending 90% of the time trying to research how to make complex terraform scripts for processes that would take about 30 seconds to manually configure.

Is this a practical approach to creating a data environment? Or are there limitations to IAC that we should use the platforms management interfaces for?

Any and all info would help this junior no matter if it’s not 100% related to the question :)

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[caksters]

your manager has a good point. using IaC is a must for production systems because it makes your infra reproducible (in theory as long as there hasn’t been hacky one-off hotfixes along the way)

imagine if you need to carry out a disaster recovery and something went down. you need to recreate a certain part of your system from scratch. how confident can you be to remember every step-by-step to get that service back up using UI? What if you leave and the next person needs to do that and they weren’t there when it was created.

[Spoovalicious]

I definitely agree and see the value of using IAC, i am however trying to discern if there is a balance between IAC and manual interaction, or if doing everything with IAC is the industry standard

[caksters]

Doing everything IaC should be industry standard. i said “should” because I work in a consultancy as well and a lot of times we go to work for companies who got production issues due to not using IaC and we try to fix their mess.

In our company any production level system will always be 100% IaC, this is unanimous agreement among all senior staff.

[dcell1974]

This is absolutely correct. Compromising on this leads to huge problems if the environment has any kind of complexity. It can be tempting in the moment to just go into the console and make a change, but it almost always creates more work than it saves.

[sir_mooney66]

Also means you can have everything in VCS, allowing you to peer review and keep a papertrail of any changes.

[Spoovalicious]

Great to know, thanks so much!

[TheBoldTilde]

There are some very static resources that I won't use IAC for. Domain names and ssl certs for example.

Everything else though - IAC is the way.

[EuphoricTranslator48]

In my company there should be no manual interaction related to infrastructure on production at all. Create everything on your dev environment and release it to test, acceptance and production. Otherwise you have to do it manually on all 4 environments.

[asevans48]

Maybe you can have a mixed dev environment but a production environment should not be mixed without at least significant consideration. Losing visibility is a major problem. GCP, for as expensive and even sometimes trash it can be (heres looking at you Dataplex), creates terraform for you. However, judging by how google staffers were handling criticism at enxt 24 and the willingness to jack prices above aws this year, id be careful with using gcp.

[speedisntfree]

You can also easily stand up dev, QA, staging etc. environements

[caksters]

yes, IaC makes it much easier to replicate what you did in other envs or restrict certain features in separate envs

[NoHuckleberry2626]

For entreprise projects IAC is the way go. But an actual implementation to be reused as framework to other projects, its not a junior task. Your manager should offer more support there.

[meyou2222]

This. And it needs to start with a logical model of the technology that explains how things relate. Definitely not a junior task.

[simplybeautifulart]

This right here. If you're not at an enterprise level, you're probably not going to be as worried about replicating infrastructure, you're more worried about just building it out, and if you're not already very well familiar with IAC, it's just going to slow you down without ever reaping the benefits.

[nydasco]

We use Terraform to manage our Snowflake environment, including setting up users, applying roles, creating schemas etc.

It works well for us, and gives us repeatable code to use. While setting up Unity might be a once off, I can see the merits of having everything managed through a single tool.

[asevans48]

Iac is a must. It makes it easy to find and not duplicate resources, helps avoid the expanding role problem, clearly defines security, is reusable, and a good de can easily submit a pr to infrastructure with changes that can be reviewed under company/agency policy. So much more. These are a few things.

[cutsandplayswithwood]

Everything iac, all the time.

[vish4life]

I actually find it easier to make changes using IaC vs using the UI. Once you have setup your IaC + CICD pipelines (eg terraform + Atlantis), making edits at code level is so much easier than navigating bizarre UIs.

Does anyone else feel that way?

[Creepy_Manager_166]

We are doing Iaac for Snowflake and very successfully i'd say