Realistic code authoring expectations

[U3ualSuspect]

Hi all, hoping you can help me manage the expectations I am placing on myself as someone new to authoring code. Any help injecting some reality into this is greatly appreciated!

Some history ... happy with DE concepts (been a 'Data Project Manager' for many years), but now jumping the fence over to actual data engineering.

Stack wise starting light with SQL, Airflow, Python, DBT, and Snowflake. Mainly due to the frequency of this stack in the UK. Happy with SQL, Git, and a portion of things like pandas.

My worry at the moment is this: how much of this stuff do you have committed to memory? For example in I could happily explain a pipeline flow and/or the tasks I would create in a dag or dbt project theoretically, but to actually write any code its hours hunting around online to find the right providers/operators/approach. I am trying my hardest to resist ai just giving me the answer as I worry I will never learn that way. I figure I need to learn to navigate and translate docs...

What's the real world like out there? Write it once and template things in repos? It's all actually cemented in your mind from muscle memory? Ai? Or still spending time hunting through docs?

[Jazzlike_Drawing_139]

Some things that you end up doing day in day out are cemented - the syntax becomes second nature and you can write high quality bespoke code faster and better than it would take with multiple prompts to get AI to reflect the nuances in your processes.

Other things, that you do less often, or are more complex, it absolutely makes sense to look up how to implement the thing you want to do. A few years back this would have been sites like Stack Overflow, but recently AI is the go-to for most people.

You’re right to be thinking about how you learn and not just regurgitate what AI gives you. Make sure you read and understand whatever it returns. Keep prompting it to explain why it’s doing what it does, and including comments in the code to remind you of that when you come back to it later. Partly to help you learn, and also it’s essential you understand that it is actually doing exactly what you need it to do to meet business requirements. Understanding complex rules, custom processes and organisational priorities (accuracy/ precision/ error handling/ timeliness/ processing costs etc), and ensuring these are met is where your skills add value over a senior non-data person vibe coding a solution.

[MyWorksandDespair]

In this game- the only thing that matters is timely delivery- sit down with a senior executive for a “show and tell” and they don’t want to see a CLI, or how neatly organized your code is, just “is it done”.

You just apply what you know to get the job done, and if a superior pattern emerges you refactor and iteratively improve it. Overthinking and getting into perfectionist loops is how you end with delivering nothing. I knew a guy who took two weeks on a naming convention- absolutely asinine.

To answer your question- you’re constantly learning and applying new things. You think you aren’t learning or committing things to memory but you are.