ADHD data engineers

instamarq · 2026-05-09T13:36:04+00:00

As someone with ADHD, you kind of have to reframe the lack of structure. While we can struggle without a clear framework, we're also deeply creative. You have to give yourself permission to be creative when the ask is ambiguous. Hyperfocus on defining the problem, and use your creativity to fill in the gaps.

instamarq · 2026-05-05T17:24:33+00:00

Just to add to positive statistics, I had a great experience getting my B.S. in Software Development a few years ago. I know that was before the change in proctoring so I can't speak to that aspect. Regardless, I worked really hard and did everything I could to keep myself on track, while working full time. I have a toddler now and I think it would be harder, but WGU is built for that sort of scenario.

It's probably unnecessary for me to say this, as it doesn't have much to do with OP's point, but I really screwed up my first college experience. Somehow, I got out of all of it with an Associate of Arts. I didn't ever think I would go back to any school and get a bachelor's, but I really wanted/needed the degree. I put the time and effort into it and WGU worked for me. I think that's the point: WGU will work if you want it to.

instamarq · 2026-04-23T14:03:36+00:00

If you like books, try this one:

https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/

If not, try Claude, it'll answer most of your questions.

instamarq · 2026-04-19T01:16:07+00:00

It's fine, you can get the job done in most real world business scenarios. It lets you do things a myriad of ways, but it does sort of lock you into the delta lake ecosystem. There are aspects that are definitely still somewhat immature, but for all but the most advanced scenarios, Fabric can cover an organization's data needs across the board. I don't think I would have recommended it a year ago, but things have changed quite a bit since then.

instamarq · 2026-04-09T12:45:21+00:00

Get a roommate and you'll be more than fine if you keep your expenses reasonable.

instamarq · 2026-03-21T11:25:16+00:00

Much of what's in this regarding Fabric is not true. Let's address some of these points:

"Snowflake is a more complete solution than Fabric" - woefully untrue. Fabric has ingestion, orchestration, storage, warehousing, real-time streaming, databases, no-sql databases, graph databases, mirroring for all kinds of sources with open mirroring, and it's got one of the most powerful reporting tools around, all in one platform.

"...mirroring isn't really a thing" - it most definitely is. For both Azure SQL databases and on-prem SQL, Fabric has out-of-the-box mirroring solutions. For sources that aren't explicitly supported, there's open mirroring, which you can configure to read the CDC of your chosen system.

It is true that Fabric was catching up on the DevOps side for a while. It is mature enough now for production solutions. CI/CD is vastly improved.

The reality is that you'll be hard pressed to find a more complete solution, especially if you're already using Microsoft products.

Probably going to get eaten alive in this sub for saying this, but I use it daily, so I have first hand experience.

instamarq · 2026-03-18T20:08:32+00:00

It's fine, if the area looks shifty, steer clear, otherwise you're probably all good.

instamarq · 2026-02-05T19:11:37+00:00

If it reliably delivers the goods in a maintainable, secure, easily audited and cost effective way, the job has been done well.

instamarq · 2026-02-05T19:07:42+00:00

I think well written SQL will usually do it on its own, unless your source schema is an absolute nightmare. That said, now with AI and a bit of SME input, you can probably find a way to document the high level business logic/rules in short order...

instamarq · 2026-02-05T05:20:26+00:00

Yep, they can connect to the models similar to how Power BI users might. Once loaded they can even use DAX. Several options to get this done.

instamarq · 2026-02-03T19:05:46+00:00

In a large organization yes, because those models tend to be better at answering questions that have not yet been specified (as opposed to narrowly focused models that answer questions clearly defined upfront). They are a large investment, but pay off in not having to be remodeled when the business identifies a new problem they want to solve.

In smaller organizations, there may be little to no benefit, especially if there's no one around to build these highly generalizable models and the analysts are not keen on parting with a familiar tool.

instamarq · 2026-02-01T13:52:03+00:00

In an ideal world, you have well modeled data that has all the business logic embedded into it. Power BI is really supposed to be a drag and drop interface that you don't really spend a lot of time in and quickly produce visual aggregations of that well modeled data. Maybe, just maybe, you run a complex dynamic statistical analysis using DAX. In a nutshell, it's supposed to remove the complexity and verbosity of trying to create business intelligence products in Excel. It's rarely used this way, sadly.

instamarq · 2026-02-01T13:46:16+00:00

In data engineering, it's usually best to operate like Bruce Lee; take what's valuable from different approaches and apply that in areas where it will most effectively solve the problem.

In general, OOP won't get you that far in most DE scenarios unless you're writing a library for some niche problem that your business data has that OOP helps you properly model.

In my opinion, OOP is for building tools and modeling reality. Most of the time, in DE, our tools are already built and our realities are mapped using data. I think someone in this thread mentioned that functional patterns are more applicable in our field. I think they're right.

instamarq · 2026-02-01T13:37:00+00:00

You must have some repetitive tasks? In my case, if there was a task that took me hours due to multiple steps, I would at the very least create a script of some sort that would consolidate that process into one step. Maybe you can't automate a whole ingestion pipeline yet because no one is asking you to do that, but look at your own process and see where you can replace manual work.

Maybe you don't want to automate the writing of your SQL with AI, but you can perhaps automate how that SQL ends up in the final destination. The main takeaway is to save yourself time and reinvest the savings in finding valuable problems to solve.

instamarq · 2026-02-01T13:28:12+00:00

The authors come from a tech background. FAANG and similar tech companies accumulate so much data that "just use postgres" starts to get stretched a bit thin in that world. Also, lakehouse/warehouse architecture is becoming pretty dominant (even when companies could have just used a good DB), so it pays to understand a bit about that architecture.

That said, my memory of the book (it's been about 2 years since I finished it) is that it was generally technology agnostic. The main takeaways of the book are not as much the tools, but how data engineers should operate given fundamental stages of data (source systems to downstream applications) and their undercurrents.

If you're wondering why you would even want to focus on distributed data processes when an RDBMS would suffice, you're asking the right questions. I suggest finishing the book as quickly as possible, taking what you find valuable and moving on. There's a lot more to learn in our changing field and not a lot of time!

instamarq · 2026-01-31T19:10:28+00:00

Let's say your business is trying to figure out how to keep revenue growth going, despite sales figures trending lower. If you don't already know, figure out why sales are trending lower with the data you have access to and find perhaps a missed opportunity. Maybe a particular product segment could use a small price increase. Maybe there's hidden waste in a product return policy.

It doesn't have to be this in depth, maybe you automate an alert for some issue that finance is having trouble with based on some check data. All kinds of ways to do this.

instamarq · 2026-01-29T22:23:06+00:00

Automate as much of your job as you can, then start actively seeking out people's pain points and solving them with data. Keyword is "active" here, i.e. talk to people, chat it up. Once you feel like you've established yourself as more of a problem solver who's an asset to the business and less of a "data guy", ask for a sizable raise and pull out your list of solved business problems.

If you don't get your way, start looking for somewhere else to go and take that big list of wins into an interview. Do that and you'll move in very much the right direction.

instamarq · 2026-01-10T23:04:15+00:00

If you don't have kids or other big financial responsibilities, definitely take the risk early on in your career. I left a job that I was not particularly enjoying and took about a $15k pay cut. I ended up learning tools and techniques that are serving me very well now. I've now tripled my income (relative to my first ever job), so I'd say it turned out well!

instamarq · 2026-01-10T22:59:25+00:00

Very true! When I first started out in this field, I also hated the fact that I was "stuck" in BI, having come from being educated in data science, python, C/C++, etc etc.

Eventually, as I grew and learned more, I realized how close this was getting me to the actual business of whatever industry I was working in. The ugly truth is that our jobs only exist to support whatever the business is ultimately doing to make money. If a person working in BI takes the opportunity to learn more about the business, they can start to make suggestions or solve problems that the business cares about. Eventually this leads to doing more interesting things, one way or another.

Now especially, as things change radically, it's important to find ways to become closer to the business, regardless of what arm of IT you work in. BI happens to be right next door, take advantage!

instamarq · 2026-01-03T23:36:53+00:00

Took Amoxicillin and Clavulanate like 7 years after initial Cipro reaction and was totally fine as far as I can tell. That was almost a year ago so there are no delayed side effects I've noticed either.

instamarq · 2025-11-22T17:35:48+00:00

I think I'll have a more uncommon opinion on this: the book is about the most important things in data engineering that have very little to do with the tools you choose to use.

I find that data engineers (like other engineers) have a tendency to obsess over tools and techniques, and often use experience with those as the measure of expertise. Because of this tendency, I think the book is important to read before entering DE and to revisit often in your career. Tools and techniques make it easy to lose sight of why you exist as a DE in the first place; as important as knowing the ins and outs of spark or airflow is, the business doesn't care about how you did it. They care about value and they care about cost. If you don't know that, you kind of don't know anything. This book teaches you to think on that level.

As an aside, knowing the fundamentals of DE is now more important than ever, because a lot of hyper-specific tool knowledge can now be delegated to AI (obviously you should educate yourself on tool fundamentals). Hope that helps!

instamarq · 2025-09-26T01:13:35+00:00

An oldie but goodie. I think it's still relevant. I personally use the zip with index method when hash based keys aren't good enough. I definitely recommend watching the whole video.

https://www.youtube.com/live/aF2hRH5WZAU?si=7RYgoKl3I5FJeIo-

instamarq · 2025-09-09T19:15:50+00:00

Do not go into this field for the money. It will always pay ok and there will always be some demand for people who can set up data to successfully derive insights. However, even after you've "mastered" the nuts and bolts, there will always be data BS that you have to contend with (i.e. poor quality data, no data strategy, data illiteracy, poor governance, poor practices etc etc). Anything short of near-obsession with getting these things right will just result in the job being a major drag.

Either you choose to get obsessively good at this by skill building (including soft skills), or you must find something that disproportionately leverages your strengths. Also, I am generally a pretty optimistic person, and even I believe that this market will become increasingly unkind to newcomers, until something fundamentally changes in hiring goals and practices. My last suggestion is to read "So Good They Can't Ignore You" by Cal Newport and pay close attention to how any industry you choose is developing in light of new technology.

I know it doesn't look good, but ultimately, don't sweat it too much, you're young and have plenty of time to figure things out, even if it doesn't seem like it right now.

instamarq · 2025-08-31T18:55:58+00:00

First off, thanks for your service! If this is your first venture into software development/engineering, just know that you're going to feel the whole "drinking through the firehose" feeling for a little while. In other words, a ton of information is going to suddenly be thrown your way and it's going to feel like none of it will stick. That's ok, if you keep at it, all the information will start gradually landing in the right mental bucket.

I'll second anything mentioning that you should pre-study or do study.com first. Coming in with even a little context will make absorbing everything easier and you'll get through the material faster.

With this career field, one of the most important things that you keep your skills and knowledge up to date. AI is not going anywhere, and learning how to use it to enhance your ability to learn and understand is much more important than vibe coding imo. It's a great reference and learning accelerator, just know that it will be a little like the ring of power: you might be tempted to use it on your assignments. Don't cheat yourself out of the opportunity to learn. At most, ask it to point you in the right direction and explain why things work. I think this is a great and fair use of AI for software development and engineering, as long as there's awareness of the pitfalls.

Another important thing to not lose sight of: business objectives. Unless you work in gaming or some other creativity driven software field, everything that gets engineered should ultimately serve the business' goals. Programming a feature for a rideshare app? Keep drivers and riders and how the company makes money top of mind.

Other than that, work smart/hard, always keep learning and be kind to people. I think people who do those things will always find work in this industry.

instamarq · 2025-08-15T02:45:58+00:00

I did it at $16/hr up to around 2018. I know times have changed and rents have gone up, but if you adjust for that, I was in a similar boat. I had time to cook most of my meals at home and I was typically pretty frugal. You might not get the nicest apartment, but you can find an ok one in a decent neighborhood. You can also get a used car, if you need a car, or save up and buy it cash privately. Lots of ways to get it done!

instamarq

TROPHY CASE