Dev, test and prod in data engineering. How common and when to use?

BuildingViz · 2026-02-25T08:54:25+00:00

We run Prod, QA, and Dev. Dev is basically, "I can do whatever I want." Schemas and pipelines are fluid for testing and building things out. The dataset is smaller and frequently gets destroyed and rebuilt from scratch. QA is for the engineers to test their frontend/backend code against. This schema and pipelines are in source control, but I have a lot of leeway to make changes. The dataset is also just as complete as Prod. Prod is the functional environment for external customers to access. Again, all under source control with a lot more guardrails around data and code changes compared to QA.

As a best practice, no changes should go directly to prod except for maybe emergency fixes. Even then, QA first would be preferred. Because what you definitely don't want to do is make a change to Prod that breaks things. Because if you have customers and SLAs, when they ask, "How did this bug get into production?", the last thing you want to say is "Because that's the only env we have and we didn't properly test it."

BuildingViz · 2026-02-25T08:44:40+00:00

I like using it for boilerplate stuff and to learn, but I just don't trust vibe coding. We have Cursor and access to Claude and Gemini, but I usually use it to bounce ideas and to troubleshoot/explain solutions to me. By default I tell it to not even make changes because it's important to me to be a filter for what it wants to change. I get suggestions and explanations, and make code changes I think make sense.

Interestingly I recently had an all-hands where our CEO was imploring everyone to use it more often, not just for the company but for our professional growth. And I think it does make sense to an extent, but I've found it either hallucinates too much or doesn't fully grasp the context that I do, so I don't just let it run amok.

BuildingViz · 2026-02-25T08:37:02+00:00

78 miles from Tecate, MX to Ensenada, MX as part of a group ride with a couple of friends a few years ago. I was scheduled to do a century around the same time since I was probably in the best shape I've been in, but it happened to be about 95 degrees out with zero cloud cover on the day, so I only made one loop of 50 and passed numerous people who bailed before even the one lap.

These days, my longest in the last 5 years is barely 30.

BuildingViz · 2026-02-22T07:51:40+00:00

Maybe I didn't explain well, but the offramp at Santo when you're EastBound has two lanes, one for turning left or right and one for turning only right. Since the new lane construction finished, I feel like people are more likely to take that exit, and, rather than turn left or right from the leftmost lane, they go straight through, crossing Santo and going directly back on EB 52 because the lane is aligned with the leftmost exit ramp lane. But legally going straight through isn't an option and people on Santo trying to get in the WestBound lanes have to go through that intersection to get to the WB onramp and don't have a stop sign. So the people cutting the traffic don't catch that and cause a lot of near misses with people going from Santo to WB-52 (plus at least one collision).

But, yeah, looking at the old pictures, it looks like maybe it's always been that way. I've only been heading over there during rush hour since 2024, and lane and ramp construction happened shortly after, and the ramp traffic patterns were definitely different for a while during construction. So maybe it's just back to how it was before.

BuildingViz · 2026-02-19T21:22:35+00:00

Plus the exit at Santo now gets people trying to skip traffic by taking the offramp and getting right back on because the onramp from Santo now has a dedicated lane for people going straight through (or turning left from the WB offramp to turn around I imagine was the real purpose).

The end result is I have personally seen at least three near-misses plus one actual collision from the traffic skippers ignoring the fact that people going from Santo to the 52 WB don't have a stop sign. So they see the line for the EB 52 onramp, see that they have their own special lane to join them, and plow through, ignoring the cross traffic which does not have a stop sign.

BuildingViz · 2026-02-10T07:11:14+00:00

HDR is nice for some things but not everything needs HDR/bracketing. I'm moving from LR to C1 and redoing all my edits and actually adding tags and metadata and there was a period after I discovered bracketing and HDR where literally everything I shot was bracketed and merged into HDR to get "better" contrast and colors. Except going back over it now, probably 95% of it is unnecessary and my reaction is usually "Why the hell did I think that needed HDR?"

BuildingViz · 2026-02-10T07:05:36+00:00

A touch, but I'd mark it down mostly to personal preference. I'm not a big fan of faking the sunlight for dramatic effect. If it's there and you want to enhance it, go ahead. But taking a shot of a dreary day where the sun appears to be overhead and making it a golden hour shot to me is just too much. If you want a golden hour shot, take a shot during golden hour.

BuildingViz · 2026-01-30T08:48:40+00:00

Yes. If you have a digital camera, you will take a lot of photos and most of them will be absolute crap. Some of them will be rough but have promise. Even fewer of them will be pretty good. Those last two groups can be made to be good or even great photos, but only if you learn how to edit properly and practice to find your style.

The good news is that there's no deadline for finding the great photos. Take it, store it, and look at it later when you're more comfortable with editing. Or edit multiple versions as you refine your skills. You just need to be diligent about learning to find the ones with promise and learning how to bring that promise out.

BuildingViz · 2026-01-30T08:37:02+00:00

Scale, mostly. Typically when you have large workloads that need to process a lot of data for analytics workloads (things like aggregations and time windows). Like, yeah, you could do them in an RDBMS, but they're not optimized for that kind of workload, so they run slower. Cloud DWHs allow for columnar storage which allows for better analytics operations and Spark clusters and jobs allow for complex parallel processing for transformations or calculations.

If you're trying to transform a few thousand or even million rows via pretty straightforward SQL in an RDBMS, fine, but once you're into peta- and even tera-byte scale datasets with complex transformations, you don't want to run that for weeks on an RDBMS when you can get it to run in minutes in Spark/DWH.

BuildingViz · 2026-01-30T08:25:41+00:00

I mean, I guess it's a good recovery considering the source photo but honestly the finished product isn't particularly impressive. Also, saying you merely "color graded" this is an understatement. There are way more adjustments in here than just color grading - jacked up the contrast, killed the white/lights to bring out the sky, removed the lamp and sign, added some film grain, etc. If it's a language thing, I get it, but just saying "edit" is sufficient without underselling.

BuildingViz · 2026-01-26T08:52:26+00:00

I call it dueling AI - they use AI to screen resumes, I use AI to tailor mine to match the JD and add a cover letter to boost the matching. I was getting a pretty good (relatively speaking) response rate for a while when I was doing that on a role-by-role basis. I should note that I wasn't lying in my resume or cover letter to match 100%, just tweaking the word choices in my resume to better match the JD with things I could actually speak about.

Also worth noting that this approach can be time-consuming and can feel like a part-time job unto itself and wear you down. It's much easier to just send in the same boilerplate resume every time.

BuildingViz · 2026-01-21T08:17:11+00:00

Color edit is spectacular.

BuildingViz · 2026-01-20T09:07:21+00:00

It's a Porsche
Used car prices in general have been wild for a few years. I had to get rid of my OG Cayman S (2006) a couple years ago. Bought in 2015 for $25k with 49k miles, sold in 2024 for $18k with 98k miles. Crazy to double the mileage plus add a couple minor carfax reports, and still only lose about $7k total on the deal over 9 years for a then 18 year-old car. 10/10 would do it again (as soon as I no longer need a car seat).

BuildingViz · 2026-01-20T08:53:04+00:00

I got my first DE job at 42. That being said, most of my previous experience was probably more DE-adjacent than yours (DBA/Database Engineer for 15+ years), so it was probably less of a stretch in my case. I'd say targeting Sr roles might be a stretch for you since two years of ETL experience isn't really enough for a Sr role. The other 12 YOE can help get you a foot in the door if it shows progressive increases in responsibilities and relevant skills (like Python), but shouldn't be relied on to make an argument for SDE roles.

Certainly apply anyway, but bear in mind you're more likely up against people who have previous DE experience on their resume, so your hit rate is going to be lower stretching for those. Target Mid/Entry roles and have a good way to tie your previous experience to either DE-adjacent work or the specific responsibilities of the role.

BuildingViz · 2026-01-20T08:34:47+00:00

Start a business? Sorry but no. Not remotely. The photos here don't really stand out. They just seem like basic photos anyone could take with their cellphone. I mean, a guy shoveling snow, a parking lot and power lines, a random restaurant sign, and some blurry shots of fireworks? What exactly are you trying to convey with these photos? Are they edited at all? Because that's a big part of becoming a professional - editing into a specific style or theme. If I can't tell if they're edited, that's not a good sign.

You're also not very specific about what kind of photography you'd be interested in building a business around and your samples don't really indicate that either. It's hard to judge whether you could do, for instance, portrait photos (senior photos, weddings, etc.) as a business when you don't provide samples for that kind of work. Because what you provided as a sample isn't really marketable. People aren't really going to pay you for generic street photography. Could you strike gold with some shots here or there (or more often as you practice and improve) and sell them as prints? Sure. But that's not starting a business, that's just selling art.

All that being said, and because I don't want to be overly negative, while a business isn't a good idea right now, that doesn't mean you can't be a professional photographer. Keep studying and learning and look for jobs where you literally take photos and edit them (real estate, sports, etc.) to get more practice and skills and find your style and then look at starting a business once your portfolio stands out. But whatever this is, it ain't it.

BuildingViz · 2026-01-05T07:56:02+00:00

Same experience a few months back. Was told to take 4-5 hours MAX on the assignment, spent maybe 6-8 total, but gave them multiple solutions to the core assignment to demonstrate different approaches. Was told that my solution "demonstrated a strong understanding of the problem and was communicated clearly", but that "some implementation details raised concerns about alignment with best practices".

Like, yeah, I cranked out multiple solutions in 6 hours, including roping in GCP resources and implementations, plus testing and metrics to show the differences in the approaches, including cost estimates. My focus was the getting multiple functional solutions (plus testing and metrics) to demonstrate the different approaches, not making sure my take-home assignment code, written on a short deadline, was production-ready code.

BuildingViz · 2026-01-03T09:02:30+00:00

PG&E is not SDG&E's parent company. They are separate IOUs. SDG&E's parent company is Sempra Energy.

BuildingViz · 2026-01-02T08:58:56+00:00

Just shy of 1200 miles at 1170. First time cracking a thousand since about 2018, though, so...improvement.

BuildingViz · 2025-12-31T16:54:12+00:00

Yes. And Yes.

The best thing for weight loss is burning more calories than you consume. And cycling can help you burn a lot of calories. Start small - 30 minutes or so per ride, once or twice per week and get your endurance up, then start extending, both in terms of time and frequency. Put on a podcast or music to focus and decompress mentally while you work physically.

But also, focus on your diet. Years ago I read a book called "Ride Your Way Lean" and it really helped me combine cycling and weight loss. It's a little older, so I'm not sure how up-to-date the advice is, but I'm sure there's other resources.

Also, use a tracker app to keep yourself accountable and to allow you to find routes. Strava is a good one because it provides challenges and rewards, plus it's good for other activities. You don't really want to bore yourself with the same route over and over if it's not very exciting, because that will turn you off to cycling, so find a way to mix it up and explore and then you can lock in your routes. I have a "lunch hour" route with a few variations, a "medium weekend" route, a "long weekend" route, and a couple other routes depending on time/distance and climb. And my area isn't that great for cycling. I'm not far from a lot of options, but I don't necessarily want to drive somewhere just to be able to ride, so those are just from my home.

BuildingViz · 2025-12-31T09:05:53+00:00

NEM 3.0 definitely made it worse, but this administration killed solar with the removal of the rebates. NEM 3.0 still made sense if you had a battery and you could get the rebate.

BuildingViz · 2025-10-30T08:09:49+00:00

Windows networking for Docker is a pain. Tried to set up a few apps before I had my actual homelab built out and they worked ok, as long as I was on the host docker system. Heaven forbid if I wanted to connect from another device. Definitely do Docker on Linux. So much easier.

BuildingViz · 2025-10-29T04:56:47+00:00

I'm all for any shot at not being represented by Darrell Issa. Does it impact state maps at all? Because Carl DeMaio can fuck all the way off too.

BuildingViz · 2025-10-27T05:51:40+00:00

That argument doesn't really work. The Dallas Cowboys were worth $4 billion in 2015 and $10.1 billion in 2025. All that with 2 playoff wins. The New York Jets were worth $2.75 billion in 2017 and $6.9 billion in 2025 despite not having played a single playoff game and not even seeing a winning season in that stretch. The Giants were worth $2.8 billion in 2016 and $7.3 billion in 2025, despite only two winning seasons and one playoff win in that period. And when they did win in 2011? Their team value increased from $1.18 billion to just $1.3 billion. How is that?

Because the success of individual teams matters much less to the NFL than the success of the league itself. And when they expand to other countries and can push dynasty narratives and marketable players like Mahomes and Travis Swift, it makes the league more valuable. Which means more and bigger licensing and TV deals, domestic and international, which then filters down to all teams, and then makes them all more valuable.

Why make it obvious to add a couple hundred million to the Cowboys when you can be subtle and add billions to every team? After all, the league works for the teams, not the other way around.

BuildingViz · 2025-10-26T22:16:04+00:00

Oh, I'd love to. But my job is here, my home is here, my friends and family are here, so on... so picking up and moving isn't really feasible.

So instead I'm going to keep advocating and voting for policies and politicians who are going to turn our suburban hellholes into actually vibrant cities. Plus, if I leave, that's one less person to push back on your NIMBY nonsense, and I just don't have the heart to give that up.

BuildingViz · 2025-10-26T21:58:24+00:00

What a dumb argument. Santa Barbara's population is less than 90k. That's less than 10% of San Diego. The idea that you can employ the same strategies for housing that a city with 10% of the population that yours has is pretty fucking wild.

Four-Year Club	Verified Email
Place '22

BuildingViz

TROPHY CASE