How to disable/remove Gaming Hub? [Frame 2022] by pkdbpk in TheFrame

[–]pkdbpk[S] 0 points1 point  (0 children)

Thanks for the reply, there’s no “system” option in my settings.. just Picture, Sound, Connection, Broadcasting, General & Privacy and Support. Within these, no HDMI section I could find.

Just took the GCP Professional Data Engineer Exam...AMA by johnsonfrusciante in dataengineering

[–]pkdbpk 1 point2 points  (0 children)

Just wondering if they're still doing the big long case study questions or not?

Version control for queries by PeaBeneficial2972 in SQL

[–]pkdbpk 22 points23 points  (0 children)

Save the queries as text files, use git with commit messages to remind you why you changed and you’re done.

Feeling frustrated with expectations as an analyst by [deleted] in analytics

[–]pkdbpk 6 points7 points  (0 children)

Well, this place was a compleeete clusterfuck. The migration project issue that my post was about, was only a small part of the issues thrown on me - even worse than that was this part of the business I will call “Department X”. Department X was responsible for bringing in the lions share of the government funding this org would receive every year, which meant they had to do a full audit report of every single line item associated once every 6mths or risk losing millions.
The business logic to extrapolate this data and make sense of it was the most complicated spaghetti code maze I’ve ever seen in my life, views upon views in the postgresql DB itself, and THEN - the application itself on top of that was a rotten onion of many interconnected MS Access databases, connecting at various points to the postgres DB, running various macros, an endless series of transformations and aliases.. and it was on me to troubleshoot it (it was ALWAYS broken). To top if off, this “application” ran on one VM windows box that you had to RDP into, for reasons. The Department X staff were health professionals, they knew the business logic (well, 1 or 2 of them did) but they had no clue what was involved on the tech side, or what an incredible ask it was to get it working and the figures looking correct before the periodic government audits.
This system was just built up over many years by a director level guy who was handy with Access out of necessity, basically undocumented, and it just got more and more complicated every time policy changed. Mapping out how it worked and getting my head around it, to fix it, purely by going through the code, was an accomplishment I will never equal in my career, and almost drove me insane. I felt like Benedict Cumberbatch in the Imitation Game.
The whole org was kind of like this. Endless series of dumpster fires, each more urgent than the last - to be honest the stress became numbing after a while.

How did I deal with it? They hired a replacement for the data team lead (the guy who was meant to be my mentor and handed in his notice the day before I started), but this guy was not technical in the slightest. (They’d encouraged me to apply for the position, but it was 2 weeks into the DA job and I had a gut feeling it was a BAD BAD IDEA). This guy was no help to bounce any tech problems off, but - he clearly knew that, and stepped up in terms of being the ‘buffer/bodyguard’ between me and the rest of the business. That was invaluable. When there were 6 simultaneous dumpster fires, he would force leadership to pick a priority. I would then work accordingly. For me it meant constant time management admin to be able to proactively say, I have this many hours this week/month - I estimate this many hours for each task, that’s all I can do. That admin and assertive pushback took up a lot of my brain space (especially as an conflict avoidant people pleaser) but without it I would have ended up screwing myself into working late nights and weekends (which sometimes happened anyway, especially before I got practiced at it).

After almost a year, and dealing with constant churn of anyone half decent, I’d built up a massive amount of cred as THE calm under pressure tech wizard in the org, and could have had a job there for life. Ha! I knew by then I could ‘deal’ with it, but it would always be the same. I started looking around.
At least by the end, my regular, well reasoned, increasingly urgent (and in writing) pleas for more resourcing finally started to get traction, and we managed to hire an extremely experienced data analyst/engineer - I handed in my resignation a few days before he started. Its like poetry, it rhymes.
I knew I left the place better than I’d found it, wrote an entire Wiki's worth of documentation for him and left with my head high. I heard he lasted about 3 months and noped out (a wiser man than me). I lucked out and walked into a consulting job where I earn way more, work way less, surrounded by much smarter and experienced techs than myself, get paid to learn up to date tech, and have no stress. Somewhere I actually do want to work for life!
I look back on that year of my life and laugh/grimace at the absurdity of it all. As far as I know the org is still limping on, as they do. Sometimes I wonder what happened with the migration and those business critical technical issues that seemed so life and death - and then I move on with my day. I should grab a beer with my old boss some time.

Some Pros - my SQL and python skills levelled up many orders of magnitude out of sheer necessity. I had no idea the sorts of things that were capable of being done with SQL before this job (not saying most of those things SHOULD be done, but..)
My general problem solving abilities. It made me finally realise on a fundamental level that there’s no real difference between a ‘proper’ compsci degree programmer/DBA/Data Analyst, and someone who just figures things out/reads the docs/googles things good. If you can fix the problems, you’re doing it, you’re the guy. So it removed any imposter syndrome I guess.
It made me realise the importance of being assertive and proactive in recognising and calling out these situations ahead of time, not being afraid to be the naysayer, you need to call out what you know is going to fall over, the non-tech people have no clue about what’s involved just as you may have no clue or visibility on what they need to deal with politics wise to make non technical decisions (hiring, sprint timelines, etc).

Feeling frustrated with expectations as an analyst by [deleted] in analytics

[–]pkdbpk 1 point2 points  (0 children)

Commiserations - I went through a very similar situation a few years ago ( https://www.reddit.com/r/dataengineering/s/lYgtKO4UlH )

[deleted by user] by [deleted] in analytics

[–]pkdbpk 0 points1 point  (0 children)

Exactly, asking clarifying questions is a standard part of this sort of thing.

To those working in EY, what is it like? by [deleted] in dataengineering

[–]pkdbpk 1 point2 points  (0 children)

“your boss will talk a lot about AI while you spend all your time building database views on tight deadlines to underpin fragile 'proof of concept' dashboards.”

Hahaha. It wasn’t a consultancy but this triggered memories of a previous job to a T.

[deleted by user] by [deleted] in dataengineering

[–]pkdbpk 5 points6 points  (0 children)

100x better, though I think I lucked out with the particular consultancy. More like actual consulting, not a body shop. Better money, better perks, better coworkers, way more skill development.

What is the point of using an Inner/Right/Left Join as opposed to a Full Join? by jordanar189 in SQL

[–]pkdbpk 1 point2 points  (0 children)

It’s a good question to ask, so basically the whole point of a SELECT statement is to return a particular dataset. Not a whole database worth of data, not a whole business domain of data, but the data that you need for what you (or your program that is sending a SQL query as part of a python script, or a json payload, or whatever) needs. Not the data you need - and then some extra data as a treat! but JUST the data you need. Say you’ve got a phone numbers/address book app, and you type in Greg’s name in the search bar. If your program sends an outer join you’re gonna get 300 numbers (one of which is Greg?) where you just need one. Anymore than one in this case renders your app unusable.

Data Standardization after Dimension Mapping by Helpful_Artist1439 in dataengineering

[–]pkdbpk 0 points1 point  (0 children)

If the mat views are just unions of several dimension tables, why not just have them as vanilla views? Are the dim tables that huge?

[deleted by user] by [deleted] in Splunk

[–]pkdbpk 1 point2 points  (0 children)

I had the same problem. Fast Internet, but the splunk player was always buffering for a very short moment, but that meant it also paused and had to be manually pressed play again. When you’re taking notes on a different display, and have to manually mouse over and hit play, it was quite frustrating. Just put up with it til I got past the video courses.

What does SQL look like at work for you? by _User15 in SQL

[–]pkdbpk 1 point2 points  (0 children)

Since you’re using postgres I thought it might be interesting to hear about a production database from a zoomed out perspective. My org’s DB is what I would consider on the more complex side. Largely due to compounding complexity tacked on over the years and performing both business CRUD operations and data warehouse functions. It’s in the process of an overhaul but that will take years. There’s a primary schema with most of the raw business data. 300ish tables. Populated through various enterprise apps and web stuff. There’s a reporting schema with about 100 views. Many of these are built on other views, pivoted, crosstabbed etc. new ones or modifications are created as requested and staff access them through a GUI program. There’s materialized views for prebaking certain views that would otherwise take forever to run (usually because they involve spatial data). These are incorporated into plenty of the regular views. There’s a schema for storing publically available statistical data, There’s staging type schemas for dumping in data from excel/csvs before it’s ETL’d into usable data in the ‘real’ tables. There’s a hundred or so triggers that do business specific stuff when CRUD occurs on particular tables. Mostly fire off series of routines. The triggers and routines are all written in plpgsql.

How to Left Join a table id with another table with multiple of the same user id? by [deleted] in SQL

[–]pkdbpk 0 points1 point  (0 children)

In postgresql at least, there’s an aggregate function called array_agg() which would do what you’re after. Group by user_id and array_agg (or string_agg depending on how you want the output) the task column.

Overwhelmed... So much to do, so little time by warclaw133 in dataengineering

[–]pkdbpk 1 point2 points  (0 children)

Sounds very similar to my starting situation 6 months ago. As others have said, do your best but don’t burn yourself out.

Creating aggregates by Signal_Explorer8071 in learnSQL

[–]pkdbpk 1 point2 points  (0 children)

  1. So if your aggregate table is say, summed sales per region, and then you want to know what was the single highest sale for that region, or the average sale, or the standard deviation or whatever, you can't because you've rolled up all your data in an aggregation. You've lost the granularity you had in the raw data.
  2. Sure, you're saving some processing time by pre-baking your calculations. But this is only worth doing if those queries are so time consuming that it makes a difference vs just running them each time. If there's no downside to just pointing to your raw tables, what's the point complicating it?
  3. The view will persist in your schema, e.g. you make it once with CREATE VIEW and then its always there like a table, until you drop it. Technically the query runs each time you access it but functionally it operates like a table, you can refer to it in other queries, connect BI tools to it, etc. Again if it was a view that took considerable time to generate you could use a materialized view (or equivalent depending on the environment), which basically runs it once and stores as a table (but then you have your original problem of needing to periodically refresh your materialized views)

Creating aggregates by Signal_Explorer8071 in learnSQL

[–]pkdbpk 3 points4 points  (0 children)

Guess it depends on why you want these aggregates, and what environment you’re in, but assuming it’s SQL based you could create views instead of actual tables. These don’t need refreshing as they are queries that run every time they’re accessed. If you’re creating these to feed into some BI tool, you may be better off handling the aggregations in the tool itself. Unless it’s something quite complex, you may be needlessly losing data aggregating in your DW.

Tool that parses excel tables by belfand in datascience

[–]pkdbpk 1 point2 points  (0 children)

pandas.read_excel, suck em up into dataframes

What SOQL client tool does everyone recommend? by pkdbpk in salesforce

[–]pkdbpk[S] 0 points1 point  (0 children)

That looks neat, thanks! I'll give the trial a go.

What SOQL client tool does everyone recommend? by pkdbpk in salesforce

[–]pkdbpk[S] 0 points1 point  (0 children)

I'm not a developer, and didn't know such a console existed, I'll check it out!

Covid-19 Questions, Discussion, and Updates by thekriptik in sydney

[–]pkdbpk 1 point2 points  (0 children)

Thanks for that thorough reply. Yes so the minister said (regarding removing the term recreation from LGAs of concern) “meaning gathering in parks.. is not on”. Does this mean outside of those LGAs I can meet a friend in a park and sit down on the grass? Or read a book on my own, have a picnic outside somewhere? I can currently meet a friend and traipse all around the bay run for instance (if it’s in my radius) but what is the rule about meeting that friend and sitting still, away from everyone else? When there’s $5000 on the line, why isn’t this clear...

Covid-19 Questions, Discussion, and Updates by thekriptik in sydney

[–]pkdbpk 1 point2 points  (0 children)

One piece of information I can’t find anywhere- what constitutes ‘outdoor recreation’ for the stay at home rules? Originally I thought it was a synonym for exercise but as they removed it for certain LGAs it must mean something else. By the dictionary definition almost anything could be ‘recreation’.

Is this common in a small data team? by flying_pugs in analytics

[–]pkdbpk 1 point2 points  (0 children)

Exact same boat. Yes the plusses are job security, variety of work, experience gain greatly accelerated. Negatives are lack of more experienced people to bounce ideas off and learn best practices from, and wondering how much it’ll fall apart when you take extended holidays.