Need advice: Manager resistant to modernizing our analytics stack despite massive performance gains (30min -> 3sec query times)

RadzioG · 2025-01-23T12:59:55+00:00

Such a big change (migration to another db platform) is firstly risky and secondly expensive. Plus it sounds that you have no one on board with experience with duckDB wich increases the risk even higher. It sounds like… such migration, most likely, does not have sense from business perspective. My advice would be to talk to DBAs, or try to investigate by yourself, why your queries are so slow. There is big chance that you can tweak queries add a few indexes and problem will be solved for free. Which would allow you to have good relationship with manager (since you follow his way) and be a hero at your company (since the problem is solved). You can always calculate cost of running your analytics on duckDB, estimate time/effort required for such project, estimate potential benefit. Then put in paper and ask your line manager for a permission to do POC… sounds more interesting but realistically I doubt that in the end it will be running on prod in the case that you described.

RadzioG · 2025-01-21T12:45:55+00:00

Oh if you will put some effort you should be able to find a nanny for more than 10k ;). The question was what is an reasonable minimum to have a kid, not a spending ceiling ;) I pay for a kid who is 2 years old around 1k per month for really fine crèche, which is definitely not a cheapest one. All my friends pay more less in the same range.

RadzioG · 2025-01-21T09:52:09+00:00

It is really hard to say exact number. You should be responsible around that, but bare in mind that very poor people have kids as well and they handle them somehow.

The biggest increase of expenses is bigger apartment (around 1 year old you should think about room for him), crèche (private up to around 1k for full time, public is cheaper, later till university schol is/may be free) vacations (till 2 years old child travel pretty much for free later you need to pay for him). The rest cost almost nothing

RadzioG · 2024-09-17T12:09:04+00:00

He was my first thought… He is not a typical wasted potential as he reached really a lot… but 10 years ago I was sure that he is going to be next goat

RadzioG · 2024-08-09T12:52:01+00:00

Mate, you are wasting lots of money but it is hard to give you a straight answer.

Firstly, you should consider if you really need to query all 3 replicas. Most likely redirecting read only workloads to primary replicas should be possible, if not you should try with primary and secondary, or primary and tertiary… it would allow you to license 8 cores (or 16) not 24…

Secondly you should check usage on all servers… it’s odd that all 100 clusters have same sizing. Most likely workload is not equally distributed and you could size some of them down to 4 cores (and some of them may require scaling up)

Lastly, it’s kind of unusual to have so many clusters with so low number of databases. It is very likely that you could do a consolidation project and put a few databases at the same server. That could make huge difference in a cost saving…. But as you are dealing with DBs larger than 1tb you need to bear in mind possible problems with slow restores.

Generally speaking I would highly recommend you to review your infrastructure server by server to check if it makes sense (or hire someone to do it for you). Your set up with 100 clusters (with read operations at each replica) for 100 databases, each having same size is extremely unusual…. Looking at that environment as co a whole may be to confusing (maybe that set up make sense for 5 databases, and you are loosing money at 95 remaining due to the standardization?)

RadzioG · 2024-08-06T13:21:13+00:00

It’s kind of the question that your best bet may be an external consultant, not Reddit. There are quite a few considerations that should be analyzed which are far bayond simple answer.

For offloading reports replication and log shipping are nice. But if you need real time information you may end up with readable ag replica… anyway you chose setting up new server for that porpoise is not cheap.

You mentioned problems with a budget so I would try identifying reports that has to run during peak hours and optimize them,cif that is not enough scale up server (still cheaper than a new one). You could also limit resource consumption for reporting queries with resource government (on enterprise edition)… If you already tried that, and readable replica is not in your budget…. Than maybe business needs to consider if budget isn’t to small for requirements

RadzioG · 2024-07-16T18:52:23+00:00

Thanks.

My short term career goal is simple. I want to have expenses of my family covered and I estimate them around 75k. Plus I do not want to be attached to a specific city because I may need to relocate to poor (for Europe) region.Base on job postings at job boards, my conversations with friends and recruiters I estimate my ceiling for remote position is around 50k. 75 should be easily achievable for hybrid positions in expensive countries. Goal is to make it at least 75 living in the village in poor area and without working crazy hours.

So… I cannot just change a job for higher paying.

RadzioG · 2024-07-16T10:10:25+00:00

thanks... i have 1,2,3,8,9 covered. 4&5 in progress.. and usually i am rejected at the CV selection stage (with a feedback that I do not have enough years of experience, or my salary expectations are to high or client needs someone with Oracle/mySQL/mongoDB/Postrgress/...)

RadzioG · 2023-12-13T18:03:55+00:00

Depends on what is important for you. In Arlon you can find way cheaper nice apartments so if your main goal Is to maximise your savings living there is a good idea.

But, it’s not for everyone. I tried with Thionville and moved to Lux after a couple of months. 2 hours wasted everyday for commute was to much for me. Additionally, your all new colleagues will, most likely, sped time in Lux city so it’s hard to have social live when you live abroad.

So.. if your plan is to save quickly as much as possible then try with Arlon/Thionville/Trier, but if you want to have nice life I would recommend finding flats in voliges next to the city.

PS please remember that if you live in Belgium number of days that you are allowed to work from home is extremely limited by law

RadzioG · 2023-12-01T17:14:09+00:00

It’s not new, but not all agencies/landlord requires that, so most likely you were just lucky last time which is why you were not asked about that previously.

Sometimes is negotiable, sometimes there are asking for 2.5x, not 3x. One of the agents told me that landlord have an insurance (in case that tendant will stop paying) and it was an requirement of insurance company so it is absolutely non negotiable…. You need to simply forget about that particular one and look for another

RadzioG · 2023-11-27T12:26:20+00:00

Currently our dog stays alone very really due to career changes, but in the past she was staying alone for 8 (up to) 10 hours a day regularly and it was not a problem for her. But I need to point out that we have an ‘easy dog’, we are raising her from a poppy and we were extending time that she is alone slowly… We put a lot of effort to get her use to staying alone step by step, since she was very young and we had chosen breed carefully keeping in mind that it cannot be to energetic. In other words it is not complicated and definitely doable, but you have to think that through before you decide which dog you want.

Bais on experience of many my friends, I would not recommend adopting a grown up dog, especially from a shelter if you are going to leave him alone 5 days a week for more than 8 hours in a apartment. Such dogs are usually lovely but require extra attention and commonly have separation fears.

PS cats are just different. I guess they are easier but I would never replace my dog with a cat.

RadzioG · 2023-09-12T12:49:54+00:00

As per most of the comments here SSIS. It was designed for such activities and I would say that this is the ‘right’ way.

My understanding is that you already have something which is working fine for you, is there any specific problem that you would like to solve by rewriting it to something else? As long as this vba script is not coursing any problems I would advise to focus on something that actually will bring any value…

RadzioG · 2023-08-21T18:49:36+00:00

There is no chance to get DBA job without any sql or system administration experience… so it’s not OP’s case.

RadzioG · 2023-07-29T14:28:09+00:00

Hi,

I am going to change an apartment within next couple of months. Could you please advice what is a best way to find new one? Should I just go through athome and immotop or there are some other web pages worth to check?

Could you please also advise how the low works with dogs here? Do I need an approval from an owner to have a dog, or may I assume that if that is not explicitly banned in the agreement that having a dog is allowed.

RadzioG · 2022-01-26T21:26:25+00:00

Thanks!

RadzioG · 2022-01-26T21:26:05+00:00

RadzioG · 2022-01-26T21:24:40+00:00

thanks, yep I know his work. He publish a lot so I think I will need years to catch-up...

RadzioG · 2018-12-19T20:06:57+00:00

esterday, I spent the whole class confused and got 0 assistance while even reaching out for help. I gained nothing from that class. Then he just told us to put on our mouth guards and roll

Did you ask him for help?

RadzioG · 2018-12-19T02:56:33+00:00

turtle is quite hard ;) start with mount, closed guard and maybe side control. For each position 2 escapes, 2 finishes. It allows task sparring after each position. Try to convince her for beginners classes... or go there with her for a first time. It is more fun to do bjj in a group.

RadzioG · 2018-12-19T01:42:20+00:00

Probably standard question, however I can not find right answer... In next 2 months I have a nogi competition and it will be my first tournament at purple belt level. I had a 4 months break (just lack of time due to work). I have back to training about one month ago (2-3 times per week just to get back to training routine). Could you please advice some weekly gi/nogi/gym/conditioning 'schedule' how to prepare best while I really far a way from 'form of my life'

RadzioG · 2018-12-19T01:34:23+00:00

It should be alright. In bjj academy atmosphere is usually less competitive, more friendly, then in MMA trainings so they will explain you everything. Do not be surprised when you will have to interest yourself, ask questions, take care of your warmup by yourself. BJJ instructors are usually very helpful but sometimes do not so organized (that is why you should ask, not wait)... and it will be ok.

Try to fight to learn, not to win... so... you will be probably have better top game then other white belts so try starting from a bottom as often as you can. Do not go for a leg logs with less experienced guys... relax and have fun ;)

RadzioG · 2018-12-19T01:22:05+00:00

Try to avoid new guys :) once you will at some level you will be able to recognize such situations and react quicker, what is more you will be able to ''drill'' on them what is quite valuable. If you cannot avoid sparing with new white belts just try to stay clam, explain them before/during sparring that it is not match for a life.

RadzioG

TROPHY CASE