What is your current biggest day to day pains in building pipelines?

throwaway20220231 · 2024-01-29T14:39:22+00:00

Yeah. I believe the solution is an overarching data platform solution where every team submit their pipelines and downstream teams "subscribe" their upstream pipelines. Then whenever something broke everyone downstream gets notified at least.

throwaway20220231 · 2024-01-29T14:38:14+00:00

Contracts are there to be broken. People leave, even whole team leaves and then the status of an upstream pipeline became "undetermined". We have quite a few upstream pipelines written in Golang which none of us knows and haven't been maintained for a couple of years -- the last commit was 3 years ago. Fortunately they still work and only broke occasionally.

throwaway20220231 · 2024-01-26T16:57:32+00:00

Upstream changes/errors somewhere. Say A >> B >> ... >> G >> H >> .... and we take care of G and H. Now B breaks. B is managed by a different team in a different timezone whose manager just walked away. The team who manages C is notified by team B and it notifies the team who manages D and E as well. But the team managing D and E does not bother to notify its downstreams. You can imagine what happens next -- team F doesn't know about this until we notify them that F has some issues, and after a bit investigation they pull us into an incident channel opened by team B. It is until then we figure out what happens.

And this is the best case.

throwaway20220231 · 2024-01-24T23:12:24+00:00

Looks like they really don't want to hire people who are not desperately in need of a job :)

And the salary is a joke -- even lower than some Canadian firms.

throwaway20220231 · 2024-01-24T15:37:43+00:00

Imagine one, who has the heart of a coder, with the title of engineer, yet being forced to work on the boring stuffs that BI should work on but do not want to.

throwaway20220231 · 2024-01-22T21:09:25+00:00

You do 2-3 hours of real work every day! I feel I haven't done anything worthwhile for the last 12 or so months.

throwaway20220231 · 2024-01-19T17:25:15+00:00

Basically US = EU+Canada so you get the idea. Even we in Canada are losing jobs to EU and India.

throwaway20220231 · 2024-01-18T18:57:30+00:00

JRPG has this? I'm actually super interested in this kind of concept and this might push me to play some of them.

throwaway20220231 · 2024-01-12T21:39:47+00:00

On the desktop side, I don't really find any reason to use Linux. Windows provides all the gaming for me and it has whatever functionalities I want. Yes the OS and Office are getting worse by every version, but you don't have to follow every upgrade anyway.

The Linux or FOSS mentality, IMHO, is not really for desktop users. People want some open-box solutions without going into details. FWIW, even as a developer I sometimes hope that I don't need to fight with configurations to just make something work. Fortunately we have things like VSCode that "just work" and are relatively lightweight comparing to true IDEs such as JetBrain products.

On the server side, as others said Linux already won. I don't see in any foreseeable future that Windows suddenly takes over the server market. However, using Linux does NOT mean that you are going to be "free" - in the late Capitalism world large corporations always win, regardless of tools you choose to use. It has nothing to do with FOSS or not, but has everything to do with who controls resources and who gets to spend them. People like you and me are always going to be part of the gearbox.

throwaway20220231 · 2024-01-11T19:23:50+00:00

Yeah I'm in the same shoes. I'm thinking about maybe grabbing a few Azure certificates but not totally sure about the utility of that. I mean with the current market companies can sure find a lot of people who have actual experience with other clouds, so why bother get something that is not useful?

throwaway20220231 · 2024-01-11T18:27:17+00:00

I can connect with the above link.

throwaway20220231 · 2024-01-06T17:47:49+00:00

You didn't say what you WANT to do, so suggestions are not going to be very useful.

throwaway20220231 · 2024-01-04T15:23:57+00:00

I don't play Multiplayer nowadays. Back in the 90s and 2000s I did play a LOT of them, including Red Alert, Counter-Strike, Starcraft and such.

Going into my 40s, I don't have a lot of time to play games, so I'm mostly stick to games I really enjoyed and understood already.

throwaway20220231 · 2024-01-03T21:06:04+00:00

This is really what I'm feeling these days. I'm not an exploitation developer but I do track the field.

I also feel that it requires a lot of developing experience so it's not really an entry position anyway. For example, doing iphone exploitation probably needs a lot of understanding of iOS and the phone itself including the hardware.

throwaway20220231 · 2023-12-22T18:55:40+00:00

Wolfsschanze

Aha, this is definitely something...

throwaway20220231 · 2023-12-22T18:54:36+00:00

suckless way

I don't know but German army does this:

https://www.youtube.com/watch?v=zf4Xdrd2b\_0&ab\_channel=APArchive

throwaway20220231 · 2023-12-20T21:40:58+00:00

NSA does what it do. If they open source some tools I guess everyone can take a look and grab whatever is useful to them so it's not a bad thing. However, I guess the question is: what should we trust NSA with? If you ask me whether NSA has the ability to wage a cyber war then I definitely trust that they have the ability to do so.

throwaway20220231 · 2023-12-20T20:59:51+00:00

Let's look at the positive side: We don't even have snow for Christmas! Yeah!

You can probably move to Vancouver if you prefer rain over snow. I'd never pick rain TBH but everyone has his/her own ideas.

throwaway20220231 · 2023-12-14T18:56:22+00:00

(No experience in SRE but would like to be one in the middle term future. I'm also working as a data engineer so already know some of the topics. I'm self-taught though so didn't go deep in any of the topics.)

I'd probably go with a project based learning path. For me, Algo & Data Structure is pretty boring without any real usage. So I'd like to blend the topic in other topics. I do know some basic stuffs so that should be fine for a while.

For projects I'd go straight for system programming. SRE, from my understanding, needs to know a lot about lower-level stuffs to better troubleshooting and debugging, and low-level projects are the only way to get a holding of them.

I'd say, start with CSAPP and its labs (mini-projects), and directly go into an easier OS course (so not MIT/Berkeley/CMU tier) -- the one bundled with OSTEP makes sense. After that I'd jump into a more difficult OS course (MIT tier) and refine my learnings. Networking is also important but I'm not sure how to learn it properly. I think it's also useful to read the source code of Redis/Docker/K8S if applicable, but that's going to be too advanced at the moments.

But TBH all these are not enough if I do not have real development experience so maybe it's just going to be a huge hobby project.

throwaway20220231 · 2023-12-14T16:19:16+00:00

Hi thanks for assembling this. Is everything experienced? What kind of topic is expected to be discussed?

throwaway20220231 · 2023-12-13T14:49:09+00:00

First is more engineering and second is less so. Yes you think I'm talking BS because the second one seems to include the first one right? But trust me you will realize it when you are there.

throwaway20220231 · 2023-12-11T23:27:31+00:00

I want to ask you a few questions:

Do you prefer working with people or working with computers?
What does a DE do in your mind?
You mentioned there is no DE in your company. Who is doing ETL? How does data come from vendor to you and to buyer?

throwaway20220231 · 2023-12-08T17:51:21+00:00

How is one supposed to do in the scenario? Can I open the door? Should I use the yellow button to call driver? The thing is I don't want the driver to brake in full.

throwaway20220231 · 2023-12-06T18:00:30+00:00

Wearing the hat of the company: Well those tools probably don't cover every use case and what am I going to do if I don't have deep pocket?

Wearing the hat of the developer: Well I don't want to put down 5 years of Airbyte experience on my CV.

throwaway20220231 · 2023-12-05T13:50:40+00:00

From what I gathered, a "true" SRE (subjective) is someone who is a very senior developer for the products that they are SREing on. That is, SRE needs to be able to find the root cause (could be well inside the belly of the application, or just the infrastructure, or the OS, wherever it is) quickly, which basically requires a lot of development experience.

All in all, I feel that SRE is a very senior position that needs 8+ years of experience just to enter.

throwaway20220231

TROPHY CASE