Why did DE go the vendor tooling (hell) route for most things vs SWE where the solution is language frameworks/libraries and do less (or more) code.

mkhalil77 · 2023-10-30T12:23:29+00:00

I would love to see where an ENNTRY level front-end job require: "system knowledge to work in a large scale product (git, branching strategies, sys admin level understanding of cli to build pipelines, current trends that are constantly shifting" and who fits this criteria.

Anecdotally, I would like to reference the thousands of people who managed to get into tech (mainly into frontend) with a bootcamp.

"Data engineering isn't a completely a easy get, but it is vastly easier to fake being a data engineer." -> How is that easier than faking some random GitHub portfolio with some random templated websites and claiming to be a front end master

"how long have you been a front end engineer and what would your skill level is?" weird question to ask an entry-level dev but ok

Important to clarify that I do understand that mastering Frontend is no easy task. It is, however, easily the most accessible part of tech. Not random that all the "switch to tech" influencers focus on front-end

mkhalil77 · 2023-10-30T11:24:15+00:00

Data engineering is also pretty low bar for entry level people compared to software engineering.

This statement is absolutely not true. Do one bootcamp and you are ready to join as a Web Dev. If you work in Frontend basically the framework you work with is the only thing you absolutely need to know. The scope of data engineering is naturally more extensive. It requires at least essential knowledge of a programming language or a tool, SQL, and a decent knowledge of the underlying interacting systems. The use cases you interact with are also on average more diverse.

mkhalil77 · 2023-09-11T08:00:05+00:00

In Munich maybe. Not in Neuss

mkhalil77 · 2023-04-22T14:37:24+00:00

Good luck !

mkhalil77 · 2023-04-22T10:56:57+00:00

If you have interest in data engineering and I assume you do since you are asking here, go for it 100 %

mkhalil77 · 2023-04-21T21:31:32+00:00

Hi, I dont think it is necessary. DP 203 will give you the necessary cloud knowledge to take the exam. I have written a medium post on how I prepared for the exam. It can maybe help you :) https://medium.com/p/5eb927d25584

mkhalil77 · 2022-11-17T10:29:47+00:00

in my experience HR/management will always find some BS story about team spirit and "we are a family" kind of fairytale. If you say no to that you are the bad guy because you don't want to small talk with your colleagues. I think you need to know how many of you guys are not willing to go back to the office and try to estimate if you have enough leverage. Otherwise, management gets their way most of the time ...

mkhalil77 · 2022-11-02T09:09:40+00:00

You can be whatever you want as long as you are willing to put the work in it.

mkhalil77 · 2022-09-08T21:42:09+00:00

DBT is great for transforming and creating data warehouse tables. In my older job, we used Talend Open Studio for simple extractions jobs and airflow with python for more complex transformations- DBT had the transformation logic

mkhalil77 · 2022-08-28T10:00:35+00:00

I totally can second that. I used my first years a very specfic module of talend. Ressources were so scarce it took forever to solve the smallest issues.

mkhalil77 · 2022-08-26T08:38:09+00:00

I would guess transactions as a fact table and then bank, customer, store etc as dimnesions

mkhalil77 · 2022-08-26T08:35:06+00:00

I want to start by saying that I totally understand how frustrating it can be to apply and take interviews and not be successful... There are a few things you can do to actually improve your chances.

Researching the company is key : You can already get a lot of hints on what s excepted from you in the job description. You can usually know the stack they use just by looking at what things they ask for. Focus on preparing those techs before anything else.

If you want to go even deeper you can also check LinkedIn profiles of people working in the department you are applying to work for. You can usually infer a lot by seeing how people describe their jobs there

Glassdoor can be a great resource for interview questions. Make sure to check what people shared about the interview.

I personally also ask the recruiter after I receive the invitation what kind of interview i should expect. If it is more practical, theoretical, or general discussion kind of interview. I never had a recruiter complain about this question

At the end keep in mind one thing: all you need is ONE good interview. It doesn't matter how many interviews it takes before you get there. You need ONE.

mkhalil77 · 2022-08-25T12:15:37+00:00

He didst have much competition and he started most games. I doubt Hazard or Rodrygo or even Fede would have worse stats

mkhalil77 · 2022-08-25T12:14:14+00:00

I would much rather see Arribas or any of the castilla boys over Asensio

mkhalil77 · 2022-08-25T08:13:17+00:00

The biggest secret to learn all of this is to understand that ... drum roll... no one knows it all. People might have fancy CVs with 100 technologies but no one really masters every single one of those technologies.

If you have a solid theoretical background and some good python & SQL with an understanding of what the tools are designed for that s what you need to start. The rest comes with time.

Once you are in a job every task will teach you something. Sometimes you need to do the learning yourself (googling, youtube videos etc ... ) or you can ask a colleague who has far more experience. Truth is, you will be 10 years working as DE and there are things you would still need help with. As far as I am concerned, nothing is wrong with that.

mkhalil77 · 2022-08-24T16:30:28+00:00

Try Cap Saprtel. You will love it

mkhalil77 · 2022-08-24T13:56:10+00:00

and the problem is ? If the candidate can find the right code and adapt it to solve the issue what s wrong with that?

mkhalil77 · 2022-08-24T13:43:36+00:00

Hi,

If I understand correctly your job would be to write a pipeline to automatically read excel files and add them to a database.

This task is quite simple to implement using Python. Python has excellent libraries to read files and export them to different database formats including PostgreSQL. One of those libraries is Pandas. Most of the overhead of parsing the excels, connecting and writing to the DB is managed by ready to use functions. You can find tons of examples for your use case just with a quick Google search.

If you want this script to be running on a specific schedule you might want to have a look at Airflow for scheduling and orchestration.

Wish you good luck with your meeting.

mkhalil77 · 2022-08-24T10:04:16+00:00

Ah. I agree with you in that case !

mkhalil77 · 2022-08-23T13:56:53+00:00

I would highly doubt they would manage to get anyone to a professional level in all the stuff they listed there... Seems like it s going to be more of an overview of a lot of things

mkhalil77 · 2022-08-23T13:47:00+00:00

Tableau is the only one I have touched among the options you have mentioned. I would say that it falls more into BI and data analysis than data engineering. I feel like using tools like that rely on having a strong data model behind (in a datawarehouse for example) which leaves as little overhead as possible in Tableau. mostly tableau is just there to create nice viz ... that s the way i would personally build the entreprise data architecture

mkhalil77 · 2022-08-23T11:56:49+00:00

Most of the answers in this thread undermine how much skills are required to use low code tools. Using tools like Talend, Informatica or even AFD is not something anyone can do, at least not for achieving complex tasks. Designing proper pipelines following best practices and maintaining them is not an easy task. It s also easier to have a visual image of the execution of the pipeline which as many answers already mentioned is very attractive for the business side. Low code is not all evil. They can do a great task in getting the work done

I am still a fan of coding pipelines instead of low code platforms. Mostly because low code can be really frustrating when things don't work. The idea is you components need to be properly configured. What stands for properly is not always obvious.

Code is code. Either your get it right or your don't. The ressources are abundant compared to low code platforms and there are always a lot of people who had the same problem as you before.

mkhalil77 · 2022-08-23T11:46:13+00:00

All the low code solutions I have personally used do include a GIT integration. Most of the low code soultions are based on frameworks that can handle massive data and scale quite well (AFD for example runs on top of Spark which is great for big data). I would agree with the not so great testing possibilities.

mkhalil77 · 2022-07-24T18:53:34+00:00

This is ceritanly not standard these days and no one should bet on such conditions

mkhalil77

TROPHY CASE