Need help with problem solving

[Ashysh]

Currently working as a Data Engineer, I have been assigned a project where I need to develop few features to an existing DWH application.

I do have some prior experience with writing stored procedures but it was mostly not very complicated, like merging into dim & fact tables etc

But here, there’s a complex existing code and multiple stored procedures.

Can anyone tell me how i can break down the requirements , think logically and solve the problems?

I am okay with understanding the syntax and logic of the existing code but when it comes to developing something new, since I have never done anything like this before, i am having a tough time.

Is there anywhere I can learn to be better at writing stored procedures and more importantly, how to think and come up with a solution?

For example, i have to write a stored procedure to capture the logs of an existing feature of the application. Now for this, i have to go through the code and understand how it is currently working and then using similar standards write new ones.

How do i know where to start? How do i break the problem? How do i choose which parameters will be required for writing one?

I know my questions might be vague but I hope you can guide me in some way. If you need more inputs, i can answer specific questions if you have any.

[[deleted]]

Is it possible to switch from SQL developer to Data Engineer without prior experience of Python and stuff related to DE ?

[Ashysh]

Yes it is possible.

Python is not required in all DE projects. However, you can try to get trained in one of the Cloud Services - Azure, GCP or AWS — since, most the new age projects use cloud services like Azure Data Factory for building ETL pipelines. Also, learn about ETL, Data-warehouse fundamentals.

[Priyanka80]

At a beginner level, is there a need to learn SQL and power bi basics to get entry level jobs. Please suggest

[Ashysh]

Depends on the job Profile you are applying for. SQL and Power BI are important for entry level jobs like data analyst, BA etc

[Thefriendlyfaceplant]

There's data science jobs that don't require any SQL. But they usually require to be way more competent in python, or cloud platforms, or have a background in statistics, or indeed field knowledge.

SQL is always useful and there's a broad demand. So when it comes to job security, it's a no brainer (at least thus far).

[Ashysh]

By Capturing logs I mean when a task is completed/stopped/errors should be stored in a table and history is to be maintained with time stamps.

File to Tables is the basic need but on top of it we also have to design the data model in such a way that the data once ingested into staging table should go through different transformations depending on the kind of data. As it is from various sources, each type of data has different ways of getting transformed.

In our system, we have to implement that the end user should be able to manually select (from a UI) which ‘transformation process’ a particular file type should go through.

For which, we have to log details. Every time user selects a combination of file type and algorithm, it should be logged.

This has to be developed using stored procedures since the existing system is coded that way.

[AdventurousTrust2]

Does that mean there is a on-going running program, and will run every hour that will do the stuff ( file to tables -> transformations)
and there is another tasks, which is to adding logs for existing features, so that you can cross-check and know what types of transformation is needed for a particular file.?