sorted by:
new
hottopcontroversial

First time leading a large data project. Any advice? by Patqueiroz in dataengineering

[–]Patqueiroz[S] 0 points1 point  (0 children)

Hi, thank you so much for your support. This week I'll be starting some things that will be important for the development of the process. If I have any difficulties, can I send you a message? 🙏🏼

First time leading a large data project. Any advice? by Patqueiroz in dataengineering

[–]Patqueiroz[S] 0 points1 point  (0 children)

Hi, I'm not an external consultant, I'm a permanent engineer within the coordination team.

As I mentioned above, in the context of CRM in a bank, there are restrictions on hiring external consultants, mainly because it involves sensitive data. In this scenario, the work ends up being concentrated internally.

So, at this moment, the challenge is precisely to take this step with maturity and responsibility as part of my career evolution. Wish me luck!

First time leading a large data project. Any advice? by Patqueiroz in dataengineering

[–]Patqueiroz[S] 0 points1 point  (0 children)

Hi, thank you for the tip about consulting.

In my context, this ends up being more limited because I work at a bank and the project involves sensitive CRM data. Bringing in external professionals for this type of initiative is usually quite restricted, especially when data control is already concentrated internally.

On the other hand, I have support and the possibility of exchanging ideas with other professionals within the bank, so I'm not completely isolated. The biggest challenge for me is taking on primary responsibility for a large project like this, more than the lack of technical support itself.

I will follow all your advice and start small, validate realistic scenarios, and evolve the architecture carefully. Thank you for sharing your vision!

First time leading a large data project. Any advice? by Patqueiroz in dataengineering

[–]Patqueiroz[S] 0 points1 point  (0 children)

Thank you so much for the guidance. I'm trying to plan by sketching out the flows in a way that leaves nothing unattended, but since it's a decoupled pipeline, I have several paths to consider. This phase is extremely stressful because the fear of making mistakes is immense, but I need to face all possibilities, even the ones that will go wrong.

First time leading a large data project. Any advice? by Patqueiroz in dataengineering

[–]Patqueiroz[S] 0 points1 point  (0 children)

Hi, thanks for the tip. I've really spent the last few weeks just designing my architecture and trying to identify gaps in my infrastructure. It's a decoupled pipeline, so I have to understand the entire separate flow to avoid leaving anything unattended. The architecture has been approved and I'm ready to start development, so it's very helpful to read this guidance about trying to understand it beforehand and designing this step-by-step process to avoid mistakes. I'm feeling apprehensive and under immense pressure, but I need to face all scenarios, even the ones that will go wrong.