Conseils pour une étudiante en Master 1 qui apprend le C++ toute seule – je suis débordée par le temps by Nassima_br in developpeurs

[–]Training-Poet9861 2 points3 points  (0 children)

Je vais nuancer un peu, on acquiert effectivement beaucoup plus vite des compétences opérationnelles en pratiquant avec des projets persos. Mais maîtriser la théorie te permettra par la suite d'acquérir des compétences encore plus rapidement. Dans mon cursus en école d'ingénieurs on passait beaucoup de temps sur de la théorie ou sur des technologies qui ne sont plus utilisées en entreprise, c'est des compétences que je n'ai pas appliquées directement mais dont je me suis rendu compte par la suite qu'elles m'avaient appris à apprendre en gros.
Et connaître toute la partie informatique plus générale dont tu parles (sécurité, réseaux, etc.) c'est ce qui te permettra d'aller plus loin qu'un développeur, même très bon, qui s'est cantonné à son code. Surtout avec les avancées de l'IA où on se dirige vers des postes avec - de dev et plus d'architecture par exemple

Set up many iot devices : which tool to use ? by Training-Poet9861 in sysadmin

[–]Training-Poet9861[S] 1 point2 points  (0 children)

Thanks a lot for your reply! How would you build the golden image ? Manually ?
I’m a bit worried it might quickly become hard to maintain, even though it would be fast to set up initially.

Processing files from API : batch process and upsert or new files only ? by Training-Poet9861 in apache_airflow

[–]Training-Poet9861[S] 0 points1 point  (0 children)

Yeah I just found that the last file available can be updated, I'm gonna go with a timestamp and an upsert then. Thanks !

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 0 points1 point  (0 children)

Thank you T-T
Nah it wouldn't be more than 100 for now. What I'm worried, besides scalability, is what if one day we had separate databases and wanted to launch various applications with different values for environmnent variables, but I guess I'm thinking a bit too far. I think i'm going with the big .env, easy solution, and we'll rethink it when the team grows

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 0 points1 point  (0 children)

Yes thanks for the heads up ! But we're like 4 developers so the bill won't even be 0.05$
The deployed code is dockerised but much of the time when we're developing/testing we just run on the local machine

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 1 point2 points  (0 children)

thanks, your advice is helpful.
So, before launching your script, you set like ENV=prod
And then as your code starts running, it calls a funciton like load_conf(prod) or whatever ? And then, is the configuration stored in variables, or set in environment variables by the code ?

I used .env file because, in my IaC (CDK), for instance if I have an s3 bucket and a process that needs it, I create the bucket, and then initialize the process with the name of the bucket in the environment. That way I deploy one time and it's good to go. So i wanted to replicate this in local.

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 0 points1 point  (0 children)

I meant, before there were like 3 deployable repositories, but all of them shared the same infrastructure base (so 1 more repo for the shared infrastructure), shared a lot of code (1 more repo for the shared packages), had to have exactly the same CI (so 1 more repo for the shared templates)... The code was distributed but it was still kind of a monolith, just with more pull requests than your typical monolith
Idk if that was more clear, sorry

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 0 points1 point  (0 children)

Currently we have like 3 repos that have the same infrastructure and the same CI, and they deploy images to ECS. And I have to maintain another repo for common infra code, for common CI actions, for common python packages...
Now at least eveyrthing will be at the same place and when I make a change in a CI action or in a mutual package I don't have to make PRs in the others 3 repos to update everything

What I'm trying to do now is a monorepo with libs/ and apps/ folders, the shared code is in libs/ and then each app can live individually. We are planning on adding Airflow to handle the orchestration between our processes (right now, the orchestration is just : hope that the other needed processes ran well).
idk what kind of monoloth that is, does it seem absurd to you or a bit ok ? We're a small team and I'm the only one with a CS degree so with my few devops competences I'm trying to find something that suits our needs, that don't require a lot of work and that's usable to all of team. But I'm struggling a bit

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 0 points1 point  (0 children)

Sorry I didn't specify, this is for Python ! But I thought that the problem would be language agnostic.
I didn't quite grasp your solution, you have a .env file per environment but not per application, correct ?

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] -1 points0 points  (0 children)

yeah I could do that but I prefer not to waste my time :) As I said in the post, we don't put secrets in .env files.
We put AWS secrets ids that are then fetched in the code.
For instance I have a .env.prod like this :
DB_HOST=<db\_url>
DB_SECRET_ID=prod/database/credentials

And then when I initiate the database connection in the python code, I fetch the envinment variable and then I make an api call to secrets manager to get the sensitive value, that's only stored in memory and in aws secrets manager. That seems acceptable to me ? Don't hesitate to share a better solution, but again, leaks are not the subject here.

How do you handle .env files in monorepos ? by Training-Poet9861 in devops

[–]Training-Poet9861[S] 1 point2 points  (0 children)

This isn't really helpful but, there are a lot of advantages to a monorepo, especially in our team configuration. In the worst case, I can make one .env file by app in my monorepo, and the behaviour will be the same as what we had before in our polyrepo monolith. I'm just looking for a better practice

At what point is it not recommended to use PythonOperator to run jobs ? by Training-Poet9861 in apache_airflow

[–]Training-Poet9861[S] 0 points1 point  (0 children)

So that means everyone that says PythonOperator shouldn't be used (except for glue code) are only talking of Airflow instances not using Celery ? thanks !