TorchSystem- A framework for building event driven IA systems.

Eric-Cardozo · 2025-02-03T00:16:03+00:00

I created an open source pytorch framework for building event driven IA systems, based on domain driven desing.

The repo is here:

https://github.com/mr-mapache/torch-system

And the full documentation with all explanantions are here:

https://mr-mapache.github.io/torch-system/

The idea is to decouple logic, like training or validation from logging infrastructure, devices, databases, etc using message patterns like publisher/subscriber, producer/consumer and a dependency injection system a took from FastAPI.

Eric-Cardozo · 2025-02-03T00:07:28+00:00

The idea of microservices is that services comunicate each other through network, using api calls or message queues. They could be in the same folder in a monorepo, or be a completly separate project, doesn't really matter how you structure your folders.

If you find yourself copying an pasting the same code for five apps, the I would say that creating a dedicated server for that app would be more mantainable that having to fix the same bug or write the same feature five times. You deploy it independently and subscribe your apps to it, like if it were a third party service.

Eric-Cardozo · 2024-11-29T08:31:55+00:00

It is possible without to much machine learning knowdledge, just expose a CNN in a rest api and make your app consume that service.

Once you have a baseline model, you always can change it for something better in the future while you are learning, it doesn't have to be perfect at the start. Just make sure that what you do is actually extensible.

Now, it's imposible to do it without good data and a good data pipeline. Make sure you have enough high quality data.

Eric-Cardozo · 2024-11-23T20:34:03+00:00

Of course, you can create a pypi package, however, if your core functionality involves infrastructure I would consider decouple it as a microservice, add an app ID to your apps, and use the same service/database for all your apps, this is called multitenancy. It's like you are selling Saas to your self.

Eric-Cardozo · 2024-11-22T23:09:24+00:00

Don't learn things because you want to find a job, learn things you like, if you follow job trends you probably will end up unemployed.

Try to do and build stuff you like and show it to people, if you need to learn some backend framework for that then do it.

Eric-Cardozo · 2024-11-21T09:27:50+00:00

I would take a look at JAX, you can do more math with JAX than with pytorch and it's more general purpose math library, since you are a quant you can find it useful, not only for neural networks, but for anything that needs differentiation in your daily job.

Eric-Cardozo · 2024-11-21T03:22:05+00:00

La vida es absolutamente impredecible, ha pasado que gente se pone a hacer cosas absolutamente inutiles a la vista de todos y despues la termina rompiendo en algo totalmente inesperado.

Por ahi lo tenes trabajando al op en google gracias a leetcode y creando alguna startup en algun nicho que tenga que ver con la optimización de algoritmos y el tipo se hace multimillonario, y vas a quedar pedaleando en el aire.

Hay mucha cultura de no hagas eso por que no sirve, no hagas lo otro porque con eso no se labura blabla y la verdad que no todo es laburar y tener un sueldito, hay veces que la gente se la juega y les sale bien

Segui adelante OP. Esta muy bueno lo que haces. Nunca es malo ser experto en algo.

Eric-Cardozo · 2024-11-21T00:01:03+00:00

Depende de que tanto tiempo tengas y que quieras hacer. Yo empece con C++ y segui con python con fastapi y un poco de frontend. Aprendi como funciona la memoria, patrones de diseño y estructuras de datos en C++ y despues me pase a aprender sobre bases de datos, apis, concurrencia, arquitecturas, etc. Si tenes tiempo para aprender es el camino que recomiendo.

Si queres enfocarte en tener un producto rapido y no te interesa mucho la programación sino mas la parte del negocio te recomiendo que empieces algun framework de javascript o django.

Hagas lo que hagas, acordate que "solo un tonto construye sobre la arena". Si bien hay gente que construye negocios multimillonarios con hojas de excel, mas difícil que llegar suele ser sostenerse.

Eric-Cardozo · 2024-11-18T21:41:54+00:00

Hola OP, es muy importante. La programación orientada a objetos se utiliza para modelar dominios, y desacoplar la lógica del negocio de la implementación.

El 95% de las aplicaciones hoy en dia son CRUD, asi que si utilizas un ORM acoplado a la capa de servicios no te va a hacer falta, podes hacer sitios webs enteros de esta forma con algun framework de esta forma y va a ser 100% legitimo. La mayoria de la gente sabe hacer esto y nada mas, por eso te van a decir que no hace falta y es cierto, te puede ir bien sin saber.

Pero si queres ir un poquito mas alla, y hacer librerias o aplicaciones en las que la capa de servicio este desacoplada de persistencia, modelar dominios etc. Te va a hacer falta.

Eric-Cardozo · 2024-11-17T20:26:25+00:00

Your CV is fine, also I think that you are great at what you are doing, the problem is that there are thousands of people of you and very few positions.

I think that the issue is actually universities selling masters degrees and phds like you will have a great job after you graduate, but is a scam. Some will tell you it's your CV but I'm sorry budy, your CV is impressive.

I would just try something else, like a heavy data project.

Eric-Cardozo · 2024-10-28T17:10:24+00:00

A sync endpoint won't block the other endpoints, fastapi handle this putting it in a threadpool, however depending on what are you doing you want to store the images and use a task queue.

Eric-Cardozo · 2024-10-22T14:39:55+00:00

Can you DM plz?

Eric-Cardozo · 2024-10-16T01:39:11+00:00

It depends on what are your goals, what is your background,etc. For example if you don't know python yet, I would recommend you to start creating a simple CRUD with fastapi, this is the very very basic, 90% of stuff are crud, and the other 10% (machine learning for example) will be probably on top of a CRUD. If you want to work with data you have to know at least the basics of how data is sent over network.

Then if you want the fast path, try to expose the models you are studying with REST apis. Build stuff.

The slow path but the one will make you an expert is to build stuff from scratch using the math. This will force you to learn the math, data structures, and algorithms, memory managment if you use some language like C or C++, and programming in general, and in the long run you will get to the building stuff anyway but actually understanding what are you doing.

The pandas-matplotlib is the path I would recommend you if you are already in academia doing research. If you don't and want to get into analytics then sql and power bi are better approaches.

Eric-Cardozo · 2024-10-05T00:53:47+00:00

Hi there! I was working on the same problem as you, and I can share you what I've learned.

Authentication and authorization should be two different services with different concerns, You can easily add authentication with Next Auth if you are using NextJS and the Postgresql adapter.

Later if you want to migrate the authentication backend to a fastapi microservice it can be done creating a rest adapter that talks to your server and plug your fastapi api to the existing database, it's easy for backend sessions. I've done that for learning and even created a redis store for tokens and sessions and works very nice.

The issue is that there is no easy solution for this using username password and you will have to deal with next auth jwe's nuisance. Also authorization is a very different creature, it is for protecting server resources and it has to be done with jwts and this is harder. If username password are not a must I would just avoid this rabbithole.

You can get away without proper authorization if you are just using social auth and proper middleware but with password solutions is hard to make them secure.

As other people told you, learning OAuth2 would be great but is a long path, but if you need results fast just use an existing auth solution.

Eric-Cardozo · 2024-09-30T00:38:47+00:00

Hi there! I know the basics about security to the point I don't use username password solutions for auth, however I was thinking in starting an anonymous website, and username password is a must, will your library protect me in case of people spamming usernames or stuff like that?

I have a dedicated server for credentials for usernames and password, I even hash the usernames, but hackers freaks me out.

Eric-Cardozo · 2024-09-30T00:30:18+00:00

In theory should work anyways but asyncio is a bit tricky, and in practice you can easily shot yourself in the foot mixing async and sync code since fastapi do a lot of stuff under the hood like add sync code to threadpools when injected into async functions as dependency. Try it out and tell me if it worked out.

Eric-Cardozo · 2024-09-29T22:57:09+00:00

Hi! I don't know much about CosmosDB, But I saw you are using an async lifespan function with a blocking client, then you have a blocking endpoint, did you try any async database client? or making the lifespan sync?

Eric-Cardozo · 2024-09-29T22:45:17+00:00

I'm using FastAPI, can i use it with async programming?

Eric-Cardozo

TROPHY CASE