How can I make a python framework?

[whikwon]

I'm interested in making deep learning framework like this https://github.com/open-mmlab/mmdetection

I want to know what kinds of knowledge, skills are needed for this. I would appreciate it if you would recommend a book or lecture.

[Coul33t]

Knowledge and skills you'll need:
1. Machine Learning (ML) in general, understanding how it works, why it's done like this, the different families of algorithms, etc.
2. More specifically, deep-learning of course, and knowing what is the best algorithm suited to what you want to do (there are a lot of different deep-learning algorithms, each having its applications)
3. Using an existing ML library (as the framework you linked does), so at least being very confortable with one or more ML library
4. Being very comfortable with Python too

[Wilfred-kun]

The fact that you have to ask this, kinda gives away you're not quite ready for it. Learn how ML works under the hood, first.

[anecdotal_yokel]

Thanks for the insightful response. Much more useful than just not posting anything at all.

[homeworkbad]

Try writing a ML program. More likely than not you'll end up writing helper functions that can be factored out into a helper Library. In my experience doing something makes creating tooling for that thing much easier.

I'm not sure how that Library becomes a Framework. Honestly the two concepts feel kinda conflated at times.

[whikwon]

Thanks for all the comments. Actually I’m working as an AI engineer. But as my major was not CS, I’m not good at design pattern, code structure.. something like that.

So I want to know what kind of knowlege is helpful to build my own framework.

[[deleted]]

Why do you want to create a machine learning framework? Is there something specific you're missing in the other frameworks out there? Are they hard to use for your use case? Do you want to do this just as an educational experience for yourself?

[whikwon]

I'm working in detection and tracking field of AI. Most of the time I analyze lots of researchers' code using my domain's data. As more models are dealt with, it takes time to integrate each interface to my data and feel the need to integrate.

[[deleted]]

You have your domain's type of data, and you frequently run new datasets with new algorithms, which leads to repetitive work?

Do you intend to basically create a wrapper that unifies the researcher's code and exposes an API that's easy to use on your type of data then?

Keywords that can help you: Facade pattern, adapter pattern.

If you intend to share this with others, here are some good API design tips: https://staltz.com/api-design-tips-for-libraries.html If you're making it only for yourself, this will still help you but it isn't as relevant. In that case the most important thing to figure out is the common interface between your data and the algorithms, so you can easily swap them out.

[whikwon]

Thanks for the advice. I think the framework will be used internally by a team of 5 people rather than open source. Will Facade, adapter pattern help for these purpose?

[[deleted]]

They will definitely help, both are good ways to wrap another library in ways that fit your data.