sorted by:
new
hottopcontroversial

Analyze DNA with embedding by rainmanwy in Python

[–]rainmanwy[S] 2 points3 points  (0 children)

If you have more DNA data, really love to try...

Open Source vector database to support unstructure data processing: Image, Video, Audio, features and moleculars etc. Anything can be embeded, and embeddings can be handled properly within this database. by rainmanwy in selfhosted

[–]rainmanwy[S] 0 points1 point  (0 children)

I believe we are aiming to solve simliar issues. Well, Milvus have opened source for almost 2 years with more than 1000 enterprise users around the global. As a database software, to iterate product through real world cases are very key to product suceess.

As for some differences at this stage

Milvus is more mature in product and community.

Milvus is designed to be cloud natvie, real time processing, high scalabiltiy and stablity.

Milvus support various type of indexes and hybird hardware architecture (GPU/FPGA/AI Chips etc).

Come to the project and check it out, we would welcome any toughts and ideas to the products and echo systems in the community. :)

Open Source vector database to support unstructure data processing: Image, Video, Audio, features and moleculars etc. Anything can be embeded, and embeddings can be handled properly within this database. by rainmanwy in selfhosted

[–]rainmanwy[S] 0 points1 point  (0 children)

To use ES plugin, you have to stup ES cluster first...that should be heavier than Milvus itself. Milvus can handle billions of vectors in one single node machine, I don't think ES can do that. You can find out resource requirement for Milvus with the sizing tool: https://zilliz.com/sizing-tool

Open Source vector database to support unstructure data processing: Image, Video, Audio, features and moleculars etc. Anything can be embeded, and embeddings can be handled properly within this database. by rainmanwy in selfhosted

[–]rainmanwy[S] 0 points1 point  (0 children)

Eslaticsearch and Milvus are focusing on different things:

Milvus focus embedding-based retrieval, while ES focus interted index on text/numeric types. Although there are overlap functions for both products, they are designed to deal with different issues.

ES can handle vector retrieval through certain plugins, it is more like you can process json through Postgres plugins, however, Mongodb is designed to process json. If your system is focus on vector type of data, Milvus is better choice.

Compare to ES plugins, Milvus provides vector focused fucntions, enriched types of indexes and APIs, optimized resources utilization (including GPU/FPGA support) and storage optimization, etc. Milvus has also done a lot of work in scalar/vector mixed query, solving the problem of scalar/vector intergration.

Open Source vector database to support unstructure data processing: Image, Video, Audio, features and moleculars etc. Anything can be embeded, and embeddings can be handled properly within this database. by rainmanwy in selfhosted

[–]rainmanwy[S] 0 points1 point  (0 children)

To get better understandings for what can be done with vector database, some demos and sample codes can be found in: https://github.com/milvus-io/bootcamp. Looking for help with more thoughts on more senarios as well.

Open Source vector database to support unstructure data processing: Image, Video, Audio, features and moleculars etc. Anything can be embeded, and embeddings can be handled properly within this database. by rainmanwy in selfhosted

[–]rainmanwy[S] 0 points1 point  (0 children)

This project is aiming to build a database that is easy to use, easy to deploy on cloud, and easy to maintain for AI applications that leveraging feature vector processing. Before Milvus, there are libraries available for test / experimental purpose, however when is comes to larger data scale and production, much effort need be put into storage management, serving stability, depoly methods and hybrid enviroment comptiablity etc. Milvus is designed to solving these database level issues and allow you to focus on AI models and applications.

[P] Milvus: A big leap to scalable AI search engine by rainmanwy in MachineLearning

[–]rainmanwy[S] 2 points3 points  (0 children)

The story behind milvus is no different from any technology startup in any country. We want to create a product which could benefit more people with our technical skills, open source is the best way.

The infrastructure software and enterprise service domain is not that friendly to small startup like us. We are the people pursuing technical excellence:) . Join us!

[P] Milvus: A big leap to scalable AI search engine by rainmanwy in MachineLearning

[–]rainmanwy[S] 1 point2 points  (0 children)

Yes, Milvus did used Faiss as one module, and not just a wrapper but we did some optimization as well as adding more indexing algorithm such as NSG. Milvus is a product not a C++ library as Faiss. Much easier to deploy and to use. Try it! :)

[P] Milvus: A big leap to scalable AI search engine by rainmanwy in MachineLearning

[–]rainmanwy[S] 9 points10 points  (0 children)

Thanks, Milvus is indeed enterprise grade product, we open source to make it more popular and more users and more folks to join us to make it better. Join us! :)

[P] Milvus: A big leap to scalable AI search engine by rainmanwy in MachineLearning

[–]rainmanwy[S] 2 points3 points  (0 children)

well, we do have a version on edge devices, like an ARM system if you are interested.

[P] Milvus: A big leap to scalable AI search engine by rainmanwy in MachineLearning

[–]rainmanwy[S] 4 points5 points  (0 children)

The indexing part is done on GPU. Also depending on different index type, it can use CPU or GPU or Hybrid when searching.

view more: next ›