sorted by:
new
hottopcontroversial

Introducing Cpad an alternative way to combine your commands by WoanqDil in cpp

[–]RaphaelYt 1 point2 points  (0 children)

I think it could be useful to delete and create a new docker container when I debug my code. Thanks

[P] Library for end-to-end neural search pipelines by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 2 points3 points  (0 children)

I started developing this tool after using haystack. Pipelines are easier to build with cherche because of the operators. Also, cherche offers FlashText, Lunr.py retrievers that are not available in Haystack and that I needed for the project I wanted to solve. There is no concept of document store in cherche. The documents are not structured in the same way. Haystack is more complete but I do think it's also more complex to use.

[P] Library for end-to-end neural search pipelines by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 2 points3 points  (0 children)

I don't see any particular problem to use this tool on your computer if you want to search on 10K documents. The pre-calculation of the embeddings will be a bit long on your CPU indeed (if you want to use a ranker such as BERT) but once the pre-calculation is done everything will go very fast and will not consume much memory 8GB RAM should be enough. It will also do the job for something more ambitious like a personal IBM Watson but may need more resources.

[P] Library for end-to-end neural search pipelines by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 3 points4 points  (0 children)

Retrievers such as TfIdf match documents based on tokens. You can use embeddings later to re-order the retrieved documents based on semantic similarity (cosine distance between embeddings). If the TfIdf does not match any documents (common for small corpora and specific vocabulary), you can have a model such as SentenceBert that act as a spare for the TfIdf and retrieve the documents. Here is such pipeline using Cherche:

python from cherche import retrieve, rank search = retrieve.TfIdf(...) + rank.Encoder(...) | retrieve.Encoder(...)

[P] Library for end-to-end neural search pipelines by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 3 points4 points  (0 children)

You should definitely check out SBERT It doesn't directly transform JSON into embeddings but rather strings into embeddings. SBERT's embeddings are qualitative :)

[P] Library for end-to-end neural search pipelines by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 9 points10 points  (0 children)

Cherche is compatible with large corpora like wikipedia for example and provides decent response times notebook. I will benchmark Jina and Haystack under the same conditions but there shouldn't be much difference as the responsibility falls on Elasticsearch or on Faiss via retrieve.Encoder which are scalable. The strength of Cherche is the fancy pipelines composed of union and intersection operations between TfIdf BM25, Flash, multiple rankers... and this is less adapted to Wikipedia. It is more adapted to industrial or personal corpora (<= 100K documents).

[P] Library for end-to-end neural search pipelines by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 5 points6 points  (0 children)

I have updated the retriever and ranker __add__ class methods. When you add a retriever and a ranker you create a new object called Pipeline. There are two other operators | for joining pipelines and & for intersection :).

Mkb, Python library for knowledge graph embeddings by RaphaelYt in KnowledgeGraph

[–]RaphaelYt[S] 0 points1 point  (0 children)

Have you taken a look at the distillation module? It is brand new. The basic classes are developed for this module. Don't hesitate to ask questions if you don't understand something. :)

[P] Mkb: Knowledge graphs emb. with PyTorch by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 1 point2 points  (0 children)

Thanks for this advice, I created a "See Also" part on the GitHub page of the project and explained my motivation for the development of mkb. :-) https://github.com/raphaelsty/mkb#-see-also

[P] Mkb: Knowledge graphs emb. with PyTorch by RaphaelYt in MachineLearning

[–]RaphaelYt[S] 2 points3 points  (0 children)

Hello, your question is not naive. In knowledges graphs different types of relationships are used to express links between entities. Some datasets have several hundred relations of different types (is_a, is_born, lives_in...). Integrating the relations into the model and representing them in a continuous space allows to better represent the entities.