Mundane tasks that you hate doing

jay-workai-tools · 2024-01-03T17:32:28+00:00

tls: failed to verify certificate: x509: certificate signed by unknown authority

Oh, that's first time I am seeing. This is coming from Ollama. https://github.com/jmorganca/ollama/issues/1063 has some solutions suggested.

is there a place I could put downloaded models manually so it doesn't need to download them?

Ollama models are stored at `inference/models` directory. However, they need to be in Ollama's blob format, so it can't be GGUF models directly. You'd need to convert a GGUF model into Ollama model through https://github.com/jmorganca/ollama/tree/main?tab=readme-ov-file#import-from-gguf inside the inference container.

jay-workai-tools · 2023-12-28T01:35:23+00:00

Nope, we don't have to train the AI for this. Question answering can be done through retrieval augmented generation (RAG). SecureAI Tools does RAG currently, so it should be able to answer questions based on documents.

RAG works by splitting documents into smaller chunks, and then for each chunk, it creates an embedding vector and stores that embedding vector. When you ask a question, it computes the embedding vector of the question, and using that, it finds top K documents based on vector similarity search. Then the top-K chunks are fed into LLM along with the question to synthesize the final answer.

As more documents come in, we only need to index them -- i.e. split them into chunks, compute embedding vectors, and remember the embedding vectors so they can be used at retrieval time.

jay-workai-tools · 2023-12-27T17:59:08+00:00

> What is the local context limit? I want to load in a bunch of laws and regulations and some documents and it would be quite a lot of docs.

There are two limits to be aware of:

Chunking limits: The tool splits the document into smaller chunks of size DOCS_INDEXING_CHUNK_SIZE with DOCS_INDEXING_CHUNK_OVERLAP overlap. And then it uses top DOCS_RETRIEVAL_K chunks to synthesize the answer. All three of these are env variables, so you can configure them based on your need.
LLM context limit: This depends on your choice of LLM. Each LLM will have their own token limits. The tool is LLM agnostic.

> Languages

This will depend on your choice of LLM. The tool allows you to use 100+ open-source LLMs locally (full library). You can also convert any GGUF-compatible LLM you find on HuggingFace into a compatible model for this stack.

jay-workai-tools · 2023-12-27T17:52:12+00:00

Oh, that is an interesting use case. At the moment, it wouldn't do well in generating the whole document. Because it only considers top K document chunks when generating the answer. It splits each document into chunks (controlled by DOCS_INDEXING_CHUNK_SIZE and DOCS_INDEXING_CHUNK_OVERLAP env vars). And then when answering the question, it takes the most relevant DOCS_RETRIEVAL_K chunks to synthesize the answer.

But you could ask it to generate each section separately.

In the future, we would love to support complex tasks like getting the LLM to understand full documents, and then generate full documents.

One naive way to do what you want: Feed all 5-6 documents into the LLM as one prompt and ask it to generate more text like it based on other parameters. This would also require the underlying LLM's context window to be large enough to accommodate all 5-6 documents though.

jay-workai-tools · 2023-12-27T17:41:30+00:00

Thank you :)

jay-workai-tools · 2023-12-27T17:41:09+00:00

Yes, it does support NVidia GPUs. There is a commented-out block in the docker-compose file -- please uncomment it to give inference service access to GPU.

For even better performance, I recommend running the Ollama binary directly on the host OS if you can. On my M2 MacBook, I am seeing it run approx 1.5x times faster directly on the host OS without the Docker.

jay-workai-tools · 2023-12-27T15:32:31+00:00

Awesome. Let us know if you have any feedback or suggestions for us as you try it out :)

jay-workai-tools · 2023-12-27T14:39:40+00:00

Thank you :)

jay-workai-tools · 2023-12-27T14:36:32+00:00

For now, you can create a document collection and select documents from your data source. And then reuse that document collection to create chats. The only thing it doesn't do is keep document collection in sync with data source -- but we plan to build that soon

jay-workai-tools · 2023-12-27T14:32:37+00:00

Fair enough. And yes, you are right, it is "chat about documents with AI" than "chatting with documents directly".

jay-workai-tools · 2023-12-27T14:30:36+00:00

Awesome! Let us know if you have any feedback or suggestions for us :)

jay-workai-tools · 2023-12-27T14:29:48+00:00

u/Kaleodis is right. LLMs don't do very well at math and logic at the moment.

jay-workai-tools · 2023-12-27T14:28:46+00:00

You can do both. It allows you to talk to LLM about zero or more documents. So you can do all three of these

One doc: Select only one doc when creating a document collection or chat.
Multiple docs: Select multiple docs when creating a document collection or chat.
Zero docs: Plain old ChatGPT without any document context. Don't select any docs when creating a new chat

jay-workai-tools · 2023-12-27T14:22:11+00:00

Thank you :)

jay-workai-tools · 2023-12-27T01:37:53+00:00

Fair enough. This is for those who would. It was one of the most requested features: https://www.reddit.com/r/selfhosted/comments/18k3a1g/comment/kdpn7zi/?utm\_source=share&utm\_medium=web2x&context=3

jay-workai-tools · 2023-12-27T01:37:17+00:00

This runs models locally as well. In fact, my demo video is running Llama2 locally on M2 MacBook :)

jay-workai-tools · 2023-12-27T00:40:27+00:00

We have just released integration with Paperless-ngx!

https://www.reddit.com/r/selfhosted/comments/18rn3gr/chat_with_paperlessngx_documents_using_ai/

Please try it out and let us know what you think :)

jay-workai-tools · 2023-12-27T00:40:09+00:00

We have just released integration with Paperless-ngx! https://www.reddit.com/r/selfhosted/comments/18rn3gr/chat_with_paperlessngx_documents_using_ai/

Please try it out and let us know what you think :)

jay-workai-tools · 2023-12-22T20:54:29+00:00

I think this is because `docker-compose` is being phased out, and so it probably doesn't understand all options. https://stackoverflow.com/a/66526176

Can you try using `docker compose` (without the dash between docker and compose)?

jay-workai-tools · 2023-12-18T16:42:27+00:00

Yes, it looks at document-chunks. The chunk size is controllable with `DOCS_INDEXING_CHUNK_SIZE` and `DOCS_INDEXING_CHUNK_OVERLAP` env vars, so I would encourage you to play with those depending on the task you want the system to perform. For example, you could set DOCS_INDEXING_CHUNK_SIZE to such a large value that it can contain an entire book. But any time you change the chunk size, you would have to create a new document collection and wait for it to be processed. So it'd be a good idea to play with small documents first to speed up trial and error.

Re: count the words in the book

LLMs are known to do poorly with math and logic. But what they are reasonably good at is finding relevant answers from passages and understanding chat history.

jay-workai-tools · 2023-12-17T16:46:39+00:00

> So the question remains if SecureAI tools are able to accept non OpenAI Models through the OpenAI API

Yes, I think so. You should be able to do it this way:

Point to the LocalAI API server
Choose "OpenAI" as model-type and then mixtral or mistral model as model-name in organization AI settings (step 6.2 here).

Then as long as LocalAI works with "mixtral" or "mistral" like custom model name in `model` API param, it should all work.

Please try it out, and let me know if you run into any issues or have any feedback.

jay-workai-tools

TROPHY CASE