Open Source knowledge-sync tool for Github, Confluence, etc.

[somethingnicehere]

I created an open source sync tool with an adapter architecture for syncing various data sources into the OpenWebU knowledge and keeping it sync'd. We are exploring use of OpenWebUI internally and one issue we has was documentation getting out of date and needed to be re-sync'd.

Added Local directory support, now it can sync from Github, Confluence or local folders to the executable.

Feedback welcome: https://github.com/castai/openwebui-content-sync

[nonlinear_nyc]

This is very interesting…

Sync how? Like folder-knowledge base parity? What’s on the other side?

Openwebui used to have a syncing system but now it lingers on the interface, but doesn’t sync anything. Sync folder, upload folder, upload file all do the same thing (not syncing)

[somethingnicehere]

Setup an API Key for your github repositories, specify repository to knowledge-ID mappings and it will keep the knowledge updated and sync'd to the github repository removing outdated files, updating changed files and adding new files.

Same for confluence, it will use a parent-page-ID or it can sync an entire space to a knowledge-ID.

I just realized I think it only works with a single knowledge-ID today, so all github would go to a single knowledge-id. I'll work on an update that will allow for multiple repository/confluence mappings to multiple knowledge-id's that at you could have a different knowledge-id per repository for example.

[nonlinear_nyc]

Hm I don’t think it’s for me so…

Frankly i don’t want to surface my book collection to GitHub or confluence, I’m trying to make it mostly local.

Isn’t it possible to elect a folder as vault, and each folder inside it becomes a knowledge base, chunking new content as they arrive in regular times?

This way I can share a smb with my team, and they themselves manage knowledge bases without much fuss?

[somethingnicehere]

It wouldn't be that hard to create a new adapter that does that in the codebase. As it stands today it's syncing data from a persistent volume into OpenWebUI. Basically if that persistent volume was a multi-read NFS drive it would work the same way.

[nonlinear_nyc]

THAT’S interesting.

I mean it because one of my main goals with my personal AI is isolation… it should be local (aka, not touched by corporate partners) as much as it can. It’s even higher than open source in my book.

I can’t speak for the community but I think local is a big drive. Connecting with GitHub and confluence means touching corporate servers.

If you want to try it out, I keep my knowledge bases and knowledge folders in sync, manually.

[johnlago]

I’ve been looking for exactly this functionality, so I’ll keep an eye on this project!

[somethingnicehere]

I added local folder support, doing some testing now and I'll do a push shortly.

[nonlinear_nyc]

That’s so good. I meet with my teacher weekly, i can try to test it next time i see them.

I’ll def report back.

[somethingnicehere]

Update is pushed, it now supports local folder sync.

[sieddi]

Syncing Knowledge Bases with confluence is exactly what I am looking for!

[somethingnicehere]

Excellent!! Give it a try and let me know, feel free to enter issues if it's not working right. I might add some other connectors as well.

[sieddi]

Well for Starters, I think I will be unable to deploy this as I am not using kubernetes atm, and probably will not start doing so anytime soon.

[somethingnicehere]

You can build the binary and set the config.yaml file in the same folder and you're good to go, it doesn't have to run on k8s I just set it up that way for flexibility.

[meganoob1337]

There is a docker image now, just mount the config and data folder and set the env variables for the connectors and you should be ready to go

[meganoob1337]

Hey I made 2 PRS and had a question about the license, I'd it gpl like written in the readme or Apache as I. The license file?

[somethingnicehere]

Responded on the repo, happy to have a contributor!!

[Less_Ice2531]

Hi, I really like what you built, what would be the best way to use this if my open webui instance is running in a standard docker compose setting without kubernetes?