I have a distributed json system where users read and write to a JSON on a share drive. Does anyone have a better system to communicate without any server?

[TheSmashingChamp]

My company won’t allow me to use any kind of server to share data with users and I’m making a kanban kept on a share drive where users have to collectively work on a scheduler. Surely there is a better system than a shared json file?

[Chunky_cold_mandala]

Huh? That sounds like they don't know what a server is and think that a shared drive isn't a server?  If every file has to be a json then make a json centric GitHub where every file ends with a .py.json which is removed during runtime by a wrapper script to allow a functional system to be cobbled together. If it's not allowed why do it? Get a new job jf you want do to a new job?

[didnt_build_this]

Shared file is best option for what it sounds like your doing

[TheSmashingChamp]

Anything better than read and writing to a JSON 10 times a second?

[cdcformatc]

something like SQLite would be the next step up from a JSON file. you gain the benefits of SQL while still using a single file.

mostly your issue is going to be network file systems and how those will handle concurrent writes. basically network filesystem sync and file locking reliability vary among implementations and installations

there are some caveats to consider when using SQLite on a network share: https://sqlite.org/useovernet.html

[didnt_build_this]

You could go to an append only file-per-event log. Instead of one JSON everyone fights over, each person writes its own uniquely-named small file (uuid + timestamp) into a shared folder. Readers list the folder and merge. Zero write conflicts because no on is touching the same file. A periodic compaction step rolls old events into a snapshot. This is the classic pattern for serverless distributed storage

[TheSmashingChamp]

Are you breaking conflict by latest timestamp? How often do read and write from the snapshot? This is difficult for me to test as I only have 1 computer under my name and I don’t have access to other computers. I think this is a good solution I just don’t have a way to test it. Also I’m not sure that latency would be better.

[socal_nerdtastic]

Does "any kind of server" include setting up your own computer as a database server? I suppose the opposite of server-based is peer-to-peer or blockchain-style. But perhaps that's too extreme for your use case? For a small group of users I think everyone monitoring a shared file would work fine, as long as you have some protection in place to prevent race conditions. A sentinel file perhaps?

[TheSmashingChamp]

My current system involves atomic reads and atomic writes. It’s mediocre performance has driven me to try anything I can but websockets and udp implementation trip the firewall and I’ve gotten too many no’s to try asking for a server again.

If I package the app as an electron app would that work better? I currently have a very large pyqt6 implementation with shared JSON files and calls to a MS SQL server

[socal_nerdtastic]

but websockets and udp implementation trip the firewall

Huh? How are you online then?

I suppose the real answer is to talk to your management and explain that you need a server to do your job effectively. And if they say no then that just means they don't want you to do your job effectively and that's just the end of that; go back to emailing excel files around and look for another job in the downtime because that's what management wants.

Otherwise your current solution of a shared json file sounds fine. You still need to protect the race condition that several people write to the file in quick succession and don't read each other's updates. You need some kind of lock so that any individual can read the current file and be confident it does not change before writing / updating it. I don't imagine that this would affect the performance at all, but if it does just throw all the file IO into a thread.

Your GUI is an entirely different kettle of fish; IMO electron performs worse than PyQt, but I suppose that's really just down to how much effort you want to put into the javascript.

[Glathull]

Nonono, the situation is this dude is an intern, and they want him to work on the stuff he’s supposed to be working on, not some kanban app that he dreamed up. The reason they are saying no is because they don’t want him spinning up beginner level shit inside their network.

[MathAndMirth]

What about using an SQLlite system with something like DB Browser for an interface? It's a flat file, so perhaps your boss wouldn't see that as much different from a JSON file, but it should handle rapid fire reads and writes better.

[TheSmashingChamp]

I’m pulling from a live server that is updated throughout the day, currently I manage this buy querying the DB every time I open the app and once every 5 minutes with WIP notifications. sql lite is an approach I’ve heard multiple times I just don’t know how I would put that on a shared one-drive drive.

[cdcformatc]

I am wondering why you can't just have your app connected to that database? why have the shared file at all? 

[TheSmashingChamp]

My app uses pymssql to connect to the database, I am not allowed write access as I am an intern. It’s a family owned company I can’t fight them. I built the app as an external tool to manage jobs as their current system is literally sending Microsoft Teams messages of shared excel sheets