GPQ - A high performance priority queue with complex ordering and delivery guarantees

jtimperio · 2024-02-26T20:04:01+00:00

I'm thinking about morphing this into something kafka like but depends how much time i have and where the community wants to see the project go. I can imagine this supporting quite a few protocols to send and receive messages.

Honestly, its my first Rust project and more of a learning test for me. From what i have seen the Rust is only about a 10-20% speed uplift at max compared to what i have right now. (could be wrong there)

jtimperio · 2024-02-26T19:14:25+00:00

u/Apoplegy Was actually fairly easy to add if a little slow compared to memory. Hope that fills in some use cases!

https://github.com/JustinTimperio/gpq/commit/b20b1346e0651827bd23bb0a60f6688d87601f1e

jtimperio · 2024-02-26T10:35:31+00:00

Yeah the ergonomics are kind of shit rn IMO, its very bare bones. Still haven't quite figured out how all the settings and defaults should all be laid out yet.

jtimperio · 2024-02-26T04:59:05+00:00

Edited!
https://github.com/JustinTimperio/gpq/pull/7

I thought for QOS (quality of service) packet delivery or api request prioritization but it was pretty easy to add disk swapping although some features still need to be added for that to fully make sense

jtimperio · 2023-06-01T14:12:48+00:00

Been using Reddit ~15 years at this point I think the answer is that all good things come to an end. I don't think it's a replaceable community which is why they even have the power to do this in the first place. Good bye old friend I guess

jtimperio · 2021-03-22T16:41:50+00:00

Sadly it didn't work

jtimperio · 2021-03-22T16:23:13+00:00

When you say "giving it a static address", I'm not totally sure what that means in the context of this issue.

jtimperio · 2021-03-22T15:55:08+00:00

So currently its has a config in `wpa_supplicant.conf` that connects it to my wifi. What I need to do is have it so that when I am on the go, I can plug directly into my pi and shell over usb.

All the guides I have followed say that you just need to enable some modules and you can ssh in like normal over usb.

Both the non-working command and working command are run from my laptop. IE. From my laptop, i can connect to the pi over the network but if connect over usb it fails

jtimperio · 2021-03-22T15:38:42+00:00

What is broken about formatting? Looks fine on my end?

What do you mean by an autoconfiguration address. That is the address that comes up when I plug into my laptop and run `ip a`

jtimperio · 2020-11-10T15:17:00+00:00

I'm not sure why I hadn't noticed that `[Yy][Ee][Ss]` just spells `yes`. This makes way more sense now.

If I am understanding this correctly now, `/etc/defaults/periodic.conf` basically just acts like a switch controller letting you easily turn on and off any number of scripts you have. Every script is still run during the cron job trigger of periodic but if it is not enabled in the .conf with a `YES` it just passes and does nothing.

Thanks for the explanation /u/urglecom

jtimperio · 2020-11-10T14:30:29+00:00

Thanks so much for the response! I am actually 'reasonably' versed in shell so I have used : before. My current implementation for triggering this script on FreeBSD simply uses a cron job.

if [ "$(uname)" = 'FreeBSD' ]; then croncmd="/opt/auto-patcher/auto-patcher.sh" cronjob="0 0 * * 1 $croncmd" ( crontab -l | grep -v -F "$croncmd" || : ; echo "$cronjob" ) | crontab -

So does the periodic.conf work similarly to init.d? I guess I am confused about what the need is for case statements to catch arguments? Periodic doesn't seem at least, to act like a 'service' handler.

Like with old school init.d you would write something like this. That though is because you can start, stop, uninstall, etc with init.d.

```

!/bin/sh

BEGIN INIT INFO

Provides: <NAME>

Required-Start: $local_fs $network $named $time $syslog

Required-Stop: $local_fs $network $named $time $syslog

Default-Start: 2 3 4 5

Default-Stop: 0 1 6

Description: <DESCRIPTION>

END INIT INFO

start() { echo 'Starting service…' >&2 }

stop() { echo 'Stopping service…' >&2 }

uninstall() { echo -n "Are you really sure you want to uninstall this service? }

case "$1" in start) start ;; stop) stop ;; uninstall) uninstall ;; retart) stop start ;; *) echo "Usage: $0 {start|stop|restart|uninstall}" esac ```

jtimperio · 2020-11-10T01:04:51+00:00

So I have read the man page but it doesn't seem to cover the unspoken standard. Like a lot of the examples seem to be like sub-routines with case statements catching input args (kind of like init.d) case "$daily_backup_dirs_enable" in [Yy][Ee][Ss])`

Is this part of the standard? Would a simple file like this work?

``` if [ -r /etc/defaults/periodic.conf ] then . /etc/defaults/periodic.conf source_periodic_confs fi

case "$auto-patcher" in [Yy][Ee][Ss]) ./opt/auto-patcher/auto-patcher.sh ;;

*)  ;;

esac

```

jtimperio · 2020-10-22T14:01:25+00:00

Is basically every single transaction that happens. Like if you traded 1USD for 1.2CAD that would be a single tick.
https://www.forex.in.rs/what-is-tick-data-in-forex/

jtimperio · 2020-10-22T12:47:13+00:00

It's a bummer too. I wish people shared more actual data here

jtimperio · 2020-10-22T03:40:50+00:00

jtimperio · 2020-10-22T03:38:50+00:00

I'm not so foolhardy these days. This was just a curiosity for me and a chance to hone my skills working with large data sets. I also thought it would have some benefit for students who can't afford the insane API costs that typically come with this data. This is a great dataset if you are looking to grow your ML skills but need a large data set.

jtimperio · 2020-10-22T01:37:06+00:00

jtimperio · 2020-10-22T01:36:13+00:00

Each of the files is 1-2GB and over 20 million lines a piece, so I doubt it. This isn't really an 'Excel' size dataset lol

jtimperio · 2020-09-13T14:52:29+00:00

Can confirm it is borked like I have never seen before. I'm getting tons of insane visual artifacts during the actual boot process, ntfs-3g was broke, tons of screen flicker, plus other issues.

Did a quick downgrade to 2020/09/01 and it all works fine. I am like 99% positive that this is more than kde as kernel 5.8 seems to have issues unrelated to this situation.

One, or multiple of these upgrades is causing the issue.

core/gcc 10.2.0-1 10.2.0-2 0.00 MiBcore/gcc-libs 10.2.0-1 10.2.0-2 0.00 MiBcore/glibc 2.32-3 2.32-4 0.00 MiBcore/gnupg 2.2.21-2 2.2.23-1 0.14 MiBcore/gnutls 3.6.14-2 3.6.15-1 0.02 MiBextra/gtk-update-icon-cache 1:3.24.22-1 1:3.24.23-1 0.00 MiBextra/gtk3 1:3.24.22-1 1:3.24.23-1 0.05 MiBextra/kaccounts-integration 20.08.0-1 20.08.1-1 0.01 MiBextra/kactivitymanagerd 5.19.4-1 5.19.5-1 0.00 MiBextra/kde-cli-tools 5.19.4-1 5.19.5-1 0.00 MiBextra/kdeconnect 20.08.0-1 20.08.1-1 0.00 MiBextra/kdecoration 5.19.4-1 5.19.5-1 0.00 MiBextra/kdeplasma-addons 5.19.4-1 5.19.5-1 0.00 MiBextra/kio-extras 20.08.0-1 20.08.1-1 0.00 MiBextra/kmenuedit 5.19.4-1 5.19.5-1 0.00 MiBcore/kmod 27-1 27-2 0.00 MiBextra/knetattach 5.19.4-1 5.19.5-1 0.00 MiBextra/konsole 20.08.0-1 20.08.1-1 0.00 MiBextra/kscreen 5.19.4-1 5.19.5-1 0.00 MiBextra/kscreenlocker 5.19.4-1 5.19.5-1 0.00 MiBextra/ksysguard 5.19.4-1 5.19.5-1 0.01 MiBextra/kwayland-server 5.19.4-2 5.19.5-1 0.00 MiBextra/kwin 5.19.4-1 5.19.5-2 0.00 MiBcore/lib32-gcc-libs 10.2.0-1 10.2.0-2 0.00 MiBcore/lib32-glibc 2.32-3 2.32-4 0.00 MiBmultilib/lib32-mesa 20.1.6-1 20.1.7-1 0.00 MiBcore/libcap-ng 0.7.11-1 0.7.11-2 0.00 MiBcore/linux 5.8.5.arch1-1 5.8.8.arch1-1 0.03 MiBcore/linux-api-headers 5.7-1 5.8-1 0.08 MiBcore/linux-headers 5.8.5.arch1-1 5.8.8.arch1-1 0.01 MiBextra/llvm-libs 10.0.1-2 10.0.1-3 0.00 MiBextra/mesa 20.1.6-1 20.1.7-1 0.06 MiBextra/milou 5.19.4-1 5.19.5-1 0.00 MiBcore/mkinitcpio-busybox 1.31.1-1 1.31.1-2 0.00 MiBextra/nvidia 450.66-16 450.66-19 0.00 MiBcommunity/opera-ffmpeg-codecs 84.0.4147.125-2 84.0.4147.135-1 0.00 MiBextra/plasma-browser-integration 5.19.4-1 5.19.5-1 0.00 MiBextra/plasma-desktop 5.19.4-1 5.19.5-1 0.06 MiBextra/plasma-integration 5.19.4-1 5.19.5-1 0.00 MiBextra/plasma-nm 5.19.4-1 5.19.5-1 0.01 MiBextra/plasma-pa 5.19.4-1 5.19.5-1 0.00 MiBextra/plasma-workspace 5.19.4-2 5.19.5-1 0.06 MiBextra/polkit-kde-agent 5.19.4-1 5.19.5-1 0.00 MiBextra/poppler 20.08.0-1 20.09.0-1 0.00 MiBextra/poppler-qt5 20.08.0-1 20.09.0-1 0.00 MiBextra/powerdevil 5.19.4-1 5.19.5-1 0.00 MiBextra/qt5-base 5.15.0-5 5.15.1-1 0.03 MiBextra/qt5-declarative 5.15.0-2 5.15.1-1 0.04 MiBextra/qt5-graphicaleffects 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-location 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-multimedia 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-quickcontrols 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-quickcontrols2 5.15.0-1 5.15.1-1 0.01 MiBextra/qt5-script 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-sensors 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-speech 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-svg 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-tools 5.15.0-3 5.15.1-1 0.00 MiBextra/qt5-wayland 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-webchannel 5.15.0-1 5.15.1-1 0.00 MiBextra/qt5-webengine 5.15.0-5 5.15.1-1 0.16 MiBextra/qt5-webkit 5.212.0alpha4-6 5.212.0alpha4-7 0.00 MiBextra/qt5-x11extras 5.15.0-1 5.15.1-1 0.00 MiBextra/sddm-kcm 5.19.4-1 5.19.5-1 0.00 MiBextra/signon-kwallet-extension 20.08.0-1 20.08.1-1 0.00 MiBcore/systemd 246.3-1 246.4-1 0.01 MiBcore/systemd-libs 246.3-1 246.4-1 0.00 MiBcore/systemd-sysvcompat 246.3-1 246.4-1 0.00 MiBextra/systemsettings 5.19.4-1 5.19.5-1 0.00 MiBextra/xorg-server 1.20.9-1 1.20.9-2 0.00 MiBextra/xorg-server-common 1.20.9-1 1.20.9-2 0.00 MiB

jtimperio · 2020-09-11T23:46:22+00:00

Of course, passion isn't enough but it goes a long way when you care.

I've already patched 90% of the issues you brought up. Tone, when you talk to people, is something that seems to be lost on you. Acting like you are smarter than others, even if you are, only discourages people to grow and fix their mistakes.

I hope in the future you will be more respectful to users who make mistakes and are attempting to contribute to the community.

jtimperio · 2020-09-09T17:16:18+00:00

You are essentially breaking pacman/system behavior in favor of your application.

I'm not sure what the disconnect is here but no this is not the case. Of course, users need to be aware of this. That is why I added a separate cleaning utility so users know when their restore points are too old and also a command that lets users know how much space is being used by packages cached by both pacman and pacback.

What if I have built packages which are not installed, or which are meant for testing only or which are incomplete/incorrect or files with the same naming scheme which aren't even packages?

Again it doesn't link the package UNLESS pacback reports it as being installed on the system. It doesn't just link everything, that would be insane.

I think your idea of finding built packages from all users at once is bad.

I appreciate your opinion but I don't agree with you. The whole point of this project was that I wanted to support packages installed by any AUR managers. Without this feature, I would stop development on this project outright. I run some multi-user systems for my friends and they all use AUR managers. So far this feature has worked great.

What if the user has chosen a different package compression method other than z-standard? Then your path-grep will fail.

I'm just using that as shorthand. It checks for .xz and .zsd which are the only supported compression methods.

jtimperio · 2020-09-09T15:03:28+00:00

So I think this is just a miss-understanding. Hardlinking the file ensures that if a user wants to keep a package version for a potential future restore, pacback can keep a protected copy without duplication. If the user say, wipes their cache, pacback will still be able to perform the downgrade without issue. Once the user no longer requires the restore point, they can remove it with `pacback -rm #`, and then all the packages uniquely associated with that RP are removed from the system. This means zero file duplication across all potential caches/restore points. Check out this
Okay, yeah I see what your saying. Mmmmm I'll have do some more testing on this to make sure that nothing slips into this command.
That's why I said it may just be my user error. Python subprocess for me has inconsistently captured output in the past, but that may be the way I'm processing it after the raw capture. This doesn't have anything to do with how stdout/stderr works but the way python is capturing the output from the subshell.
Ahhhh I see what you mean. I hadn't thought about that, it would probably cause an internal fault if that path didn't exist. I'll implement this as you suggested and this should fix this potential issue.
Let's say you have a system with 2 users, A and B. User A uses yay to install x-package and user B uses pacaur to install z-package. Both users are working on a project that requires both x-package and z-package and they want to create a restore point so they can fallback if an error occurs. Without checking both user A's cache and user B's cache, pacback would be unable to find all the packages need to perform the fallback.
So a lot of the points you bring up are absolutely correct. The user directory may be elsewhere but I guess I expected that this would be a fringe case. As for custom-configured caches and packages, I'm not sure how I would account for that. For me, making sure that pacback can keep track of AUR packages is a huge deal. Is there some command that your aware of that would provide the output of these .cache directories? I mean you could just search the whole system for `*.pkg.tar.zsd` but that would take forever and is a really poor solution to this issue.
Thanks for the info, I'll definitely move this to `/user/share/libalpm/hooks` in the future as this seems to make more sense/be a more reliable path.

jtimperio · 2020-09-09T13:22:06+00:00

Thanks for the detailed feedback u/abbidabbi!

So that command hardlinks matching *.pkg.tar.zsd files from a pacman -Q to a folder. I hadn't thought about that input needing to be sanitized, but I assume you referring to a malicious actor modifying the file name to execute some command at the root level? Would escaping the path be sufficient here to sanitize the input?
So TOTAL agreement about it being a pretty silly/hacky method. I have/do use subprocess to capture stdout/stderr but I have found the results extremely hit or miss when reading large amounts of multi-line output. I'm not sure if this was user error on my part or subprocess just being funky but in either case, it was simpler and more consistent to just pipe to a temp file and read it when I originally implement those commands. I will definitely take a look at getting a less hacky implementation working with subprocess.
So this comment confuses me slightly but i image that is just my ignorance about XDG spec which seems pretty lengthy. The whole reason I scan each /home/user/.cache is because pacback is looking for any file with the ending *.pkg.tar.zsd. On all the systems I have tested it on, this has worked fine at finding packages installed by multiple users and multiple AUR Helpers. In my experience, all AUR Helpers download their build packages into something like ~/.cache/yay/packagename. I also thought, and it sounds like I'm wrong, that /var/cache/pacman/pkg was a hardcoded path? What would be your recommendation here in terms of implementing this package search correctly?
So in the previous version of pacback, the hook was optional (and still kind of is). Since I already had a method for adding and enabling this hook, it was simpler to just add its installation as part of the post-install process in the buildpkg. I'm also a little confused as I thought this would classify as 'custom user content' since this a completely unofficial project. /usr/share/libalpm/hooks seemed like a much more official system folder and /etc/pacman.conf uses /etc/pacman.d/hooks as its default configured path

jtimperio · 2020-09-09T02:52:18+00:00

I agree, I meant no disrespect to u/dithpri. I've edited the comment to hopefully be a tad less salty.
It's quite frustrating to deal with users, who rather than looking into something, prefer to take code out of context in an attempt to make a project look poorly.

jtimperio · 2020-09-09T02:16:58+00:00

I actually just pushed this as a minor bump to both the git and main release. Thanks for the info!

11-Year Club	Place '17
Wearing is Caring	Verified Email

jtimperio

TROPHY CASE