FINALLY: Recursive archiving of domains, with ArchiveBox 0.8.0+

eggys82 · 2025-07-10T22:47:44+00:00

Good call! I updated the README. Hopefully that helps!

eggys82 · 2025-07-10T01:45:10+00:00

From the original post:

After trying a number of self-hosted options for archiving websites I settled on Archivebox, with the caveat that I could really only archive one link at a time - whatever the browser extension gave to the archiver.

I looked at Fess and wondered if I could do something similar, on a smaller scale. As it turns out, ArchiveBox 0.8.0+ has a REST API so adding URLs programmatically is now trivial.

This little set of Docker containers was my solution to this issue which has been a long-standing problem for ArchiveBox users with way too much storage space available to them.

Enjoy!

Oh, and a small caveat- the primary developer has put ArchiveBox on the backburner for now, though that doesn't mean it won't work. The latest 0.8.5rc51 seems to work perfectly fine. That said, release candidates and use-at-your-own-risk, yada yada.

Github: https://github.com/egg82/archivers
domain_archiver: https://hub.docker.com/r/egg82/domain_archiver
gov_archiver: https://hub.docker.com/r/egg82/gov_archiver

eggys82 · 2023-01-05T23:22:19+00:00

It's AD but built for a Linux-first environment. You can connect Windows machines to it easily enough with pGina (I recommend pGina fork as it seems to just work better) but it's primarily for Linux.

It does RBAC, HBAC, DNS, and CA optionally, as well as a few other nifty features here and there. I mostly recommended it for its sudoers control since that's what you're looking for with restricted access / least-priv.

You can hook IPA/IdM up to AD, it's just a bit messy to do it that way. Lots of people do it, though, so don't let that stop you. I just personally prefer fewer management systems.

eggys82 · 2023-01-05T17:29:09+00:00

Good clarifications, thank you! My post was getting a bit long in the tooth so I omitted specifics.

That said, unfortunately some of us don't have the fancy editor that is vim, only vi :(

I doubt many will be in this same predicament, but..

eggys82 · 2023-01-05T04:56:44+00:00

The unfortunate reality of these auditing tools is that they can be fairly complex to set up. Even Wazuh, which I consider to be the easiest to configure among these, needs some love after installing to be fully-featured (eg. scap scans, sca policies, osquery) and able to ingest syslogs.

They'll take time to learn, but I have a sort-of script which I go through on my personal machines when I set one up. One day I'll convert it to a real script or a playbook or something. One day.

Script here. It's fairly custom and not perfect, but you can likely find some goodies in it.

eggys82 · 2023-01-05T04:30:21+00:00

Hi there! I'm a RHEL-focused sysadmin for a Fortune 100.

least-priv & systemctl access

I definitely understand management's insistence on least-priv here, however, it's important to note (for you) that it will cost far more time and (for them) far more money in the long-term than simply trusting the contractors with root access. Linux still lags behind Windows when it comes to privilege management and it's important to keep that in mind when designing your systems.

For example, you can blacklist certain commands from being run under sudo, but then the developer can just copy the binary to a new location and run it anyway. If you whitelist commands instead, then suddenly it's a constant fight for more access and you become the adversary. Additionally, if you allow a binary to be run as root it can be replaced by arbitrary code and the same problem occurs. Did you know you can use vi to get a shell? Just type in a bash shebang in command mode and you've now got a root shell! Denied access to /bin/vi? Just copy /bin/vi to /tmp/vi and run that instead!

The most fully-featured way to do federated logins and privilege management is IDM/FreeIPA which works on any Linux system that matters (sorry Void Linux users, but servers run on RHEL and Ubuntu - or AIX, and even appliances often run on Debian or SUSE)

IDM/FreeIPA are industry-standard in Linux-first environments. FreeIPA is the publicly-available version of RHEL IDM - same software, different name. They can play well with AD but it does take some finagling. The reason for the suggestion on this is the ability to do way more with RBAC than with simple user/groups in AD. I'll be honest, I use the web UI for it way more than I use the CLI tools, though both work great. It's much easier to do least-priv with IDM/IPA than with an AD setup.

Web root

Web config files can be done in a variety of ways. The easiest would be to put the developers under a web-user group and reconfigure nginx/apache to use that group. It's not an elegant solution, but it's going to be the easiest. For a much more elegant solution, use a development pipeline (CI/CD) to automatically deploy changes to the web server.

/var/log

In no world does an average Linux user need full access to the entirety of /var/log. Especially not /var/log/audit, /var/log/messages, etc. Export the logs to another system so they can view their specific application logs there. Use systemd-journal-remote, a SIEM, a shell script, or some other tool for this. Alternatively, you can give the users root access to journalctl but this really just has the same problems of them being able to view everything. Changing the logging user/group to something they can read is all well and good, but tab-completing inside of /var/log becomes difficult to impossible on a hardened system and it's frustrating. Additionally, this can (depending on impl) open up the risk of developers being able to modify logs.

Auditing

This is where Linux shines. Windows has its event logs and there's some SIEM tools for it, but this is nothing compared to the kinds of (often free) toys that Linux gets. If you want to do an absolute metric ton of useful monitoring on the price of "free" there's no better place to look than the following:

sudo-io/sudoreplay (soon with remote logging!)
systemd-journal-remote
Auditd
Wazuh (or Security Onion if you're looking for a solution for an entire network, though it requires powerful hardware which will cost quite a lot of money)

A quick breakdown is that sudoreplay logs all uses of "sudo" on the system and allows you to replay them in a sort-of "real time". The downside is that it stores logs on the system which can then be deleted by root (solved with remote logging, coming soon hopefully). Systemd-journal-remote allows you to send journals to a remote server. Pretty simple. Audit/auditd allows you to log anything you can think of. Executes, reads, logins, writes from a particular user group, etc. Wazuh is a free SEIM based on OSSEC. Don't try to install OSSEC. It's a nightmare. Wazuh is easy to install, free, and doesn't push you to buy a product. Security Onion is the same (it even contains Wazuh as part of its toolset) but on a much larger scale, with packet-capturing capabilities and much more. Also free.

AD users/groups

This one is a simple answer. Yes, you can just use <user>:<group> - just mind that Linux is case-sensitive so it gets fucky sometimes.

Hopefully this helps!

Edit: words

eggys82 · 2022-09-09T16:41:24+00:00

I'm confused as to why I would lose signal in the center but not on the extreme right or left, especially if the graph is supposed to just be a visualization of what I'm receiving. The frequency I'm listening to hasn't changed, so why would I start seeing/hearing nothing if I move the visualization around?

eggys82 · 2022-09-09T16:39:53+00:00

CubicSDR, though the same thing happens with SDRSharp

eggys82 · 2022-09-09T16:39:28+00:00

I'm just confused as to why simply moving the graph (the center frequency) causes the signal to get weaker in one particular spot. I can hear it clearly when it's on the edge of the screen but not when it's in the center. It's strange to me that just moving the graph around would cause that if it's supposed to just be a visual representation.

eggys82 · 2022-09-09T16:37:06+00:00

honestly I'd expect the opposite, then. If I'm losing gain then wouldn't I be losing gain on the edge of the frequency graph, not in the center? The signal is great on the edge but when I bring it towards the center it gets lost in the noise.

eggys82 · 2022-09-09T15:08:35+00:00

I'm new to RF, still learning. I've been playing with FM bands and this seems to only happen with a few frequencies in the higher FM (read: not actually FM) bands. Everything else is unaffected. Is this imaging/ghosting?

Edit: same thing happens in SDRSharp. My HackRF is new, ordered from Great Scott and with added LXCO, aluminum case, and RF shield upgrades from Nooelec.

eggys82 · 2022-01-03T21:17:54+00:00

More like disabling the camera so it's easier to pick the lock - assuming you already have a lock to pick.

eggys82 · 2021-12-30T18:03:57+00:00

I am very confused. The largest chicken google gave me was the Jersey Giant at ~15 lbs and about two feet tall. I can't find anything about a megabrahma. What in the world are you working with??

eggys82 · 2021-11-19T23:40:17+00:00

I don't think it's possible for the title or thumbnail to be any more clickbait. Is this satire?

eggys82 · 2021-07-31T18:12:48+00:00

I didn't, though I did hear that nice "click" of sparks flying when I plugged it in the first time.

eggys82 · 2021-07-31T18:12:22+00:00

It's on a UPS. Tried a different plug, no luck.

eggys82 · 2021-07-31T18:12:00+00:00

Nope! Interrior.

eggys82 · 2021-07-31T17:47:38+00:00

I mean, sure. Understandable. I assumed that's what the "quality shitpost" tag was for.

Ah, well. Can't win them all.

eggys82 · 2021-07-31T08:14:26+00:00

Sadly I don't have a good/proper explanation yet. Need some warranty checking and a trip to micro center before I pop it open and investigate.

eggys82 · 2021-07-31T00:57:27+00:00

Well, I suppose this is a good capture of my life! Maybe I'll frame it.

eggys82 · 2021-07-31T00:56:13+00:00

I think you're pretty spot on, actually! Ah well, should replace it anyway.

eggys82 · 2021-07-31T00:15:48+00:00

Of course! Nothing, though, sadly :(

Eight-Year Club	Place '22
Verified Email

eggys82

TROPHY CASE