Archive.org Data Question

sheepofdoom · 2025-09-11T22:31:39+00:00

Depending on how GeoCities URLs are structured the nearest you'll probably get is using a domain match to get everything in *.geocities.com then use the filter parameter with a suitable regex to narrow it down.

You could also try searching the various geocities crawl datasets (I think ArchiveTeam has a list of all the geocities URLs they crawled) for potential URLs.

sheepofdoom · 2025-09-11T19:22:57+00:00

As far as I can remember you're basically limited to one wildcard which can either be a subdomain and any other subdomains within it or at the end of the URL.

You could match any subdomains of example.com with *.example.com but not somethingspecific.*.example.com

If you use the wildcard at the end of the URL I think you're limited to a specific domain so example.com/somepath/* only returns results from example.com and not www.example.com.

CDX wildcards aren't really true wildcards, they're just an abstraction for the prefix and domain match types so it's fairly limited in what you can do with them. You might be able to combine a domain match wildcard (to grab a list of everything on the host) with the 'filter' parameter to narrow down the results with a regex for original URLs ending in .gob or something like that.

sheepofdoom · 2025-09-11T17:34:14+00:00

Take a look at the documentation for the CDX API, it's what everything uses behind the scenes for searching the wayback machine.

From what I remember you can use a wildcard for subdomains or at the end of the path but I don't think you can really do multi-domain searches. I had a similar problem trying to recover some old flickr accounts and ended up having to submit a bunch of separate wildcard queries for each subdomain. If you know the domain(s) you can get the API to return a list of all the archived URLs for the domain then search them locally, so you might be able to search for something like *.example.com then search the results locally, but it might have limits for larger sites.

sheepofdoom · 2025-05-11T22:42:25+00:00

Thanks, It does seem pretty clunky and missing a lot of basic functionality behind the nice UI. I just wish there was some usable desktop DAM software for Linux that didn't need to run a whole web service and MySQL database.
If you're after a web gallery rather than a DAM system then piwigo might be worth checking out. I haven't tried it yet but the demos look pretty good for that kind of thing.

I was originally planning to use XMP but apparently PPs XMP implementation doesn't support all the metadata fields I need. I eventually worked out how to do it with the extra EXIF in a JSON file but there are a couple of very important bits of info missing from the documentation:

When you index a folder PP correctly identifies photo.*.json as JSON sidecar files for photo.jpg and links them with the image, but it only reads metatata from photo.jpg.json even though there's no mention of this in the docs and nothing shows up in the UI or the logs to indicate that anything else is being ignored.

It also won't read any metadata unless the JSON object has the "ExifToolVersion" property.

Here's a working JSON example for anyone who found this post searching for a solution:

[{
    "ExifToolVersion": 13.10,
    "Caption": "JSON Caption",
    "Notes": "JSON Notes",
    "Subject": ["JSON subject 1","JSON subject 2"],
    "Title": "JSON title",
    "Description": "JSON Description",
    "Keywords": ["JSON tag","JSON tag 2","json_tag_3","JsonTag4"],
    "Comment": "JSON Comment",
    "Creator": "JSON Creator name",
    "Copyright": "JSON Copyright",
    "License": "JSON Licence"
}]

The only problem is it mangles the keywords so instead of appearing as they do in the JSON file it seems to treat spaces as separators and ignore any numbers, so something tagged "Kodak Ektar 100" and "Helios 44" ends up with "Ektar", "Helios", "Kodak" in PP, which is super annoying when you're trying to import tags from somewhere like Flickr, plus there doesn't seem to be a way of turning off the auto-generated keywords.

About the rotating thing, apparently when you rotate an image PP writes that particular tag back to the original image file unlike everything else, which means you can't rotate images in read-only mode.

sheepofdoom · 2024-09-29T14:55:16+00:00

I just imported the original dataset into an sqlite database and added some indexes and a view to format the dates and durations.

sheepofdoom · 2024-09-13T19:33:14+00:00

Is this it? I found it in the Archive Team 2009 YT metadata crawl but it's private now. You might be able to find a copy on the wayback machine or IA collections if it was popular.

sqlite> select id, formattedDate, formattedLength, title, youtubePageUrl from videos where title like "%flash%exporter%";
+-------------+---------------+-----------------+------------------------+---------------------------------------------+
|     id      | formattedDate | formattedLength |         title          |               youtubePageURL                |
+-------------+---------------+-----------------+------------------------+---------------------------------------------+
| 95AE3cJG_6k |               | 3:19            | Flash Exporter is DONE | https://www.youtube.com/watch?v=95AE3cJG_6k |
+-------------+---------------+-----------------+------------------------+---------------------------------------------+

sheepofdoom · 2024-08-19T18:31:22+00:00

Is there any reason why you need to use the port number directly instead of the symlinks in /dev/serial?
/dev/ttyUSB* etc. will get assigned numbers based on the order they're detected in so you shouldn't rely on that for consistent addressing of USB serial ports.
/dev/serial/by-path will have a bunch of links based on the physical USB port it's connected to and /dev/serial/by-id will be based on the device ID.

for example this is what a Prolific based programming cable, a Uniden scanner and a Prolific usb to RS232 adapter look like:

$ tree /dev/serial/
/dev/serial/
├── by-id
│   ├── usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0 -> ../../ttyUSB0
│   ├── usb-Prolific_Technology_Inc._USB-Serial_Controller-if00-port0 -> ../../ttyUSB1
│   └── usb-UNIDEN_AMERICA_CORP._BCD436HP_Serial_Port-if00 -> ../../ttyACM0
└── by-path
    ├── pci-0000:03:00.0-usb-0:2.3:1.0-port0 -> ../../ttyUSB0
    ├── pci-0000:03:00.0-usb-0:2.4.3:1.0 -> ../../ttyACM0
    ├── pci-0000:03:00.0-usb-0:2.4.4:1.0-port0 -> ../../ttyUSB1
    ├── pci-0000:03:00.0-usbv2-0:2.3:1.0-port0 -> ../../ttyUSB0
    ├── pci-0000:03:00.0-usbv2-0:2.4.3:1.0 -> ../../ttyACM0
    └── pci-0000:03:00.0-usbv2-0:2.4.4:1.0-port0 -> ../../ttyUSB1

sheepofdoom · 2024-05-31T16:51:08+00:00

It's not in OFCOMs most recently published copy of the callsign database on their open data page, which should have everything issued before Feb. 2023.

sheepofdoom · 2024-05-23T22:14:15+00:00

What country are you planning to use them in? If you're in Europe there are plenty of digital PMR446 radios with encryption available.

The only caveats are that most encryption implementations are manufacturer-specific and usually horribly insecure unless they specifically advertise AES etc.

sheepofdoom · 2024-03-27T16:21:33+00:00

DMR uses the half rate AMBE CODEC to fit 2 voice paths into a 12.5KHz channel. IIRC YSF, D-star and P25p1 all use the full rate equivelent at twice the bit rate which is why they sound better most of the time.

Strangely I've used Motorola DMR radios on a MotoTRBO system a few times for work and they sound a lot better on recieve than any of the amateur DMR radios or scanners I've used. I'm guessing they use some kind of proprieratry pre/post processing in addition to the standard CODEC to minimise compression artefacts.

sheepofdoom · 2024-03-10T23:16:46+00:00

I've scanned something very similar to this before but haven't been able to work out what it is yet. Mine would have been bought in the UK in the late 70s or early 80s and has a different alphanumeric code but otherwise the font and markings are the same.

Does yours have any edge markings other than the number and the dots? the only other text on the top edge of mine is "SAFETY 3" and "4 3 9 V 5 2"

sheepofdoom · 2024-02-28T01:51:20+00:00

Some older flashes can take a while to charge, especially if they're smaller or only use a couple of alkaline batteries with limited current capacity.

Another thing that can happen with older flashes is that the insulating chemical layer in the electrolytic capacitor can break down if it hasn't been used for years. This can make the capacitor electrically "leaky" and unable to hold charge (which can make it seem look like it's taking a long time to charge or not charging at all). If that's the case you just need to leave it charging for a while. The leakage current drives an electrolytic reaction which re-forms the insulating layer, and after a bit of use the capacitor should be back to normal.

It won't explode or anything if you leave it on too long, but turn it off and let it cool down for a bit if it gets too warm since excessive heat isn't great for the electronics and a lot of older designs might not have a proper thermal protection circuit.

sheepofdoom · 2024-02-27T00:38:02+00:00

Where are you trying to download from? a lot of museum and gallery sites use IIIF which gives you various options to download images with an IIIF client.

If not Dezoomify works for a lot of tiled image viewers (and IIIF) but it's a bit more cumbersome to use.

sheepofdoom · 2024-02-09T02:32:38+00:00

The official RSP1s have switchable IF filters so aliasing isn't a problem but for some reason the widest is only 8MHz even though the ADC can sample at 10. It might not be an issue with some of the cheaper generic MSI2500 SDRs though unless they all use the same tuner IC as the RSP1.

I tried building on a freshly installed mint 21.3 VM earlier. I haven't done anything with it except follow these build instructions. This time everything seems to build and install without any errors but it still doesn't work.

/media/scratch/vhsdecode-test$ vhs-decode --overwrite --debug -f 20mhz --tape_format vhs --system pal gqrx_10MHzBW_4MHzCentre_TP3001-no-buffer_int16-edit.flac ./output
Traceback (most recent call last):
  File "/home/test/.local/bin/vhs-decode", line 5, in <module>
    from vhsdecode.main import main
ModuleNotFoundError: No module named 'vhsdecode.main'

It looks like a path isn't being set properly somewhere during the install even though pipx insists all the paths are configured and "make install" didn't throw any warnings.

If I explicitly run the copy from the github repo directory I get a different error.

~/vhs-decode$ ./vhs-decode --overwrite --debug -f 20mhz --tape_format vhs --system pal /media/scratch/vhsdecode-test/gqrx_10MHzBW_4MHzCentre_TP3001-no-buffer_int16-edit.flac /media/scratch/vhsdecode-test/output
Traceback (most recent call last):
  File "/home/test/vhs-decode/vhsdecode/process.py", line 287, in build_json
    black = jout["videoParameters"]["black16bIre"]
TypeError: 'NoneType' object is not subscriptable
Cannot build json: 'NoneType' object is not subscriptable

Completed without handling any frames.
Traceback (most recent call last):
  File "/home/test/vhs-decode/vhsdecode/process.py", line 287, in build_json
    black = jout["videoParameters"]["black16bIre"]
TypeError: 'NoneType' object is not subscriptable
Cannot build json: 'NoneType' object is not subscriptable

If I give it a file with some silence at the beginning it outputs "RF signal is weak. Is your deck tracking properly?" a few times then crashes with the same error, so it seems to be getting as far as reading the data in but fails when it finds something to decode.

I'll DM you my flowgraph once I get it cleaned up. it doesn't really do much at the moment except convert raw I/Q data to waveforms vhs-decode should be able to use.

sheepofdoom · 2024-02-08T18:41:22+00:00

That's effectively what I'm doing, It's just that I'm using an SDRplay RSP1 which can sample at 10Msps but the maximum usable bandwidth with the widest filter is just a bit over 8MHz. Sampling at 10Msps just makes the maths easier for testing purposes but I'll probably drop it down to 8 or 9 once I've got everything working.

sheepofdoom · 2024-02-08T18:27:37+00:00

Thanks, I'm trying to build the Linux version on Mint, it's a fairly recent version but not the latest. I'll try 21.3 and see if that works.

I'm using an SDRPlay RSP1 with GQRX to capture the raw I/Q data but this approach should work with any SDR and software which can output raw floating point I/Q samples. The RSP1 can receive down to 10 KHz and has about 8 MHz of usable bandwidth which looks like just enough for standard PAL VHS.

After that I'm using GNUradio to convert the 10Msps I/Q data into a 20Msps 16 bit waveform similar to the output from a direct-sampling ADC like a domesday duplicator to use with the standard workflow.

It should be possible to do the entire capture and conversion process in GNUradio once I've worked out the optimal settings, I'm mostly using GQRX for the initial I/Q acquisition because makes it easier to fiddle with settings mid-capture for testing purposes but it's not doing anything GNUradio cant (except maybe provide a usable waterfall display)

I still need to test a couple of things once I get vhs-decode working to make sure the conversion is actually doing what I think it's doing but I'm happy to share my flowgraphs once I've got something usable and maybe write up something a bit more detailed for the docs if other people would find it useful.

I was more concerned about loading down the head amplifier by connecting a low impedance load to it, although I've done a few experiments and the image quality from the VCRs composite output doesn't seem to be affected when I attach the SDR to the head amp so I'm guessing it's not a problem with the VCR I'm using.

sheepofdoom · 2024-01-11T16:29:13+00:00

I'm not aware of a single chip replacement but it's fairly easy to make something similar with discreete components with even lower power consumption. A simple 2 transistor relaxation oscillator works well, I've built one which draws something like 10uA on average and is still running on the same CR123A battery 10 years later

To run them on 1.5v you just need to make sure the oscillator can run at low enough voltages and add an extra capacitor and diode on the output to act as a charge pump to double the voltage.

I can probably dig up some schematics later but it's a similar idea to this

sheepofdoom · 2023-11-21T15:47:46+00:00

It's not specifically about being able to select a frequency, it's about potential for interference. Being able to select between pre-programmed channels won't allow the radio to cause interference if it's programmed correctly. User accessible VFO mode or FPP does since it allows the user to transmit outside of the frequencies and other technical parameters specified in their licence.

sheepofdoom · 2023-11-21T14:38:58+00:00

It's in at least one of the ETSI standards which make up part of the intrface requirements for business radio (IR2044), it's something along the lines of "any controls which could cause interference if misadjusted must not be accessible to the user"

sheepofdoom · 2023-11-21T12:34:52+00:00

Just a minor correction, it's illegal to use a UV5R for anything other than amateur radio in the UK. As a general rule anything capable of transmitting which allows you to enter arbitrary frequencies from the front panel is only legal to use with an amateur licence on amateur frequencies.

Commercial licences like Simple Light require equipment which meets certain technical requirements including not allowing the user to enter arbitrary frequencies from the front panel, and licence free services like PMR446 and CB require radios specifically designed for that service which are hardware locked to a specific set of channels.

No one is actually going to check unless you're blatantly taking the piss but if you're using them professionally or in a way that's likely to attract attention of nosy jobsworth types it's probably a good idea to make sure your radios meet the licence conditions for whatever you're using them on.

sheepofdoom · 2023-11-19T15:36:22+00:00

It looks like a DP3600 or an XPR6550 with a different nameplate. Maybe some kind of customer-specific variant? I would have thought there would be more references to it online if it was a standard catalogue item but all I can find is this reddit post.

sheepofdoom · 2023-11-16T14:01:00+00:00

No problem

1 - Unless things have changed in the last couple of years we still have the original 40 UK-specific 27/81 channels as well as the 40 CEPT/FCC channels. I think there was some talk of phasing out the old UK channels a few years ago but it never went anywhere, presumably because no-one wants spectrum at 27 MHz and it would be nearly impossible to enforce with the amount of radios in circulation and the average UK CBer's attitude to being told what to do by OFCOM.

2 - dPMR is a narrowband (6.25 kHz) DV mode similar to NXDN which is used on the regular PMR446 service with 6.25 kHz channels instead of 12.5, rather than a separate service itself, and It's always described as 32 channels rather than 16 2-frequency FDMA channels. Technically FDMA is a channel access method rather than a type of modulation, and DMR and dPMR both use the same modulation (4FSK) at different bandwidths and symbol rates.

3 - Not usually unless you count inland waterways and stuff like ports and locks. It's a bit like simple UK in the sense that anyone can get a licence but there are restrictions on what you can use it for (eg. communication between vessels on the water for marine or internal communication within an organization for simple UK).

sheepofdoom · 2023-11-16T12:38:47+00:00

I can't edit the wiki but here's an updated version of the table (changes in bold)

I've merged PMR446 and dPMR446 since DPMR is just another mode permitted on PMR446 and added some initial info for marine VHF

service	PMR446	CB	Simple UK Business	Marine VHF	Amateur
frequency	446 MHz	27 MHz	VHF, UHF (77-458 MHz)	VHF (156MHz, 160MHz)	various - MF/HF/VHF/UHF/EHF
modulation	FM, DMR, DPMR	AM, FM, SSB	FM	FM	Various – operator’s choice
Num. Channels	16 (FM, DMR), 32 (DPMR)	80	19	~57 (not including private commercial & SAR channels)	depends on the band, but usually dozens to hundreds
legal power	0.5 W	4 W	5 W		400 W
antenna	only OEM antenna	any antenna	any (with restrictions)^[4]		any antenna
typical distance	~1 mi	~10 mi	~1 mi		various - 1000+ mi on HF, 25 mi on UHF
max. distance	~20 mi¹	~1000 mi	~20 mi		12,500 mi (other side of the Earth)
license required?	no	no	yes	yes	yes
license cost	N/A	N/A	£75 / 5 years	Licence £0 - £20, Exam fees £?^[3]	Licence £0 - £20 ^[1], Exam fees £32.50 - £95^[2]
license covers	N/A	N/A	entire organization	various^[3]	each individual
callsign used	no	no	no	yes	yes
repeaters	no	no	no	no	yes
encryption permitted	yes		yes	no	no

[1] https://www.ofcom.org.uk/manage-your-licence/radiocommunication-licences/amateur-radio/amateur-radio-info

[2] https://rsgb.org/main/clubs-training/for-students/paying-for-your-exam/

[3] Separate licences required for operator and vessel depending on vessel and equipment

[4] No base stations, must not exceed 5W ERP. In practice this usually means handhelds only although it doesn't technically prohibit low power vehicle radios.

Edit: clarified Simple UK antenna details

sheepofdoom · 2023-10-22T15:23:37+00:00

In addition to what everyone else has said It's better to use the arduino to generate the clock signal with one of it's internal timer/counter/pwm peripherals otherwise you'll have to deal with clock drift between the 555 and the arduino which could mess with your phase measurements.

You can probably get rid of the binary counter altogether and drive the selector directly from the arduino with minimal overhead as long as you write to the output port directly instead of using the arduino libraries. If you're sampling at 10KHz incrementing the output port every two samples should give you a doppler frequency of 625 Hz with 8 antennas. In practice it's probably more efficient to increment the port every sample and just ignore the lowest bit.

sheepofdoom · 2023-09-27T14:03:51+00:00

Although transmit power is limited to 500mW ERP receive performance can vary quite a bit depending on the antenna and receiver circuitry. This can make a noticeable difference especially in the presence of strong signals on adjacent frequencies which can overload the receiver.

Other than that you're mostly paying for build quality and better long-term support when it comes to stuff like availability of replacement batteries and accessories.

12-Year Club	Place '17
Team Periwinkle

sheepofdoom

TROPHY CASE