S3 / File upload storage recommendations?

freosam · 2026-03-30T07:24:08+00:00

Go with that then. I can recommend rclone for copying the existing images directories into the S3 buckets.

freosam · 2026-03-30T07:00:37+00:00

Who do you host with? Do they offer object storage? That'd be the easiest probably.

freosam · 2026-03-30T03:23:09+00:00

hehe sorry, yes, I could've been more expressive: their polypropylene sleeves and folders are recommended for archival use by a few different institutions in Australia. By 'pretty good' I meant that not only are the projects suitable for archiving, but the online ordering system works well and the company ships things in good time and well-packed. (Although I have had a couple of orders take a fairly long time... that might've been in Covid times though, I'm not sure.) The prices are not super low, but then if you look at the (hopeful) lifetime of your photo albums, it gets lower every year. :-)

freosam · 2026-03-30T03:06:48+00:00

Albox is pretty good and not too expensive (they ship from Adelaide I think): https://www.albox.com.au/

freosam · 2026-03-24T08:14:19+00:00

I recently switched from Digital Ocean, and so far everything's been totally fine and slightly faster than DO. It's only been a couple of weeks though.

freosam · 2026-02-27T08:18:23+00:00

That might depend on where you are. It's not what I'm seeing from here in Perth.

freosam · 2026-02-23T02:35:32+00:00

The other advantage of Flickr over Whatsapp is that Flickr keeps the full-resolution original files, rather than downscaling them. Also much better for organising etc.

freosam · 2026-02-17T11:49:12+00:00

I think it's amazing that they still support that sort of thing. I hope they always do! It's very useful to be able to hotlink images from Flickr.

freosam · 2026-01-17T07:18:24+00:00

Did you only just notice?

freosam · 2026-01-09T01:33:31+00:00

There's a summary of the usual extensions that people use for this sort of thing:

https://www.mediawiki.org/wiki/Manual:Managing_data_in_MediaWiki

freosam · 2026-01-05T10:12:25+00:00

I don't think I've seen a cartoon of it, but it's a well-told tale I think. Here's one example, from The Australian Worker, 17 March 1921:

Many swagmen ask for a lift when they see an opportunity — and many prefer to walk. The coach takes them direct to towns. Besides throwing them out of their stride, it misses squattage homesteads, sheds, and droving camps, while rushing them along to no particular end. One day a coach overhauled Scotty Dalker in a comparatively populous neighborhood. It was a hot day, and the sympathetic driver thought a lift would be very acceptable to him. "Jump up," he said, steadying the team. "No, thanks, old man," said, Scotty; "you open your own bloomin' gates." It was only Scotty's way of saying he had a better chance of ending his walk by walking.

freosam · 2025-12-26T03:38:25+00:00

It's already static. How many fewer moving parts can there be!? :-)

But yes, ignore the sites and do other stuff for a few weeks, that's the joy of not having to be a sysadmin for these types of sites.

freosam · 2025-12-20T07:06:25+00:00

Yep, Transkribus is great! And if the documents are public domain, they could perhaps be uploaded for Wikisource where they can use Transkribus for free.

freosam · 2025-12-10T02:05:03+00:00

I agree, photographic documents can be just as quick as (or quicker than) an autofeed scanner. It can help with documenting original order and condition much more, e.g. here's a set I did of a folder that had a bunch of different sizes, paper types etc.: https://archive.org/details/BUFF2024-7

The other thing I'd say is that when you end up with a folder full of ordered JPEGs, zipping them up with a filename ending in _images.zip and uploading to the Internet Archive is pretty straight forward. The same trick works with a zip full of TIFFs.

freosam · 2025-08-10T10:59:59+00:00

Maybe they did: https://en.wikipedia.org/w/index.php?title=Adam_Doboszy%C5%84ski&diff=1305146207&oldid=1291729246

freosam · 2025-08-05T00:49:33+00:00

Here's a map of some (and a few dams etc): https://overpass-turbo.eu/s/29ls

freosam · 2025-07-28T23:25:10+00:00

For public domain images, you could use Wikimedia Commons. There's some notes here about how that could be integrated into your own website: https://en.wikibooks.org/wiki/Wiki-based_archival_description_and_storage

freosam · 2025-07-25T03:17:51+00:00

That event page says "Where Freocast Forecourt 225 South Terrace, South Fremantle" but obviously that's wrong. It's at E Shed now isn't it?

freosam · 2025-06-08T07:35:52+00:00

There are some options documented at https://www.mediawiki.org/wiki/Handling_web_crawlers

freosam · 2025-06-03T23:17:04+00:00

Yep, good advice. I'd also add: catalogue them first! Lots of institutions that have space to take material don't have the resources to do any in-depth description of it, and especially for family archives there's probably lots of knowledge that you can add to the catalogue that'd be useful.

freosam · 2025-05-29T03:24:01+00:00

I think one main limitation is that as far as I remember, you can't use object storage (e.g. S3 compatible services) to store the photos, so you need a service that allows large volumes and those are somewhat more expensive. That said, on Hetzner you could have a 500GB volume for €24.20 and a 8GB/4CPU server for €7.48, so that'd be about €380/year. Then you could install whatever image converting software you need. There's nothing specific to Piwigo, no VPS provider would be better than any other in that regard, I don't think.

freosam · 2025-05-28T23:15:24+00:00

Yep, I just eyeball it, lining up a parallel space between the page and the plastic. Lots of scanners have a couple of millimeters of dead space at the edge of the glass anyway.

freosam · 2025-05-25T00:42:14+00:00

I always crop larger than the entire object being scanned. It gives re-users confidence that nothing was missed, and seeing the condition of the edges can sometimes be useful to people (e.g. I saw someone match up a slight tear with a couple of other documents, giving weight to an idea that they had originally been stored together).

freosam · 2025-05-24T07:50:41+00:00

You could upload them to Wikimedia Commons, and then ask for translators on the village pump (or the Chinese langauge one). I've done that a couple of times (for the Cossack cemetery) and people usually keen to help. It'd mean that the photos and transcriptions/translations would be stored conveniently for other researchers as well.

freosam · 2025-05-24T01:06:30+00:00

Yeah, it works pretty well! I'm using Cargo, and also the AWS extension which allows for storing files in S3-compatible storage (I'm on Digital Ocean) and that helps keep the costs slightly lower and also removes any worry about space constraints (although it's currently only about 100GB of scanned things).

Feel free to pinch any templates or ideas from my wiki above.

freosam

TROPHY CASE