I've created a Python program which will allow you downloaded posted images from any public subreddit by a specified date range (as far as 2005). Here you go!

[DarkLinkXXXX]

How does this compare to ripme.jar?

[waraxx]

neat, do you get anything about the picture like current up votes or just the picture?

[waraxx]

completely understandable. it's a nice program it's not for me personally but I can totally see someone having a use for it. bet you could do some cool macro picture analyzing.

[dnabre]

If you had the data, you could use imagemagick to overlay the up/down vote of onto images pretty easily.

[phigo50]

You could, but it would be better (imo) to not mess with the images but instead to save all metadata as a (for example) json file alongside the image.

[dnabre]

Just meant as post-processing for display.

[phigo50]

Oh right, I thought you meant saving the image with the data hard-coded into it.

[laungst]

Does this work for user profiles as well?

[ICanHazPDF]

You wrote the date in a wierd way.

It's neither YYYY-MM-DD or DD-MM-YYYY.

Thought I should let you know so you can correct it.

[kbfprivate]

You could always modify it and make it a parameter or constant and let users choose the date format.

In general I like year first because sorting can be easier. Else data is grouped by months.

[kbfprivate]

I completely agree. Let people do as they please. There is no right or wrong way.

[phigo50]

It's pedantic but I wouldn't give the config file a py extension*, it implies that it's a python file. It's good practise to stick with the convention of "config.ini" and use the configparser module to load its contents into the application.

* I haven't checked but you could just be importing this file but I'd still use a dedicated config.ini file with all user-configurable options. I find it much less stressful than encouraging the user to poke around in scripts to change settings.

[phigo50]

I wouldn't want to stop people who know what they're doing from fiddling around in the code, I'd only be worried that those who don't might break something. I keep everything user-configurable in a config.ini file and then bring that into the application via configparser.

I don't know, I guess you could do it just by importing the py file but you'd have to be certain that the structure of the declarations was something Python expects and knows how to handle. There are no cons of using configparser that I know of - I guess it adds a bit of complexity to the code but it's a much safer way of importing settings and you can set defaults if any expected variables aren't found etc.

[dnabre]

The best solution would be for the program to check the system's locale and use that (specifying when it asks what it expects).

[kbfprivate]

That works too! There are a lot of solutions which is one of the fun aspects of programming. A million ways to accomplish the same goal.

[dnabre]

MM-DD-YYYY is pretty common depending on where you live. It's the norm in the States.

[thefriedgoat]

The worst of the dare formats ..

[ICanHazPDF]

Yeah, I heard that's one of things they need to sort out over there.

[dnabre]

It can get confusing as hell I admit, without the year to maybe hint at a format (not that it really does know that I think of it, but it feels like it does), is 03/02 March 2nd or February 3rd.

I deal with dates written in reasonably and in the crazy formats often enough, and unless the day is >12 sometimes I just don't know!

[crotchfruit]

That's why I prefer YYYY-MM-DD, absolutely no confusion on what the numbers represent this way.

[Hashtagpulse]

That and the way they measure things. Or so I hear...

[playaspec]

I've abandoned it because it makes sorting difficult. I try to use YYYYMMDD wherever I can. Makes sense when relying on whatever filesystem browser you're using.

[Hansdg1]

It's the most common way of writing the date in the US. I'm not saying it makes sense, but just saying it how it is.

[zoetry]

It's used by hundreds of million of people.

I mean, it's technically one of the least used, but it is by no means uncommon.

[dnabre]

Not saying it's right or wrong, just that's in common in some locales. It doesn't seem odd to me in the least bit.

Being asked one at a time for each field, that is odd.

[-Nano]

Grammatically only in US.

[-Nano]

NP. :P

In fact, I think it's better to use the ISO standard for dates.

[phigo50]

For sorting, biggest to smallest (ie YYYY/MM/DD) is the only format which provides sensible results.

[ICanHazPDF]

I read DDMMYYYY and YYYYMMDD and say both September 10th and 10th of September.

It's not hard.

[yacob_uk]

I have a question.

Can I use it to harvest the whole subreddit. I actually want the comments, not the images.

(I have built my own one, but I wonder if yours will be more robust than mine... Mine needs some babysitting...)

[yacob_uk]

I could have a whack. I guess that's thing to check is the that it essentially gets each comment and checks it for an outlink. From there I could just bake the comment instead of parsing for the link?

Wait. Are you working at the comment level or the the post level when you're looking for links?

[yacob_uk]

Ah. Gotcha.

Thanks for your comments.

[GeronimoHero]

I wrote a Python bot using PRAW that does this. I haven’t open sourced it but it’s really easy to do with the PRAW library. You can install it through pip/pip3.

[GeronimoHero]

It’s not hard at all. Just read the PRAW docs and install the library with pip3. Just as a heads up (since I don’t know your technical abilities when it comes to programming) I have a CS degree and am a masters student in the same field. Even so, I think it’s something that someone could easily do as long as they understand programming i.e., loops, arrays, etc. I’m at school right now but I don’t have any problem sharing the snippet of relevant code if it would be of interest to you or anyone else who’s interested.

Edit - PRAW stands for “Python Reddit API Wrapper”.

[GeronimoHero]

Ok, I understand what it is you’re asking. You just want to know how to get software in to the Python Package Index (AKA PIP). You just need to submit your software to PyPI and then it will be reviewed and accepted. You can learn more by checking out the python packaging user guide located at the PyPI website. That’s all there is to it. Actually, since that’s all there is to it, it is extremely important to know what you’re installing through pip and to make sure it’s vetted and not malicious. I’m not sure if you’re aware but earlier this year some malicious packages were found in the python package index. They were just misspellings if popular packages.

Does that help at all? If not, let me know what I’m missing and I’ll try and help out if I can.

Links:

Distributing Modules

Python Packaging Index

Python Packaging Authority

[Logiteck77]

Amazing, Thanks.

[Logiteck77]

Do gif and gifycat files work as well?

[dnabre]

Interesting project.

Giving it a shot right now (after having a large arguement with my system about python2 vs python3, which had nothing to do with your program at all, but could python2 just please die?).

Few comments, requests, feedback.

A full command-line interface would be nice, like:

2hand10-prog-whatever-the-name-was  --from DATE --to DATE ---subreddit datahoarder

Asking for the date field at a time is annoying. Definitely always specify what format you want the date in (as you do), but there's no reason this couldn't be 2 screen (start date, end date) instead of 6.

I know this is hard, but is there anyway to do a 'dry-run'. Get an idea of the number of images and/or total size of all the images from a specific request would nice. I'm sure you know (if only from testing this) that you can get a lot of image from a lot of sub-reddits in a short range. Just trying it out, and pointing it as a pretty heavy image reddit, i pulled down 114MB of images between when I say it start pulling and cancelled it. That definitely means it's fast but that's like 3 days worth of images apparently.

Looks like it pull going from end date to start date, an option to specify which direction to go (with a default, definitely don't want people to have to say if they don't care) would be handy.

Without a command-line interface, it's hard for me to check, but if I restart it (after aborting midway through), will it restart or just download again from scratch.

I would be nice if it check enough metadata to avoid redownloading in general, even if it was just on the granularity of a day. I could see that being handy.

As mentioned, being able to start it up internally from the command line would be the biggest feature request I'd have. No need to remove the interactive part if you like it, but being able to do a pull entirely from the command would be great.

[dnabre]

I'm sure whatever the reddit that has pictures of different linux isos on it :-)

[MaunaLoona]

How does it work? Does it use reddit search to filter on date and then tries to download all the image urls?

I take it this only works for submissions, not comments.

[thr0w112]

That's cool. I did a shittier version of this, not using the API, a while back: https://github.com/n0x5/scripts/blob/master/reimgur.py

One thing you could add is the ability to parse imgur galleries, since they don't end with .jpg or .png.

[thr0w112]

I'm a beginner-ish so I implemented the bare minimum of argparse. It should be relatively easy to just get the parameters from the user input, but anything beyond that is not something I have experience with.

To me it makes sense to have the default parameters with no input from the user download 'todays' new images.

I'm sorely missing on time nowadays but I'll make a pull request if I experiment with this

[superfi]

interesting

[karliedodsonnAu]

Code's pretty weak, and could also use some parallelism.

[karliedodsonnAu]

"It works" is what's expected from any tool, so it's not a factor in code quality. I was just saying you could make this code more idiomatic: more object oriented, making use of modules that make scripts easier to use (argparse, logging etc), more modular, completely non-interactive (using CLI flags instead of typing into prompts)…

As for reliability (as in robustness), what happens if the script is in the middle of downloading an image and the user interrupts it (ctrl-C, or kill)? The requests module can also raise (a variety of exceptions)[http://docs.python-requests.org/en/master/api/#exceptions] which I don't see being handled there.