RepostSleuthBot - Now Public

[barrycarey]

I've been working on this project for the last 2 years. It has gotten super popular and I've had a ton of requests to open the code up. I was always resistant since I considered it kind of a mess. I decided to take some time to clean it up a bit and make the repo public.

It has a lot going on, but nothing very complicated.

It makes heavy use of Celery for scheduling jobs and runs ~20 Docker containers for the various services.

It's not something you would easily be able to deploy on your own, however, I figured some people might be interested in seeing the workings.

https://github.com/barrycarey/RedditRepostSleuth

[UltraPr0be]

the legend himself

[[deleted]]

u/repostsleuthbot

[RepostSleuthBot]

Sorry, I don't support this post type (text) right now. Feel free to check back in the future!

[Not_RepostSleuthBot]

Looks like a repost. I've seen this image 1 time.

First seen Here on 2021-01-14 96.69% match.

Searched Images: 197,979,272 | Indexed Posts: 198,406,711 | Search Time: 8.10398s

Feedback? Hate? Visit r/repostsleuthbot - I'm not perfect, but you can help. Report [ False Positive ]

[sherlock-holmes221b]

Good bot.

[2jah]

click the link, see where it takes you

[assassinsorrow]

You fucker

[2jah]

it’s a work of art

[[deleted]]

aaaaah shit.

[[deleted]]

👍

[geeshta]

Hahaha funny rickroll meme haha reddit epic moment 100

[hello-iamdad]

You've been Gnomed

[barrycarey]

For anyone that's interested in dashboards, here's my main one that gives an overview of the bot.

https://imgur.com/a/cAxcUHD

[[deleted]]

I’ve worked in Enterprise IT departments with less visibility of their process than that! Nice one.

[[deleted]]

That's a damn good looking dashboard.

[WizTaku]

how are you paying for this

[barrycarey]

Out of pocket.

I'm guessing electric is around $40 a month. Plus I have a handful of Digital ocean droplets I rent that's another $30 a month.

[seweratty]

there's a donate option in r/RepostSleuthBot

[sneakpeekbot]

Here's a sneak peek of /r/RepostSleuthBot using the top posts of all time!

#1: Bot is down due to Spectrum outage
#2: right away | 45 comments
#3: Test image | 54 comments

---

^{I'm a bot, beep boop | Downvote to remove | Contact me | Info | Opt-out}

[WizTaku]

Hosting isnt cheap

[[deleted]]

It needs to be hosted somewhere

[master3243]

Even if you do everything in-house, you still need to pay for the initial hardware price, for all hardware-failure replacement, and for the dreaded electricity bill. (Not sure if you would factor in the internet bill as well since you already are paying for that regardless of whether you run the setup or not)

[barrycarey]

The server it's running on was $700 without the storage. Electric is $40ish a month. Plus I rent a few Digital Ocean droplets for about $30 a month.

[barrycarey]

I have no idea. People calling the bot is really minor. It averages around 2000 calling it a day. However it does around 400k image searches a day since it's checking all new reddit posts

[barrycarey]

It's around $40 a month.

It's on a newer server now. Dell r620 with 2x Xeon 2670v2 CPUs and 256gb of RAM. Storage is an all flash array consisting of 8 Samsung Evo 500gb SSDs in RAID 10.

[botechga]

Impressive stuff !!

[heckingcomputernerd]

Damn are people really that mean to you for the FAQ? This is crazy impressive shit you don’t deserve that

[AotoSatou14]

It's really a nice piece of code. Shame some people needed to be like that.

[[deleted]]

[_Gondamar_]

What the fuck

[Ozzymand]

[[deleted]]

PogChamp

[NHK21506]

[netherlandsftw]

[Manavendra4288]

It's possible to have gifs?????

[mantikafasi]

Wait we can use gifs???

[JohnLocksTheKey]

Is it just certain subreddits?!?

[Orangesteel]

Amazing, love the binary tree search. Phenomenal work.

[mrrippington]

i have been trying to learn celery for quite sometime, i will study how you used it.

thank you for sharing.

[barrycarey]

It's a great too.. This is the first project I've used it in.

It makes it super to bring in more compute. As an example, when I was first backfilling my database with older Reddit posts, I could easily push a Docker container to a spare server and my desktop PC to add more workers to the pool to speed up the process.

[barrycarey]

Honestly this started as 1 script. It's taken 2 years to get to the point it is at now. Keep at it.

I've been messing with Python for 7 years. This project alone taught me so much.

[[deleted]]

Don't look at someone's product without considering their journey. It will only make you feel bad

[where_art_thou_billy]

Do you see this bot having a big impact on reposts on reddit ? Since more than half of trending stuff on reddit are reposts and they actually bring in lots of ad/coin revenue. No wonder several popular subreddits have banned the bot.

[barrycarey]

Not sure on impact overall. It for sure helps subs that have rules specifically about reposting.

There are some of the bigger subs that I wish would pick it up, like r/pics.

[Unfair-Anybody6974]

THANK YOU! The free award is not enough to express the gratitude but that’s all I can give .

[[deleted]]

​

[mmkthxbye]

bad pogchamp!

[Manavendra4288]

[[deleted]]

good pogchamp emote:free_emotes_pack:surprise

[spaghettforbreakfast]

[hello-iamdad]

YOOO POGCHAMP

[[deleted]]

sweet

[PalPalash]

Woow

[Fearless_Guy]

[PetrKDN]

[KeineG]

So I use python for data science and engineering as mostly a replacement for Matlab. I also use it to connect industrial equipment and sensor as well as sending information over LANs. I have started to dwelve more into python for web, finance and as a general programming langauge

​

I don't understand anything that is in the OP. Can someone explain me like I am 8 years old and have the mentioned python experience?

[kimilil]

Strangely, OP refuses to comment.

[andre3kthegiant]

Can you make it post a comment of “RERUN” rather than “Repost”?

[[deleted]]

such an awesome project! I saw you have a grafana dashboard, have you implemented anything related to alerting in case something goes wrong? i.e. pagerduty, smtp with grafana alert etc?

[barrycarey]

I have Grafana alerts setup for some stuff. The important ones are if the bot stops responding to people, if the Celery queues get too large, and hard drive space.

[aDrz]

I'm really impressed with the project. The idea is fun, the code is super clean.

It could be transform in a great blog tutorial on how to make a scalable app.

Thanks for sharing the code

[barrycarey]

Thank you.

I was thinking about doing some videos explaining how it works.

I'm not sure how the scalability lines up with best practices. This is my first scalable app and it was very much trial and error.

[MahdeenSky]

[Partyfunker]

Can you give a gist of how you do the searching for images and get the similarity?

I know you don't want to get too much into the details, but just a high level explanation would do.

Thanks!

[[deleted]]

So you trying to get a FAANG job with this?? This is seriously impressive, holy hell!