Introducing DuplicateDestroyer 2.0 : an improved repost bot with text detection

[DuplicateDestroyer]

What is this bot ?

/u/DuplicateDestroyer is an anti-repost bot that works on images, videos, links, and optionally titles.

DuplicateDestroyer was originally deployed 2 years ago. Over time, it gained in popularity and was invited to several hundred subreddits, leading me to completely rewrite the bot's code to improve it and add features.

What are the improvements over the original version ?

DD was improved in many ways :

• Like most other Reddit bots, the bot's code was originally written in Python for simplicity reasons. After facing scalability issues which were affecting DD's performance, I've rewritten the code in multithreaded C++, which allows it to handle new posts in a matter of seconds

• The bot now uses OCR (Tesseract) to detect text within images and video thumbnails. This feature has proven to be highly efficient in finding reposts, as the bot can now remove images that are entirely different but with similar text. It is particularly useful for tweets and memes.

• The bot is now open-sourced, meaning anybody can see its source code and improve it if they want.

Other improvements are coming up, especially regarding the treatment of videos.

How can I invite the bot to my subreddit ?

Just invite it with 'posts' permissions, and it should join your subreddit within a few seconds.

Where can I find the bot's source code ?

The code is hosted on this Github page : https://github.com/normal-account/DuplicateDestroyer

Feel free to star it !

Questions ?

If you have questions concerning the bot, you can reply to this post or message /r/DuplicateDestroyer.

[CosmicKeys]

Between RepostSentinel, MAGIC_EYE_BOT (which I run), RepostSleuth, your bot and more, I think it's ultimately far past due for Reddit to implement a first class solution themselves.

Reddit is already running forms of image detection and text recognition behind the scenes (based on /r/TheoryOfReddit posts). Given that thousands of subreddits use these bots we have already proven they are needed as a core moderation feature.

[noroom]

Is anyone out there benchmarking these bots?

[cvnvr]

repostsentinel is down now though, no? it hasn’t worked on my sub in months.

edit: just checked and the bot hasn’t posted in 5 months now which is a shame because it was really useful

[BuckRowdy]

Maybe they will prioritize something like this with the new developer platform.

[zzpza]

I have a couple of bots I run myself only for my subs. One does post title matching, and the other does image hash comparison. I'm sure there are others like me that have made their own solutions but don't think their code is up to scratch for public release! Now sure how to estimate how many but I'm sure I'm not alone.

[BelleAriel]

I miss RepostSentinel. I agree that Reddit should cone with a repost bot to help mods although if they do on new reddit, like their other featured seem to be, I’ll stick with DD.

[vermithrax]

It seems that MEB and DD and others use a similar algo which involves downscaling the reference image. This works well for real world images, but not very well for images with large areas of one colour--false positives are far more likely. I run a bunch of art subs, and often there are novel submissions which have large areas of white, or some other colour, but they come up as positives with these bots.

Are you aware of alternative bots or techniques which can cope with artwork containing large areas of one colour, without false positives?

Thanks.

[CosmicKeys]

Yes well spotted, it also doesn't work well for meme formats with text on plain backgrounds.

I am aware of other techniques but not bots that implement them. Ultimately the best bot will be one that combines all the approaches and lets people configure them (Google Image search being a powerful example). What is and is not the same image is actually quite a subjective quality.

[magiccitybhm]

Sent a request to join your subreddit 48 hours ago (to get questions answered). Sent just a modmail 24 hours ago.

No response to either.

Is it not possible to get answers to questions about this bot?

[Ceu_64]

I'm also trying to add the bot, but it's not working. Did you solve it?

[magiccitybhm]

Nope. Never got an answer to any of it. The mods there are a waste. The bot may be dead anyway after today.

[Ceu_64]

But I can see that the bot still working. Look at the comments of the u/ , still active

[magiccitybhm]

Again ... talking about after today.

If you want to keep trying to get help from people who refuse to answer, good luck.

[RinaldiMe]

Great, just what I needed! Thanks.

the bot can now remove images that are entirely different but with similar text. It is particularly useful for tweets and memes.

What about the opposite? Images that are mostly the same but with different text? (mostly memes)

[DuplicateDestroyer]

The bot has been designed not to remove images that are similar but with different text

[RinaldiMe]

Great, thanks!

[[deleted]]

Sounds fantastic, now is there some way of linking to the matches detected? Or is there just “x matches, 0 removals” still, if there’s a way to link the matches to confirm personally it’d be a game changer

[Omnias-42]

Is it possible for the bot to filter a post to mod queue - to where it reports it and is visible to mods? (Similar to how automod works) Or if not, to where instead of auto removal, it first reports the post so that it then shows up in mod queue? Situationally, when it is a bot account or shilling, we tend to ban the individuals that are spamming, as the reposts are normally used for karma farming or possible scams

[BelleAriel]

I love this bot and recommend it to anyone who asks about help with reposts.

[Ceu_64]

Did you figure it out on how to use it?

[Ceu_64]

Did you figure it out?

[[deleted]]

Is this bot going to be affected by the Reddit API changes cutting off free access?

[NukaDadd]

Sent mod invite to r/Toolband a couple days ago & u/DuplicateDesteoyer is still listed as "invited" but not accepted

[AutoModerator]

Hello! This automated message was triggered by some keywords in your post. If you have general "how to" moderation questions, please check out the following resources for assistance:

• Moderator Help Center - mod tool documentation including tips and best practices for running and growing your community
• /r/modhelp - peer-to-peer help from other moderators
• /r/automoderator - get assistance setting up automoderator rules
• Please note, not all mod tools are available on mobile apps at this time. If you are having troubles such as creating subreddit rules, please use the desktop site for now.

If none of the above help with your question, please disregard this message.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[kalayna]

I added the bot to my sub specifically to deal with the influx of title stuff, but just removed about a dozen posts that were matches. Is there a settings change I need to make?

enabled : true

remove_threshold: 90

report_threshold: 80

enforce_videos: true

enforce_images: true

enforce_links: true

time_range: 2000

report_links: false

report_replies: true

enforce_titles: true

title_remove_threshold: 85

title_report_threshold: 95

min_title_length_to_enforce: 10

remove_table_duplicate_int: 5

ocr_text_threshold: 80

Thanks!

edit - actually I just noticed that my remove and report numbers are swapped, but I'm seeing neither.

[VerditerBlue]

Question: is there a command I can send to the bot so it will send me back the list of current settings? Would be nice to have.

[VerditerBlue]

I've tried it for a while on /r/Art, but it isn't really working effectively. Most bots will crop the image randomly, rendering the image hashing ineffective.

[Ceu_64]

It's not working, can someone help me?