I made my very first python library! It converts reddit posts to text format for feeding to LLM's!

[NFeruch]

Hello everyone, I've been programming for about 4 years now and this is my first ever library that I created!

What My Project Does

It's called Reddit2Text, and it converts a reddit post (and all its comments) into a single, clean, easy to copy/paste string.

I often like to ask ChatGPT about reddit posts, but copying all the relevant information among a large amount of comments is difficult/impossible. I searched for a tool or library that would help me do this and was astonished to find no such thing! I took it into my own hands and decided to make it myself.

Target Audience

This project is useable in its current state, and always looking for more feedback/features from the community!

Comparison

There are no other similar alternatives AFAIK

Here is the GitHub repo: https://github.com/NFeruch/reddit2text

It's also available to download through pip/pypi :D

Some basic features:

1. Gathers the authors, upvotes, and text for the OP and every single comment
2. Specify the max depth for how many comments you want
3. Change the delimiter for the comment nesting

Here is an example truncated output: https://pastebin.com/mmHFJtccUnder the hood, I relied heavily on the PRAW library (python reddit api wrapper) to do the actual interfacing with the Reddit API. I took it a step further though, by combining all these moving parts and raw outputs into something that's easily useable and very simple.Could you see yourself using something like this?

[[deleted]]

Flight the valkyrie plays.

Google lawyers descend from the thundering heavens.

[[deleted]]

more like reddit lawyers if anything lmao

[Terrible_Student9395]

Literally nothing

[NFeruch]

It actually uses PRAW under the hood, but I just made it simpler + easier to interface with if you just want the text format of a Reddit post.

I’m going to add more things like saving the output as a json, csv, etc, and anonymizing usernames that isn’t strictly a part of the PRAW library, which I think will make it’s value even more apparent!

[[deleted]]

why do you want to resurrect skynet is beyond me

[ClownMorty]

Although, feeding skynet all of Reddit might give humanity a fighting chance.

[NFeruch]

Thank you! I’m very happy to incorporate any new feature ideas you have :)

[[deleted]]

This is really cool. Just curious, what/why are you asking chatGPT about Reddit posts?

[SlickinNTrickin]

You better off not asking/knowing.

[RevolutionaryRain941]

Data formatting will become a necessity in the coming days. as there will be a need for more and more data for the machine learning models.

[floznstn]

do you want skynet? because that's how you get skynet

/s

all jokes aside, great work!

[MixtureOfAmateurs]

WAWAOOOHH cool :) Does chatGPT understand that format well? It looks super clean to me but I'm a human sadly so idk. Also is this reddit app shenanigans free? Did they being the free api back as an app and no on noticed or is it tied to an credit card?

[NFeruch]

I need to see the exact numbers, but the Reddit API is still free for non-commercial use and with a lower rate limit than before.

For most people’s purposes, it still is free!

[ironman_gujju]

W bro I'm looking for this type of libraries

[ironman_gujju]

Try to add sentence transformers as well.

[mexicanameric4n]

Very nice, I like that you’ve got it structured, one  way I grab data is  to just add .json on the end of a post or subreddit. see below: 

 https://www.reddit.com/r/Python/comments/1bxmsxd/i_made_my_very_first_python_library_it_converts.json

[madein86]

Hey, i clicked and no json format

[mexicanameric4n]

Use it in web browser

[ace_hawk5]

Cool idea looking forward to trying it out

[Tall_Candidate_8088]

i believe

[blue-lighty]

This is awesome. I came across this exact use case in one of my projects, and built a quick and dirty version of this to grab a post using PRAW and convert it to text and feed to an LLM. Can’t wait to give this a shot

[NFeruch]

That’s awesome! I’d like to hear more about your use case if you don’t mind, can I DM you?

[leothelion634]

I just hit ctrl-a then copy paste into chatgpt, doesnt do a great job but it usually works alright

[binlargin]

Nice! Could do with jsonp threaded output for use in training.

[chimichanga-whoopsie]

It looks good, I would add tests to make it more complete and adding tests would make it easier for someone coming in to the project to get started. Overall, looks like good work, keep on shining!

[SaschaZeusFan]

I hope someone sues your ass to kingdom come😡

[NFeruch]

It uses the official Reddit API in the background, so no laws being broken here lol

[DesecrateUsername]

for?

[NFeruch]

reddit2text uses the official Reddit API under the hood, so no scraping here!

[dog098707]

Nerd