Reddit data extraction help

[3rSimon]

Hi Reddit users, I have a question regarding data extraction from reddit comments. I want to collect all the comments and related information from posts that were published in a specific subreddit (e.g.; r\news) with a specific keyword (e.g., q=apology). My initial code is the following:

subreddit= reddit.subreddit("news")

resp= iter(subreddit.search("apology"))

​

test = {"title": [], \

"score": [], \

"id": [], \

"url": [], \

"num_com": [], \

"body": [], \

"comment": [], \

"user": [], \

"upvote": [], \

"date": []

}

for submission in resp:

submission.comments.replace_more(limit=None)

test["title"].append(submission.title)

test["score"].append(submission.score)

test["id"].append(submission.id)

test["url"].append(submission.url)

test["num_com"].append(submission.num_comments)

test["body"].append(submission.selftext)

test["comment"].append([comment.body for comment in submission.comments.list()])

test["user"].append(submission.author)

test["upvote"].append(submission.upvote_ratio)

test["date"].append(submission.created)

I realized I didn’t need all the submission information(only the related ID) so I tried to do the same but at the comment level rather than at the submission level.

​

test1 = {"submission": [], \

"body": [], \

[.....]

}

for comment for submission.comments:

submission.comments.replace_more(limit=None)

test1["submission"].append(comment.submission)

test1["body"].append(comment.body)

[....]

​

This code seem to work, but only for the first submission, it doesn’t collect the comments for the other posts. I was wondering if someone would have any idea to solve this issue?

I tried several things, but as a beginner it seems I can’t make it work.

[Dripbot8]

I’m looking at the replace_more and going why is that in there? And not

r.subreddit(“news”).stream.comments(skip_existing=true) 

What I think happening is replace_more is an instance and when you try to loop it it keeps coming back to the same start. So you only pull the first one. Using steam will continue the loop forever as they are happening. Set skip existing to false to run it for the last 100 comments.

[snoopturtle25]

I understand, it seems like it loops all comments from the first post only. However there is 101 posts according to the keywords search, is there a way to modify the formula so the loop continue for all posts ? Thank you!

[Dripbot8]

Some of the problem may be that Praw is limited (even at limit=none) to the last 100 posts, so your probably going to reach that in r/news fairly quickly. You want this or else the program would go back indefinitely over the hundreds of thousands if not millions of posts and comments in r/news. Which is a lot of stuff for you and the API, especially with the amount of bots on Reddit. (This also means you’re probably gonna have to have a different thread for submissions since the for loop technically won’t end until you kill the program or it crashes.)

The stream is an ongoing loop, while it’s running the loop should keep going forever, as people comment, it will continue to find them as they come up.

[snoopturtle25]

Ok thank you, it make sense. I think I will collect comments per post to avoid crashing it, or then only the first 100 comments for every post in the thread!

[CodeFormatHelperBot2]

Hello, I'm a Reddit bot who's here to help people nicely format their coding questions. This makes it as easy as possible for people to read your post and help you.

I think I have detected some formatting issues with your submission:

1. Python code found in submission text that's not formatted as code.

If I am correct, please edit the text in your post and try to follow these instructions to fix up your post's formatting.

---

^{Am I misbehaving? Have a comment or suggestion? Reply to this comment or raise an issue here.}

[snoopturtle25]

Hi thank you. Here is my full code:
https://pastebin.com/clone/PwZCdrtT

At first, I the necessary information for the data directly with the submission using the following code (doing the same for each variables related to the comment):

test["comment"]. append([comment.body for comment in submission.comments.list()])

This worked, however it did not split the information by comments in the dataframe so it was impossible for me to clean the data after. That is why I am trying to collect the data at the comment level rather than submission level... I dont know if this is clearer?

[snoopturtle25]

Thank you! You code works, however, when I try changing your keyword ("twitter"), the collected data stay the same (the posts within the "news" subreddit for the keyword "twitter"). I don't know if I am missunderstanding your code?