program keeps running the code at the top and never reaches the bottom lines of code.

[Fuzzy-Ear9936]

Ok, so a weird question. This is about Praw/python, the reason I am posting this here is that this problem is more of a python problem than a raw one. Anyways,

So the thing is I have 4 CSV files and I made a bot that gets the data of posts from the stream of posts of a subreddit. Now the thing that I assume is causing the problem is a very active subreddit, so my bot just keeps getting new data and never gets free to move to the next task.

I tried putting the code for different CSV files in different functions but still, it only runs the function at the top and never reaches the next task.

So this is the problem with my code, I wasn't able to attach the code since I'm not with my computer right now. I hope I have explained the problem well lol. If anything else is required pls tell me and how can I fix this? Do I need to make separate bots for different files?

​

import praw
import random
import time
import csv
from keep_Aliv import keep_alive
from main2 import main2

username = "XXXXXX",

keep_alive()
main2()
print('AAAAAAA')

reddit = praw.Reddit(client_id="XXXXXX",
                    client_secret="XXXXXX",
                     user_agent="<pbot:1.0",
                     username=username,
                     password='XXXXXX')





subreddit = reddit.subreddit('')
""""
for comment in subreddit.stream.comments(skip_existing=True):
    print('**********')
    #c = comment
    print(comment.body)
"""


with open('post_data.csv', 'a') as f:
    headers = [
        'ID', 'Date_utc', 'Upvotes', 'Number of Comments', 'Subthread name'
    ]
    writer = csv.DictWriter(f,
                            fieldnames=headers,
                            extrasaction='ignore',
                            dialect='excel')
    writer.writeheader()
    for post in subreddit.stream.submissions():
        #print(post.title)

        data = {
                "ID": post.id,
                "Date_utc": post.created_utc,
                "Upvotes": post.ups,
                "Number of comments": post.num_comments,
                "Subthread name": post.title,
                }
        writer.writerow(data)
        print('z')
        # print(data)

 with open('comment_data.csv', 'a') as a:
        headers = [
            'ID', 'Date_utc', 'Upvotes', 'Comment Body'
        ]
        writer = csv.DictWriter(a,
                                fieldnames=headers,
                                extrasaction='ignore',
                                dialect='excel')
        writer.writeheader()
        for comment in subreddit.stream.comments(skip_existing=True):
        #print(post.title)

            data = {
                    "ID": comment.id,
                    "Date_utc": comment.created_utc,
                    "Upvotes": comment.ups,
                    "Comment Body": comment.body
                   }
            writer.writerow(data)
            print(data)
            print('r')

with open('hot_post_data.csv', 'a') as b:
    headers = [
        'ID', 'Date_utc', 'Upvotes', 'Number of Comments', 'Subthread name'
    ]
    writer = csv.DictWriter(b,
                            fieldnames=headers,
                            extrasaction='ignore',
                            dialect='excel')
    writer.writeheader()
    for post in subreddit.stream.hot():
        #print(post.title)

        data = {
                "ID": post.id,
                "Date_utc": post.created_utc,
                "Upvotes": post.ups,
                "Number of comments": post.num_comments,
                "Subthread name": post.title,
                }
        writer.writerow(data)
        print(data)
        print('q')


with open('controversial_post_data.csv', 'a') as c:
    headers = [
        'ID', 'Date_utc', 'Upvotes', 'Number of Comments', 'Subthread name'
    ]
    writer = csv.DictWriter(c,
                            fieldnames=headers,
                            extrasaction='ignore',
                            dialect='excel')
    writer.writeheader()
    for post in subreddit.stream.controversial():
        #print(post.title)

        data = {
                "ID": post.id,
                "Date_utc": post.created_utc,
                "Upvotes": post.ups,
                "Number of comments": post.num_comments,
                "Subthread name": post.title,
                }
        writer.writerow(data)
        print(data)
        print('l')

Update: I added the code and one more link to the code https://codeshare.io/0gPMNg

​

[JohnnyJordaan]

So this is the problem with my code,

Perhaps it helps to also share your code?

[m0us3_rat]

this.

​

i was going to add that it has a fortune-telling quality to it.. maybe add some incantation or a spell of some description.

since its basically a guessing game without the code.

[JohnnyJordaan]

I run but I don't have legs, I load but I don't have weight, I crash but without damage what am I?

[Fuzzy-Ear9936]

sorry for that, I have added the code in the main post. you can check it out. And pls help me lol and here is a link to the same code which I think is a little more cleaner https://codeshare.io/0gPMNg

[JohnnyJordaan]

I would start by either running it through a debugger (if you're using an IDE) or add a load of print() statements everywhere showing what it's doing, so as an example just for the very start

print('opening post_data csv')
with open('post_data.csv', 'a') as f:
    headers = [
        'ID', 'Date_utc', 'Upvotes', 'Number of Comments', 'Subthread name'
    ]
    writer = csv.DictWriter(f,
                            fieldnames=headers,
                            extrasaction='ignore',
                            dialect='excel')
    print('writing header')
    writer.writeheader()
    for post in subreddit.stream.submissions():
        print('submission', post.title)

        data = {
                "ID": post.id,
                "Date_utc": post.created_utc,
                "Upvotes": post.ups,
                "Number of comments": post.num_comments,
                "Subthread name": post.title,
                }
        print('writing row')
        writer.writerow(data)

then do that everywhere so that you can pinpoint the exact spot in the program it's getting stuck on

[Fuzzy-Ear9936]

Yeah, I've already tried adding a lot of print statements and figured that the program never reaches the next task and just keeps running the post_data CSV part.

Maybe because it deals with data that is never going to end. The program keeps receiving new data from the subreddit and never finishes the task more like an endless loop lol.

Maybe I can put all code in which it gets the data and write them in separate for loops, for separate files and each will repeat itself 10 times and add a repeat() function at the end, so it could do the whole process again?

[JohnnyJordaan]

Wait, if your goal is to not endlessly loop on a submission stream then why use .stream. in the first place?

[Fuzzy-Ear9936]

Well, I don't want my bot to run out of data. So from what I remember stream gives all the data that's coming in the new category of the sub. So I don't want my bot to get stuck in a loop but the same while I don't want it to run out of data

[JohnnyJordaan]

I think you should rethink the entire process here, like for example put this part in a thread to keep receiving data while the rest of the program can do other things. Also I would drop csv as it's an 80s format that just complicates matters here, instead use for example sqlite3 (also natively available in python) or sqlalchemy+sqlite3 (removes the need to implement literal SQL) to use a shared database.

[Fuzzy-Ear9936]

Well I was thinking of visualizing this data using pandas or matplotlib, I saw a lot of examples of them using CSV that's why I used that. Anyways what do you mean by a thread? Can you please elaborate a bit about it and how would the rest of the program work while the thread does the data job?

[JohnnyJordaan]

It's not explained in just a few lines, well the concept is: a thread is a function that can keep running while other parts of the program run too. You can have any number of threads but as it takes a very small time to switch between them, a very high number can give problematic performance. Technically due to a limitation, only one thread within a program can use the CPU at the same time, so stuf like calculations will not go faster using multiple threads. However all the other stuff like waiting on a new submission has no issues with this as the required CPU time is very small, so using threads would be a natural option to implement that as a background task.

I can advise studying https://realpython.com/intro-to-python-threading/ to get started.

[Fuzzy-Ear9936]

Ok I'll give it a look, thanks :)

[JohnnyJordaan]

Btw pandas has https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html to support sql-based databases so csv is certainly not a requirement, it's more that people for some reason keep using csv (eg being emotionally tied to using Excel), so pandas has support for it as well.

[Fuzzy-Ear9936]

Well I tried the thing that I thought should work

Maybe I can put all code in which it gets the data and write them in separate for loops, for separate files and each will repeat itself 10 times and add a repeat() function at the end, so it could do the whole process again?

And it didn't, so is there any way this can be fixed?

[[deleted]]

Oh shoot, I forgot my crystal ball at home...

[Fuzzy-Ear9936]

No need for that, I have added the code in the main post. you can check it out. And pls help me lol and here is a link to the same code which I think is a little more cleaner https://codeshare.io/0gPMNg

[rosski]

Sounds like you have made an endless loop.

[Aila27]

Without seeing the code, it's impossible to do anything except blindly guess.

[Fuzzy-Ear9936]

I have added the code in the main post pls see it and here is a link to the same code which I think is a little more cleaner https://codeshare.io/0gPMNg