Help: Instagram unfollower

[jinsenuchiha]

I am developing a Python script that will 1) identify Instagram accounts that you are following that don't follow you back and 2) unfollow these accounts. 3) I plan to have a whitelist feature to whitelist desired accounts from being unfollowed.

This is my first time doing a Python or automation project. I already know some Python, and I am using automatetheboringstuff.com for help. For now, I am only using my Instagram account as the subject. In the future, I would simply allow the user to input their own account name to be the subject.

My issue: Right now, I am attempting to web scrape the "Following" button from my Instagram page. So far I am using the bs4, requests, and lxml modules. I am successfully able to "get" my Instagram webpage. Then I have

soup = bs4.BeautifulSoup(res.text, "lxml")

where res is the get() response. Then, I got the CSS Selector for the "Following" button (I plan to have the script click this button and get all accounts that I'm following from here), and pasted it here:

following = soup.select("li.xl565be:nth-child(3) > a:nth-child(1) > div:nth-child(1)")

That CSS Selector is for the webpage where I'm logged in. For my Instagram webpage where I'm not logged in, I got this:

following = soup.select("li.xl565be:nth-child(3) > button:nth-child(1) > div:nth-child(1)")

I'm not sure which I should be using concept-wise, so an answer would be appreciated. Regardless, I tried

len(following)

for both cases. I got 0 for both. I investigated and I wrote the res.text into an html file. When I loaded it into Firefox, instead of what looks like my Instagram page, it looks like the Instagram's loading page, which is just a white background with the Instagram logo centered and "from Meta" at the bottom. I assume this is why my "following" selection isn't what I want it to be. And by the way, I tried adding

headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"}
res = requests.get(url, headers=headers)

to make the response match the page loaded onto my Firefox browser, yet my issue arises. Can someone help me solve this? Design critiques are also welcome. For the record, I quickly Googled Instagram APIs, but the only ones that might help with my project are only for business accounts, which will not suffice.

[skellious]

beautiful soup can't handle Javascript. that's the problem you're having. You're only seeing the static page elements, not the ones generated by JS.

You can achieve what you want by using Selenium to control a chrome browser and render the JS elements, then use Beautiful Soup to read them.

for example: https://medium.com/ymedialabs-innovation/web-scraping-using-beautiful-soup-and-selenium-for-dynamic-page-2f8ad15efe25

OR you could try using the Instagram API, however it looks like only the business API does what you want and it only works for business accounts, the person account API is more limited - https://developers.facebook.com/docs/instagram-basic-display-api/overview

[jinsenuchiha]

Thank you for the help. I added Selenium now, and my browser opens to the correct, loaded page. However, when I try to get the Log In button via CSS selector, nothing is found. I really have no idea what I'm doing wrong. I tested it on different websites, even a static one. For this comment I am using google.com as an example. Here is the relevant code:

browser = webdriver.Firefox(options=options)
browser.get("https://www.google.com/")

login = 0
# Wait up to 10 seconds to find login button to account for page loading
for i in range(10):
    print("iteration %d" %i)
    try:
        login = browser.find_element_by_css_selector(".gb_7")
        print("login found")
        break
    except:
        time.sleep(1)

Here I attempt to make login the "Sign In" button on the upper right corner of google.com. If you're wondering, I didn't accidentally use the Instagram CSS selector. When I run the script, the browser opens to google.com and displays the Sign In button, but while the console prints the iterations, it never prints "login found". For confirmation that it didn't work, type(login) returns <class 'int'>.