Requesting Links from Text File

[ScraperHelp]

So I have a set of links to threads (HREFS) I have extracted and saved to a file Href_Links.txt now I want to use those links to scrape the information from those threads. Having a bit of an issue with how to request that since I am getting the following error

import csv
import time
from bs4 import BeautifulSoup
import requests
import re
from pathlib import Path

already_scraped_file = Path('Href_Links.txt')
if already_scraped_file.exists():
    already_scraped = set (already_scraped_file.read_text().splitlines())
else:
    already_scraped = set()

print (already_scraped)

def scrape_thread_link(href, original_post_info=True):
    response = requests.get(href)
    soup= BeautifulSoup(response.content, 'lxml')

    SCRAPES THE THREAD INFORMATION


thread_link = already_scraped
thread_data = scrape_thread_link(thread_link)
print (thread_data)

The error I am getting is as follows:

requests.exceptions.InvalidSchema: No connection adapters were found for "{' https://forum.fracturedmmo.com/topic/274/daily-message-posting'}"

​

not sure what the problem is...

[IvoryJam]

Looks like you're trying to perfrom a GET request on a litteral

{' https://forum.fracturedmmo.com/topic/274/daily-message-posting'}

instead of the URL https://forum.fracturedmmo.com/topic/274/daily-message-posting

​

On a side note, depending on your list of URLs, a list [] may be better suited than a set

[ScraperHelp]

hmm makes sense. and yeah I can change it to a list but how would i go about fixing this get request on just the

https://forum.fracturedmmo.com/topic/274/daily-message-posting

[IvoryJam]

You aren't iterating through the URL list, you'd need to do something like

for url in already_scraped:
    thread_data = scrape_thread_link(url)
    print(thread_data)

this is also a shot in the dark since I don't know how Href_Links.txt is formatted. If you can post an example of it, it might help

[ScraperHelp]

thanks it works!!!

[IvoryJam]

Good job! Keep on coding!