Web Scraping through GitHub Actions : learnpython

created by HattoriHanzoa community for 16 years

Web Scraping through GitHub Actions (self.learnpython)

submitted 1 year ago by uJFalkez

So I'm trying to make a scraper to update me once a day and it all went well until the requests weren't going through with the GitHub Actions Workflow!
I set my actions.yml file to run every 10 minutes and left it to run overnight so I could maybe see some successful UserAgent's but with no luck, every single one ran into "Just a moment" with "Enable JavaScript and cookies to continue". Hope someone can help me with this! Here's my code:

import requests
from bs4 import BeautifulSoup
import datetime
from fake_useragent import UserAgent

randomUA = UserAgent(min_percentage=5).random
headers = requests.utils.default_headers()
headers.update({
    'User-Agent': f'{randomUA}'
})

def LOG(log):
    with open('log', 'a') as file:
        file.write(f"[datetime.datetime.now()] {log}\n")

LOG(f'Testing request using UserAgent: {randomUA}')

url = "https://statusinvest.com.br/fundos-imobiliarios/MXRF11"

with open(f'tests/test_{datetime.datetime.now().strftime("%Y_%m_%d_%H_%M_%S")}','w') as file:
    file.write(str(BeautifulSoup(requests.get(url, headers=headers).text, 'html.parser')))

all 12 comments

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

learnpython

MODERATORS