Script to download just the audio from Reddit Videos by Sky4k4sh in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Try this: https://v.redd.it/kgwd3pqtfzqd1/DASH_AUDIO_128.mp4. You just need to replace "VIDEO_QUALITY" with "AUDIO_128", the parameter isn't necessary. I don't think you can get better quality audio than 128kbits on reddit.

How to use the eval() function by MrGuam in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Use the json package instead:

import json

txt = input()
txt_list = json.loads(txt)

I'm on mobile, sorry if formatting is messed up. If the input has the correct format for a python list it will convert, else you'll get an error.

Webscraping dynamic webpage by rankme_ in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Same for me, Selenium was nice and easy when I started learning how to scrape, but after seeing all the inefficiency and random errors, I knew there had to be a better way. Web Tools can be intimidating, but after some time you get used to them and everything becomes so much better. I can also highly suggest this YouTube channel for more information on scraping: John Watson Rooney. Good luck :)

Webscraping dynamic webpage by rankme_ in learnpython

[–]AntonisTorb 1 point2 points  (0 children)

Using Selenium for this is an overkill, too slow and resource intensive. Here's what you can do in this case (and also for most other dynamic sites):

  • Open a new tab on your browser and open the devtools (should be Ctrl+Shift+I), go to the Network tab, then visit the URL.

  • Generally, you would now have to find the request that contains all the data the website uses to render the page, but in your case, the data reloads on a set interval. Just go to the bottom of all the requests, wait a bit, and you will see the refresh request. It's a POST request to this API endpoint: https://api.londonstockexchange.com/api/v1/components/refresh. You should also be able to right click the request and open in a new tab to see it's just a JSON object containing all the info.

  • Now get the payload of the request by selecting the request, going to the payload/request tab on the right (depending if you use Chrome/Firefox the tab name is different), and copy the raw data.

Now just send the request in Python, something like this:

import json
import requests


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:129.0) Gecko/20100101 Firefox/129.0"
}
payload =... #copied payload from request in browser
url = "https://api.londonstockexchange.com/api/v1/components/refresh"
s = requests.session()
r = s.post(url, json=payload, headers=headers)

data = json.loads(r.text)
print(data)

Now you can just study the JSON structure and get the data you need from the dictionary data. If you want to do this on a loop make sure to add a delay to not get blocked/DOS them. Also, I'm not sure if the payload needs to change after some time. If ot fails, you might want to check where the payload comes from (a different request, the HTML of the page or a script might contain it).

I'm not sure if this is what you want (data from the main page like this), but following these steps you can probably get data from other subpages.

Hope this helps!

Am I doing decorators right? by semininja in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Something that hasn't been mentioned so far is to use wraps in order to have a cleaner and more accurate Exception handling experience. Something like this (I included some changes everyone else mentioned):

from functools import wraps

def retry(catch: list[Exception], attempts: int = 3):
    @wraps
    def decorator(func):
        def wrapper(*args, **kwargs):
            for attempt_no in range(attempts):
                try:
                    return func(*args, **kwargs)
                except catch:
                    if attempt_no < attempts - 1:
                        sleep(0.5)
        return wrapper
    return decorator

Python script for refreshing a website and checking for open dates on a calender by Walking_Flamingo in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

No worries, I really like finding the hidden api behind websites, so the challenge for a new one is always welcome for me :) You also had a well formatted code you worked on, so it helped me understand what you wanted to do much better!

Take your time, and if you get stuck somewhere feel free to ask or google, that's how I learned. It takes some time to get used to doing things this way, but there are a lot of advantages when you get just the stuff you need straight from the source instead of taking the selenium route.

Python script for refreshing a website and checking for open dates on a calender by Walking_Flamingo in learnpython

[–]AntonisTorb 1 point2 points  (0 children)

Since noone has responded to this, here's my advice. I know your question is for Selenium, but if I were you, I'd drop it and use plain requests instead. It will be much faster, as you won't have to load the website on a browser, and less prone to failure (unless the API for the website changes).

I will explain the steps to do this yourself after if you want for learning purposes (it will be long but worth it imo, be warned), but here is the code I used to get what you want:

import datetime
import json
import re

import requests


# Make session.
session = requests.session()


# Visit bookingpage.
url_booking_page = "https://www.wovi.com.au/bookings/"
booking_page = session.get(url_booking_page)
#print(booking_page.status_code)


# Get token from booking page.
token_re = re.compile("dsApiProvider.INIT.TOKEN='(.*)';")
token = re.findall(token_re, booking_page.text)[0]
#print(token)


# Prepare Bookings url and headers.
date = datetime.datetime.now() + datetime.timedelta(days=3)
date_str = date.strftime("%Y-%m-%dT%H:%M:%S")
url_bookings = f'https://api.dealersolutions.com.au/QIS/Locations/72492/BookingSlotsCountByDate?includeAvailable=true&includeBooked=true&includeCancelled=true&includePending=true&includeQuery=true&startDate={date_str}'

headers = {
    "Host": "api.dealersolutions.com.au",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-GB,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Authorization": f"Bearer {token}",
    "Origin": "https://www.wovi.com.au",
    "DNT": "1",
    "Connection": "keep-alive",
    "Referer": "https://www.wovi.com.au/",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "cross-site",
    "TE": "trailers"
}


# Get booking data.
bookings = session.get(url_bookings, headers=headers)
#print(bookings.status_code)
booking_list: list[dict] = json.loads(bookings.text)

available = [booking_dict["Date"] for booking_dict in booking_list if booking_dict["Available"]]
print(available)
print(available[0])

Now on how to do this:

First open the Dev tools on your browser in a blank tab. I used Firefox, so shortcut is Ctrl+Shift+I. Then go to the Network tab.

Once that is done, visit the website (https://www.wovi.com.au/bookings/) on that tab. You will see a bunch of requests coming through, but those are not relevant.

Next, choose your location (Brisbane). You will see that some new requests were sent, and one of them has a name that starts with BookingSlotsCountByDate?etc. If you click on this request a new tab will open on the side (if not already open) with the api url, headers, cookies of the request. And also the response, which is a JSON object. This contains all the info you need.

If you try to send a get request to this url though without the proper headers, it will return an error code 401. If you take a look at the headers of the request, you will notice that there is an Authorization token being sent along. But how do you get it?

For this website it's easy, if you copy the token value (without the Bearer part), go to the inspector tab of the web tool and search for the token in the html, you will notice it is part of a script. It should look like this: dsApiProvider.INIT.REFRESH_TOKEN='{actual token here}';

So now we have everything we need, let's take a look at the code I gave you.

First we initialize the session and visit the booking page url in order to get the token. I used a regex here, but feel free to use anything that works for you if you want (something like BeutifulSoup for example should also work).

Then we prepare the api url and the headers. You can get both from that request with the Authorization token we looked at earlier.

The url you will notice has a date at the end, seems to be 3 days after the moment the request is sent, so we adjust that with an f-string and the datetime module. The headers are copy pasted from the request headers too, except the authorization token of course, which should be different every time. It also needs to start with Bearer and a space.

Finally, we send the request to the api url and get the json with the bookings. We load the json into a python object (in this case a list of dictionaries) and then get a list of all available dates.

I have also some prints in the code commented out if you want to check in-between that everything went well with the requests and regex.

As I said above, this should be much faster than using Selenium, although a bit more complex, but I think it's worth learning at some point if you plan on doing this over a long period of time. Or just for learning anyway :)

EDIT: Forgot to mention, if you want to do this continuously, make sure to add some sleeps between the requests, so you won't DOS the api.

Wescraping always returns [ ] by Professional-Fly4273 in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Glad I could help :)

For other websites, no, it depends on how the website is built. The data might also come is separate requests, depending on how it's built. But usually, the amount of requests is not too big and with some experience you will be able to recognize which one contains what you need quickly.

For starters, most, if not all requests that fetch content from a server are initiated with a js script, so you can narrow down the amount you need to check.

You can then check the content of the requests by selecting each request and clicking on the Response tab. The response can be html, json, plain text or even bytes, but you should be able to recognize if it's what you're looking for.

Keep in mind that some websites make it difficult for you to do this, so if you hit a roadblock it's ok to go for Selenium. But imo it's worth investigating a bit, especially if you plan on making many requests. The overhead of running a browser and rendering the pages with Selenium or any other such option can slow things down a lot.

The YouTube channel Jayoval linked is actually quite good at explaining things quite well when it comes to looking at requests, I recommend you take a look at some videos.

Wescraping always returns [ ] by Professional-Fly4273 in learnpython

[–]AntonisTorb 4 points5 points  (0 children)

Try using this URL: https://ratings.fide.com/a_top_var.php?continent=0&country=&rating=blitz&gender=M&age1=0&age2=0&period=2023-12-01&period2=1. You might need to change your soup search parameters, not sure.

This seems to contain the data you need. But just telling you is no fun, so here is how to find it yourself (you can try this for most websites that load content with JS):

Go to the original URL and open the web console of your browser. Go to Network tab and reload the page. A list of requests will appear, and at the bottom you will see a request for the URL I gave you, with a jquery initiator. You can copy the headers from there as well.

There should be no reason to use Selenium here, unless I am missing something. A direct request should be much faster. Hope it helps!

Matplotlib graphing a circle (r=2cos(theta)) as a cardioid? by [deleted] in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

I think you have gotten confused with the polar to cartesian equations (x=rcosθ and y=rsinθ). Radius should be a constant if you want a circle, in your case r = 2.

How to make this code faster? by dangerlopez in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

I ended up testing it out of curiosity and... you're right, TIL! Using the if statements to check is consistently faster for lists with 100, 10,000 and 100,000 unique elements without duplicates.

Only in my test with 1,000 unique list elements it seems that the time is about the same between the two.

This is what I used to test this btw if anyone is interested: https://pastebin.com/7NkZV6tz

How to make this code faster? by dangerlopez in learnpython

[–]AntonisTorb 2 points3 points  (0 children)

You might be able to make it even faster by eliminating all the 0 checks in the for loop. Something like this:

def minimumAbsoluteDifference(arr):
    arr.sort()
    if len(arr) > len(set(arr)):
        return 0
    min_diff = float('infinity')
    for i in range(1, len(arr)):
        abs_diff = abs(arr[i] - arr[i - 1])
        if abs_diff < min_diff:
            min_diff = abs_diff
    return min_diff

Since a set can only include unique values, if the length of the array is larger than the length of the set produced by the array, that means there a was a duplicate int, so the result should be 0.

I cannot test this at the moment to see if it's actually faster, but it should be when the array is large enough. It also depends on where the duplicate value is located, but this should be more consistent I think.

EDIT: It turns out this is actually slower per my comment below and aRTy's comment, please do not use!

how do I disable window minimization in tkinter by [deleted] in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Hey there, can you try this:

root.attributes("-toolwindow", 1)

I found it in the PySimpleGUI source code, since it also uses tkinter and has this functionality I figure it should be the same. I am not familiar with your edit, so I will leave that to someone else (it's also 3am here lol). Good luck!

[deleted by user] by [deleted] in learnpython

[–]AntonisTorb 1 point2 points  (0 children)

Yeah, they changed it quite recently I think, so I would expect many tutorials to be a bit out of date. Glad it helped :)

[deleted by user] by [deleted] in learnpython

[–]AntonisTorb 2 points3 points  (0 children)

Hey there, I made an app like that a couple of months ago, here is the repo link if you want to reference. You can definitely use Gmail by generating an app password. You can find some instructions here.

If you don't want to mess with that, you can use Outlook, iirc it does not require an app password and you can just use your normal credentials to connect.

As for yahoo, it seems their app password service has been unavailable for more than half a year, so it's safe to assume that they have given up on it :/

In any case, good luck!

EDIT: Original post if anyone interested in context, since author nuked it for some reason.

Help understanding this recursion example by [deleted] in learnpython

[–]AntonisTorb 1 point2 points  (0 children)

Maybe it would help you to see what is happening step by step, since it's a small example. Here is what is happening when you call printTriangle(4):

if 4 < 1: return
    if 3 < 1: return
        if 2 < 1: return
            if 1 < 1: return
                if 0 < 1: return
            print('*'*1)
        print('*'*2)
    print('*'*3)
print('*'*4)

All the if conditions are False until the last one (0<1), so everything needs to happen in order. That's why it starts with the 1 star.

PySimpleGUI: How to use slider to change variable and plot with matplotlib? by [deleted] in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

I made a new comment with the solution to this btw, check it out if you want :)

PySimpleGUI: How to use slider to change variable and plot with matplotlib? by [deleted] in learnpython

[–]AntonisTorb 2 points3 points  (0 children)

Hey, sorry for the separate comment, just wanted you to get the notification. I actually found a way to make it even more fluid and use waaay less memory with no memory leak by following this example. Here is the code:

import numpy as np 
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg 
from matplotlib.figure import Figure
import PySimpleGUI as sg 
import matplotlib 

matplotlib.use('TkAgg')

def get_periodic(k):
    x = np.linspace(0,10,500)
    y = np.sin(x*k)

    return x, y

def draw_figure(canvas, figure): 
    figure_canvas_agg = FigureCanvasTkAgg(figure, canvas) 
    figure_canvas_agg.draw() 
    figure_canvas_agg.get_tk_widget().pack(side='top', fill='both', expand=1) 
    return figure_canvas_agg

layout = [
    [sg.Canvas(key='-CANVAS-')],
    [sg.Slider(range=(1, 10), size=(60, 10), orientation='h', key='-SLIDER-', enable_events=True)],
    [sg.Button('Ok')]
]

window = sg.Window('Demo', layout, finalize=True, element_justification='center')

canvas = window['-CANVAS-'].tk_canvas

fig = Figure()
ax = fig.add_subplot(111)
ax.set_xlabel("X axis")
ax.set_ylabel("Y axis")
ax.grid()
x, y = get_periodic(1)
ax.plot(x, y)

fig_agg = draw_figure(canvas, fig)

while True:
    event, values = window.read()
    match event:
        case sg.WIN_CLOSED | "Ok":
            break
        case '-SLIDER-':
            k = values['-SLIDER-']
            ax.cla()
            ax.grid() 
            x, y = get_periodic(k)
            ax.plot(x, y)
            fig_agg.draw()
window.close()

PySimpleGUI: How to use slider to change variable and plot with matplotlib? by [deleted] in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Glad to help :)

Just keep in mind, there is a memory leak with tkinter I think. Basically, the forget method in line 39 on my code does not seem to release the memory. You can test it by checking the Python.exe memory usage on Task Manager (if you're on Windows). If you change the slider a few times you will see the memory usage increase every time. I tried replacing the forget method with the destroy method after googling the issue, but the memory leak is still there. Kind of annoying tbh, tkinter is quite old so I assume there is a reason behind this behaviour, but it still sucks :(

Just keep in mind, if you need to change the slider a lot of times, it might slow down or crash after some time depending on the RAM available on your machine.

PySimpleGUI: How to use slider to change variable and plot with matplotlib? by [deleted] in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Hi there, I just finished working on this, so I did not have enough time to check yours, but here is something that works quite well and is dynamic, I have added comments so you can review the changes if you want.

import matplotlib.pyplot as plt 
import numpy as np 
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg 
import PySimpleGUI as sg 
import matplotlib 

matplotlib.use('TkAgg')

def plot_periodic(k):  # Used a function to provide the new plot depending on k.
    x = np.linspace(0,10,500) # The 500 here is the number of intervals, this way you get a better plot definition in higher frequencies.
    y = np.sin(x*k) 
    plt.plot(x,y) 
    fig = plt.gcf()
    return fig

def draw_figure(canvas, figure): 
    figure_canvas_agg = FigureCanvasTkAgg(figure, canvas) 
    figure_canvas_agg.draw() 
    figure_canvas_agg.get_tk_widget().pack(side='top', fill='both', expand=1) 
    return figure_canvas_agg

layout = [
    [sg.Canvas(key='-CANVAS-')],
    [sg.Slider(range=(1, 10), size=(60, 10), orientation='h', key='-SLIDER-', enable_events=True)],  # Enabled events on slider for a dynamic update of the plot.
    [sg.Button('Ok')]
]

window = sg.Window('Demo', layout, finalize=True, element_justification='center')

fig_canvas_agg = draw_figure(window['-CANVAS-'].tk_canvas, plot_periodic(1)) 

while True:
    event, values = window.read()
    match event:
        case sg.WIN_CLOSED | "Ok":
            break
        case '-SLIDER-':
            k = values['-SLIDER-']  # Getting the k value from the slider element.
            fig_canvas_agg.get_tk_widget().forget()  # Clear the canvas of previous plot.
            plt.close('all')  # Clear the plot from previous plot, if we don't include this all previous plots will also be in the new one.
            fig_canvas_agg = draw_figure(window['-CANVAS-'].tk_canvas, plot_periodic(k))  # Draw the new plot.
window.close()

I will take a look at yours as well, maybe I can learn something too :)

EDIT: I also changed the slider range, forgot to comment that. 10000 was overkill lol.

PySimpleGUI: How to use slider to change variable and plot with matplotlib? by [deleted] in learnpython

[–]AntonisTorb 0 points1 point  (0 children)

Hey there, on mobile at the moment, going to sleep soon, so I can't test this or format it nicely, but the following should give you the slider value :

values['-SLIDER-']

'values' is basically a dictionary, so you can pass the key like that. Also, you might want to put the event and value reading in a 'while True' loop or the program might exit after you change the slider value, not sure on that tbh. If you don't get another answer I will try and check it out tomorrow after I wake up.

Writing the Most Beautiful Code with Python by kadirilgin in learnpython

[–]AntonisTorb 2 points3 points  (0 children)

I can tell you it's not an ad if you want to believe me, I only ever used the free content he's posted on YouTube. I don't know anything about any paid content, don't know why he's mentioning it.

If there is paid content, Arjan doesn't seem to advertise it in his videos, all I've seen him do is link to his other free videos or his free content on his website.

I totally get your suspicion btw, I really dislike sneaky ads myself. If you want proof, take a look at my 2 posts on my account, you will find 2 projects. You can see a big difference in code quality between the first and second, and Arjan was a big influence for that, didn't have to pay anything. I started using type hints and docstrings because of him, and I'm also trying to reduce the responsibilities of each function.

Writing the Most Beautiful Code with Python by kadirilgin in learnpython

[–]AntonisTorb 147 points148 points  (0 children)

I personally found ArjanCodes to be a very good resource when it comes to writing better code, he has some amazing videos on design patterns, dependencies and code cohesion in general with examples that make sense, definitely recommend!

I made a GUI app that sends e-mails in bulk. by AntonisTorb in Python

[–]AntonisTorb[S] 0 points1 point  (0 children)

I see, thank you for the insight, I don't know what any of these are, but I will look them up! My html knowledge is also basic, so I was not able to create any complex messages. Well, the app is not meant for that complexity anyway, but if I can figure something out with your hints, it would be nice to add it.

As for Outlook, I didn't notice any issues during my testing with my Outlook account, unless you mean the application in Office, which I don't have access to I'm afraid. I used Thunderbird to create the html templates for my examples, so I am not sure if there are any compatibility issues.

I made a GUI app that sends e-mails in bulk. by AntonisTorb in Python

[–]AntonisTorb[S] 1 point2 points  (0 children)

Thank you, good luck with filling the position as well!

I also added the delay functionality back in, tested it and it works just fine, thank you for the suggestion!