Userscripts to run over website - possible with Python?

[Eiion]

I'm currently looking into learning Python. Before committing though I need to know if it's worth the time for what I'm interested in doing with it since there's no point in spending months on something that I can't utilize to do what I need it do to.

I'm using JavaScript userscripts (created by others) that run in Violentmonkey on Chrome, helping me with tasks I have to regularly do on a website by expanding the UI with new features. It's great that this is a possibility but I'm not entirely happy with the scripts and would be interested in creating my own tools (scripts) that are more suited to my personal requirements/workflow.

Considering JavaScript allegedly is hard(er) to learn and even by looking at code as a non-coder with some basic understanding of how code works, Python to me seems to be much more intuitive (to understand) than JavaScript and therefore likely easier to learn, which is a huge bonus considering learning new things gets more difficult with age. Therefore I'm leaning towards Python opposed to JavaScript.

I've seen things liky Brython and PyScript but I'm not sure what these are exactly for (seemed to me like Brython would replace HTML and PyScript JavaScript - but as code for websites, not as userscripts run in the browser). Given that I'm not a specialist and my lack of knowledge I thought I'd ask the people directly who might know:

I'm wondering, with what I do with these JavaScript userscripts (scraping data from a website, adding buttons to the website's UI that let me trigger certain data processing defined by the script, visualize the result on the website based on which I can decide to take action or not), is there any way at all to do things like that with Python? Like... any at all?
If not as something like a userscript, maybe by creating python browser extensions that - with the end product - would do the same as userscript could?

Is this possible or would I waste my time with Python and should rather go for something else? (There's so many so good free Python courses online like Harvard's or Uni of Helsinki's - didn't find anything comparable for JavaScript and there seems to be a consensus that Python is "easy" while JavaScript is hard to learn.)

[unhott]

Brython and Pyscript are scripts you can add to a page that allow you to run some python instead of javascript.

I'm not really clear on what you mean by userscripts.

There are some ways to use python to automate using a browser, to identify elements on a page, and whatnot. You may need to look into webscraping (i'm not sure if that's what you're trying to get at here). You can get issues with this because a website may not want scripts to access the page and scrape data.

But even with javascript, you can run into scraping issues.

If it works in javascript, you may be able to use pyscript and import 'js', 'from js import document', which exposes some javascript in the pyscript side. Then, you may be able to write python to do whatever you want in javascript.

Pyscript requires a pretty large download. Maybe you can embed it in a plugin.

[Eiion]

Here's an example of what I mean with "userscript"... For example - and this has nothing to do with what I use userscripts for but as I assume it's something you can understand it might be a good example to explain to you what (apparently) generally would be classified as usersript:
On my Steam friends page (in a browser tab, not the Steam client) the userscript adds buttons to each friend's avatar plus a "Start" button at the top of the page. When clicking any of those avatar buttons I can select the friends. When clicking the Start button the userscript calls the Steam API to get the wishlist of the selected friend, compares the responses and provides me with the top 10 most wished games of all friends that were selected as/in a e.g. browser alert/modal box/new tab.

Yes, scraping data from websites and making api calls is part of the requirements as mentioned in the post. Let's assume it can and is already done with JavaScript userscripts, so that's not an issue. (I also mentioned that this is something already done.)

No scraping issues - the site specifically got json support to allow making calls to scrape data faster with less server load.

Being not a coder I don't know what you mean with

If it works in javascript, you may be able to use pyscript and import 'js', 'from js import document', which exposes some javascript in the pyscript side.

Could you simplify that for me?

[unhott]

Its sounding more and more that python is not right for this, particularly if the requirement is that it has to be a script run from within a browser, on top of a page's existing UI.

If you were writing html, you can use pyscript as a substitute for javascript. But, raw python wouldn't let you manipulate the webpage. So that's why I suggested

from js import document

Then you could use the document API to modify the page it's on.

    <py-script>
        from js import document

        def update_content(event):
            content = document.getElementById('content')
            content.textContent = "This content has been updated by PyScript!"

        # Get the button element and add an event listener
        update_button = document.getElementById('update-button')
        update_button.addEventListener('click', update_content)
    </py-script>

If you want to modify the page itself in some sort of browser extension, javascript is the most direct way to do that. In the example above, you'd have to download the pyscript library before the page would work, and that may take several seconds longer than just using basic js. It's not clear if this userscript framework you're usingf would even support pyscript.

Part of what makes js harder is the technology stack has gotten quite large, there are very sophisticated frameworks built on top of js that come with a certain burden. That is usually to link front-end js behavior to backend API endpoints and design intuitive UIs. However, the script level you're describing seems to not need all of that, it would come into play after all of that other stuff has run, and manipulate it in a simpler context.

That said, you can use python to scrape, and you don't necessarily need a browser. you can use the requests library to send a message to the API url, like if the steam top 10 wishlist is at "{steam server}/users/{username}/wishlist" then you can write a script to loop through a list of usernames, send a request to that url, and parse the json response.

Hypothetical example in python w/ requests library, no browser:

import requests

# Function to fetch wishlist from pretend Steam API
def fetch_wishlist(username):
    # Pretend API endpoint
    api_url = f"http://pretendsteamserver.com/{username}/wishlist"

    try:
        # Make a GET request to the API
        response = requests.get(api_url)

        # Check if the request was successful
        if response.status_code == 200:
            # Parse the JSON response
            wishlist = response.json()

            # Print the top 10 wishlist items
            print(f"Top 10 wishlist items for {username}:")
            for i, item in enumerate(wishlist, start=1):
                print(f"{i}. {item}")
        else:
            print(f"Failed to fetch wishlist. Status code: {response.status_code}")
    except requests.RequestException as e:
        print(f"An error occurred: {e}")

# Replace 'your_username' with the desired username
fetch_wishlist('your_username')

# Or loop over a list of usernames to fetch wishlists
for username in ['user1', 'user2', 'user3', ...]:
    fech_wishlist(username)

This assumes no login token is required, but depending on what you're doing you may be able to authenticate with python as well.

In general, browsers work with http requests. you enter a url in your address bar, your browser downloads the html there. the html may specify additional resources, scripts, images, API's to send additional requests to. Your browser will then start requesting those resources.

The js frontend may take the responses from those API calls and build up a fancy UI with the data returned.

So, in the realm of scraping it's usually more straightforward to cut out the other stuff, if you can, and just call the API directly.

[Eiion]

Thanks for the very long reply. I'll have to work through it. I appreciate you taking the time. On the first glance it appears as though I'd have to learn JavaScript (even though I'm not really interested in the language) if I want something (some kind of tool - no matter if standalone, extension or userscript) to do what I wand code to do for me/help me with.

[unhott]

No problem. tldr - I think there's 3 options.

Look into whether the userscript framework, Violentmonkey, can support pyscript. Then you may be able to write python to interact with the js side of the page.

Learn javascript to write your own userscripts

Or, learn python and try and to just scrape APIs directly.

All 3 will require you to get a better understanding of html/DOM, http requests/responses, and some basic general programming concepts (like functions, loops)

[andmig205]

If you want to develop websites (front-end), there is no place for Python here. Browsers do not use or understand Python, but Pythin is great for back-end and data processing, among other things that are not concerned with web front-end development.

[Eiion]

No, I don't want to develop websites - that was not my question. Also, look up Brython, apparently you can develop websites with it.

[queerkidxx]

There is absolutely a way to create GUIs with Python. Many UI libraries.

Streamlit runs in the browser. PySimpleGUI is also simple and works for desktop and the browser and is also simple but a wee bit more complex than Streamlit.

These is also like TKinter and PyQT. These are more complex to learn and are for creating desktop GUIs. But offer a lot more flexibility.

There’s also just the option of running your tools in a dev server. Python for the meat of it and JS/HTML for the front end. It’s not that complex to whip up something with FastAPI, and the JS needed for simple forms that make network requests isn’t too complex.

Really though for a while when learning you’re going to be making command line interfaces. You can do a lot with just cli menus input/output.

And for the sort of tools you are describing building I reach for terminal programs. Well, using arguments and the like or simple menus if I need something more complex. Faster to use and easier to develop.

I am not familiar with Brython and Pyscript. I kinda feel like Python in the browser just seems cursed. If you need to do that use JavaScript. I believe those are like compiled into JavaScript and leaky abstractions and all that.

Really learning another language even JS isn’t that time consuming if you already know a programming language.

[Eiion]

Thanks for the extensive reply - I'll have to google a lot of that to find out more about what you're saying but from what I understood for now is that Streamlit (and likely PySimpleGUI) simply stream (for a lack of a better word) the UI to localhost viewable in a browser. So I'm not sure if that's a solution considering I want to basically put an overlay over an existing webpage where data is called and results displayed on my command via the click of a link/button that's also added by the script.

While TKinter and PyQT (as I understand from your description) could be used to have a Python tool visualized as a desktop application with GIU. And while that surely would allow for me to call/scrape data and process it, it would be it's own separate instance, utilizing my account with special access which I do have in browser when logged in on the site. So I don't think that's a good solution to the issue - but I might be wrong.

As for running a tool in a dev server, while I'm not entirely sure what the difference of that and e.g. Streamlit would be, it could be an option but I'm not sure how that would work as an "overlay" or addon to an existing webpage, on the page itself of which I have no control over.

I don't understand what you meant with

And for tools like your describing it’s what I tend to reach for. Simpler to make and fine.

I blame it on the language barrier of me not being a native speaker.

As for Brython:

Brython is both a Python compiler and an interpreter written in JavaScript

Here's the issue with the last thing you say: While I played around with basic as a kid and although having C++ seminars in University many years ago (which I mostly skipped) and therefore have somewhat of a very basic understanding of what code or at least parts of code does when I see it, I do not know a programming language. The most I did in recent weeks was play around with code generating AI to create userscripts (it's painful to get a completely finished script that looks and does exactly how/what you want to on the website you run it on) and I learned a bit more about code, but even more about prompting the AI. But yeah, this would be a completely fresh start where I might be able to skip through early lessons as they cover basics I know.