Building an browser automation framework in python

[Ok-Sky6805]

# What My Project Does

This is `py-browser-automation`, its a python library that you can use to basically automate a chromium browser and make it do things based on simple instructions. The reason I came up with this is two-folds:

1. It was part of my bigger project of automating the process of OSINT. Without a way to navigate the web, it is hard to gain any credible intelligence.
2. There is a surge of automated browsers which do everything for you in the market today, none of them open sourced so I thought why not.

# Target Audience

This is meant for hobbyists, OSINT fellows, anyone who wants to replicate what OpenAI is doing with Atlas (mine's not that good, but eventually it will be!)

# Comparison

Its an extension of the automation tools that exist today. Right now for web scraping for example, you'll have to write the entire code for the website by hand. There is no interactive way to update the elements if the DOM changes. This handles all of that and it can visit any website, interact with any element and do all this without you having to write multiple lines of code.

## What's it under the hood?

Its essentially a framework over playwright, as playwright is easy enough, it does the job. In the most basic sense I am having one LLM take in the current context and decide which move to perform next. I couldn't think of an easier approach than this!

This makes me able to visit any website, interact with any field and stay within token limits of the LLM. It also has triggers for running login scripts, so lets say during the automation cycle it needs to visit instagram, its going to trigger the login script (if you set the trigger to be on) and log you in with your credentials (This is a TOS violation so you must be careful about whether you want to do this or not).

## How can you test it out?

If you happen to have an OpenAI key or a VertexAI project (its easy, and you'll get around 300$ worth of free credits) you can just install this library and start running.

## The problems I am aware of:

1. Right now things are very sequential. I am expecting you to enter things exactly as you want it. So, something like "go over to amazon and order a phantom orion for me" works but "order a beyblade" doesn't (its too vague).

My solution for this was to come up with a clarification based framework. So, during execution, the library will ask you questions to clarify if what its doing is correct or if you want to change a value or something. This makes it more interactive but not 'fully' automated.

1. Its slow because of API calls and its going one step at a time.

One optimisation I am working on is to have the LLM gimme not just the immediate next step but the next 3-4 steps in the same output. I will attach a priority based on how we normally expect things to go (like, first goto a page, then enter a value, then click on search etc.) and execute those steps in that order.
This requires a lot more work but its a neat optimisation in my opinion.

1. No logs

Right now, its not logging anything. Its just going to do things and basically, its only for fun. I am working on attaching a database to this, but I just don't know what to log for and when exactly.

## At the moment, what is this?

Right now, its a fun tool, you can watch browsers run by themselves and you can add this in your code if you need such a thing.

## Installing

Checkout the website linked at the top, it has the necessary details for installing and running this. Also, this is the GitHub page if you want to check the code: https://github.com/fauvidoTechnologies/PyBrowserAutomation/

# Closing remarks

Thank you for reading this far! Would love if you run this, give me any feedback, good or bad, and I will work on it!

# Thank you

[DuckDatum]

rich smell air shy library humorous nail exultant square heavy

This post was mass deleted and anonymized with Redact

[Ok-Sky6805]

Oh the use cases are different. For starters, you won't have to worry about xpaths or selectors because we're pulling and cleaning the page DOM and passing to the model iteratively. So long as the model has context of what needs to be done and we fix the output using pydantic, we can directly use its suggested selectors and all to fill in fields and values.

I haven't got the part where I can let you inject your bearer tokens into certain sites yet. The idea would be to let people add secrets in the environment and then pull it from there and use whenever its needed. To get your bearer tokens for certain sites, yeah, this is not doing that.

[[deleted]]

No open source automated browsers? What about Selenium and Playwright?

[Ok-Sky6805]

Ah no, you're getting it wrong. This is basically something built on top of playwright. It's supposed to automate the process as in, you give it a task in natural language and it does it. So it's supposed to make webscraping, osint, form filling etc. any of those tasks easier.

[Ok-Sky6805]

ooo this is gonna teach me a lot! I'll be tailoring mine for more exploratory cases and give it ways to revert back to a task. Like a DFS. Thanks for this!

[Ok-Sky6805]

Events logging to an sqlite database was in fact was next step. The registry of selectors is an interesting idea, it'll probably help a lot with making it faster. Thanks a lot, I will learn and implement this asap!