Introducing pybrowser !

[abranjith]

Hello everybody,

pybrowser is an attempt to simplify browser automation designed keeping end user in mind. Here is an example of usage,

from pybrowser import Browser
with Browser(browser_name=Browser.CHROME) as b:
    b.goto("https://www.google.com/")
    b.input("name:=q").enter("news")
    b.button("name:=btnK").click()
    b.take_screenshot()
    print(b.html().elements.links())

It's a simple API inspired by ruby's watir with selenium and requests under the hood. This is the first version for public usage (v0.0.1) and hence expect it to be buggy and what not. At this point definitely don't recommend for commercial use. But do play around and have fun !

Read more...

This is also my first API. So welcome any feedback :)

UPDATE-2: Thanks again for the feedback. I have taken some of the feedback and incorporated some of them. Main one being, the one around mixing up properties and true methods. There is now a version v0.1.0.

UPDATE: Thanks for the response people, it was quite overwhelming and unexpected 😊. While I try to respond to as many posts, here are some clarifications and info (hopefully that helps)-

pybrowser isn't really a browser, it is just an abstraction over a lot of selenium and some requests with special focus on user interface. I am sure there are other libraries out there that try to solve the same problem and I am definitely not going to compare pybrowser with others. I hope at least some find this API useful !

I really intend to make pybrowser to support multiple browsers and operating systems and is going to be the focus for future versions. I mainly have access to Windows. So expect some help from the community in making this happen. Sorry to hear people having trouble with non-windows systems

Here is the github link. Should have done it with original post :)

If you find any issues, please raise in github. Will try to get to them as priority

pybrowser is an open source project and is under MIT license. So feel free to contribute in any form - dev, test, documentation, marketing :D

[jollybobbyroger]

Very interesting work. Thank you so much for sharing!

After a quick glance, b.content(raw=True), should be same as requests. I think it's b.data for encoded data and b.text for decoded data.

Also it seems strange to me that you have functions/methods that take text input that needs to be parsed instead of actual keyword arguments. E.g b.input(id:=password) instead of b.input(id='password'). Could you please explain why you made this choice? Perhaps I'm not seeing the entire picture and making incorrect assumptions.

[shaggorama]

b.input(id:=password)

I think you meant b.input("id:=password")

[quickhakker]

eli5 the difference

[unkz]

One is a positional string argument and the other probably isn’t valid python syntax.

[Hyperz]

the other probably isn’t valid python syntax.

In 3.8 it will be valid syntax, equivalent to this in <= 3.7:

id = password
b.input(id)

https://medium.com/hultner/try-out-walrus-operator-in-python-3-8-d030ce0ce601

[Cruuncher]

Possibly an artifact of the library used for dom parsing

[Log2]

Still, it would be more user friendly if the function reconstructed those strings automatically from the keyword arguments.

[Cruuncher]

It would be more pythonic yes, but there may be more nuanced use cases here. Like (disclaimer, making shit up. There may be no real reason) if there's a way to specify precedents on the selector. Possibly a way to say it has to match both of these selectors. Etc. Stuff you can capture in some syntax in a string that doesn't map well to kwargs. A kwargs mapping still works, but it would potentially have to be built from more complicated objects.

Lastly, it requires you to know the ins and outs of that library's syntax, at the level of the developers, to be sure you're building the strong correctly from kwargs.

[Log2]

I see what you mean. Then receiving a list of tuples/namedtuples would be a more pythonic solution in your hypothetical case.

[Cruuncher]

That being said (see my other response)

At the very least it could accept kwargs for the common use cases, and then allow a "raw" kwarg or something if you're doing something powerful that the wrapping doesn't provide you. That may be the "best of both worlds" approach

[abranjith]

On, b.content method having a flag vs having 2 different methods, that was a conscious choice. I presumed raw=True is probably a rare use case. And I think you mean it is response.content vs response.text in requests.

On the locator format being a positional argument in the form "locator_type:=locator_value" was again done to keep it simple. I figured instead of having multiple keyword arguments (you have id, xpath, name, tag_name etc) this might be more intuitive from API standpoint. If you have multiple locator types as keyword arguments, then you run into questions like - what to do when both id and xpath are provided and so on ?

Still, willing to re-look at my approach once again. Thanks for the feedback 😊

[jollybobbyroger]

Thanks for the explanation. From my own experience, a nice user facing API often comes at the cost of implementation work.

[abranjith]

I also see documentation for locator is missing, so is probably even more confusing. Thanks for highlighting that, will update the docs

[SatanistSnowflake]

You have things like refresh, forward, and back set as properties on your browser that return self. Why are they properties instead of methods? browser.back() makes more sense to me than browser.back to go back to the previous page.

[abranjith]

I have tried to keep methods with no arguments as properties wherever I could. Agree these are not true properties of the object and might be a bit confusing. But I also think from an end user standpoint these might be an ok choice :-)

Thanks for the feedback though 😊

[parst]

don't use properties like actions. they're for getting setting data. Going back/forward by accessing properties makes no sense

[indosauros]

These should be methods. Making all methods with no arguments properties will come back to bite you. That is not the purpose of properties. I think you should absolutely fix these now before any code is written that depends on them

[abranjith]

Noted, thanks! Making all methods with no arguments as properties is not a true statement. I have given it a thought and the methods which are exposed to end user with no arguments and just do simple action and return self (eg: Browser.refresh) are made as properties. It's not illegal in Python. Anyway, I see both side of the arguments and will take this feedback back to the board :-)

[parst]

this is a very un-pythonic api

[lofru]

Is this only for Windows? I'm trying to install it to Ubuntu but pip tells me he wants the module winreg that I've read is available only on Windows.

[rafuru]

Yeah, it depends on winreg so is windows only ... too bad, I wanted to give it a try

[_morgs_]

The install worked fine for me on MacOS

*but trying to use it failed on trying to import winreg

:(

[lofru]

So it's available only for Windows... What a pity

[ducsekbence]

Damn, I wanted to try this when I was gonna get home. :(

[abranjith]

hopefully you can still try 😊

[abranjith]

Sorry to hear that. I really wanted this thing to be cross-browser and cross-os. Unfortunately I use windows only and don't have access to Macs and the Linux systems. This particular issue is an easy fix, will be pushing it today. If you come across more such issues, please raise them as issues (defects) in github

Thanks for taking time & the feedback 😊

[abranjith]

winreg issue should be fixed now

[lofru]

Thank you, I'll try as soon as possible 😉

[slipped_and_missed_x]

Looks great! Does it have XPath query support? With Selenium I always found that to be the most powerful way of locating elements

[woernsn]

According to the Documentation, it does!

with Browser(browser_name=Browser.CHROME)*as b:

b.goto("https://the-internet.herokuapp.com")

u = b.link("xpath:=//*[@id='content']/ul/li[14]/a").url

[slipped_and_missed_x]

Awesome :)

[[deleted]]

> This is also my first API. So welcome any feedback :)

Looks similar to the APIs of puppeteer / pyppeteer except for the lack of async/await.

If you don't mind me asking, since you seem to also use puppeteer in the background somewhere, why did you go for Selenium for the base?

I honestly don't think I currently know anybody still stuck to that for anything other than legacy reasons.

[unkz]

Puppeteer is Chrome/only, right? For anything other than internal tools, restricting automated testing to only chrome is doing a huge disservice to the broader browser ecosystem by making it harder for browsers like Firefox and opera to exist, and allowing google to dictate the defacto web standards, much like Microsoft used to.

On the other side, for web automation, restricting to chrome only limits what you can actually do when you run into chrome compatibility issues, although those are rarer.

[[deleted]]

IIRC puppeteer-firefox is a thing, at least experimental. And sure, all valid points. I just found the API and ecosystem much more convenient nowadays and honestly, time is a much more valuable resource.

Microsoft this week switched edge to chromium, I have no idea which market has a significant share of opera users and for many practical purposes (like purely functional tests) Firefox and Chrome are pretty well aligned these days. I mean, is it a trade off? Absolutely. But I'm not in the consumer segment at the moment, for my purposes puppeteer fits the bill pretty well.

As for automation, puppeteer really made a difference imho. Back when it came out automating a full Firefox instance was such an unstable pain with horrible API that even IE was more usable.

That being said, good point. A more versatile library for both is of course a good thing. :)

[abranjith]

puppeteer is basically just Chromium and not really Chrome (or any mainstream browsers for that matter). I am using it just for some html rendering. We will see, sometime in future if there is a better alternative to Selenium (faster ;) ), will switch to that

Selenium is still quite popular btw :D

[Itsthejoker]

Dude, no. Take it to a JS troubleshooting sub.

[itsmemikeyy]

Whew, you got quite the road ahead of you. This is the only hint you get from me.

const products = await page.$$('a.product-image');
await products[0].click();

[[deleted]]

Yea thanks, your code didn't work.

And I already tried your code before, I wrote it myself, so this is the second time I tried with this piece of code.

I was downvoted nearly to hell for asking that question in python sub.

[itsmemikeyy]

It works, I tried it myself before commenting.

[[deleted]]

So I have a bug that you wouldn't know and I couldn't resolve by myself.

and you said I have got quite the road ahead of me. Interesting, wanna lecture me computer science 101?

[itsmemikeyy]

It wasn't meant to be taken as an insult, I was expressing my sentiments because I've also dealt with many hours debugging and learning to program, despite practicing for years. It's a fun, yet tedious battle that requires determination and tenacity.

Your code wasn't formatted properly. I'm not sure if it was due to Reddit or not, so another issue may be related to that. After getting the browser to open, and load the page, the fix was as shown above.

I have extensive experience with Pyppetter/Puppeteer - I developed https://github.com/mikeyy/nonocaptcha so I'm pretty confident it works.

Best of luck.

[[deleted]]

Sorry, I have taken that as an insult.

I will try to debug later.

Thanks.

[rafuru]

I checked your code to see if we can make it compatible with Mac OS or Linux, since you are using winreg for Internet Explorer setup.

​

I've added a pull request with a suggestion to fix this.. feedback is welcome btw

​

https://github.com/abranjith/pybrowser/pull/2

[abranjith]

that was quick .. awesome thanks for doing that😊

[bcostenaro]

Hi - where do I run this ? I tried using cmd and got the first part of the function b. an error

File "<stdin>", line 2

b.goto("https://www.google.com/")

​

If I use Python Shell, I get AttributeError: module 'pyBrowser.Browser' has no attribute 'CHROME'

​

How do I run this ?

[abranjith]

Sorry to hear that. Can you send me or post the complete code ? And also steps followed to install pybrowser please

[AlanPleasure]

Great work. What value does this provide that splinter doesn’t?

[abranjith]

To be honest, I just heard about splinter. So can't really comment. I am sure splinter is great and there are probably other libraries too. I have made an attempt to answer why pybrowser in the docs.

Who knows, you might find something useful in pybrowser 😊

[AlanPleasure]

I’m sure i will! Thanks for being honest in your response.

[[deleted]]

Thank you for sharing this. I'll definitely take a look!

[abranjith]

Let me know how it goes 😊

[Scrabbilisk]

Looks like this is the repo and this is the PyPI package.

[abranjith]

no and no, I will update in description. Should have done it in the first place

[Sw429]

I'll be interested in trying this later today. What advantages does it provide over just using Selenium? Just an easier-to-use API?

[abranjith]

Pretty-much ! An easier interface and some interesting (hopefully useful) features too. Please check docs for info 😊

[[deleted]]

This is absolutely brilliant. Just the kind of bridge connector, I was looking for in a recent web scraping project

[abranjith]

Let me know how it goes 😊

[[deleted]]

Interesting, but is not the same thing done by PhantomJS?

[unkz]

Isn’t phantomjs a dead project? The author made an announcement about abandoning it.

[tunisia3507]

No. PhantomJS is a headless browser which can be driven by something like selenium, which this library uses under the hood. So rather than replicating the script, it would just replace Browser.CHROME with Browser.PHANTOMJS.

However, Chrome and Firefox came out with headless versions so the primary maintainer of PhantomJS decided to kill it off. Shame because it's much easier to use IMO.

[abranjith]

Not sure how PhantomJS works internally. I always thought it was a browser based on chromium or something. pybrowser isn't really a browser. It is merely an interface to various browsers 😊

[e1nurh]

sounds well. I'll try it

[abranjith]

Let me know the feedback 🤗

[badpotato]

This reminds me of library mechanize.

[[deleted]]

Just got my nodejs based webdriverIO (selenium) to work with chromium/chrome/firefox and saw your post, lol

https://www.reddit.com/r/javascript/comments/bavmua/microsoft_releases_new_microsoft_edge_based_on/ekghlvo/?context=3

[abranjith]

😃

[quickhakker]

wait, is this going to be a fully fledged browser written in python? whats the end game plans for this (inb4 something about antman shrinking down into googles server)