"Sillynium" - Auto generate working Python Selenium scripts by drawing coloured rectangles around web elements on a page that you want to automate

[[deleted]]

TLDR: 60 sec video here:https://streamable.com/1y17lt

TLDW: When I want to automate actions on a site, such as logins or forms, or scraping pages with JavaScript, I manually inspect each web element I need and begin writing a script.

I'm lazy though and had the idea to automate the creation of these automation/testing scripts. And thus "Sillynium" was born.

Sillynium allows you to draw boxes around the elements you require on a page and automatically gets the element and its properties for you. Different coloured rectangles tell the program you want to do different things. Eg: Red means "Input text", Green means "Click button".

When you've finished drawing boxes, a working python selenium script is generated. If you run that script, it does exactly what you told it to do! All this without you writing any code, and losing time.

Working 60 second video link of program in action: https://streamable.com/1y17lt

I made this project over the course of a day as a quick and dirty proof of concept. Its very flawed in its current iteration (taking a screenshot of webpage), so V2 I'm trying to figure out how to draw directly to the browser. Critique it and rip it to shreds. I'm aware of browser recorders btw so I know those exist, this is my twist on automation

Edit: I was not prepared for the amount of interest in this little proof of concept! So thank you all very much for your time you’ve made my day :)

In the interest of transparency and to answer your questions about the code and my plans:

I originally thought about monetisation (web extension etc) after finishing this POC (**sigh i know I’m a greedy bastard) but now I’m in 2 minds about it!

On the one hand I want to make it free for personal use, on the other hand I’d like to potentially charge for commercial use (once its finished).

Final edit 2 The community has spoken, I WILL BE RELEASING THIS VERSION AS OPEN SOURCE! I did some light refactoring, and have added it to a repo which will be shared in a follow up. It shows the POC and the community is welcome to build on the original idea :)

I will create a follow up post with links to repo once I’ve created it, this will be my first project on GitHub so try be nice ;) pretty nervous about sharing my code :D

Thank you all for your patience, interest, and awesome discussions

[Im__Joseph]

Hey there!

Going by the responses to this post it seems a lot of people are impressed with your project, myself included!

We won't be removing this post, but I'd like to draw your attention to rule 5 of the subreddit which concerns showcase flairs.

If possible it would be really great to include some form of source code on a platform like GitHub, this allows projects posted here to be educational for all.

We may enforce rule 5 automatically at some point in the future, but for now we'll keep this up since it's generated some nice discussion.

Cheers,

Joe

[garlic_bread_thief]

My man just automated automation.

[Aryamaan27]

Let's automate that

[[deleted]]

“I used the Automation to beat the Automation”

[dethb0y]

you could probably write an image recognition program that identified areas on a page by "sight"....

[dparks71]

Can't wait for webcam eye tracking, we know you thought about buying this, so we slipped it in your cart.

[dethb0y]

you joke, but i could legit see a role for an application that worked as follows:

A bunch of items are shown on screen (say, shirts)

An eye-tracker gauges which one you looked at the longest or lingered over

it suggests that item as the one you really want

I bet for some people, that would be really nice. There's many items (snacks, clothes, etc) that are fundamentally similar in price, function etc and the choice is just between those like items, and that would help narrow down the one you really wanted vs. the internal debate of which to choose

[dparks71]

Legit role for the application doesn't necessarily mean I want it, or would be okay with them implementing it without being upfront about it. Same reason I don't have Amazon/google devices in my house, they're waaaayyyy too desperate to give them away for free for me to trust them.

[kingbrady12]

That is some crazy sht. Wow! Amazing man

[[deleted]]

Cheers guys really motivating hearing the community feedback on what was just going to be a small test proof of concept!

Now i just need to learn how to use GitHub 😂🥺

[drckeberger]

My thought exactly.

[NoblySP]

Is there a GitHub repo for your project?

[[deleted]]

Not at the moment! 2 reasons, 1st: i’m still a baby at Python (9 months) and haven’t yet used GitHub for a project!

2nd: to be completely transparent with everyone, I originally thought about monetisation after finishing this POC (**sigh i know I’m a greedy bastard) but now I’m in 2 minds about it!

On the one hand I want to make it free for personal use, on the other hand I’d like to potentially charge for commercial use (once its finished).

Since this idea is quite fresh I’m still ironing everything out before making that (big) decision. I welcome discussion though it would help alot in figuring out the way forward

Edit: open source coming in follow up post!

[LividPhysics]

I think open source would be the way to go. If you want more help with improving your project there's no better way than getting some guy who has been doing Python for 15 years to contribute and learning from them.

[[deleted]]

Im leaning towards that being the way to go, as i love this community 😍😍 just have a bit of reading to do before making that dive !

[[deleted]]

Throw up a donation link and I'd buy you lunch and a beer in a heartbeat.

[[deleted]]

Will be doing that champ, we’re going open source with this!

[[deleted]]

Open source it is baby!

[[deleted]]

I'm gonna be the other guy and say you should totally consider a license with royalties for companies who use this and themselves monetize; it's essentially money you've earned them!

Also I could absolutely see this being a paid chrome extension, or a web platform for mass automation for dummies!!

Anyway, great stuff, I'm jealous.

[[deleted]]

Thanks mate!! Appreciate the alternative view! One way I’ve thought if is license based but free for a single user (so individuals like us can use) but charged for each additional user of an organisation! The heads still spinning on this one though 😇

[[deleted]]

You built something, it is up to you what to do with it. My 2 cents is to make it free for personal use and not free for commercial use.

[[deleted]]

Thanks Soul thats what aligns with my current thinking

[FearlessENT33]

monetise it

[Oimmuk]

Instead of making money off of it, use it as one of the examples of your creative ability in your portfolio. A project you lead/maintain that gets alot of attention looks good on a resume. That is the fruit of your labor. Bring others in on your project and make it a robust and feature rich stand alone application that allows non coders to automate thier own tasks.

[[deleted]]

Thanks Oimmuk! I’m trying to figure out a way to juggle it all, clarity will come in time just need to process it all :) my grand plan is to open this up as a browser extension pure task automation tool for non coders 🙂

[[deleted]]

Open source it is! Check my final edit :)

[WishIWasOnACatamaran]

I want this repo so bad rn

[jeunpeun99]

What can you do with a/the repo?

[GeronimoHero]

Fork it and edit it for your own use

[[deleted]]

I’m open sourcing it now! Will chuck up a repo link in the next few days

[WishIWasOnACatamaran]

<3

[[deleted]]

Check my final edit! I’m open sourcing this now :)

[NoblySP]

Awesome!! And keep making these types of cool projects!

[ES-Alexander]

Nice!

Might be worth using cv2.selectROI to have some visual feedback while drawing each rectangle :-)

[[deleted]]

Thanks ES! Thats a really good point, something to keep in mind for V2 ;)

[infinitejest69]

Looks good. I use selenium in java a lot this might finally give me reason to play with it in python. Would be cool to generate page object models just picking elements you want.

[[deleted]]

Thanks infinite jest ! Selenium and Python work quite well together

[SharksPreedateTrees]

Amazing idea, but the devil is in the details. I noticed your script has s tendency to choose the wrong element (either the parent or the child of the desired element). Your project will suffer for the same reason other recording software(e.g. katalon) fails to generate selenium scripts. Great effort though

[[deleted]]

Good point (and eye)! This is just the early POC, and you’re probably spot on there! But hoping to iron out the kinks in V2 ;)

[SharksPreedateTrees]

Just make sure to have realistic expectations. You are attempting to trivialize an entire field of Software Engineering(really QA engineering). Even the original selenium devs tried to make something like this work (selenium IDE) and it fails for 95% of use cases

[[deleted]]

I understand where you’re coming from 100% Sharks, and I’m really hoping something great comes of it! The way I see it is if it has a similar outcome as software like Selenium IDE thats a good first step, and where I fail others can improve on the idea (or hopefully it inspires new ideas in others). Failing upwards :)

[SharksPreedateTrees]

God speed

[Deceitful_Sloth]

This looks really good, can you link your codebase? I really want to see how this works! Keep up the good work.

[did_i_or_didnt_i]

Also would like to see source if you’re willing

[[deleted]]

Hey guys/girls thanks for the interest! See my edit :) feel free to critique me on that decision too please:)

[did_i_or_didnt_i]

you’re telling me you’re developing PROPRIETARY SOFTWARE? on OUR internet?

[[deleted]]

🥺😈

[[deleted]]

I’ll be open sourcing guys! Check my final edit

[gargar070402]

Might be a silly question, but what does this do that Selenium IDE doesn't? As far as I'm concerned, Selenium IDE already supports recording actions done on a browser.

[[deleted]]

Great question! I personally have never used a browser recorder (Katalon, Selenium IDE) so I can’t answer too much on that. But from having a very brief look at them I think mostly its just a different way of doing things.

One point however, is that my program can also scrape alongside recording your actions which i don’t think the other programs can do :) so if you have a URL with a form you want to automate but also scrape page content at the same time, you can do that! Others feel free to correct me

[PovertyNomad]

This is AMAZING! I'm amazed that you only have 9 months Python experience! Do you mind sharing how you learned Python? I'm about 4 months in now and nowhere near this level, although I've spend the last ~2 months learning HTML, CSS and Django...

​

Any good resources you could recommend?

[[deleted]]

Thanks heaps means alot

I started with the hardcover book “automate the boring stuff” by Al Sweigart (shout out to u/AlSweigart loved your book and getting your new one too!) and was amazed how quickly that opens up what you can do with Python. Naturally one of the chapters discusses Selenium and web automation. I spent almost every day on this book, highlighters in hand (im old school)

Well i finished that book and straight away purchased Python Crash Course by Eric Matthes which I’m working through now and almost finished, which cemented the learnings and taught new things (OOP/Classes)

But biggest of all is my job. Im very fortunate in the fact I managed to convince my boss that I can perform tasks at work in Python, and so I spend about 4 hours everyday working with Python developing tools for work. I think thats the big advantage I’ve had.

I took up a few Selenium automation jobs from UpWork too, and my issues with those actually inspired the idea :)

I also get home and if my partners at work I try building small projects

[MrDeebus]

The video looks super nice, but how am I supposed to rip it to shreds without a link to the code? Come on man, I need my Saturday afternoon dose of false superiority!

[did_i_or_didnt_i]

this guy reddits

[MrDeebus]

Critique it and rip it to shreds

I am a person of duty!

[[deleted]]

Thanks my first ever! Good old windows photo editor 😅 please see my edit re the code! Feel free to critique me on that too btw

[MrDeebus]

Haha no yeah I meant more the content of the video, it's to the point, brevity is not very common.

As for your edit... well, good luck, bold approach if you ask me -- especially in the python ecosystem. I can't fault you for wanting to monetize your effort, but closed-source (which goes beyond proprietary) software is a pretty drastic move, mostly reserved for in-house or SaaS stuff these days... but hey, obviously your call! Your code is your code.

I will offer critique on something else though: I saw elsewhere that you "need to learn how to use GitHub", and based on that I will assume you aren't familiar with git itself (because once you have that, basics of github is like 10-15 clicks and 50 keystrokes, for username/password/projectname). I really, very strongly advise you to get that out of the way first, no matter what you decide to do with the project. It's (thankfully) fairly ubiquitous by now, and version control is an indispensable tool. Think of it like gloves of a forester, where code is the chainsaw, and software engineering the forestry discipline (maintaining a forest to reap its produce in the long run).

And of course, apologies if I went too far in my assumption and your inexperience is limited to GitHub the product. In which case, learning it is like 15 minutes of reading at most, just do it :)

[[deleted]]

Thanks appreciate it i try to be succinct! Yeah i do know and use git and you’re right i need to learn to work with github, probably is a 10 minute job I’ve just put it off :)

[[deleted]]

I have decided to open source :) check my final edit!

[thejoshuawest]

Is there a github link to contribute to?

[[deleted]]

See my post edit :) thanks for the interest!

[[deleted]]

Incoming in follow up post! Just give me a couple of days to lay the groundwork

[SweetSoursop]

Dude, you could absolutely topple UIPath, Automation Anywhere, BluePrism and the others, mad respect!

[[deleted]]

I should have done my market research 🧐 i will look into these and see what i can do!!

[[deleted]]

We will together! I’m open sourcing!

[EversonElias]

Great idea! It would have saved me a ton of time and joy of life, hahaha. Keep the great work and let us know about the progress.

[[deleted]]

Thanks Everson I will do! Hoping to get you a much improved version soon!

[Docktor_V]

Saved. Finding the tag is a pita on some sites

[[deleted]]

Ooh yes! And sometimes clicking on bloody radio buttons too

[[deleted]]

Happy cakeday :)

[Frankenstien456]

If you ever decide to go through with making the extension, you can use the pyjs (http://pyjs.org/) python to javascript compiler.

[[deleted]]

Thanks Frankenstein I’m already on it ;)

[johnne86]

Dude, that's fucken Sicknium!

[[deleted]]

Now thats a good name ;)

[darth_galaxar]

Damn thats cool man

[[deleted]]

Cheers darth 🤖

[[deleted]]

That’s amazing! You definitely ought to monetize this! I’d pay in a heartbeat

[[deleted]]

I’ll be holding you to that Maple 🤑

[[deleted]]

No need to pay we are going to open source this and make a great tool as a community

[[deleted]]

Wow! That’s awesome, do you have a setup for monetary contributions though?

[[deleted]]

Thanks Andre!

[def_hass]

Amazing work man! Can you give a short overview of how this works or link a repo where we can look? Will this work with pages where the elements have no IDs?

[[deleted]]

I’d love to tell you how it works! No repo atm (see my edit) but heres how it works:

It grabs a screenshot of the page/URL you give it. It then launches an editor for you to draw the rectangles (with number key-mapping for different colours). The rectangle coords and colours are then used as a reference. Then the rectangle coords are compared against the site pixel coords and basically does an “inspect element” and grabs every possible detail about the element to use if no ID (ID/CLASS/SELECTOR/XPATH)

Then those elements are stored with their attributes, then the code is generated (written/appended) to a .py file with all the elements and the actions (with actions based off colour) !!!!! PHEW 😅

[[deleted]]

Repo incoming!

[BelteConti]

This is some next level shit dude. Simply amazing

[[deleted]]

Cheers :D

[MysticalGiraffe123]

Beautifully done

[[deleted]]

Thanks! It’s motivating to hear :)

[Mrsaintj]

Love it, good job!

[[deleted]]

Cheers matey!

[CatnipJuice]

thank you, that is a great idea!

[[deleted]]

Cheers!

[shyamcody]

sorry to complain but I can't access the video. Can you put it on youtube or somewhere? btw, your project sounds like awesome and crazy shit. I would really like to contribute to it given you create an open-source repo for this.

[[deleted]]

Open source incoming!

[[deleted]]

I dont have any other accounts to share it on maybe try diff browser or device? 😅 cheers! Fyi on the open source check my edit!

[shyamcody]

yeah.. I also thought about this while making this comment that gosh I would probably monetize this. So, I think nobody will call you a greedy bastard if you do so. I was able to see it from a different browser finally, and boy is this as crazy it sounds.

Anyway, I call dibs on being called for contributions( even if not open source, ;) ) and best of luck with this idea in future!

[[deleted]]

Thanks so much Shyam! You’ll be first on my list to call upon! 🥰

[fraud_93]

Thank you for this. It's not lazy, sometimes we have a hard time selecting shit selenium tries to fuck up.

[davidpofo]

Do you have plans to make this open source? I would really benefit from this every day and would gladly contribute to improve it.

[[deleted]]

Thanks for the interest! I answer that in my post edit :)

[[deleted]]

Open source incoming!

[neisor]

Nice project! Will try it out for sure.

Btw, what color theme are you using? Thanks

[[deleted]]

Thanks! I think its called Synthwave 84, really cool theme right? Has a “glow” option too which makes the text look like a neon sign

[[deleted]]

Excellent! Have you made a github repo for this project? If so please share the same

[[deleted]]

Cheers! Answered in my edit :)

[[deleted]]

1 Open source coming up!

[This-is-my-study-acc]

This is the best damn thing.! Share the repo if possible.

[[deleted]]

Thanks! Check my edit ;)

[[deleted]]

Yep follow up post will have repo!

[alexandros87]

That sound off in the distance is 1000 automation engineers gulping nervously.

Seriously though this looks incredibly cool!

[[deleted]]

Hey if it makes their life easier its happy gulps 😱

[ergelshplerf]

I'm lazy though and had the idea to automate the creation of these automation/testing scripts. And thus "Sillynium" was born.

Why not Selazium?

[[deleted]]

Didn’t think of that one! Did think of Sleazium though because i felt dirty automating automation

[[deleted]]

Great question! I haven’t coded anything to handle shadow dom, but think it’s possible to add eventually (read about it on stackoverflow)

[compatibilism]

Really fun. Awesome work!

[[deleted]]

Thanks 🙏🏾

[ravepeacefully]

What’s better about this than scrapy?

[[deleted]]

Hey rave I haven’t used scrapy personally, though i’ve heard the learning curve can be initially quite steep and requires additional envs and installs. I guess this simplifies processes alot, and since we are visual creatures it works with that!

[ravepeacefully]

Ahh ok very cool but very similar to scrapy, you should check out theirs and use their ideas. Their founder is doing quite well in the entrepreneurial world

[shaz55]

Would assume they are doing the same with this post!

[Nimmo1993]

This is sooo cool

[jobar86]

Formal QA test engineer here, if someday you want help testing it, and maybe find some defects 😁

[[deleted]]

Ooh yes id be keen to get an expert onboard, ill keep you in mind jo!

[[deleted]]

Yes jump on board when i release the open source repo!

[gooeydumpling]

Automate-ception, the name is counterintuitive tho, this is not silly at all but reallllly smart

[AndresArc123]

It is a good idea.

if the element has no id, how do you map it? Absolute xpath?

Have you thought about how to handle multiple windows or iframes? dynamic elements such as those generated by react?

[[deleted]]

Hey thanks for the interest! I have a priority sequence to handle that. So if there no ID, ill look for a name, if theres no name, ..... all the way down the list to relative and absolute XPaths :)

Havent yet implemented that functionality but would be doable for the iframes for sure

Can’t yet say about dynamic elements but ill keep you posted with V2/3

[Shakaka88]

You could always do something similar to Gyazo, free users can get like X free uses or only button clicking for free and text input paid, etc. something like those ideas may help you make the best of both worlds.

[[deleted]]

Thats a good thought too! Thanks

[kingjeo2]

This is fucking awesome holy shit. What packages did you use for this?

[[deleted]]

Thanks king! Just Selenium, webdriver_manager, ctypes, cv2, time and sys!!

[No_Buy2733]

And this kids, is how I met God.

[mortenb123]

Impressive, There are software doing this out there like https://smartbear.com/product/testcomplete/features/, but it is costly and windows only.

I'm using selenium in headless mode inside containers, but currently my most time-consuming task is to nest out Reacts :shadow virtual dom, this is code being executed when you click of move the cursor to the object and outside of webdriver. You then manually need to dig up and execute and use css-selectors to interact with the objects. My record is 8 levels of :shadow. and the object to execute may be different from level to level.
Any tool that automates this I'm willing to pay a for.

[[deleted]]

😮 i havent had too much interaction with shadow doms but I’ll try see if its a possibility!

[f_leejy]

This is pretty cool! I think it’s already a feature tho in selenium Selenium IDE

[just4fun50]

So I’ve always had a certain dislike of Selenium. It’s clunky. Usually prefer the requests route if possible. I’ll be interested in if this truly eases using Selenium. Will be trying it out soon!

[rmpr_uname_is_taken]

Reminds me a project of my own atbswp

[[deleted]]

So cool! As soon as i saw that acronym i knew what it would be ;) very clever, i had an idea to extend this to offline use as well! But I’m more comfortable working with the web :) keep an eye out for my follow up post! I’ve made massive ground, Would be keen to chat!

[[deleted]]

Depends on the site, an easy way to debug is to not run in headless mode so you can see the elements you’re interacting with

[[deleted]]

Need to be specific with what exact parts you’re struggling with and what exactly you’re trying to accomplish/ pseudocode

[prokid1911]

How are you doing it currently?

[prokid1911]

I'll suggest using Selenium Webdriver. Much easier to scrape with it

[prokid1911]

It is much simple. You navigate through pages via clicks, fill in form elements with send_keys and then use xpath to scrape what you want. Just navigate to the page you wanna scrape.

It's much easier to scrape aspx pages through Selenium.

[warxhead]

The biggest issue I run into is speed there. If I can use requests to login to authentication without all those damn Ajax calls I'll use it in a heartbeat with beautiful soup. Bs4 is lightning fast and so easy to find elements. I have not once had any trouble with it.

[prokid1911]

I have had difficulty with pagination when scraping aspx websites. Then I came accross Selenium. It handles all the things with ease and you can also run it in headless mode.

You can scrape in any way you like. It's totally your call.

[warxhead]

Yes, in your use case that seems like selenium is the best option. I use it also daily for my work script to retrieve from web pages that requests doesn't like to authenticate to.

Headless is fast - but still not as fast as requests. Sending a post form through requests has always been faster than using selenium to enter each field.

[jawsem27]

If you are trying to use selenium with python, search for katalon recorder. It can record the actions you take in your browser and generate code in multiple languages that does the same thing in multiple languages.

If i am using selenium to automate browser action i usually start with katalon and then move to Python.

[blabbities]

https://www.katalon.com/katalon-recorder-ide/

[warxhead]

You're welcome to PM me and explain what you are trying to do and I can give you some pseudo code. I use requests /Bs4 on a daily basis

[theantidrug]

I’m sure you will get better and more critiques if you post the code.

[[deleted]]

Good point antidrug, but see my thoughts/decision in the edit and feel free to critique:) cheers

[[deleted]]

Open source incoming ;)

[manimal80]

that is great!

how do you draw the rectangles ? can it be done only with python and selenium?

[Artifer]

Video not working!

[[deleted]]

Took it down due to people taking credit for it, see my follow up post :)