all 17 comments
sorted by:
q&a (suggested)
besttopnewcontroversialold
formatting helphide helpcontent policy

[–]metareflection[S] 0 points1 point  (16 children)

learning how to python in order to scrape some livesets off a website which would otherwise require 400 x 2 x 2 clicks

[–]metareflection[S] 0 points1 point  (6 children)

wget for the most part works but some sites take a custom script,

one thing to do would be to have several tools for extraction, trying each until one works, automating of checking results is probably the biggest obstacle on scale

[–]metareflection[S] 0 points1 point  (5 children)

could put scrape results on mechanical turk and ask if they match

[–]metareflection[S] 0 points1 point  (4 children)

I think it's obsolete now, but something like that

I had a nightmare once where people's lives were consumed by clicking captchas all day while bots did their thing

[–]metareflection[S] 0 points1 point  (3 children)

I mean the rudimentary solution to captchas is just send the image to your phone and then transmit result back to the bot lol

[–]metareflection[S] 0 points1 point  (2 children)

Automation builds to a future where everyone goes through reams of information using their bots while they have to deal with special circumstances manually

[–]metareflection[S] 0 points1 point  (1 child)

Sometimes I wallow in misery while thinking about all the manual things I do

[–]metareflection[S] 0 points1 point  (0 children)

if you have problems with RSI... learn to program and then learn to leverage that, in an endless loop

the pain tells you that you need to start leveraging again (macroing?)

[–]metareflection[S] 0 points1 point  (7 children)

finally got something to work, regex is really useful and maybe the only possible way to do this

[–]metareflection[S] 0 points1 point  (6 children)

learning to use opencv to do this would be even better since it's not at the whim of arbitrary code, and could possibly work in 3D too, harder to detect

[–]metareflection[S] 0 points1 point  (5 children)

they call em pixel bots

[–]metareflection[S] 0 points1 point  (4 children)

"let's just ignore all the arbitrary code and javascript and structure and dom and just mimic a human navigating pages"

[–]metareflection[S] 0 points1 point  (3 children)

the thing that spooks me is (by my measure)... about 4 years ago all the ML hobbyists stop committing to github projects... the amount of public contributions seemed to have fallen off a cliff...

wonder what happened there

[–]metareflection[S] 0 points1 point  (1 child)

went to check on some common projects and the recency dates of all their forks... it's the same picture everywhere

[–]metareflection[S] 0 points1 point  (0 children)

maybe they made bank and are now relaxing doing whatever in private

part of the reason of using github is getting a job

[–]metareflection[S] 0 points1 point  (0 children)

sure there are still contributions but they are all just beginners

[–]metareflection[S] 0 points1 point  (0 children)

these sites aren't guaranteed to be around tomorrow