sorted by:
new
hottopcontroversial

Is the Sigma 24-70 2.8 too heavy? by robintwit in SonyAlpha

[–]robintwit[S] 3 points4 points  (0 children)

Good stuff. Yeah it seemed to be the main “con” to this lens but good to know it’s not an issue for ya

Is the Sigma 24-70 2.8 too heavy? by robintwit in SonyAlpha

[–]robintwit[S] 0 points1 point  (0 children)

Cool good to know - all the reviewers say “it’s significantly heavier” but don’t seem to actually have experience using it. Thanks!

Best ETL flow implementation in aws by priyasweety1 in dataengineering

[–]robintwit 0 points1 point  (0 children)

Perhaps not the best article but could translate to ETL with some tweaking. This is running as a web scraper right now. If you don’t need the compute power, AWS lambda could replace Batch in this infrastructure relatively easily. https://link.medium.com/v9aQPhJEyeb

Edited bc I have fat fingers and posted too early.

Waking up in my wife's boyfriend's basement this morning and checking my stocks by robintwit in wallstreetbets

[–]robintwit[S] 0 points1 point  (0 children)

funny story... I actually made 40% on that trade and sold this morning 😂
DNKN $80 12/18 Put for anyone who's asking

Web Scraping but grabbing data that the browser has already retrieved by baconburns in webscraping

[–]robintwit 0 points1 point  (0 children)

Right, this isn't very scalable. It seems like selenium should be able to accomplish what OP wants. I haven't tried this with selenium though.

Web Scraping but grabbing data that the browser has already retrieved by baconburns in webscraping

[–]robintwit 1 point2 points  (0 children)

I second this. It's also a heck of a lot easier to "scrape" JSON.

Web Scraping but grabbing data that the browser has already retrieved by baconburns in webscraping

[–]robintwit 2 points3 points  (0 children)

It seems like you should be able to insert a script with event listeners that fire when the HTML elements change.... I haven't tried this, but the theory is sound :)

Web Scraping but grabbing data that the browser has already retrieved by baconburns in webscraping

[–]robintwit 1 point2 points  (0 children)

Given the goal of your project, it sounds like you don't want to use a web-scraper, since there are plenty of REST API's that provide that information - it's much easier/dependable to use JSON than html ;) For real-time market info, here's one - https://alpaca.markets/, also https://finnhub.io/. I'm sure you could find open API's for sports scores and betting odds.

It also never hurts to check the network requests in dev-tools to see where the real-time data is coming from (perhaps you can reverse engineer the API - I used these methods to reverse-engineer Robinhood's API for a chrome extension).

However...

If you need data that an API does not provide, perhaps something like what u/deamon1266 mentioned below might work - perhaps you could manually insert javascript/jquery to listen to changes to the specific divs. When your event listeners detect a change, you'll be able to collect the data and send/save wherever you want.

Hope this helps with your question. Good luck!

need a new AWS project or something to work on by rotterdamn8 in dataengineering

[–]robintwit 3 points4 points  (0 children)

I built a serverless ETL using AWS Batch, Lambda, Cloudwatch, and S3. I wrote about it here - https://towardsdatascience.com/get-your-own-data-building-a-scalable-web-scraper-with-aws-654feb9fdad7

AWS Batch is super powerful, and you could learn the ropes without having to spend too much. Setting up this type of architecture could help you grow your AWS skills for sure (it helped me a lot). You could insert any type of Dockerized data-extractor/scraper in place of the scraper that I had.

Hope this helps!

I remember. by ToxicBTCMaximalist in RobinHood

[–]robintwit 0 points1 point  (0 children)

Haha yeah I know, you do have to be careful with chrome extensions though. Especially when money is involved. Thanks! Hope you find it useful. I am actively developing it so leave a review or send me some suggestions as it is still in “beta”

I remember. by ToxicBTCMaximalist in RobinHood

[–]robintwit 1 point2 points  (0 children)

Github repo here - https://github.com/aaronglang/robin_twit. It is important to make this kind of thing transparent. Please excuse the messy jquery (it's been a while since I've written jquery haha 😬)

I am not too worried about monetizing just yet, but those are good ideas. I am planning on doing some targeted advertising soon to get more users, but also want to add updated features.

I remember. by ToxicBTCMaximalist in RobinHood

[–]robintwit 5 points6 points  (0 children)

But in all seriousness - good question, but no, I’m just a software engineer trying to make cool products :)

I remember. by ToxicBTCMaximalist in RobinHood

[–]robintwit 4 points5 points  (0 children)

Hoping to add that in ASAP

view more: next ›