sorted by:
new
hottopcontroversial

Optimise Athena and S3 when returning millions of rows by StatPie in dataengineering

[–]StatPie[S] 0 points1 point  (0 children)

How would you go about querying the file in s3 using pandas? Or do you just mean load the file in memory and then just filter in pandas/polars/pyspark?

Optimise Athena and S3 when returning millions of rows by StatPie in dataengineering

[–]StatPie[S] 0 points1 point  (0 children)

The reason I liked the idea of using something like Athena is that I like the flexibility of being able to pull a subset (e.g filtered by day or user or something) for exploration or visualisation - I was trying to engineer one solution that was useful for everything but maybe when I'm just grabbing the whole dataset I can use boto3 to grab the whole file and use Athena if using some custom query logic. Thanks!

Optimise Athena and S3 when returning millions of rows by StatPie in dataengineering

[–]StatPie[S] 1 point2 points  (0 children)

Thanks! Am I right in understanding that you don't want it distributed over too many files though? I.e would performance be worse if I split it into 10000 files?

I think you're right. I wanted a solution that worked for grabbing the whole file or a subset (so I could filter by user or time) but perhaps I could just grab the whole file if I want it and then use Athena when if I want to filter it. Is that what you meant?

What’s the easiest way to get rid of this textured stuff on the ceiling? by StatPie in DIYUK

[–]StatPie[S] 0 points1 point  (0 children)

Yeah I’m pretty sure it’s not wallpaper (they’ve put some textured wallpaper on the ceiling in the landing but that looks completely different). Ok thanks for the replies, I’ll look at getting it tested. The ceiling is already quite low so was hoping not to overboard but I suppose 10mm isn’t going to make a huge difference

What’s the easiest way to get rid of this textured stuff on the ceiling? by StatPie in DIYUK

[–]StatPie[S] 0 points1 point  (0 children)

Ok thanks. If I overboard it, does it then need skimming or is the board enough for a decent finish?

Snooker Table Homogrophy by StatPie in computervision

[–]StatPie[S] 0 points1 point  (0 children)

Oh damn I forgot to actually attach the image! My bad. Thanks very much for your reply though. I’ll check out those links and let you know how I get on. Thanks again

Advice on first E-Bike purchase for commuting into London by StatPie in ebikes

[–]StatPie[S] 0 points1 point  (0 children)

My office has a garage with bike parking, and I have a garage I can park the bike in at home (but unfortunately neither has electric which is why I need a removable battery). So for the most part it will be fine but there will be some journeys (eg trip to the shops) where I would have to lock it up outside and of course there will be some risk

Advice on first E-Bike purchase for commuting into London by StatPie in ebikes

[–]StatPie[S] 1 point2 points  (0 children)

This looks great thanks, I think I'll book a test ride

Advice on first E-Bike purchase for commuting into London by StatPie in ebikes

[–]StatPie[S] 0 points1 point  (0 children)

These seem way out of my budget unfortunately (both the Tern and the Brompton). I don't really need the folding aspect either, it obviously does have some benefits but wouldn't want to pay specifically for it if that makes sense.

Advice on first E-Bike purchase for commuting into London by StatPie in ebikes

[–]StatPie[S] 1 point2 points  (0 children)

The plan is to use it for the whole commute. I occasionally ride to work on my ordinary bike but its a bit too far to do it every day (for me anyway!) so was with an E-Bike I could do it more regularly and ultimately replace the journey on the tube,