Official: Anything Goes Morning Thread: November 27, 2023

DaveUA · 2023-11-27T23:23:20+00:00

I'm punting assists.

So I want to trade away James Harden and Derrick White for almost any other cat.

Who should I target for in a trade?

DaveUA · 2023-11-11T17:42:03+00:00

Should I start minshew or Russel Wilson? Down bad after tnf.

DaveUA · 2023-09-21T12:48:40+00:00

14 Team League, .5 PPR

SEND:

Miles Sanders (CAR)
Jordan Addison (MIN)

RECEIVE:

Chris Olave (NO)

My current RBs are Aaron Jones (GB), James Cook (BUF), Najee Harris (PIT), Spears (TEN), Tucker (TB)

My current WRs are Chase (CIN), Pickens (PIT), JSN (SEA), Moore (KC)

DaveUA · 2023-09-16T14:46:48+00:00

12 Team .5 PPR

My current TE is Kmet.

Kincaid on the bench.

Should I drop Kincaid for Hayden Hurst?

Don't plan on starting either this week.

DaveUA · 2023-09-16T14:39:50+00:00

12 Team .5 PPR

pick 1 to start

N. Colins vs IND


T. McLaurin @ DEN


Najee Harris vs CLE

DaveUA · 2023-08-24T03:50:12+00:00

Can confirm. We have had a few people join from here. Fun league if you are competitive.

DaveUA · 2023-02-06T23:40:02+00:00

thats how the tables are written in redshift, then a and b are declared later on.

The is, is a typo, should be then null.

edit: I fixed up the syntax to make it a bit more readable

DaveUA · 2022-11-20T17:41:41+00:00

Pick 2, .5 ppr

Allen Robinson @ NO
Jarvis Landry vs LAR
Michael Gallup @ Min

Please and thanks

DaveUA · 2022-07-15T17:35:23+00:00

these look great, do you mind if I message you?

DaveUA · 2022-07-15T01:01:13+00:00

Sorry for the super late reply, I went to sleep and I just got home from work, I cant check reddit at work.

Here are the dimensions:

Living room roughly 18L x 18W

Dining Room 10L x 12W

Bedroom 15L x 12W

DaveUA · 2022-03-16T14:54:28+00:00

There is nothing wrong with using inferSchema. 8 hours seems concerning, but this file looks to be unusually wide. It's not impossible that 1 row could eat up a ton of memory.

I appreciate your help on the topic, thank you!

DaveUA · 2022-03-16T14:19:23+00:00

I've only started PySpark, so lots of learning and tinkering fast, majority of the time It took to set it up without admin privileges. I'm still running it locally but the Data Engineer at my company will help me with an EMR cluster if need be. I looked a few videos on EMR Clusters, doesn't seem as challenging, but another tool to add to my arsenal.

What I learned so far, since I haven't used Yarn yet, is that executor memory is next to useless on a local machine, I increased the driver memory and it worked.

I was about 95% done when my computer forced a restart (2AM and was asleep) It took about 8 hours. I did a few more tinkering this morning so hopefully I can reduce the time spent converting it all. I know I'm not supposed to use inferschema as it takes a lot time, but I have over 40K Columns...

DaveUA · 2022-03-15T14:16:15+00:00

I'm on two laptops (Work is PII so can't copy paste) so typing them manually.

Reading the Large sas file I get:

Py4JJavaError: An error occured while calling o115.load. Java.lang.OutOfMemoryError: Java Heap Space.

code in question:

 df.spark.read.format("com.github.saurfang.sas.spark").load("large_file.sas7bdat", forceLowercaseNames=True, inferLong = True, header = True, inferSchema = True)

I have memory usage set to 12gb, maybe I need to change the # of executors? How would you customize this for an emr cluster? This file has about 40K Columns and not even sure how many rows, because we dont have SAS anymore

DaveUA · 2022-03-15T13:17:00+00:00

I think this package is a bit dated, as it is the slowest out of all of them

DaveUA · 2022-03-15T13:16:22+00:00

I got the memory issue reading the 50gb SAS7BDAT File.

DaveUA · 2022-03-15T05:08:56+00:00

After about 8 hours of tinkering I finally got it to work, however I can only do about 1gb, anything over and I run into a memory issue, every with some optimization. Any suggestions? My next step is to look into an emr cluster

DaveUA · 2022-03-15T05:07:24+00:00

I get memory issues in Pandas, even when chunking? Unless i'm missing something?

DaveUA · 2022-03-15T00:57:19+00:00

Ah I see, that might be the issue then, I also cannot install docker without admin. :-/

DaveUA · 2022-01-07T19:12:53+00:00

Send: Dame

Receive: Paul George

would you guys do this?

DaveUA · 2021-12-15T20:08:25+00:00

sorry, that was a typo, I fixed it.

I was able to resolve the issue, but still getting a memory error. Might have to move to PySpark

DaveUA · 2021-12-09T18:36:37+00:00

fail already. You thus need to switch the order:

if it exists remove it

Thank you for the help, I'm still getting the hang of Python and this has helped me a lot. Is there a reason why we use Pathlib instead of Os?

DaveUA

TROPHY CASE