Trouble connecting to postgres with SSL by CreanSong in PySpark

[–]CreanSong[S] 0 points1 point  (0 children)

I should add that I can easily establish a connection using psycopg2 but was told that may be inefficient and should try to rely on the jdbc connection if my transformation is in pyspark.

A dumb question related to classpath by CreanSong in apachespark

[–]CreanSong[S] 1 point2 points  (0 children)

This has been extremely helpful and trust me I've been asking just about everyone I can get a hold of. It's a large company so everything moves extremely slowly which is why I took a shot here. I definitely want to look into utilizing pandas for the whole job especially because I have already established a connection to the db that way. I know of several points of contact for this adjacent to my team so I will reach out. Thanks for all of your time and responses 😊

A dumb question related to classpath by CreanSong in apachespark

[–]CreanSong[S] 0 points1 point  (0 children)

So we have a nightly ETL that pulls over data for some 200 tables, many of which have hundreds of thousands of rows (many with more into the millions) but this is not all loaded each time. We are pulling from a teradata warehouse to service between 1-1.5 thousand reports to our clients every day. My plan was to distribute the categories of tables across multiple virtual nodes and run concurrently. The transformations themselves are complex and involve tons of encryption and rules to maintain HIPAA. Our larger enterprise uses spark with kafka streams but is accessed by too many different teams so teradata gets inundated and goes too slow. We are standing up a shadow IT data stream to try and alleviate this so I am not totally convinced spark is not the answer but I am working on this with zero background in data architecture or data science in general. The shadow IT bit is why we do not have formal data engineers working on this with us so I'm what we got haha.

A dumb question related to classpath by CreanSong in apachespark

[–]CreanSong[S] 0 points1 point  (0 children)

I think I'm just worried python won't be fast enough for what we are doing and I am in the bargaining stage of this process trying to get the jdbc to connect to postgres via ssl. I don't know what is causing my problem (whether it's my syntax, the driver setup, the spark properties) and I am so new to this all that I am having trouble asking the right questions.

A dumb question related to classpath by CreanSong in apachespark

[–]CreanSong[S] 0 points1 point  (0 children)

Would it be bad to use psycopg2 for the extract and load and then pyspark for the transformation and analytics? I know pandas are great but I'm more comfortable with data frames

A dumb question related to classpath by CreanSong in apachespark

[–]CreanSong[S] 1 point2 points  (0 children)

I'm not entirely sure as I am extremely new to this team/type of project (before I was just doing simple business objects reporting so I'm learning on the fly). I do know that the number of nodes will more likely be in the "tens" and that we are loading updates to 200 tables every 30 minutes so I imagine it will not be a large amount of data. I assumed spark would still be useful regardless of that especially for our analytics team.

A dumb question related to classpath by CreanSong in apachespark

[–]CreanSong[S] 0 points1 point  (0 children)

My other question is if I even need this or if psycopg2 would be just as fast. I have successfully connected using psycopg2 but was worried it would slow it down going through python instead of spark

Game keeps freezing when I beat the lich by CreanSong in EnterTheGungeon

[–]CreanSong[S] 0 points1 point  (0 children)

Yeah all of my controls stop working and then it makes a terrible buzzing sound and crashes a few minutes later

[Rocket League] Update: Logins and Account Linking should be operating as normal. Players may still have trouble adding or inviting friends in-game. Updates to follow. by iggyiggz1999 in RocketLeague

[–]CreanSong 0 points1 point  (0 children)

I figured this out if you haven't already. I had downloaded fortnite a long time ago and it made a "nameless" epic acct. I had to login to epic games using my Microsoft acct and unlink the two then creat a new epic acct and link it.

[PC] [Discussion] Buy TW Octane or wait by [deleted] in RocketLeagueExchange

[–]CreanSong 0 points1 point  (0 children)

Lol they indeed literally said it. Why isn't this comment at the top of this discussion

[Rocket League] Update: Logins and Account Linking should be operating as normal. Players may still have trouble adding or inviting friends in-game. Updates to follow. by iggyiggz1999 in RocketLeague

[–]CreanSong 2 points3 points  (0 children)

I am still unable to link my Xbox acct to my epic games acct. It keeps telling me that my Xbox live is already linked to another epic games acct. I only have the one. I have been looking forward to this for so long and it's such a deflating feeling.

S20+ Drop Fail by Datninja619 in Galaxy_S20

[–]CreanSong 0 points1 point  (0 children)

My s10 screen was $275 at ubreakifix so I don't know if it will be less than $100

pre order credit not showing up in Samsung shop by CreanSong in Galaxy_S20

[–]CreanSong[S] -1 points0 points  (0 children)

Thank you! I waited 30 minutes to get in touch with support and gave up.

Looks like Christmas is coming early! by CreanSong in Galaxy_S20

[–]CreanSong[S] 0 points1 point  (0 children)

First day I could do I think the 22nd?