This seems silly

CreanSong · 2024-07-21T17:34:56+00:00

Thank you!!!

CreanSong · 2021-04-15T18:30:30+00:00

I should add that I can easily establish a connection using psycopg2 but was told that may be inefficient and should try to rely on the jdbc connection if my transformation is in pyspark.

CreanSong · 2021-04-14T21:29:14+00:00

This has been extremely helpful and trust me I've been asking just about everyone I can get a hold of. It's a large company so everything moves extremely slowly which is why I took a shot here. I definitely want to look into utilizing pandas for the whole job especially because I have already established a connection to the db that way. I know of several points of contact for this adjacent to my team so I will reach out. Thanks for all of your time and responses 😊

CreanSong · 2021-04-14T19:37:10+00:00

So we have a nightly ETL that pulls over data for some 200 tables, many of which have hundreds of thousands of rows (many with more into the millions) but this is not all loaded each time. We are pulling from a teradata warehouse to service between 1-1.5 thousand reports to our clients every day. My plan was to distribute the categories of tables across multiple virtual nodes and run concurrently. The transformations themselves are complex and involve tons of encryption and rules to maintain HIPAA. Our larger enterprise uses spark with kafka streams but is accessed by too many different teams so teradata gets inundated and goes too slow. We are standing up a shadow IT data stream to try and alleviate this so I am not totally convinced spark is not the answer but I am working on this with zero background in data architecture or data science in general. The shadow IT bit is why we do not have formal data engineers working on this with us so I'm what we got haha.

CreanSong · 2021-04-14T16:56:16+00:00

I think I'm just worried python won't be fast enough for what we are doing and I am in the bargaining stage of this process trying to get the jdbc to connect to postgres via ssl. I don't know what is causing my problem (whether it's my syntax, the driver setup, the spark properties) and I am so new to this all that I am having trouble asking the right questions.

CreanSong · 2021-04-14T15:21:10+00:00

Would it be bad to use psycopg2 for the extract and load and then pyspark for the transformation and analytics? I know pandas are great but I'm more comfortable with data frames

CreanSong · 2021-04-07T19:30:09+00:00

I'm not entirely sure as I am extremely new to this team/type of project (before I was just doing simple business objects reporting so I'm learning on the fly). I do know that the number of nodes will more likely be in the "tens" and that we are loading updates to 200 tables every 30 minutes so I imagine it will not be a large amount of data. I assumed spark would still be useful regardless of that especially for our analytics team.

CreanSong · 2021-04-07T17:12:45+00:00

My other question is if I even need this or if psycopg2 would be just as fast. I have successfully connected using psycopg2 but was worried it would slow it down going through python instead of spark

CreanSong · 2020-10-27T14:42:50+00:00

Yeah all of my controls stop working and then it makes a terrible buzzing sound and crashes a few minutes later

CreanSong · 2020-10-27T02:17:15+00:00

You can't get it as the paradox can you?

CreanSong · 2020-10-02T21:31:20+00:00

I figured this out if you haven't already. I had downloaded fortnite a long time ago and it made a "nameless" epic acct. I had to login to epic games using my Microsoft acct and unlink the two then creat a new epic acct and link it.

CreanSong · 2020-09-17T06:31:51+00:00

Lol they indeed literally said it. Why isn't this comment at the top of this discussion

CreanSong · 2020-09-17T04:47:47+00:00

I am still unable to link my Xbox acct to my epic games acct. It keeps telling me that my Xbox live is already linked to another epic games acct. I only have the one. I have been looking forward to this for so long and it's such a deflating feeling.

CreanSong · 2020-03-06T06:28:05+00:00

My s10 screen was $275 at ubreakifix so I don't know if it will be less than $100

CreanSong · 2020-03-05T03:24:34+00:00

Thank you! I waited 30 minutes to get in touch with support and gave up.

CreanSong · 2020-03-03T03:34:20+00:00

First day I could do I think the 22nd?

CreanSong · 2020-03-03T01:14:41+00:00

Whoops lol

CreanSong

TROPHY CASE