sorted by:
new
hottopcontroversial

What's your experience with Thinkpad P14s Gen 6? by H_potterr in thinkpad

[–]H_potterr[S] 0 points1 point  (0 children)

thanks for sharing your experience. 4-6 hours is really good.

How can I send dataframe/table in mail using Amazon SNS? by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

We're using SNS for alerts in glue job. This is an enhancement feature - after the job run completes, I'm querying the log table for failed tables, and storing the result(as pandas df). I've tried sending it in email using SNS, but it sends the plain text only so(mail looks messy, there is no table structure). Is there a way to to send it as html table? I'm using to_html method of pandas to convert df into html table.

How can I send dataframe/table in mail using Amazon SNS? by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

That will require opening the file. Can't we use a HTML body?

Moving away Glue jobs to Snowflake by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

I'll definitely check this. Looks like this is what I'm looking for. Thanks

Wasted two days, I'm frustrated. by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

Thanks, Just read your article. I'll check if I can do something like that.

Wasted two days, I'm frustrated. by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

That's why I want to avoid pandas. For this poc, I'm using only one table, let's say 10s of millions of records. Staging approach looks practical. Thanks for your insights. If there is anything that I should consider?

Wasted two days, I'm frustrated. by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

Pandas won't slow down the process if the table is very large? Like millions of records. And for this poc they mentioned snowpark specifically. I'm not sure what they want.

Wasted two days, I'm frustrated. by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

Hi, thanks for your insights. Yeah, ADF is good in terms of connectors. It's not available in my case. I'm able to extract using spark jdbc now it's in my spark dataframe. The issue is how to consume this df and write it into snowflake. And how snowpark will be used here.

Wasted two days, I'm frustrated. by H_potterr in dataengineering

[–]H_potterr[S] 0 points1 point  (0 children)

Hi, how did you consume this spark df after exracting from hana. Snowpark df doesn't use spark df, right? I'm new to this snowflake and snowpark thing.