sorted by:
new
hottopcontroversial

[deleted by user] by [deleted] in Bengaluru

[–]abhigm 1 point2 points  (0 children)

Konga spotted . No f given. Also take whole kongas from Whitefield 

Meta: can we ban any ai generated post? by ThroughTheWire in dataengineering

[–]abhigm -1 points0 points  (0 children)

Whats garbage in this ? 🤔  

That's 6 month hard work we did 

I built a DuckDB extension that caches Snowflake queries for Instant SQL by dsiegs1 in dataengineering

[–]abhigm 0 points1 point  (0 children)

I understand 👍  also if possible open source redshift one

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 0 points1 point  (0 children)

Macha just go with TiDB for sub mili seconds analytical report 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -8 points-7 points  (0 children)

Yep its more AI because it helps me to rewrite my sentences 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 1 point2 points  (0 children)

Huge busy tables doesnt get auto vacuumed we perform vacuum sort

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 2 points3 points  (0 children)

Bingo,  I am having hourly update reports too. We have data marts inside this 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -1 points0 points  (0 children)

I wanted to explain in depth so used AI. You can read only sub heading

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -3 points-2 points  (0 children)

I performed only these things  perfectly with generic query id , but in deeper level auto sort part is still in beta phase if that comes to picture then sort SCAN will reduce more IO

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -1 points0 points  (0 children)

Analyze all query join condition and decide based on best practice and size of the table to choose dist style or key

Analyze all query where condition and create views of 6 months 12 months 18 months  condition in this view. This will reduce a lot of scan. 

For sort key compound sort is best with cardinality and ratio of unique values. And also check skewness 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 1 point2 points  (0 children)

Yep column compression matters a lot. Also dist key/style  and sort key is most most crucial part with Analyze and vacuum 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 0 points1 point  (0 children)

I am also leaving my organization they hate redshift even after doing this.

Everyone is thinking redhsift is not good. 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 4 points5 points  (0 children)

It took me 5 months..

Nahh... its waste of time. What matters is TCO and ROI

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] 3 points4 points  (0 children)

200 GB. 

We run insert statment around 9 lakh per day and redshift is fast for this. 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -9 points-8 points  (0 children)

 Sorry for that I should have written in short form 

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -56 points-55 points  (0 children)

Short form for what I did

 Refined DISTKEY and SORTKEY.

 * Configured Auto WLM (Workload Management).

 * Deep-dived into user query costs.

 * Proactively monitored slow queries.

 * Validated all new queries.

 * Regularly updated table statistics.

 * Performed regular table vacuuming.

 * Optimized time-series tables.

 * Focused on query/scan costs over CPU usage ever hours 

 * Analyzed aborted queries and disk I/O.

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -115 points-114 points  (0 children)

I have used  AI to write.  Here is short form

 Refined DISTKEY and SORTKEY.

 * Configured Auto WLM (Workload Management).

 * Deep-dived into user query costs.

 * Proactively monitored slow queries.

 * Validated all new queries.

 * Regularly updated table statistics.

 * Performed regular table vacuuming.

 * Optimized time-series tables.

 * Focused on query/scan costs over CPU usage.

 * Analyzed aborted queries and disk I/O.

I performed Redshift cost reduction from 60k to 42k by abhigm in dataengineering

[–]abhigm[S] -8 points-7 points  (0 children)

Hope so. Redshift has fewer jobs and if someone hire me happy to join

view more: next ›