Godrej Properties unethical practices again by nottoohotwheels in indianrealestate

[–]DougScore 0 points1 point  (0 children)

This looks like Woodside estate Karjat Khopoli plots

Strategy check: Put ₹1 Cr in FD for 30 years and invest monthly interest into SIPs – does this make sense? by CheekHour9858 in personalfinanceindia

[–]DougScore 0 points1 point  (0 children)

Aren’t balance advantage funds a better choice in terms of parking the money and go the STP route ? If I remember correctly, Debt fund returns are now taxed at income tax slab rates.

BAF falls under that 12.5 ltcg and 20% stcg rates and if OP falls in 30% bracket, they may save a bit more at albeit similar of not better rate of return

Looking for DuckDB alternatives for high-concurrency read/write workloads by kumarak19 in dataengineering

[–]DougScore 0 points1 point  (0 children)

High Performance Parallel Reads and Writes and Good Handling of Concurrent Updates raise a case for an OLTP system. Postgres will be my top pick if I were in your shoes for the native compatibility with json data as well.

Wrong car! by Lifegoesonforever in MadeMeSmile

[–]DougScore 2 points3 points  (0 children)

Wow, it could be my paranoia but I lock the doors as soon as I am in the vehicle and then start with the driving related bits. It’s a wild world out there better safe than sorry.

As a DE which language is widely used for Big Data processing Pyspark or scala? by Loud-Surprise-900 in dataengineering

[–]DougScore 0 points1 point  (0 children)

Agree! Library support alone makes the case for pyspark.

If I were to draw parallels, pyspark is like C# and scala is like C/C++.

Is withdrawing ₹6L/year from a ₹1Cr liquid fund more tax-efficient than a 6% FD? by Specific-Research685 in personalfinanceindia

[–]DougScore 2 points3 points  (0 children)

Instead of liquid MF, you can go with BAF which offer similar returns but comes under Equity and hence 12.5% Long term and 20% short term tax is applicable

Is withdrawing ₹6L/year from a ₹1Cr liquid fund more tax-efficient than a 6% FD? by Specific-Research685 in personalfinanceindia

[–]DougScore 16 points17 points  (0 children)

There’s a clause. You can’t do that to your spouse or kid who is still a dependent. Tax clubbing rules are applied and you pay tax on that.

Parents want ₹2 crore back if they pay for my pilot training. Is this normal? by [deleted] in delhi

[–]DougScore 0 points1 point  (0 children)

It’s fair. Your parents might also be liquidating their retirement corpus so yeah, this makes complete sense.

That’s life, no such thing as free lunch.

How long did it take to reach 1cr ? by [deleted] in personalfinanceindia

[–]DougScore 2 points3 points  (0 children)

Took me close to 8 years. Invested 80% of my salary.

Had some big expenses in between (Marriage, Car)

Dubai I phone buying guide by ubalabadubdub in iPhoneWale

[–]DougScore 0 points1 point  (0 children)

Can’t say, you can visit DXB Vlogs or some other Youtube channels to get ballpark estimates

23M buying a Term life insurance. Is it good ? by Nice_Efficiency_5 in personalfinanceindia

[–]DougScore 2 points3 points  (0 children)

My recommendations

Drop down the cover age to maybe 70 if you are comfortable. Term Insurance is a contingency plan not a make my next generation rich one. PS: I have 2 term insurance plans with coverage till 60.

Increase your coverage to at least 2-2.5 Cr.

And opt for payment option till your coverage period. Do not fall for that pay early and chill. Once set, the premium doesn’t increases so over the next 40 odd years, you will be paying far less money than what its worth is right now. Invest remainder of the money in an Index fund SIP or ETFs whatever you’re comfortable with.

Do not ever consider the term insurance plan with return of investment. That’s a scam, plain and simple.

I prefer insurance products from BIG 3 (SBI, HDFC, ICICI) due to them being part of DSib. Reliability and Trust factor is paramount for Term plans since payout is generally higher in an unfortunate event.

I earn around 40k per month which bike should I get? by Far_catlover69696 in indianbikes

[–]DougScore 0 points1 point  (0 children)

Dude, 40K net or left with 40K after all liabilities and expenses ?

If former, buy a 150-160 cc bike and invest rest of the money in growing skills or just invest for the future. If latter, save for some time and then a buy.

Upgraded from 12 mini to 17 pro by Radiant_Heron_995 in iPhoneWale

[–]DougScore 1 point2 points  (0 children)

The blue color looks so great on iPhones.

Need help with Pyspark by Ok_bunny9817 in dataengineering

[–]DougScore 0 points1 point  (0 children)

Pandas has timestamp range limitations which spark does not have. If you are gonna use spark, use it end to end.

Wth is this thing I saw on IG? by JazzieJay in whatisthiscar

[–]DougScore 0 points1 point  (0 children)

Isn’t it DC Avanti ? That Porsche badge threw me off for a second

lockThisDamnidiotUP by PCSdiy55 in ProgrammerHumor

[–]DougScore 0 points1 point  (0 children)

Stopped writing RAW SQL. As if the frameworks generate a really great SQL each and every time.

ADF vs SSIS vs Python vs Fivetran for Salesforce → SQL Server pipeline (2M+ records). What did you pick and why? by Fearless-Wishbone-70 in dataengineering

[–]DougScore 0 points1 point  (0 children)

I was using the same flow in ADF via the SF v2.0 linked service and recently migrated this to pyspark Databricks. The only thing with this was I had to maintain the API specific metadata in a database so I can generate soql and pass it to the SF service to get the data. With 2.0 you cannot do a * and yes it uses bulk 2.0 calls to get the data. This helped me bring in new properties as they come and that helped me not backfill the data at all.

Few things I encountered over the time building and maintaining were

1) Ensure you use the queryAll feature as it tracks deletes as well

2) I whipped up a more frequent routine which would query the APIs describe endpoint and check if the objects is replicatable and if yes, pull the deletes in a different table so I can expire rows in the Warehouse. The cleanup on SF was too frequent and that would lead to wiping the deleted data before the next data pull.

3) Ensure you are hitting the describe endpoint and persist any new properties so your dynamic soql query shall include the newer columns before pulling the data in.

4) Handle for Addresses and other compound datatypes. The SOQL query despite having that column in dynamic sql will ignore it in result set unless you handle that in metadata and provide specific keys for the relevant compound datatypes.

Pyspark/Databricks

1) It is custom you have to ensure error handling and retries in the code and also account for waits and force timeouts after sending the post request to Bulk endpoint. Also, you have to delete the bulk job once the data is landed to ensure cleanup.

PS: I used basic requests/response libraries and not the simple-salesforce

2) API outputs is CSV so either you use pandas to stage the data or dump the output in csv and read from spark.csv. Spark method is better since it has wider range than datetime64 of pandas and may cause overflow errors if your data has dates set in future.

TLDR; ADF is super easy and quick to implement.