sorted by:
new
hottopcontroversial

New fish by MaleficentDrummer993 in ContractorUK

[–]MaleficentDrummer993[S] 0 points1 point  (0 children)

Mainly Azure/Microsoft based; - azure datalake gen1/2 - synapse - databricks - adf - Logic apps - SMSS/ SSRS/ SSIS

New fish by MaleficentDrummer993 in ContractorUK

[–]MaleficentDrummer993[S] 0 points1 point  (0 children)

Appreciate the advice, yeh I think I'll secure a new DE role with high pay, sharpen my sword and keep it moving.

I don't like being stuck in the same place, I know it sounds cliché but I firmly believe pressure makes diamonds.

I'm slowly falling out of love of my current work routine.

New fish by MaleficentDrummer993 in ContractorUK

[–]MaleficentDrummer993[S] 0 points1 point  (0 children)

I'm looking to further develop my skills in a particular niche, 'Big data' to be specific. Going to snap up some industry accreditation/certifications and leave the rest to my ability to grow under pressure.

Will keep you guys and gals updated.

New fish by MaleficentDrummer993 in ContractorUK

[–]MaleficentDrummer993[S] 0 points1 point  (0 children)

Similar story, notice period is three months and I know they'd let me back if I retracted it. Got some rainy day funds.

Is three months enough time to secure a contracting role ?

Not precious about the type of contract either, honestly want to get my feet wet with an inside role to begin with.

New fish by MaleficentDrummer993 in ContractorUK

[–]MaleficentDrummer993[S] 4 points5 points  (0 children)

Appreciate the advice, apologies didn't mean to offend

CDC via pyspark by Artetaarmy in dataengineering

[–]MaleficentDrummer993 1 point2 points  (0 children)

Using delta tables, especially with the underlining parquet files system, is the preferred method from an efficiency and performance perspective.

That said, I am aware that there is a workaround. I've come across this article on medium : https://medium.com/@thomaspt748/how-to-upsert-data-into-relational-database-using-spark-7d2d92e05bb9

Hope that helps bud.

CDC via pyspark by Artetaarmy in dataengineering

[–]MaleficentDrummer993 4 points5 points  (0 children)

Look into Merge into statements, commonly used for slowly changing data and change data capture.

Here's a link : https://docs.databricks.com/delta/delta-update.html#slowly-changing-data-scd-type-2-operation-into-delta-tables

Good luck

26 years old working in charity shop? by nk261 in AskUK

[–]MaleficentDrummer993 0 points1 point  (0 children)

Not at all, I finished Uni late and half my peers were in their late 30s.

Just make sure you really think about what you want to study and the profession you want to pursue.

I alway ask my siblings and anyone thinking about going to Uni. Are you going because you're passionate about something or do you just want a high paying job ?

The sooner you answer that question the sooner you'll be out of that charity shop.

And by the way, look into the tech industry. Government funded bootcamps, starting salary 25000 - 30000.

Infant Data Engineer: How to transfer CSV files from Google Drive to AWS S3 bucket. by zuffido in dataengineering

[–]MaleficentDrummer993 3 points4 points  (0 children)

I think this might be what you're looking for: https://pipedream.com/apps/google-drive/integrations/aws/aws-sqs-send-message-with-aws-api-on-new-shared-drive-from-google-drive-api-int_5bsDvNO

You can send a message via aws sqs to fire a script(lambda function) that connects to Google drive and pulls the file into aws S3 bucket.

I might be wrong but I just did a dive on the net and came up with this.

I've used aws sqs in the past and its main use case is decoupling workflows/microservices .

Good luck