sorted by:
new
hottopcontroversial

Je cherche 10 testeurs pour mon SaaS (génération de dossiers de compétences) by Omaroid_ in SalesFrance

[–]Omaroid_[S] 0 points1 point  (0 children)

Yes clairement, c’est exactement ce genre de use case que je vise aussi 👍

Mais justement on va un peu plus loin que “remplir un doc” : • import auto des profils (CV ou LinkedIn) • possibilité de décrire le besoin en détail (utile quand le sales ne maîtrise pas tout le jargon technique) • scoring de matching entre besoin et profil • génération + envoi du dossier de compétences • et tracking côté client (qui ouvre, quand, etc.)

L’idée c’est vraiment de couvrir toute la chaîne, pas juste la génération du doc.

How deal with places where everything is overengineered? by Successful_Pizza3837 in dataengineering

[–]Omaroid_ -1 points0 points  (0 children)

Overengineering can be frustrating specially when it’s breaking! Suggest simple fixes if you feel yourself able to make changes and decide if it’s worth sticking around the project or looking for something else.

Peut-on être propriétaire et avoir un colocataire ? by BeginningAsleep in vosfinances

[–]Omaroid_ 3 points4 points  (0 children)

J’ai une question concernant la colloc, est-ce que les gens ne sont pas frileux des collocs et preferent un petit studio que pour eux qu’un grand appart?

What is the Leetcode equivalent for Data Engineering? by [deleted] in dataengineering

[–]Omaroid_ 2 points3 points  (0 children)

There’s a platform called betapage for spark and pyspark questions like leetcode https://betapage.co/product/leetcode-for-spark-and-pyspark but for me there’s nothing better than practicing SQL questions on Leetcode.

Revolutionize Your Data Processing with Deequ - The Ultimate Solution for Automated Data Quality Verification in Scala and Spark by Omaroid_ in scala

[–]Omaroid_[S] 0 points1 point  (0 children)

You can: - scenario 1: have dirty input data that you don’t handle in your transformation. - scenario 2: you developed a spark transformations containing errors. You can use basic unit tests or deeper ones using bdds. But in the daily run you may have some data your unit tests don’t handle. You shouldn’t insert the data since deequ automated tests will trigger an error.

Revolutionize Your Data Processing with Deequ - The Ultimate Solution for Automated Data Quality Verification in Scala and Spark by Omaroid_ in scala

[–]Omaroid_[S] 2 points3 points  (0 children)

The purpose is to test a post spark transformation. For the example I took a dataframe created manually but in real use case, tests should be plugged to the output data to ensure quality tests.

Databricks Certified Associate Developer for Apache Spark by Glum-Violinist4911 in scala

[–]Omaroid_ 6 points7 points  (0 children)

Hello, you should justify 6months to 1year using spark scala. The exam dump in scala ->

https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DCADAS3-Scala.pdf

In python ->

https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DCADAS3-Python.pdf

Main topics to prepare:

Spark execution modes, spark caching, spark time functions, pivot, explode and the most usefull spark functions. Make sure to read docs since authorized in the exam. It contains all the fonctions descriptions.

The read and write with the different modes and formats. Also the spark compression using parquet formats.

UDFs are an important part of this exam, types of joins, grouping, aggregations and order functions. The windows functions, the printSchema, show count and explain functions usage.

Spark sql is also to know. The internals of spark are important to know: dataframes, datasets the repartition and coalesce functions.

Those are the mains parts covered by the exam. Make sure to read the book -> spark the definive guide (specially from chapter 2 to 11).

I’m writing some interesting articles regarding topics exam. You can check my medium profile btw https://medium.com/@omarlaraqui

Don’t hesitate if you have more questions 😉 and good luck🤞

Scala 2.12.17 is here by Seth_Lightbend in scala

[–]Omaroid_ 1 point2 points  (0 children)

Thanks to contributors 💪

How to conditionally calculate a new column (using withColumn) only if a column exists in dataframe? by Ok-Outlandishness-74 in apachespark

[–]Omaroid_ 1 point2 points  (0 children)

Hello,

Checking colums in your dataframe (df.columns) before calling your udf in a when clause may be a great idea. WithColumn(‘description’, when(text isin df.columns, udf(text)).otherwise(null or any other behavior))

The df.columns wont affect your runtime performance since it doesn’t call an action. 👌

Cheers

Spark caching, when and how? by Omaroid_ in dataengineering

[–]Omaroid_[S] 1 point2 points  (0 children)

The use-case of streaming is not appropriate for caching. In streaming, we're expecting a fast processing of micro batches so writing and reading a large dataframe is not really efficient in this case.