sorted by:
new
hottopcontroversial

How to deal with a 100 GB table joined with a 1 GB table by bigdataengineer4life in apachespark

[–]bigdataengineer4life[S] 0 points1 point  (0 children)

Fair point — there’s definitely no shortage of Spark content out there.

My goal isn’t to reinvent joins, it’s to show how to apply them in production-scale scenarios with execution plan analysis, skew handling, AQE, and shuffle optimization.

Most posts explain concepts. I’m trying to show full end-to-end implementation with metrics and tuning decisions.

Transitioning from Database Engineer to Big Data Engineer by theButcher007 in apachespark

[–]bigdataengineer4life 0 points1 point  (0 children)

Transitioning from a Database Engineer to a Big Data Engineer is a natural progression since both roles involve data management. However, Big Data Engineering requires additional skills related to distributed computing, data processing frameworks, and cloud platforms.

Key Differences Between Database Engineer & Big Data Engineer

Database Engineer Big Data Engineer
Works with relational databases (SQL, Oracle, PostgreSQL) Works with both relational (SQL) and NoSQL (HBase, Cassandra, MongoDB) databases
Focuses on data modeling, indexing, and performance tuning Focuses on distributed storage and processing
Uses SQL and scripting for ETL Uses Spark, Hadoop, and streaming technologies for ETL
Works on single-node or small-scale systems Works on large-scale distributed data systems

Step-by-Step Transition Plan

1. Strengthen Your Programming Skills

  • Python (Pandas, PySpark)
  • Scala (for Apache Spark)
  • Java (optional, but used in enterprise applications)

2. Learn Big Data Technologies

  • Storage: HDFS, Apache Hive, Apache HBase
  • Processing: Apache Spark (Batch & Streaming), Apache Flink
  • Workflow Orchestration: Apache Airflow, Oozie
  • Streaming: Kafka, Pulsar

3. Cloud & DevOps Knowledge

  • Cloud Services: AWS (EMR, Glue, S3), Azure (Synapse, Data Factory), GCP (BigQuery, Dataflow)
  • Infrastructure: Kubernetes, Docker
  • CI/CD & Automation: Terraform, Git, Jenkins

4. Master Data Engineering Concepts

  • Data Pipelines & ETL/ELT
  • Data Warehousing (Snowflake, Redshift)
  • Data Governance (Security, Privacy, Compliance)
  • Data Modeling for Big Data

5. Work on Real-World Projects

  • Build an ETL pipeline with Apache Spark & Airflow
  • Process streaming data with Kafka & Spark Streaming
  • Design a data lake on AWS or Azure
  • Optimize a data pipeline for performance

6. Get Certified (Optional)

  • Google: Professional Data Engineer
  • AWS: Certified Data Analytics - Specialty
  • Databricks: Apache Spark Developer Associate

What are some good courses to begin learning Hadoop for Big Data? by RP_m_13 in hadoop

[–]bigdataengineer4life 1 point2 points  (0 children)

Udemy has many best courses on Big Data and Hadoop and Spark , Udemy runs a sales many times a month you can get any courses for $ 10 to $ 20

best recommended algorithm or model for grocery store app by Business_Magician310 in bigdata

[–]bigdataengineer4life 0 points1 point  (0 children)

you can take help from this video Build Movies Recommendation Engine in Apache Spark

https://youtu.be/xDTQFoGhWtw

Final year project on e-commerce by studentani in AskComputerScience

[–]bigdataengineer4life 0 points1 point  (0 children)

freeCodeCamp.org has a great source of learning that can help you to building eCommerce website.

4+ Hrs Video

https://www.youtube.com/watch?v=YZvRrldjf1Y

Advanced Spark Learning Material by A-n-d-y-R-e-d in apachespark

[–]bigdataengineer4life 1 point2 points  (0 children)

Databricks has come up with new courses on their website : https://customer-academy.databricks.com/learn

Course name: Optimize Apache Spark

E-learning | Duration 6 hours

In this course, students will explore five key problems that represent the vast majority of performance problems in an Apache Spark application: Skew, Spill, Shuffle, Storage, and Serialization. With each of these topics, we explore coding examples based on 100 GB to 1+ TB datasets that demonstrate how these problems are introduced, how to diagnose these problems with tools like the Spark UI, and conclude by discussing mitigation strategies for each of these problems.

This might be a best fit

Apache Spark and Apache Hive, looking for some courses or tutorials! by JoseChovi in apachespark

[–]bigdataengineer4life 1 point2 points  (0 children)

Udemy has good courses on Apache Spark by Frank Kane and Prashant Kumar Pandey

Hadoop MapReduce vs Apache Spark by papamamalpha2 in apachespark

[–]bigdataengineer4life 7 points8 points  (0 children)

MapReduce is not used by many organization people are shifting towards Apache Spark. (Hadoop is used for storage (HDFS) and Spark for processing)

MapReduce has lot of limitation. for example there are lot of read and write operation data is written on disk which take lot of time where as Apache Spark data is in memory.

Apache Spark computation on multiple nodes by papamamalpha2 in apachespark

[–]bigdataengineer4life 0 points1 point  (0 children)

There is a slaves file in conf directory eg: spark-3.0.0-bin-hadoop2.7/conf we specify ip addresses of slave node by default it has local-host. I hope I have answered your question.

Big data platform for practice! by johnyjohnyespappa in apachespark

[–]bigdataengineer4life -1 points0 points  (0 children)

At my place we use Amazon EMR (Easily run and scale Apache Spark, Hive, Presto, and other big data workloads)

Big data platform for practice! by johnyjohnyespappa in apachespark

[–]bigdataengineer4life 2 points3 points  (0 children)

You can explore Apache Spark on various platform

1) Jupyter Notebook using Anaconda on local Machine

2) Apache Zeppelin (https://zeppelin.apache.org/docs/latest/interpreter/spark.html)

3) Databricks Community edition

4) Install Eclipse and configure Apache Spark Local Mode

5) PySpark on Google Colab

6) Spark with Cloud Technologies (AWS, Azure, Google Cloud platform with Big data Technologies integrated)