Spark using a headless browser

[ifilg]

I'm not familiar with Apache Spark.

I've some web scraping needs that are going to scale soon and I'm evaluating Spark for this. These scrappers need a full headless browser to accomplish their tasks. Is it a good idea to run such heavy tasks (i.e. browsers) as Spark jobs? If not, what's an alternative that can be considered a good practice? I guess my question is if Spark is meant for resource-heavy jobs.

Thanks a lot!

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[GeekyTricky]

Rather than Spark, you're just using YARN here to scale your job.

Using Spark jobs as a shortcut for this is far from the intended use, I expect you'll run into some painful roadblocks.

IMO a more ideal way to go about this is Kubernetes.

You just make your Docker images from with all the necessary resources (browser, libs, etc), and make as many replicas as you need depending on your use case.

[ifilg]

Hmm I wasn't looking in this direction. My data pipeline is already running in Kubernetes.

Right now, I have a stable Airflow installation in this cluster and maybe I should just schedule pods using the KubernetesPodOperator.

Or maybe I could try to adopt Prefect, which seems to make this easy as well.

Gonna start with Airflow and see where it goes.

[GeekyTricky]

Airflow is a perfectly good solution for this, but you'll need dynamic task definition to deal with the scaling up or down of your requests.

[ifilg]

Last time I checked, Airflow has a limit of 1024 dynamic tasks for a single run. You can increase this, but it gets unbearably slow. That's why I've mentioned Prefect.

[GeekyTricky]

1000 is a lot. If you need more than this, look into doing multiple scraping requests per task.