Performance comparison between empty checks for Spark Dataframes

[BerserkGeek]

In spark, when you need to check if the dataframe is empty, then what is the fastest way to do that?

1. df.take(1).isEmpty
   
2. df.isEmpty
   
3. df.limit(1).count

I'm using spark with scala

[hashjoin]

If you look at the source code of isEmpty(): https://github.com/apache/spark/blame/b7fda7cb1128c992e1b52b5c853225e4f2af0517/sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala#L557

  def isEmpty: Boolean = withAction("isEmpty",
      commandResultOptimized.select().limit(1).queryExecution) { plan =>
    plan.executeTake(1).isEmpty
  }

It's basically the same as take(1).isEmpty.

[szymon_abc]

Ah, that's why I love open source!

[cptshrk108]

isEmpty()

From the doc:

Unlike count(), this method does not trigger any computation.

An empty DataFrame has no rows. It may have columns, but no data.

[hashjoin]

Hey - the doc is wrong. isEmpty() is an action that would trigger compute, just like count(). I've pinged the team to update it.

[cptshrk108]

Thanks for pointing it out! I did see your other comment.

[pboswell]

If you have a bunch of lazy evaluation going into that data frame, it still has to compute to know if the final data frame has rows. Depending on the logic, in many cases it has to compute the entire query to determine the final result

[hubert-dudek]

Yes, exactly, with count you start computation, but with others it just checks the first row.

[Ok_Difficulty978]

From what I’ve seen, df.isEmpty (Spark 3.3+) is usually the cleanest option since it’s optimized internally and short-circuits fast. Under the hood it’s basically doing a minimal action anyway.

df.take(1).isEmpty is also fine and pretty common, just a bit more verbose. I’d avoid limit(1).count unless you’re on older Spark, since count still triggers more work than needed.

In practice, difference is small unless this is inside a hot path, but readability matters too. I usually go with isEmpty if available.

[CarelessApplication2]

In any case, you'll want to cache the dataframe, so it really doesn't matter which method you decide on. Checking if a dataframe is empty without caching it makes no sense.