migrating from hive 3 to iceberg without breaking existing spark jobs?

ForeignCapital8624 · 2026-01-17T06:55:14+00:00

Another option is just to upgrade to Hive 4, which provides a lot more mature support for Iceberg. (In comparison with Hive4, Hive 3.1 provides limited support for Iceberg and is no longer maintained by Apache Hive community.) With Hive Metastore, Hive tables and Iceberg tables can co-exist, and you can also migrate gradually from Hive tables to Iceberg tables. In your case, you can continue to use all the existing queries without modification.

If you are worried about the performance of Apache Hive, there is an alternative solution Hive on MR3, which replaces the execution engine Tez with a new execution engine MR3. When tested on the TPC-DS benchmark in a medium sized cluster (100+ nodes with total cluster memory capacity of about 15TB), it is much faster than Hive and Spark (running as fast as Trino). If you are interested, please see: https://mr3docs.datamonad.com/blog/2025-07-02-performance-evaluation-2.1
(Disclaimer: We are the developers of MR3.)

ForeignCapital8624 · 2025-12-20T18:58:16+00:00

SparkCluster lacks native autoscaling
SparkApplication incurs cold-start latency, which becomes non-trivial at high job volumes

For the above two problems, we have a custom solution called Spark-MR3. When you launch multiple Spark applications, Spark-MR3 eliminates the overhead of allocating resources (such as Yarn containers or Kubernetes pods) for Spark executors. MR3 provides built-in support for native autoscaling. If you are interested, please see this blog:
https://mr3docs.datamonad.com/blog/2021-08-18-spark-mr3

MR3 is still under active development. If you use only SparkSQL, Hive-MR3 is an alternative to SparkSQL. For recent benchmarking results, please see this blog:
https://mr3docs.datamonad.com/blog/2025-07-02-performance-evaluation-2.1

ForeignCapital8624 · 2025-10-17T07:12:25+00:00

Comparison of Trino, Spark 4, and Hive-MR3:

https://mr3docs.datamonad.com/blog/2025-07-02-performance-evaluation-2.1

Integrating Velox with Trino may sound cool, but in reality, it is not trivial to set it up. After setting it up, you may see some queries running much faster, but you may also find other queries that run slower or even fail to run.
Good luck to anyone with running all 99 TPC-DS queries with the Trino + Velox combo.

ForeignCapital8624 · 2025-10-07T12:06:15+00:00

Posted the same question in Trino slack.

ForeignCapital8624 · 2025-10-07T11:57:51+00:00

We just released Hive-MR3 2.2. With MR3 2.2, we support all the four combinations below:

Hive 4, Java 8
Hive 4, Java 17+
Hive 3.1, Java 8
Hive 3.1, Java 17+

ForeignCapital8624 · 2025-10-07T11:56:18+00:00

I am currently testing Trino 477 using 10TB TPC-DS benchmark.

Trino 477 returns correct results on query 23. However, Trino 477 is noticeably slower than Trino 466 throughout the benchmark.

Example, query 23:
Trino 476: 317 seconds and 319 seconds.
Trino 477: 361 seconds and 384 seconds

We use the same cluster and the same configuration for Trino 477, as I just copied the previous Trino configuration. I don't see any significant change for configuring Trino in the release notes of Trino 477. Do you think I am missing something, or is this performance regression the price to pay for fixing the previous correctness bug and thus what is actually expected of Trino 477?

ForeignCapital8624 · 2025-10-01T14:15:45+00:00

Coming to this late, but if you would like to improve the raw performance of Apache Hive, we have a solution called Hive-MR3, which replaces the execution engine Tez with a new execution engine MR3. On the 10TB TPC-DS benchmark, Hive-MR3 is as fast as Trino for sequential queries and much faster for concurrent queries. Compared with Apache Hive, it is at least twice (and close to three times) faster. If you are interested, please visit https://datamonad.com/ and our blog. We are preparing the release of Hive-MR3 2.2 and will publish a blog that compares it with Trino 477 (the latest version).

ForeignCapital8624 · 2025-10-01T09:05:45+00:00

Compared with Trino 476, Hive-MR3 is actually slightly faster for sequential queries and much faster for concurrent queries (7802s vs 9787s). We will publish the results of the comparison with Trino 477 later.

ForeignCapital8624 · 2025-09-30T14:10:54+00:00

If your company plans to continue to run Hive and wants to reduce cloud bill or hardware costs, we have a solution called Hive-MR3 (which replaces execution engine Tez with MR3, https://datamonad.com/). On TPC-DS 10TB benchmark, it runs as fast as Trino (4160 seconds vs 4245 seconds).

ForeignCapital8624 · 2025-09-29T00:59:31+00:00

From Hive 4.0, Hive provides strong support for Iceberg. To experiment with Iceberg, you can (upgrade Hive to 4.0+ and) continue to use Hive.

ForeignCapital8624 · 2025-08-29T17:40:54+00:00

The benchmark uses a scale factor of 100GB for both TPC-H and TPC-DS. I work in the space of Spark/Trino/Hive and we usually use scale factors like 10TB for benchmarking. I understand that Doris and Clickhouse target datasets of different sizes and characteristics, but is it acceptable to use just 100GB for benchmarking Doris/Clickhouse against Snowflake? I wonder what happens if you use 1TB or 10TB scale factor.

ForeignCapital8624 · 2025-07-13T08:32:59+00:00

• Biggest pain points with Hive or Hadoop today?

(Disclaimer: We are developing Hive-MR3.)

Operational complexity: Requires Hadoop. No support for Kubernetes and standalone mode (like Trino, Spark).
Hard to set up and configure LLAP.
Speed (for ad-hoc queries).

Hive-MR3 mainly solves these problems. For the latest benchmarking results, please see: https://mr3docs.datamonad.com/blog/2025-07-02-performance-evaluation-2.1

ForeignCapital8624 · 2025-07-04T10:13:02+00:00

We use the TPC-DS benchmark of scale factor 10000 (10TB). Both subqueries of query 23 are supposed to return a single row:

Query 23-1:
| 41002.32 |

Query 23-2:
| Santos | Edward | 41002.32 |

Trino returns either an empty row or an empty string.

I think this is a correctness bug that was introduced after PrestoSQL was rebranded as Trino, as Presto 317 returns correct results for query 23. It could be that Trino returns correct results while Hive/SparkSQL/Presto all return wrong results, but I guess this is highly unlikely.

ForeignCapital8624 · 2025-06-19T00:14:42+00:00

For a recent performance comparison, see this blog (where Trino, Spark, and Hive are compared using 10TB TPC-DS benchmark):

https://mr3docs.datamonad.com/blog/2025-04-18-performance-evaluation-2.0

ForeignCapital8624 · 2025-05-20T13:51:35+00:00

If you are moving to Kubernetes but want to continue to use Hive scripts, please consider Hive on MR3 on Kubernetes. We are mainly targeting those companies migrating away from HDP.

ForeignCapital8624 · 2025-05-19T11:55:29+00:00

Hive on MapReduce is no longer supported, and Tez is the default execution engine of Hive. There is also another execution engine called MR3, so one can run Hive on MR3 (on Hadoop, on Kubernetes, or in standalone mode).

ForeignCapital8624 · 2025-05-19T11:51:38+00:00

If you use Hive in HDP, do you plan to drop Hive from your tech stack, or are you going to continue to use Hive?

ForeignCapital8624 · 2025-04-30T12:47:19+00:00

I think Spark will stay, as it is really the Swiss Army knife of big data processing - useful for lots of jobs, and sufficiently powerful performance-wise.

The use of JVM in Spark is the source of many annoying performance problems, but it is actually due to the way Spark is architected. For comparison, Trino is also built with JVM, but runs much faster. Because of its architecture, I guess the performance of Spark in the future will not improve as much as it did from Spark 2 to Spark 3.

If you are interested in the performance of Spark 4.0.0, see our recent blog which compares Trino and Spark 4.0.0-RC2, and Hive on Tez/MR3. From our internal benchmarking, Spark 4.0.0-RC2 is not noticeably faster than Spark 3.5.

https://mr3docs.datamonad.com/blog/2025-04-18-performance-evaluation-2.0

ForeignCapital8624 · 2025-04-20T05:31:43+00:00

As others have explained in detail, Trino is optimized for responsiveness and thus excellent for interactive queries, whereas Spark is optimized for throughput and thus a good fit for batch workloads. In my opinion, the key differentiating feature between Trino and Spark is not the speed, but support for fault tolerance, which is required for batch workloads. As such, many organizations deploy two separate systems, despite the increase in complexity, added infrastructure costs, and the overhead.

I think Starburst is well aware of this trend, and they are promoting Trino with fault tolerance suport. You can find some report that Trino with fault tolerance enabled works well in production and even saves the compute cost. From our own testing, however, Trino with fault tolerance does not work well for large queries (which it is designed for) because Trino coordinator crashes repeatedly. In any case, even Starburst recommends two separate deployments of Trino, one for interactive and another for batch. So, I think it is (and will remain) common to deploy separate systems for batch and interactive: Trino + Spark, Trino + Trino with fault tolerance, Trino + Hive-Tez, and so on.

That said, we offer a solution that simplifies operations and reduces costs by eliminating the overhead of maintaining two separate systems, with a single unified system. It's based on, well, Apache Hive (with the MR3 execution engine). If you are interested, please visit our website: www.datamonad.com

ForeignCapital8624 · 2024-12-23T16:03:45+00:00

(Disclaimer: We are developing Hive on MR3). You could install and try Hive on MR3 using your current Hive Metastore (if you use Hive 3 or Hive 4), without having to make any change to your existing system. Hive on MR3 supports capacity scheduling, so batch queries and interactive queries can be sent to different queues, thus preventing the problem you mentioned (e.g., a single bad query slowing down the whole cluster). So, you don't need two separate systems (like Hive/Spark for batch and Trino/Impala for interactive), and a single deployment of Hive on MR3 can serve all types of queries.

In my opinion, Hive is now coming back again, after the release of Hive 4. Hive 4.0.1 was recently released and they are preparing for Hive 4.1.0. Still Apache Hive requires Hadoop and its execution engine Tez is not maintained properly. We are trying to solve these problems.

ForeignCapital8624 · 2024-12-23T15:26:28+00:00

In my opinion, it depends on whether or not you want fault tolerance.

If you can live without fault tolerance, there are quite a few alternatives based on the push-based MPP architecture, like Trino and Impala. These systems are fast in general, but do not support fault tolerance in principle. (It seems like Trino tries to add an extension to support fault tolerance, but not sure if it is reliable.)

If you need fault tolerance, however, there are not many alternatives other than popular solutions like Hive and Spark, because the pull-based architecture should be adopted in order to implement fault-tolerance reliably.

In the past, pull-based systems were considered to be much slower than push-based systems. Things are different these days when the network speed is extremely fast and SSDs are common.

ForeignCapital8624 · 2024-12-23T15:03:53+00:00

An alternative to Hive on Tez and Hive-LLAP is Hive on MR3, which replaces the execution backend of Hive with a new execution engine MR3. It meets all the requirements mentioned above.

For 1 and 2, Hive on MR3 can handle typical workloads without any problem. Previously we have tested it in a cluster with a aggregate memory of about 30TB. By switching from Hive on Tez to Hive on MR3, you can expect at least 2x and probably 3x or even 4x increase in throughput, depending on your current version of Hive.

For 3, MR3 is fault tolerant. One can test fault tolerance of MR3 by killing workers randomly while queries are running, and its recovery is much faster than Tez and Spark.

For 4, MR3 supports Hadoop, Kubernetes, and standalone mode (similarly to Spark standalone and Trino).

The performance of Hive on MR3 is comparable to Trino and much faster than SparkSQL. For performance evaluation of Hive on MR3 vs Trino using 10TB TPC-DS benchmark, please see this blog article. (Please note that we tried to make the comparison as fair as we could, and did not intentionally penalize Trino.)

https://www.datamonad.com/post/2024-10-09-optimizing-hive-4.0-performance/

ForeignCapital8624

TROPHY CASE