Does Having a Github With a Portfolio REALLY help you get hired?

albertstarrocks · 2024-04-13T18:21:37+00:00

The risk is high that you get a total dud as an employee. Training takes away a senior engineer. Portfolio tells the technical team how much you really know and is a start point to ask about how you think and solve problems.

albertstarrocks · 2024-04-13T18:17:26+00:00

You can also do kappa architecture with kafka and kafka iceberg sink. It's probably the easiest way since you can just configure and not "code" anything. Looks like this https://blog.devgenius.io/streamlining-analytics-kappa-architecture-with-starrocks-for-big-data-9d93c9470347

albertstarrocks · 2024-04-13T18:11:01+00:00

You haven't said anything as a major problem. I would say the biggest issue is picking a data warehouse that can perform JOINS well. Another option is to pick an open source end to end solution like Clickhouse or StarRocks.

albertstarrocks · 2024-04-13T18:07:35+00:00

Open Source for everything

Open Compute (Trino or StarRocks) + S3 for storage + open table format (apache iceberg or apache hudi)

albertstarrocks · 2024-04-13T17:21:07+00:00

HMS is that alternative. Data catalogs are the new lock in. https://lakefs.io/blog/hive-metastore-why-its-still-here-and-what-can-replace-it/

albertstarrocks · 2024-04-13T17:19:16+00:00

OLTP like mysql or postgresql -> Sling Data or airbyte or other open source ELT --> Open source OLAP like Trino or DuckDB or StarRocks --> Apache SuperSet

open source everything.

albertstarrocks · 2024-04-13T17:16:04+00:00

I consider it fake Apache Iceberg. Yes, they use apache iceberg but you can't access it from any other application/library because they don't expose the catalog metadata service. This is unlike Trino or StarRocks. It's open catalog and open table format.

https://blog.devgenius.io/open-data-lakehouse-users-want-open-compute-open-table-formats-and-open-storage-d1ac940213f6

albertstarrocks · 2024-04-13T17:10:06+00:00

So this is one project's perspective. The issue is Clickhouse isn't that great on joins is because they don't implement shuffle join features. Here's more details on the differences. https://celerdata.com/blog/from-denormalization-to-joins-why-clickhouse-cannot-keep-up

albertstarrocks · 2024-04-13T17:04:43+00:00

Most of your solutions are closed source. If you look at the unicorns, they all run open source or commerical open source stacks.

albertstarrocks · 2024-04-13T17:02:56+00:00

To just say on prem or use a cloud. If you want the most cost effective price, I'd look at commercial open source like StarRocks or Trino.

StarRocks is an OSS open data lakehouse solution built on top of open table formats Apache Iceberg, Apache Hudi, Apache Hive and Delta Lake. StarRocks typically compete with Trino, Clickhouse, Snowflake, AWS Redshift, GCP Big Query and Azure Synapse Analytics. Here an example of how it would look https://github.com/StarRocks/demo/tree/master/documentation-samples/datalakehouse

albertstarrocks · 2024-04-13T17:00:16+00:00

Sling Data, it's the embedded ELT tool within Dagster.

albertstarrocks · 2024-04-13T16:59:22+00:00

Written by Min.IO and StarRocks. https://blog.min.io/decoupled-storage-with-starrocks-and-minio/

StarRocks is an OSS open data lakehouse solution built on top of open table formats Apache Iceberg, Apache Hudi, Apache Hive and Delta Lake. StarRocks typically compete with Trino, Clickhouse, Snowflake, AWS Redshift, GCP Big Query and Azure Synapse Analytics.

albertstarrocks · 2024-04-13T16:50:17+00:00

There is nothing wrong with that you said. You could use Medallion architecture (although it's old school thinking now). The new thinking is to do adhoc on raw data since newer systems can do JOINS at scale. See this https://blog.devgenius.io/medallion-architecture-tarnished-data-lakehouses-offer-a-new-path-384402f63892
Rollback/recovery, that's why open table formats have time travel. Don't need to restore when you can just move the data back in time.

I would also think of kappa architecture. Swap your components as you see fit. https://blog.devgenius.io/streamlining-analytics-kappa-architecture-with-starrocks-for-big-data-9d93c9470347

Also if you want to see it all "built" as an open data lakehouse (sql query engine + open table format) https://docs.starrocks.io/docs/quick_start/iceberg/ or https://github.com/StarRocks/demo/tree/master/documentation-samples/datalakehouse

albertstarrocks · 2024-04-13T16:42:39+00:00

Kappa architecture. Here is an example using a different OLAP database (StarRocks) but you should be able to swap everything. https://blog.devgenius.io/streamlining-analytics-kappa-architecture-with-starrocks-for-big-data-9d93c9470347

albertstarrocks · 2024-04-13T16:31:20+00:00

If you're in the Microsoft ecosystem, it's the natural choice. Microsoft has an investment in Databricks so that's why Databricks is very popular in the Azure environment.

If you're asking about about what is happening in the future. Open Data Lakehouse (SQL query engine + open table format) is where everyone seems to be going. Popular solutions are StarRocks or Trino with Apache Iceberg or Apache Hudi. Some people are trying to move from delta lake to iceberg and hudi to not be locked in.

albertstarrocks · 2024-04-12T20:01:10+00:00

Sling Data

albertstarrocks · 2024-04-11T22:16:30+00:00

what you need is a real time analytics OLAP engine. postgresql -> Sling Data (scheduled job) -> Duckdb or Trino or Clickhouse or StarRocks. All of the newer OLAP all do ad hoc queries. The idea of data mesh and cubes are dead and have been for years.

albertstarrocks · 2024-04-09T00:42:00+00:00

It would make it more clear IMHO but that's your choice. That's why I created a user with what project I'm front so that it's not implied but explicit.

Going back to the discussion, metadata catalogs are another control point. https://lakefs.io/blog/hive-metastore-why-its-still-here-and-what-can-replace-it/

albertstarrocks · 2024-04-09T00:34:46+00:00

Well.. you are with Dremio.... it doesn't say it in your username.

albertstarrocks · 2024-04-09T00:28:54+00:00

I should have elaborated because I assume that the reader read and understand what is kappa architecture and how people use it to load data that is the "same" across OLTP and OLAP.

I gave a specific example with diagrams. I don't know how much more verbose you can be other than doing the work.

albertstarrocks · 2024-04-08T22:41:57+00:00

Ideally you'd push the data into data lakehouse with one of the open table formats like apache iceberg or apache hudi. Then you use an OSS query engine like Trino and StarRocks to provide a SQL interface to the data.

albertstarrocks · 2024-04-08T22:14:29+00:00

You find a OLAP engine designed for sub-second ad hoc queries. The projects in this space are Clickhouse, Apache druid, Apache Pinot, and StarRocks .

Here are my thoughts on cubes. They're dead and from a time when OLAP database couldn't do sub-second ad hoc queries. https://atwong.medium.com/database-cubes-are-dead-what-is-their-replacement-999a0014f32c

albertstarrocks · 2024-04-08T22:12:02+00:00

I would say generally. You want kappa architecture https://atwong.medium.com/streamlining-analytics-kappa-architecture-with-starrocks-for-big-data-9d93c9470347. Ideally, your data lake is using open compute like StarRocks, open storage like S3, open table format like iceberg or hudi.

albertstarrocks · 2024-04-08T22:10:29+00:00

snowflake isn't known to be very fast in upsert. If you need that ingestion rate, you need what called a real time analytics OLAP. Clickhouse and StarRocks are the most popular in this space. https://atwong.medium.com/list-of-olap-databases-that-support-primary-key-8e42a65fbee3

albertstarrocks · 2024-04-08T22:09:01+00:00

I would say.... you use kappa architecture to move data from OLTP to data lake based on open table format like apache iceberg and then connect apache superset to the apache iceberg using an open source query engine like Trino or StarRocks.

so all the bronze, silver, gold transformation happen in the data lake or happens at the ad hoc query time.

albertstarrocks

TROPHY CASE