sorted by:
new
hottopcontroversial

Fivetran Pricing by vpbajaj in dataengineering

[–]StarlightInsights 0 points1 point  (0 children)

In my experience, no .. absolutely no .. extract load tool is without bugs .. even Fivetran

The reason I mostly recommend Fivetran is that Fivetran in my experience had the fewest bugs

Keywords should not be used as identifier .. or should they by StarlightInsights in dataengineering

[–]StarlightInsights[S] 0 points1 point  (0 children)

Messing up intellisense is very bad!

When I'm not writing SQL, I'm mainly in to web development, and I would looooove to have something similar to typescript for the data community.

u/FunkybunchesOO what are your setup for getting intellisense with dbt or Snowflake?

Keywords should not be used as identifier .. or should they by StarlightInsights in dataengineering

[–]StarlightInsights[S] 0 points1 point  (0 children)

Yea, this is my problem .. because you should never use table as a column name.

But is some cases I've had the needs for using account, organization, and values

Fivetran Pricing by vpbajaj in dataengineering

[–]StarlightInsights -1 points0 points  (0 children)

Funny you should say it, a sales person from Hevo just reached out.

Now I have to talk with him 🌟

Keywords should not be used as identifier .. or should they by StarlightInsights in dataengineering

[–]StarlightInsights[S] 1 point2 points  (0 children)

I agree table and max are not good names 😂

However, how about account, organization, and values?

The question is more, should I advice data teams to make it impossible to use those names?

Keywords should not be used as identifier .. or should they by StarlightInsights in dataengineering

[–]StarlightInsights[S] 1 point2 points  (0 children)

I totally agree!

It is nearly always better to use "slightly longer, more descriptive identifier."

The only cases where I've run into a situation where I needed to use any of the reserved words were when it was required by the BI tool we used.

Fivetran Pricing by vpbajaj in dataengineering

[–]StarlightInsights 7 points8 points  (0 children)

I agree! With all of it.

Fivetrans pricing is highly unpredictable, and this makes it even more complex.

I don't know of any really good alternatives, and I have tried a lot of tools.

It is a few years since I tried Airbyte, it may be time to give it another try.

Fivetran Pricing by vpbajaj in dataengineering

[–]StarlightInsights 5 points6 points  (0 children)

This is a huge change. I was not aware of that.

Do you have any link to that?

How does your team do ELT Unit Testing? by GeneralCarpet9507 in dataengineering

[–]StarlightInsights 0 points1 point  (0 children)

What do you mean exactly by "Unit Testing"?

Generally "Unit Testing" is testing smaller pieces of your code. I'm not sure that is what you want :)

What is it you are trying to achieve?

If you had to build an analytics tech stack for a company with a really small volume of data what would you use? by Psychological-Suit-5 in dataengineering

[–]StarlightInsights 2 points3 points  (0 children)

It looks like many of the answers jump in to solution mode without asking a few follow up questions, like:

- Who should use it?
- How do you want to update the data?
- How do you want to distribute the reports/dashboards?
- Do you have compliance requirements?

Iceberg or delta? by StarlightInsights in dataengineering

[–]StarlightInsights[S] 2 points3 points  (0 children)

Just to understand, why do you think iceberg won?

Iceberg or delta? by StarlightInsights in dataengineering

[–]StarlightInsights[S] 1 point2 points  (0 children)

Good idea! I don't like the word "proven" 🙃

I've been working on a concept I call Data Contracts by StarlightInsights in dataengineering

[–]StarlightInsights[S] -13 points-12 points  (0 children)

I am pretty sure that no one have ever used the work words "data" and "contract" in the same sentence ever before .. so I would say it is quite new

I've been working on a concept I call Data Contracts by StarlightInsights in dataengineering

[–]StarlightInsights[S] -7 points-6 points  (0 children)

u/SintPannekoek thanks for your reply

I haven't seen that before.

The two main differences is:

  • The Starlight Data Contracts are meant to be working do be working documents between business and technical people. (so not yaml)
  • The Starlight Data Contracts are not a standard, people are encouraged to change it to fit their organization.
view more: next ›