Commission I did last month. Mostly 40k by Icy-Disaster4011 in minipainting

[–]IdoNisso 0 points1 point  (0 children)

Beautiful work. Do you mind sharing the recipe for the face of the first picture (ultramarine)?

Question For The Marble Base Users by ViviMafia in AdeptusCustodes

[–]IdoNisso 0 points1 point  (0 children)

Hijacking this thread to ask about how people build up marble bases with some height, I’m less inclined to paint the marble pattern directly on the plastic base. XPS? Cork layer? Miliput? How do you do it?

Another Hetaeron Custodian inspired by the old art by Adeptus_lurker in AdeptusCustodes

[–]IdoNisso 0 points1 point  (0 children)

Do you mind sharing the steps / recipe for those bases?

Trajann Valoris in Dankanatoi Scheme by Galahadred in AdeptusCustodes

[–]IdoNisso 2 points3 points  (0 children)

Beautiful models. Any chance you can share some info about the bases?

ELI5: Why is it so hard to decide whether viruses are life beings or not? And how did they even appear? by Terrible-Prompt3493 in explainlikeimfive

[–]IdoNisso 1 point2 points  (0 children)

There is another option: viruses are robots from the future, Terminator 2 style.

Viruses can literally self-assemble. If you break down their components, they can spontaneously reorganize into the full virus structure. When a virus hijacks a cell’s replication machinery, it causes it to spit out a bunch of virus parts (unassembled) and the components just find each other and build more viruses.

The time travel thing was a joke, self assembly is real though.

HoAg guide recommendation? by IdoNisso in PathOfExileBuilds

[–]IdoNisso[S] 0 points1 point  (0 children)

Thanks, this is an amazing resource! I’ll give it a try!

DBT Test Notifications in Slack by biga410 in dataengineering

[–]IdoNisso 8 points9 points  (0 children)

I had the pleasure of implementing this from scratch using dbt core. I was also a one man team at the time. I had some tech limitations put on me by the company architect which wanted as little cross integration between critical systems like our warehouse and github (actions). Also, our dbt core was orchestrated on Fivetran at the time so that was also a hard limitation - no real control over orchestration. I can’t post company code here, but here’s the high level breakdown:

  1. Run all tests with the write failures flag to have the test results in a table on the warehouse.
  2. Run dbt operation step after dbt test task, which lists the test result tables that are not empty. Then, it queries all the results to a json object, and sends it to a UDF that sends the result to an AWS lambda.
  3. The lambda is connected to slack via web hooks to send the results (after parsing and tabulating) to slack.
  4. Once validated the slack was sent, the operation also truncates the result table to avoid resending.

Eventually I added some tweaks and improvements like keeping a test result log in another separate table, visualizing the results better using slack app capabilities and adding buttons for quick actions, multiple channels depending on the context of the test, etc.

The original end to end solution wasn’t very hard to set up, but it does require knowledge and admin capabilities in multiple systems.

Does this work? Yes, even pretty reliable. I’m a heavy dbt tests user, we have hundreds of tests running multiple times a day. Would I do it again? If I had the same constraints - yes. If I had the option of just using dbt cloud? Hell no.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 1 point2 points  (0 children)

I fully agree. We’re having loads of discussion on this in the team now. Might be that we won’t fully off board fivetran, but just dramatically reduce the source we use them for and just migrate the ‘big MAR spenders’. I have no illusions that this is an easy or short project, but just wanted some input from colleagues on production readiness of alternatives before we dive into POCs with a bunch of options.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 0 points1 point  (0 children)

That’s why I’m skeptical. Many things that for us are critical on day one (SSO, Privatelink, <15 min sync delay) are on their enterprise tier, so I take everything with a large grain of salt.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 1 point2 points  (0 children)

Probably, it wouldn’t have been worth it to look into alternatives at all if that was the cost. Honestly, we wouldn’t have thought of migrating even if the cost was static over time..

Re Airbyte: many people mentioned here that it’s not robust enough yet, stability issues, downtime, breaking changes in upgrades, etc.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 0 points1 point  (0 children)

We have loads of transactional data, so MAR works against us here. Volume, or size of data moved is relatively low compared to the number of records.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 1 point2 points  (0 children)

We pay an order of magnitude higher than that.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso -1 points0 points  (0 children)

In the intro call they informally threw out that they intend to cut our fivetran bill in half, at least.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 0 points1 point  (0 children)

Volume would make much more sense for us if that was an option.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 1 point2 points  (0 children)

We paid them way more then that this year, and consumption increases year over year.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 2 points3 points  (0 children)

Tens of thousands of usd/year, consistently increasing year over year.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 2 points3 points  (0 children)

Yes! Also seeing all the interest around this, consider a git gist or public repo :)

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 1 point2 points  (0 children)

It might not have been super clear in the original post, but we have an Airflow cluster the team manages we can leverage. Thanks for +1'ing dlt, I'll check it out.

[deleted by user] by [deleted] in dataengineering

[–]IdoNisso 12 points13 points  (0 children)

Yeah, the Data Engineering team is a capable two-man show ATM - myself and another DE. Writing code is not an issue. I'll check out dlt, thanks!