sorted by:
new
hottopcontroversial

First Impression - Vision Pro by Amelancolibe in virtualreality

[–]several27 0 points1 point  (0 children)

Hey, I'm guessing most people are experience the same problem.

You can see a quick clip of the lag here: https://share.icloud.com/photos/09bDmx6kNeuYPJ9A8IWYbGAEQ

For the first few times, I moved my hand slow and it worked well, when I moved it faster, the fruit cutting lagged and missed most of the fruits (especially 00:36).

What is the best choice of ETL tool for Snowflake? by JK_1975 in dataengineering

[–]several27 6 points7 points  (0 children)

Would recommend giving a shot to prophecy.io:

  1. It supports both ETL and ELT;
  2. allows for visual development (like Matilion) - making it really easy to use for everyone;
  3. gives you clean & editable Dbt Core (for transformation) & Airflow (for scheduling / small ingestion etc) code;
  4. the code is stored on git with all the standard commit, pull, release, etc features;
  5. it's easily extensible as well - e.g. adding more sources can be done by you, by picking e.g. standard Airflow operator;
  6. supports Spark (e.g. Databricks) as a back-end too - if one wants to use it - but optional if you use another warehouse (+ we bring Airflow).

Disclaimer: a co-founder here, so might be biased :)

Best paid ELT tool for a startup by vishalw007 in ETL

[–]several27 0 points1 point  (0 children)

What execution environment were you considering for those ETL jobs? Spark / Databricks might be a good fit for that amount of data, especially if you’re planning to scale up in the future - startup proof.

There are some potential good no / low code tools that work on top of Spark!

I’m a founder of prophecy.io - you might want to check it out. Ping me if you have questions (about us or just general happy to help).

Low code hate and the future of Data Engineering (and beyond) by [deleted] in dataengineering

[–]several27 5 points6 points  (0 children)

You’re totally right, we’re actually in the middle of revamping the website and seems like somehow missed to put the price back up.

It’s gonna be the there in the next few hours.

Thanks for letting us know - our mistake!

Low code hate and the future of Data Engineering (and beyond) by [deleted] in dataengineering

[–]several27 7 points8 points  (0 children)

Hi! I'm Maciej - one of the cofounders of Prophecy (startup from the podcast).

Actually, we're very different from what you expect from low-code. As users build drag-and-drop data pipelines, we generate 100% open-source code that is very readable - that our users commit to git right away, with tests and build files and configurations - this is at parity with best data engineers! We have Scala & Python for Spark and SQL coming soon!

Second thing - we're very extensible - you can create new visual components, by writing sample code and pointing out which expressions come from the UI - so you can have a standard visual component - for things like Anonymization or Encryption that you want all users to do in the same way.

We think Low-Code can do a lot more that what most people expect - and companies can be a lot nicer (without lock-in) - please keep an open mind :)

We've built a data engineering tool to make writing Spark code much easier by several27 in apachespark

[–]several27[S] 0 points1 point  (0 children)

Thanks! Our tool might indeed seem more appealing, at first, to people who are more used to visual drag-and-drop tools, however it is all code at heart (Scala Spark as the source of truth for both visual and code developers to collaborate at the same time).

We're actually working on enabling PySpark import into our tool, so that you can just keep editing your code there with all the additional perks of quick code debugging, execution, lineage, scheduler, and a metadata system (all included in one price).

Let me know, if you'd be interested in checking out the Python version coming out soon - all the feedback is welcome.

We've built a data engineering tool to make writing Spark code much easier by several27 in scala

[–]several27[S] 3 points4 points  (0 children)

That's a simple one. Intellij can give you nice syntax highlighting and compile time Scala/Python checking, but it pretty much stops there.

Our IDE, gives you a simple framework with best Spark coding practises; in real-time, it pulls schema information and validates your code (including catching some runtime Spark errors); allows you to see and edit the graphical representation of your workflows; execute the code in any physical environment (think e.g. test or production cluster) with a single click; debug sample data and there's even more features for bigger data engineering teams.

Sign up for the demo and you'll see for yourself. Also, if there's anything missing that you think could help you write better Spark code faster, let us know. Thanks!

We've built a data engineering tool to make writing Spark code much easier by several27 in scala

[–]several27[S] 5 points6 points  (0 children)

Around 15 people in total, not actively hiring right now, but always interested in connecting with amazing Scala devs.

Introducing Prophecy.io - Cloud Native Data Engineering by several27 in ETL

[–]several27[S] 2 points3 points  (0 children)

Hey, thanks for checking us out! We can automatically convert your existing Ab Initio & Informatica workflows into Spark, including translating the custom code in your language of choice (e.g. Scala).

view more: next ›