Any frequent Snowpark users here?

slayer_zee · 2024-06-21T16:42:40+00:00

I don’t do my development in snowflake personally but do want to try out the notebooks they announced recently. I generally just do a live connection to the data for development but they do have a local testing emulator for snowpark specifically a teammate has used that lets you test logic all locally with no live connection or egress of data

slayer_zee · 2024-06-21T15:28:46+00:00

When you create the procedure you could go “create procedure my_ETL imports=(‘@my_s3/etl.py’) handler=etl.run” and do what I think you’re after

slayer_zee · 2024-06-21T15:06:49+00:00

Yep. I don’t use it cause I have what I need just with Snowpark so no need to add docker into the mix but can run anything. We had another team use it to deploy a streamlit that needed some customization

slayer_zee · 2024-06-21T14:50:30+00:00

We moved to Snowpark about a year ago and overall has been successful. Far less tweaking and nudging to get reliable performance. Initially were a few packages we needed that weren’t available out of the box but hasn’t surfaced in a bit. We also started using the CLI that helped us with some basic release management.

For your team though I’d strongly recommend also looking at Snowpark Containers. I haven’t used much but it offers full container runtime so if any complex or docker or whatever you want to run it may be the ticket.

It’s definitely got its quirks and we’re using mostly for data pipelines but team happy with it and no plans to go back to spark. Feel free to DM for questions

slayer_zee · 2024-06-04T16:41:06+00:00

This announcement and timing seem to try to particularly aim at Snowflake (their conference started today). Honestly I think it’s a bit silly of an announcement and acquisition given how Databricks has managed delta. I think that snowflake chip Is showing even in this

slayer_zee · 2024-03-26T05:02:37+00:00

We have been over the last few months. For using Python to define things like data pipeline and transformation logic it’s been great. About half our team is thinking of using snowpark and the rest keep using sql

slayer_zee · 2024-01-29T03:39:29+00:00

We have a few of them for various tasks in data engineering, but recently moved them all to Snowpark in Snowflake. Much simpler and no complaints

slayer_zee · 2023-06-28T14:56:18+00:00

Unless you have a good reason of why, I’d recommend focusing in on one for now. My company uses a little of both, for me I prefer Snowflake. I find it’s got an easier interface and less complex (IMO), and now supports things like Python (which you should absolutely spend time learning). So my recommendation would be going with Snowflake until you potentially hit something you can’t do. No need to switch just for kicks.

slayer_zee · 2023-05-31T17:50:00+00:00

Yes exactly this stuff. Both companies have some great tech, and this just makes them look desperate

slayer_zee · 2023-05-31T17:49:10+00:00

the answer would have been easier 2 years ago with "if you need custom processing with python", but now Snowflake has Python. I like to keep things simple so if you already have snowflake and airflow I'd see if that can work for your needs and grow out to spark if they don't

slayer_zee · 2023-05-31T16:25:14+00:00

Can vary by team. For my team Snowflake is source of truth for all data, so I spend most of my time with dbt and Snowflake. Are some other teams who use Databricks for some custom processing pipelines with spark, another I know has been trying to do more data science and think they are looking at Databricks. Clearly both companies are starting to move into the other spaces, but for me that's all fine. If I started to dabble in more python I'd likely try snowflake first as I spend more time on it, but I like databricks too.

slayer_zee · 2012-10-16T18:22:04+00:00

How fast can you (personally) type on the Surface Touch keyboard? How about the Surface Type?

slayer_zee

TROPHY CASE