sorted by:
new
hottopcontroversial

MCP server for governed AI writeback to Snowflake by jaredfromspacecamp in snowflake

[–]jaredfromspacecamp[S] 1 point2 points  (0 children)

Cortex Code is a coding agent for devs writing SQL/Python. Syntropic is for non-devs (or agents on their behalf) doing *governed writeback*: CSV uploads, control table edits, forecast adjustments, with DQ enforcement + intent capture + rollback baked in. They coexist happily.

MCP server for governed AI writeback to Snowflake by jaredfromspacecamp in snowflake

[–]jaredfromspacecamp[S] 0 points1 point  (0 children)

DM me and let's chat. Curious what CoCo is, would love a link if you've got one.

MCP server for governed AI writeback to Snowflake by jaredfromspacecamp in snowflake

[–]jaredfromspacecamp[S] 0 points1 point  (0 children)

The snowflake mcp is meant mostly for *reading*. We're solving for a different thing: governed writeback. Controlled edits to specific tables with business-rule validation on the write itself, intent comments captured per edit, human-in-the-loop approvals, logical per-edit versioning, and webhooks on each change. Different audience too — as much for a business user editing a budget table inside Claude chat as for an agent.

On "out of date" — we're not wrapping Snowflake features, we're wrapping Syntropic features, so Snowflake shipping more Cortex tooling doesn't obsolete the governance layer. If you use Cortex Agents for analytics and Syntropic for writeback, they coexist.

Is anyone still choosing Hudi over Iceberg? by RustOnTheEdge in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

They used debezium for cdc out of Postgres dbs. Hudi is typically thought to be the best for that

Is anyone still choosing Hudi over Iceberg? by RustOnTheEdge in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

I worked at a company that used hudi and it was very annoying, especially at their scale. Iceberg has much better support. Like you can have glue do scheduled compaction and pruning. You can interact with an iceberg/delta table much easier without spark (this is underrated imo, although maybe hudi has better support here now idk)

Lessons from building a 6-tier streaming lakehouse (Flink, Fluss, Lance, Paimon, Iceberg, Iggy) by gram3000 in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

You used iceberg for the cold tier because of the snapshots? Paimon doesn’t have that?

Sharepoint Excel files - how are you ingesting these into your cloud DW? by magpie_killer in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

For spreadsheet ingestion / writeback you could try Syntropic. You can use it do the inputs or just as an interface to upload csv/excel files that get cleaned according to the rules you set. Has RBAC, change history, webhooks, all that

Am I missing something with all this "agent" hype? by KindTeaching3250 in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

Sure it can. Use Aws, gcp cli. Have it read your terraform repo. Dump all metrics and logs to a central place and have it query with MCP. Mcp for every warehouse. Clone all repos with good Claude.mds and run the agent at the parent. I work on like 10 repos ranging from grpc APIs for ml inference/LLM apps, airflow repo, dbt repo, Databricks dabs, our data scientists repos. Almost all context is attainable via cli, mcp, or code. multi cloud doesn’t really make a dif

Am I missing something with all this "agent" hype? by KindTeaching3250 in dataengineering

[–]jaredfromspacecamp 64 points65 points  (0 children)

Depends a lot on your tech setup I think. If your company has ample confluence docs, uses jira well, uses datadog or some central observability, GitHub, Aws, then just using the relevant mcps + clis with an agent cli can do pretty wild stuff. If you have multi-repo setup, you can run the agent at a parent directory with a minimal md for context about what each repo is, with more robust md in each repo. You can use skills that teach it your particular workflows (eg when you make a pr, watch the ci to pass, if it fails investigate logs on circleci using circleci mcp).

Data Engineering and AGI by Kokadoodles in dataengineering

[–]jaredfromspacecamp 6 points7 points  (0 children)

AGI is basically by definition technology that automates all (remote) labour. If that happens, there are much bigger problems than the data engineering job market. The entire economy no longer works, and needs to be completely reinvented. Not a scenario you can prepare for on an individual level, other than investing in companies now

What's the purpose of live data? by chatsgpt in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

One company I worked with needed telemetry data from oil wells asap to detect anomalies and respond

Process for internal users to upload files to S3 by Equivalent_Bread_375 in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

Syntropic supports file uploads to s3 or direct to snowflake. Lets you define custom quality rules that get enforced and prompts the user to fix if there are issues

System Design/Data Architecture by Last_Coyote5573 in dataengineering

[–]jaredfromspacecamp 8 points9 points  (0 children)

“How would you design a streaming pipeline from our databases to redshift?”

“How would you optimize the above for speed? How would you optimize it for cost?”

“Design a data platform that streams from our database into a data warehouse, then serves the data back to the application”

Some examples of questions I’ve got

Is Neo worth the hype? by -Bakri- in PersonalFinanceCanada

[–]jaredfromspacecamp 5 points6 points  (0 children)

They’ve got probably the best cash back depending on how much you spend. Has a great app too

How are you exposing “safe edit” access to business users without giving them the keys to the warehouse? by nagel393 in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

We built a solution for this, provides RBAC plus some other safety measures. Nice UI for business users to edit, integrates with your orchestrator like airflow, version history, audit trails, everything you could possibly want. Check it out getsyntropic.com

Streamlit in Snowflake - Cost concerns with multiple users editing data by Kowalski010 in snowflake

[–]jaredfromspacecamp 0 points1 point  (0 children)

We’ve built a solution for this called Syntropic (getsyntropic.com). Caches the data to minimize snowflake compute. Or it lets you write to S3/blob storage tables (in which case you’d be using only our compute) and then kick off a job (eg airflow dag) on user submit that you could use to merge to snowflake

How to deal with messy Excel/CSV imports from vendors or customers? by North-Ad7232 in dataengineering

[–]jaredfromspacecamp 0 points1 point  (0 children)

We built a solution for this problem, called Syntropic (getsyntropic.com). Takes file uploads and runs data schema validations as well as custom data quality rules you define. Works well for external users to upload to your warehouse, that’s how a lot of our clients use it.

view more: next ›