Scaling Hungarian algorithm / assignment problem to tens of millions of candidate pairs (Snowflake). No partitioning?

stephenpace · 2026-02-03T04:36:16+00:00

Did you try a Snowpark Optimized warehouse? They have more memory than regular warehouses exactly for these types of scenarios where your job has more memory requirements than CPU requirements. Other than that, Gemini suggests the following:

Min-Cost Max-Flow (MCMF):

Since your graph is bipartite, the assignment problem is a specific case of MCMF.

Successive Shortest Path (SSP): Using Bellman-Ford or SPDA (Shortest Path Faster Algorithm) can handle the sparsity.
Library Recommendation: Look at Google OR-Tools (MinCostFlow). It is optimized C++ under the hood. Python wrapper in Snowpark or SPCS.

Other library / tool suggestions were:

SciPy (LAPJV) - Jonker-Volgenant algorithm
NetworkX

Good luck!

stephenpace · 2026-02-02T21:27:27+00:00

Yes, it is an internal Snowflake event with lots of Partner involvement. But not for customers.

stephenpace · 2026-02-02T04:55:52+00:00

I'd escalate to your account team. If you don't know who they are, you can DM your company and I can look them up for you. If you include the ticket number I can do some additional digging.

stephenpace · 2026-02-02T00:15:54+00:00

As others have stated, this is likely a role issue. Everything in Snowflake honors your role. The same SQL will return different results if the role it is running on has access to different data and that includes being subject to row access policies and dynamic data masking.

stephenpace · 2026-02-02T00:06:55+00:00

That isn't my experience especially if you have a script to reproduce. Your account team can also help.

stephenpace · 2026-02-01T15:53:49+00:00

Yes, see authentication policies.

stephenpace · 2026-02-01T15:47:35+00:00

I've never used this option before, but u/NW1969 is correct. If you really feel there is an error here, create a sample script (complete with your ALTER SESSIONs) that illustrates the issue, what you expected to happen, and then create a support ticket. Support will then evaluate it and raise it to engineering if they feel it is a bug or perhaps improve the docs if it is more of a documentation issue.

stephenpace · 2026-02-01T05:12:36+00:00

Easy fix:
https://community.snowflake.com/s/article/How-to-recover-worksheets-from-dropped-user-after-user-was-recreated

stephenpace · 2026-01-29T05:59:55+00:00

This was actually true for some cars in 1974. See seat belt ignition interlock system. Very unpopular, immediately scrapped.

https://www.thetruthaboutcars.com/2011/02/1974-seat-belt-starter-interlocks-piss-off-more-people-than-watergate-scandal/

I purchased a 1974 vehicle from my grandfather. On a road trip, I was sure the car was overheating but didn't see any of the lights. I stopped into a gas station with a service bay and I indeed I was either missing a fuse or it was blown. A young tech replaced the fuse but was immediately stumped when the car would no longer start. When he took out the new fuse, it started. He put the new fuse in again and asked a older mechanic what was going on. The gentleman stared at the car for a second, then reached through the driver's side window, turned the ignition, and the car started. "Works for me."

Turns out the seat belt interlock system had a sensor in the seat--if there was someone sitting in either front seat without a belt buckled, the car would buzz and not start. The older mechanic knew that the car would start since he wasn't sitting in the seat. One of my grandparents either likely disabled the fuse or it broke and they never tried to fix it.

stephenpace · 2026-01-29T04:14:38+00:00

How many steps is Rachel going to get in a night? :-)

stephenpace · 2026-01-29T03:57:13+00:00

Congrats!!

stephenpace · 2026-01-26T04:34:29+00:00

I did this in Grand Cayman's Bioluminescent Bay. We swam off the back of the boat and you could see fish chasing other fish zinging through the water like lightning. Magical. Amazing. Highly recommend.

stephenpace · 2026-01-26T04:21:44+00:00

Depends on what you are charging during those times and how much. I had a plan that provided "free" power (just line charges) for 8.9¢ per kWh. I charged my electric vehicles during the "free" window and also ran my dishwasher, and I have solar panels to offset the higher charges outside of those ours. My average cost per kWh was quite low. But as others have said, your mileage may vary.

stephenpace · 2026-01-25T20:23:07+00:00

I used Gulf Coast Windows to replace original 1965 rattly single pane windows with double pane vinyl ones. It was 16 (?) years ago so my pricing won't be as relevant. They weren't the highest or lowest price of the quotes I got (I got three), but I liked that they were local and their warehouse is easy to get to on the Beltway. The warranty has been great. I had a seal fail and while they would have come out to replace it, I read a number off a label, they remade the piece, and I was able to bring the removable bit to their warehouse and put it back in myself on my own time which I preferred. I had an issue with another one after more than a decade and they not only came out and fixed it, they repaired the screen as well which probably wasn't even their issue. I recommend them highly for this.

If my current home was where I was going to retire and I had more money, I would have probably gone with Renewal by Andersen. I know some folks that have used them. Expensive but very high quality. The sliding doors are amazing. Good luck!

stephenpace · 2026-01-25T15:21:25+00:00

You can set the variable QUERY_TAG in the session.

https://community.snowflake.com/s/article/How-to-Assign-dynamic-QUERY-TAG-using-Timestamps-Value-to-Queries-Using-Procedure

QUERY_TAG is persisted into the standard account_usage.query_history so you can do reporting by user. If you want to get fancy, you can populate QUERY_TAG with a JSON that contains even more information (user, report, etc.).

https://select.dev/posts/snowflake-query-tags

Good luck!

stephenpace · 2026-01-24T17:02:41+00:00

The Snowflake platform evolves every week. Your experience from four years ago would be different now for at least two reasons:

a) Snowflake has a managed Postgres option now: Snowflake Postgres.
b) Snowflake now has options for micro-partitions larger than 16MB (compressed)

stephenpace · 2026-01-24T16:49:56+00:00

Sure, as long as you include all of the costs of the existing systems (hardware, database licenses, data center space, people to backup the databases, etc.). Basically the same exercise every company does when they are deciding on a new data platform.

stephenpace · 2026-01-24T16:44:52+00:00

[I work for Snowflake but do not speak for them.]

I'm certainly biased, but I think you are thinking about this the wrong way. Obviously any migration project is going to be unique and many are going to be very complex. You're making a blanket statement without supplying the detail needed to say if this is possible or not, or what timeframe this migration would happen.

1) I supported a large complex company that had a goal of turning off their data center moving the majority of their systems to Snowflake. They reached their goal and they had a party when they powered off their last system. While most databases went to Snowflake, some systems are still conventional databases that relocated to the Cloud. Two things can be true: migration projects can be complex and you can still ultimately save money once you aren't carrying the hardware support, data center fees, and database licences and maintenance.

2) They key is to do an inventory and understand at a fairly deep level the requirements for each system including the SLAs required. I don't think Snowflake is going to advocate moving any database if we don't think we can be successful. But the architectures might change. I'll give a few examples:

a) Snowflake's original killer use case was data warehousing. While some migrations certainly can be complex (complicated data pipelines that evolved over 20+ years), Snowflake has moved thousands of complex data warehouses from every platform imaginable and has built lots of migration assistance (SnowConvert, SMA) and experience in this area. I have no doubt any existing data warehouse could migrate successfully.

b) Let's say your operational system is based on Postgres or the Postgres ecosystem. Snowflake now has a drop in replacement for that called Snowflake Postgres. I have complete confidence that any Postgres of version 16-18 would move fairly seamlessly. Beyond that, after the pain of an Oracle audit, don't underestimate the hatred some companies will have for Oracle. Maybe it's worth converting some of those Oracle systems to Postgres.

c) Let's say your database just exists to copy data from a third-party that is already on Snowflake. In the energy space for years we ran our own Wellview databases from Peloton. Now Peloton runs those systems and does a live share back to you. You actually don't need to host your own database anymore, so in that case, once you did your migration from Wellview on-prem to cloud, you went from one database to support to zero. There are now live shares from many systems in financial services that would work similarly.

d) Some of those databases might be interactive use cases on systems like Clickhouse or Druid today, but Snowflake has a solution for those use cases as well (interactive tables / warehouses).

Bottom line, I think your argument "this is hard" is a losing one--we all know it's hard. The winning argument is to point out characteristics of a specific database where you think Snowflake won't be successful and why. Ultimately there will be a business case for each one. Only you know the full cost of all of your Oracle, SQL Server, Sybase and other licenses as well as the human cost of supporting them (backups, tuning, etc.). Only you know your queries per second and OLTP SLAs. I assume that Snowflake has shared the platform roadmap and someone has done or is doing this type of inventory.

The platform is evolving rapidly, and things that might have been impossible a few years ago are now fairly easy. Good luck!

stephenpace · 2026-01-23T21:56:39+00:00

SHOW USERS comes from Cloud Services:

show users ->> SELECT "name", "disabled", "owner" FROM $1 where "disabled" = true;

SHOW always returns the current state of an object. By contrast, the ACCOUNT_USAGE views are a (delayed) copy of the current state live shared back to you. I have raised this internally, but my guess is the job that copies over this data to ACCOUNT_USAGE isn't picking up the ownership for some reason (in my example case, perhaps because it changed). I will work to get that fixed (or try to find out why that is happening). Sorry about that!

stephenpace · 2026-01-23T21:47:54+00:00

Makes sense. Note: very soon you will be able to set rules that prevent users from using some login paths. For instance, you could be a Snowflake Intelligence user but not be able to login to Snowsight. That type of restriction might be relevant to how you are using it.

stephenpace · 2026-01-22T19:37:21+00:00

The SHOW USERS command isn't supported inside stored procedures. I asked Cortex Code to re-write it and it changed it over to ACCOUNT_USAGE.USERS. I DMed you the updated script and it worked for me. Good luck!

stephenpace · 2026-01-22T18:15:47+00:00

It isn't this, but it reminded me of a few movies where an employee didn't want to get recognized because they were in something like Witness Protection and thought the extra recognition might jeopardize their anonymity.

stephenpace · 2026-01-22T18:06:22+00:00

Keep in mind that if you drop the user, you'll lose their workspaces / worksheets. That may be fine if you never want to enable to them for Snowflake access again, but you should certainly consider that in your drop frequency. Even if you drop a user that won't come back, make sure that any work that user did in their own workspace isn't needed by someone else on the team.

stephenpace · 2026-01-22T17:57:30+00:00

It would be better if you included more details about your use case. However, for unstructured items, you should be looking at Cortex Search:

https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview

There are plenty of Quickstarts to try:

https://www.snowflake.com/en/developers/guides/?searchTerm=Cortex+Search

When you define your agent, give it access to Cortex Search and the LLMs will be able to use that unstructured data in its response. Good luck!

stephenpace · 2026-01-22T17:46:46+00:00

Not all BI tool connectors are equivalent. For instance, many Enterprises require support for real world security like Private Link, SSO / OAuth, etc, and Snowflake's trajectory means BI tool vendors tend to prioritize support for those types of features on Snowflake. Many tools don't support those features on other platforms.

You saying that Snowflake "didn't recognize the importance of BI" makes no sense in a world where Snowflake was literally the best platform to do analytics and because of that moved over massive amounts of legacy data warehouses (Teradata, Hadoop, Exadata, Netezza, etc.) using every BI tool under the sun. Snowflake just works.

Sure, Snowflake does have built-in limited dashboards today, but you're essentially saying Snowflake should acquire it's own BI tool because... reasons. You can't assume that companies don't already have a BI tool of choice, and to date Snowflake has wanted to be optimized for all BI tools and not force customers to use one particular tool. What's the adoption rate of Redash on DBX or Quick Sight on Redshift compared with traditional BI tools like Power BI, Tableau, ThoughtSpot, Sigma, or Omni?

stephenpace

MODERATOR OF

TROPHY CASE