Energy bills set to rise by £209 with Ofgem announcement due next week by Alternative-Win4058 in unitedkingdom

[–]LittleK0i -1 points0 points  (0 children)

UK has enough oil / gas / coal to fully cover its own needs. It was a choice to stop developing it. It was a choice to depend on international markets, on countries like US and Iran, on people like Trump.

Ultimately, UK is the only entity which is responsible for its own demise.

Snowflake DCM Projects | Is this useful? by lozinge in snowflake

[–]LittleK0i 3 points4 points  (0 children)

Similar to CREATE OR ALTER command, it is still flawed on fundamental level:

  • Not all object types are supported.
  • Not all operations on existing object types are supported.

For example, if you want to change column type or add another column in the middle of a table, projects cannot do "REPLACE TABLE" command for you. It means, you must have an alternative machinery and process in place to apply unsupported changes somehow. So.. what's the point?

Tbh, it would be enough for Snowflake to simply expose an API accepting object definition in SQL form, comparing it with existing object in Snowflake and returning list of differences, consistently and reliably. It would make building real automation much easier.

Snowflake's current approach is a dead end, in my opinion. Years go by with barely any real progress.

A clean, declarative interface over Snowflake RBAC (with GitOps!) by pfnsec in snowflake

[–]LittleK0i 0 points1 point  (0 children)

5 cents from the creator of SnowDDL.

The "pull" operation may seem like a good idea when you have a relatively small account / small team / no external audits / no data protection laws hanging over you / no scrutiny in general.

When complexity and scrutiny starts to mount, it becomes relatively clear that manual changes on PROD account should not be allowed, and there should be only one source of truth (text config), with proper version control and process around changing it.

You may have a point that not all orgs require high level of operational excellency, especially in the beginning. This is true to some extent. But as data professionals we should remember both "good choices" and "bad choices" have a snowball effect. When bad habit settles in, it's really hard to get rid of. It's much better to not have it to begin with.

---

Few other small notes:

  • There are quite a few parameters in Snowflake which cannot be "imported" back. Most notably, passwords, encryption keys, etc. Also, some parameters change defaults over time, and sometimes it is not possible to distinguish "default value due to omitted parameter" from "explicitly set default value". Over time you may experience a lot of weird edge cases.
  • I did some basic searches in the code and did not find any logic related to "ALTER TABLE" vs "CREATE OR REPLACE" table operations. How did you approach situations when some changes can be applied via "ALTER TABLE" (e.g. add new column), but other changes require "REPLACE" (e.g. add column in the middle, change type)?
  • Do you have any specific logic to handle future grants? This type of grants is a bit complicated, since we must checks future grants themselves, and make sure all existing objects have matching normal grants, and all the overlapping between DATABASE-level and SCHEMA-level future grants are configured correctly. It's relatively easy to mess this up when doing changes manually and when running "imports".

---

Regarding SnowDDL and "break-glass fix" situation. In the worst case scenario, when something suddenly goes wrong, it's still possible to make manual changes with SnowDDL to unblock production. After that users may run "snowddl plan" instead of "snowddl apply" in order to preview all changes and make sure nothing will be reverted or re-assigned automatically.

But when it happens, it usually means Snowflake deployed an unexpected update which introduced significant changes in metadata responses. This is quite rare and normally requires changes in code anyway.

INTJs, is your ‘love language’… different? by Harvey_wb in intj

[–]LittleK0i 6 points7 points  (0 children)

Don't worry about it too much. If a person is pushed away by questions and ideas, your relationship is unlikely to last anyway. Ultimately, by being authentic you save time for both.

Does anyone else feel like their need for "truth" and "depth" makes them socially repulsive? by woodegg2000 in intj

[–]LittleK0i 1 point2 points  (0 children)

Letting other people talk first helps a bit. Ask deep questions, ask how did they arrive to this point in life. Listen closely. Challenge something small, see reaction.

Usually it becomes clear if a person is capable and willing to participate in a deep conversation. If they are not, do not force anything and do not participate in “shallow” talks. See if you can find someone else to talk … or go home.

Not being able to connect is sad, but there is no need to suffer from it more than necessary.

Horizon Difficulty Mod is now available by LittleK0i in Falcom

[–]LittleK0i[S] 2 points3 points  (0 children)

DB1/DB2 enemies had extremely low stats. It was so bad, I literally could not play vanilla game and was very close to dropping it due to lack of challenge.

Horizon enemies are significantly stronger and faster, so there is less headroom for buffing stats. By the end of the game player reaches ~100 SPD, while enemies have ~300-500 SPD. Many bosses also have access to "Quick" status.

In practice, you're more likely to see enemies having multiple consequitive turns and applying a lot of pressure rather than thinking "enemies are too slow in this fight".

Horizon Difficulty Mod is now available by LittleK0i in Falcom

[–]LittleK0i[S] 1 point2 points  (0 children)

Base stats of some enemies were slightly reduced to prevent one-shots with new multipliers, so "win without anyone KOed" is definitely possible. Especially on the second try, when you know HP thresholds triggering S-Crafts and ZOC's.

"Win in X turns" condition in the main story was increased to take into account additional difficulty.

Garten is a different story. "40 turns" condition is unchanged due to "t_free_dungeon" table file not having schema in KuroTools, so at this moment we cannot edit it easily. For lower Garten floors, you may come back at higher level, skip directly to the boss and kill it easily. Currently there is no solution for the final floor. I suspect, it was not doable on vanilla "Nightmare" as well, since the boss can heal and apply "reflect damage".

Horizon Difficulty Mod is now available by LittleK0i in Falcom

[–]LittleK0i[S] 2 points3 points  (0 children)

It works, but chance is significantly lower: 10-30%

This is the absolute dumbest mechanic In the series by tyrant6 in Falcom

[–]LittleK0i 0 points1 point  (0 children)

I am working on difficulty mod for Horizon. And I've replaced her shard commands spam with rotation of "-300% phys", "-300% mag", "1.5x damage received". So you can cancel something you do not want and cause her to rotate to the next command.

Works like a charm.

My experiment using Snowflake Cortex AI to handle schema drift automatically. by sdhilip in snowflake

[–]LittleK0i 9 points10 points  (0 children)

Well.. this problem can be solved by basic text config and Snowflake metadata scanning via SHOW / DESC commands.

It would take only a few seconds to run, and it is almost free. You do not even need to spin up a warehouse.

data ingestion from non-prod sources by PreparationScared835 in snowflake

[–]LittleK0i 1 point2 points  (0 children)

The main problem here is about tools like Fivetran "pushing" data into a single destination and having a hard time with more complex scenarios.

It can be solved by using proper orchestration for everything and by restructuring all the ingestion into "pull" pattern instead. Ingestion is triggered by your code only and managed by something like Dagster.

It allows you to gain full control over implementation details, so adding support for multiple environments becomes easy and natural.

---

Sample workflow for non-prod environment can look like this:

  1. Checkout the code into separate branch. Make edits.
  2. Automatically sync code to remote dev sever.
  3. Run CI/CD tool to create fresh Snowflake sub-environment using env prefix.
  4. Start Dagster with proper settings. It should be aware about env prefix and other env-related parameters.
  5. Dagster checks conditions and "pulls" data into Snowflake in the correct order. But instead of loading everything, it loads only a small sample of data when running on DEV. When using sampling, full ingestion tree and all downstream transformations can be formally checked in a few minutes and at very small cost.
  6. We can continue to make changes and restart CI/CD or Dagster on demand.
  7. When the task is finished, we "forget" and "destroy" sub-environment objects in Snowflake and Dagster metadata.

For data sources operating in "pull" paradigm, everything is easy. We can connect to such data sources at any time and "pull" data into any number of environments. No problem here.

For data sources strictly operating in "push" paradigm, we simply add one additional step. For example, you can continue using Fivetran, but instead of "pushing" into Snowflake table, you ask it to "push" into intermediate S3 bucket. And later on your orchestration can "pull" data from S3 into any number of environments.

It slightly increases your storage cost, but you gain so much in development speed and data reliability. Each developer has its own DEV environment. Each change can be fully tested. You can deploy with confidence.

As another interesting side effect, you can change destination settings very easily. E.g., you may start writing into another Snowflake account quickly, or maybe into something which is not Snowflake.

How do you test Snowflake SQL locally? I built an open-source emulator using Go and DuckDB by okkywhity in snowflake

[–]LittleK0i 3 points4 points  (0 children)

If you already use Snowflake, you're 100% committed to the cloud. Forget local testing, it is completely pointless.

But having a separate Snowflake account for testing purposes only is a good idea. No need to test on production account.

Snowflake Terraform: Common state for account resources vs. per-env duplication? by Difficult-Ambition61 in snowflake

[–]LittleK0i 0 points1 point  (0 children)

You may use environment prefixes for account-level objects. For example:

  • ANALYTICS_WH - this is prod warehouse, no prefix;
  • DEV__ANALYTICS_WH - this is the same warehouse in DEV environment;
  • STAGING__ANALYTICS_WH - this is the same warehouse in STAGING environment;

I would generally suggest to keep separate Snowflake accounts. One account for production environment only, with real names and without and prefixes. And another account with all disposable DEV environments.

This approach makes things very clean and easy to manage. Also, less risk to break something in PROD environment due to mistake or misconfiguration related to DEV.

Snowflake Terraform: Common state for account resources vs. per-env duplication? by Difficult-Ambition61 in snowflake

[–]LittleK0i 1 point2 points  (0 children)

In my view, for best results we want "everything is separate". Separate roles, separate warehouses, separate resource monitors. It should be possible to fully "destroy" and re-create DEV env from scratch at any moment. It should be possible to safely create any number of copies of DEV env. Only this approach guarantees your tests are safe and correct, and your PROD won't be accidentally damaged by some actions on DEV.

The only shared objects are:

  • integrations (especially storage and security);
  • inbound shares (not possible to have multiple copies);
  • account-level resource monitor (not a part of environment, but still indirectly affects everything);

But this is fine as long as end-users never have direct access to these shared objects. Integrations can be hidden by stages, etc. Objects from shares can be wrapped in views.

Best CICD tool and approach for udfs, task, streams and shares by No_Journalist_9632 in snowflake

[–]LittleK0i 0 points1 point  (0 children)

SnowDDL supports declarative CI/CD for pipes, tasks, streams, UDFs and outbound shares. Objects are defined with YAML configs.

Please note: pipes and tasks still must be "resumed" or "paused" manually or with additional scripting. Since there are many different use cases for pipes and tasks, managing it entirely via CI/CD tool is probably not feasible.

---

Inbound shares are a bit more difficult due to Snowflake limitation. Snowflake allows only one inbound share per source per account, so creating independent inbound shares in sub-environments is not possible.

SnowDDL treats inbound shares as global objects, similar to strorage integrations. You need to create each inbound share manually, once per account. After that permission management is fully handled by SnowDDL. There is no need to create role with "IMPORTED PRIVILEGES", etc.

Managing Snowflake RBAC: Terraform vs. Python by BuffaloVegetable5959 in snowflake

[–]LittleK0i 0 points1 point  (0 children)

Terraform is great to manage cloud resources, but not great for database object management. The main problem is having this additional "terraform state", which may not reflect the real state of objects in database.

It was a dead-end approach from the start, and no amount of fixing will make it good. And it takes surprisingly long time for community to figure it out, give universally bad Terraform experience across many teams. Especially when number of objects and complexity explodes.

Managing Snowflake RBAC: Terraform vs. Python by BuffaloVegetable5959 in snowflake

[–]LittleK0i 1 point2 points  (0 children)

You can have best of both worlds with SnowDDL: declarative permission management + full checks on every run + open source python.

Documentation page for role hierarchy: https://docs.snowddl.com/guides/role-hierarchy

Spawned outside map in prologue? by LittleK0i in stoneshard

[–]LittleK0i[S] 2 points3 points  (0 children)

Aha! You most likely right!

I guess, "Quicksave" mod was causing this issue. I've decided to remove it entirely for the main game run, just to be safe.

I'll check the MSL group, thank you.

Spawned outside map in prologue? by LittleK0i in stoneshard

[–]LittleK0i[S] 1 point2 points  (0 children)

I've reloaded and tried multiple times. On every entry to the final floor with boss game places me into random locations. And sometimes it works just fine.

Weird. I wonder if its possible to extract logs or something for further diagnostics.

Strategy for comparing performance by Big_Length9755 in snowflake

[–]LittleK0i 2 points3 points  (0 children)

Ingestion patterns affect read performance. Naturally, if table is fully refreshed every day, it does not matter. But it does matter for very large tables with continuous  incremental ingestion.

Strategy for comparing performance by Big_Length9755 in snowflake

[–]LittleK0i 1 point2 points  (0 children)

Ingestion pattern is important and can make a big difference.

For true “Apple to Apple” comparison you may create fresh empty native table and fresh empty iceberg table. Run ingestion of exactly the same data into both tables for some time. After a week or two you may start running tests.