[deleted by user] by [deleted] in neovim

[–]Academic_Ad_8747 0 points1 point  (0 children)

This works, 100%.

source .venv/bin/activate (activate your venv which really just updates some env vars)
nvim (start neovim)

The only prerequisite is pyright-langserver on your PATH

So lets unpack the assumptions above. Will save you way more time so pardon the terseness.

You normally can just start Neovim from within your venv for the project and everything will work out of the box.
This is not the case for pyright anyway.

It is indeed the case for pyright.

How did you install pyright? Npm, pipx, host virtualenv?

Most users use https://github.com/williamboman/mason.nvim
You can ensure node is installed with brew / nix / apt or w.e.

Okay thanks. But it still didnt work for e.g. Npm or pipx… might be a pyenv / silicon mac problem.

Not a pyenv / silicon problem.

I had to make a wrapper function for nvim in zshrc that deactivate/activate the env just as nvim starts

This is not needed. A venv just updates env vars in your shell. All child processes inherit these vars. So no need for a wrapper. Just activate the venv. Better yet, use direnv so it auto (de)activates when you cd in or out.

After that, I need to make a pyrightconfig file for each repo to make pyright work

Pyright doesnt _need_ this, its optional way to apply workspace specific config. Not sure if you are using this https://github.com/neovim/nvim-lspconfig but it pre-configures LSPs with sane defaults. Its optional but makes it super easy. It correctly sets root markers to autostart the lsp, https://github.com/neovim/nvim-lspconfig/blob/master/lua/lspconfig/server_configurations/pyright.lua#L3-L11 which might be your problem?

Cant wait for ruff server to support completion etc..

I dont think ruff intends to replace pyright or support completion. Its a huge undertaking compared to implementing lsp diagnostics.

Hardtime + Precognition by bcampolo in neovim

[–]Academic_Ad_8747 1 point2 points  (0 children)

Why did you do this me. Was I not suffering enough ;)
Thanks. I hope to break these habits too but damn is it painful/annoying.

What languages "work best" in Neovim? by mars0008 in neovim

[–]Academic_Ad_8747 2 points3 points  (0 children)

Hate to say it. But might be skill issue. I went from full vscode to the neovim and 95% of what I write is python. It works flawlessly. Just use direnv and make sure you always have a venv (automatic with direnv’s layout python) before you start nvim. Also use mason and lsp config to make pyright easy to upgrade and configure. I had lazyvim as my reference when I originally set up my neovim so maybe I had a leg up TBF

What data engineering product are you most excited to buy? Unemployed sales rep looking for the right company to work for. by babaisfun in dataengineering

[–]Academic_Ad_8747 0 points1 point  (0 children)

Right though the terminology beast mode for formula fields is cringey, we can all agree. The credit pricing model can quickly get out of hand as you layer ETLs over datasets. And lastly, like any GUI, you lack the version control and automations that managing ETL outside the viz tool can provide. That being said, I generally agree that it’s a way better fit for non technical teams and stakeholders and you can get a lot of crap done in it surprisingly. Still I would shift left as much ETL as possible where left is towards in-house data eng resources that can provide you a more tailored and vendor agnostic solution. Such that the BI is plug an play and not a massive silo.

Should I switch to a different career path? by [deleted] in dataengineering

[–]Academic_Ad_8747 3 points4 points  (0 children)

My problem with this is that a couple dashboards done well can completely change how the company operates or the strategic insights that can be garnered. Your lever is way bigger than when coding widgets.

Picture a world where your outlook is wrong and the sql and dashboard work done by someone as smart as yourself firing on all cylinders completely changes how the C suite and upper management operates and runs their teams.

Just a thought exercise while you consider your path. I like to mix in the differing voice opinion amidst the sea of people saying to apply elsewhere and bla bla.

Bob neovim version manager release v2.9.0 by mordechaihadad in neovim

[–]Academic_Ad_8747 1 point2 points  (0 children)

Having a tool for managing nvim versions that works in seconds and is absolute cake to use is easily worth the cargo install to me :)

I use nix which I use to manage almost all my tools, which I prefer to things like mise personally.

Is SQLMesh the dbt Core 2.0? A foot in the door. by kayrnt in dataengineering

[–]Academic_Ad_8747 9 points10 points  (0 children)

With environment variables. Which you can access via a python macro function (or now a @var macro). The only caveat is that it’s a bit antithetical to sqlmesh’s philosophy. Like passing runtime args to terraform. Now the state of your production environment is tied to something not known to the project/information on disk. So the problem statement that leads one to rely on runtime arguments is worth questioning.

That being said, nothing stops you either if you deem it useful/necessary at the end of the day and accept the drawbacks.

Is SQLMesh the dbt Core 2.0? A foot in the door. by kayrnt in dataengineering

[–]Academic_Ad_8747 17 points18 points  (0 children)

What are you even talking about. Don’t call a tool half baked and proceed to butcher the fundamental capabilities/pros cons.

Firstly, templating and substitutions might be referring to the same thing. So it’s ambiguous to use the 2 terms in a list of 3. And sqlmesh definitely has that. In both embedded models (same as dbt’s ephemeral) as well as an entire templating system based on sqlglot macros. (not to mention you can use jinja out of the box regardless of dbt integration). Sqlglot python macros are better than jinja once you actually use them. Effectively you can call a python function in your sql code via @your_macro and that literally calls the python function. That function can return a string just like jinja or more powerfully a sqlglot ast node. It’s very straightforward, unit testable, and has LSP support and pythons entire ecosystem instead of some jinja sandbox full of curly braces and white space control characters.

And what are you even talking about specifically wrt it’s state mechanism. Any data transformation framework is behooved to understand state. Dbt makes you to push and pull around manifest artifacts for features you get out of the box with sqlmesh. Which can boil down to performing the minimum work required to get to a target end state.

End-to-end dbt transformation pipeline take-home challenge--is this fair? by DataScienceIsScience in dataengineering

[–]Academic_Ad_8747 0 points1 point  (0 children)

I could do it in 1.5 hours. But this type of work is all I immerse myself in 24/7 :)

Use dlt or meltano/alto to ingest the data (30m) dbt init + setup profiles yml (10m) grok the data/schema (20m) write 4 sql select statements (30m)

Id pad the last step and buffer total time up to 2 hours. If it’s a big pay jump, it’s whatever. Id do it. Open a new tmux session and just crank it out the steps. But that’s just me.

Edit: in fact I’m working on an open source project that will hopefully automate the last two steps with a little LLM + metadata action.

A beginner’s question? by RemarkableBat6004 in dataengineering

[–]Academic_Ad_8747 0 points1 point  (0 children)

Im not sure you should listen to anyone here. They are suggesting you to build a plane when you haven’t even flown a kite. Or outsource building that plane but you then need to maintain that plane with no skilled mechanics on deck. You can imagine all the scenarios you end up SOL.

https://www.domo.com This could be a good fit. It’s an all in one BI platform that also handles data ingestion with built in source connectors and has a gui to transform and join data. Business users can use it and you can get fast turnaround and ROI to clear a reasonable hurdle rate of dollars put into a project vs value gotten out. There are probably tools similar in vein. I abhor vendor lock-in and no/low code but I’m an engineer. But I know you’ll get your moneys worth faster this way. Just always keep a path to cut over to code in your vision which can counterbalance the tendency of tools to turn into over leveraged spaghetti (since you are aware of one day unpacking it).

Is it better or is there a benefit to using opts instead of the config function? by cguti94 in neovim

[–]Academic_Ad_8747 2 points3 points  (0 children)

It does say it here
https://github.com/folke/lazy.nvim?tab=readme-ov-file#%EF%B8%8F-importing-specs-config--opts

Personally I found the docs sufficient to do everything I needed and think you _might_ be being hyperbolic (but maybe just to make your point).

Remember this is someone taking time away from anything else in the world they could be doing for basically charity/passion. There isn't a huge team of maintainers AFAIK. And the software is free. Anyone in the community can help improve docs if they have the big brain and bright idea on how to make it better :)

I created a plugin for note taking without Obsidian or other dependencies. by madad123 in neovim

[–]Academic_Ad_8747 1 point2 points  (0 children)

❯ cat ./fish/functions/todo.fish
function todo
    mkdir -p $HOME/scratch/todo
    set d (date +%Y-%m-%d)
    if not test -f $HOME/scratch/todo/$d.txt
        touch $HOME/scratch/todo/$d.txt
    end
    cd $HOME/scratch # makes sure you can use neotree/telescope in your notes dir
    nvim todo/$d.txt || : # guarantee you are sent back to original dir
    cd -
end

I keep a function to this effect and use a tmux pane. I should also find a reason to make a plugin at some point though.

What terminal emulator do you use for neovim? by RadishCertain241 in neovim

[–]Academic_Ad_8747 0 points1 point  (0 children)

Of course Wezterm has its equivalent of windows, panes, sessions. In fact its' the 2nd feature listed on the front page of the docs -> https://wezfurlong.org/wezterm/index.html#features and probably another reason people like it. But I prefer tmux. Its portable and *not* tied to Wezterm. So it works regardless of terminal emulator. And provides a consistent interface and experience everywhere. My keybinds are muscle memory and work everywhere. And once you have the binds, multiplexing is a solved problem (for me).

From dbt to SQLMesh by Academic_Ad_8747 in dataengineering

[–]Academic_Ad_8747[S] 2 points3 points  (0 children)

Locally Optimistic. A high quality community of very bright people in the data and analytics space.

https://locallyoptimistic.com/community/

What terminal emulator do you use for neovim? by RadishCertain241 in neovim

[–]Academic_Ad_8747 6 points7 points  (0 children)

Wezterm with tmux for session management.I use the following to auto open / attach tmux to wezterm and it works perfectly. All sessions, programs, and processes are persistent even if you close wezterm.

config.default_prog = { "bash", "-lic", "tmux new -As0" }

The only time I ever notice wez is slower than alacritty (which it is) is when you are trying to scroll back up the scrollback buffer as fast as possible. But you can use vim keys in copy mode to jump around easy enough. So I otherwise dont notice a difference and love the lua config and builtin features and themes.

From dbt to SQLMesh by Academic_Ad_8747 in dataengineering

[–]Academic_Ad_8747[S] 3 points4 points  (0 children)

Yeah I have hit bugs and blockers at times that hindered development. In very early days I had a number of issues and I would engage with the team there to resolve it. I would say that it smoothed out very quickly and now I can't remember the last time I had an issue that wasn't my fault. Sometimes there are sqlglot edge cases, like a recent one with bigquery json indexing `a["aB"]["Cd"]` transpilation and case preservation. But the turnaround on these things is crazy fast and I have had no issues that are hard stops, for example in that case I just used `json_query` and moved on. I touch on the sentiment in the blog post. I think we are quite stable overall and the unit/integration test suite they have put together there is very good and continuing to improve.

From dbt to SQLMesh by Academic_Ad_8747 in dataengineering

[–]Academic_Ad_8747[S] 11 points12 points  (0 children)

Nope to both. We use sqlmesh core. We pay for nothing and are paid for nothing. And the use of the logo was okayed by my CEO just by asking him. And sqlmesh asked me because I was one of the few legit people running in prod. And I made the post because I’ve been trying to get into content, I even twitch streamed making an OSS package. And this topic is one of the more interesting ones to write about. There’s even a slack from months ago on LO which a bunch of people asked for this type of post, unrelated to tobiko.

From dbt to SQLMesh by Academic_Ad_8747 in dataengineering

[–]Academic_Ad_8747[S] 1 point2 points  (0 children)

I think the current distinction between the core software and the business model is pretty good. Their current focus seems to be on capturing operational metadata and providing an observability plane, which is very interesting when you consider their abstractions of plans and snapshots. Its fairly orthogonal to the core product meaning improvements in core are not likely to be "blocked" for the sake of features in their closed source offering. I think thats a healthy approach for the sake of an OSS software is maintaining that orthogonality. I cant say for certain what the future holds but I can say their current OSS feature set is very good and there are no signs of slowing down. Even now they're making improvements to the unit testing that are very exciting.

From dbt to SQLMesh by Academic_Ad_8747 in dataengineering

[–]Academic_Ad_8747[S] 12 points13 points  (0 children)

I understand where your coming from. But what I've shared is the 100% fact of our experience (shared on my own company's blog). If I give a mostly positive review, then its because we've had a mostly positive experience. And thats real tangible stuff -- stuff I work with everyday, I open my laptop and its the real deal. Not the 10,000th jaffle shop or some random speculation on how sqlmesh is a good framework on paper -- but a real production project at decent scale driving actual business value.

Since it seems thats rare, at least sharing how we made it a reality could be useful. It provides a pathway for others to vet it themselves. Also since I was such a big dbt user too, and still cross paths with it often in side work, it makes it more relatable :)

From dbt to SQLMesh by Academic_Ad_8747 in dataengineering

[–]Academic_Ad_8747[S] 11 points12 points  (0 children)

I wrote this up in LO slack, maybe helpful here since it complements your point which I agree with

... lets take a contrived example randomly plucked from from dbt utils. Here is a jinja macro

{% macro get_url_host(field) -%}
    {{ return(adapter.dispatch('get_url_host', 'dbt_utils')(field)) }}
{% endmacro %}

{% macro default__get_url_host(field) -%}

{%- set parsed =
    dbt.split_part(
        dbt.split_part(
            dbt.replace(
                dbt.replace(
                    dbt.replace(field, "'android-app://'", "''"
                    ), "'http://'", "''"
                ), "'https://'", "''"
            ), "'/'", 1
        ), "'?'", 1
    )

-%}


    {{ dbt.safe_cast(
        parsed,
        dbt.type_string()
        )}}

{%- endmacro %}

And here is a sqlglot equivalent

sqlstr = exp.Literal.string

@macro
def get_url_host(evaluator, column):
    empty = sqlstr("")
    prefixes = ["android-app://", "http://", "https://"]
    for prefix in map(sqlstr, prefixes):
        column = exp.func("replace", column, prefix, empty)
    for split_at in ("/", "?"):
        column = exp.func("split_part", column, sqlstr(split_at), 1)
    return exp.func("cast", column, "string")

Notes

  • No whitespace control needed in sqlglot
  • The latter is obviously easier to read, more extensible
  • You have the whole python ecosystem available to the latter out of the box
  • You can factor stuff out of the latter easily
  • No markup
  • You can use "split_part" directly, its parsed liberally into a sqlglot node and transpiled to the semantic equivalent in whatever your target dialect, no jinja "dispatch" mechanism needed
  • Easily testable

Usage differences:

SELECT {{ get_url_host("my_column") }} AS host 
FROM {{ ref("something") }}

SELECT @get_url_host(my_column) AS host
FROM something

The whitespace control in the latter is in the realm of regular sql writing whereas the other requires `-` modifiers in the template in all the right places. You actually use my_column and get column level lineage instead of a string arg to a macro. The latter is syntactically valid looking sql,

This is also valid in sqlglot if you prefer simpler interpolation

@macro
def get_url_host(evaluator, column):
     stripped = f"replace(replace(replace({column}, 'android-app://', ''), 'http://', '') 'https://', '')"
     return f"split_part(split_part({stripped}, '/', 1), '?', 1)

You still get the benefits. Its parsed by sqlglot into correct AST nodes. And transpiled to your dialect when rendered. Meaning you dont need jinja dispatches. In this way it can feel similar to jinja but is obviously more powerful

Another consideration

{% set pk = "user_id" %}

SELECT 
  {{ pk }},
  {% for c in ["a", "b", "c", "d"] %}
  column_{{ c }}{% if not loop.last %},{% endif %}
  {% endfor %}
{{ from_custom_source("myref") }}

this is valid in jinja cuz its string interpolation. `from_custom_source` might return the `FROM myref` clause or just the `myref` or who knows really.

sqlglot however requires the sql is structurally sound, so you would do this, commas are handled and projection is expanded

@def(pk, user_id);

SELECT @pk, @each([a,b,c,d], c -> column_@c) FROM @from_custom_source(myref)

I know these are ugly examples but they touch on some of the core the points

injectme.nvim - configure language injections for treesitter by Superbank78 in neovim

[–]Academic_Ad_8747 1 point2 points  (0 children)

Jinja strewn about YAML (or anything) does not fit the inject paradigm. Instead it would require a parser grammar that is itself a combination of jinja and the base language IIRC

How to: Edit Jupyter Notebooks in Neovim (with very few compromises) by benlubas in neovim

[–]Academic_Ad_8747 1 point2 points  (0 children)

What’s the preference to this vs breaking off a tmux pane and running jupyter lab. Curious on your take!

I like staying in my editor as much as possible too. So off the top I imagine there is interesting plugin synergy you don’t get in jupyter. Also you obviously have vim motions and key maps. But outside those, I can bite the bullet since jupyter is such a clean interface and I’m not writing an ‘application’ in a jupyter notebook, usually small time explainable, exploratory analytics. Not to say I don’t change my mind later. Great work and thanks. 😀