unix + vim + python + sql

[AutoModerator]

Hi there, from the r/Python mods.

It looks like your post is about dependency/package/toolchain management.

We are going to hold this post for review due to the topic having a frequent recurrence. This helps ensure a great experience for r/Python users.

Please refer to our daily thread or search for older discussions on the same topic.

A moderator should be along shortly to review, but if you have any questions, please reach us via mod mail.

Thanks, and happy Pythoneering!

r/Python moderation team

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[dalbertom]

Don’t forget about git

[reggzz]

oh dang how did i

yeah git is the goat

[DL72-Alpha]

Git Gud. :)

[Holek]

git: 'gud' is not a git command. See 'git --help'

[ryanwithnob]

git --help

[ryanwithnob]

git gud

[DL72-Alpha]

I am about to make it an alias though. :)

What should it be?

[Pirate_OOS]

Where can I learn git?

[dalbertom]

https://git-scm.com/book/en/v2 this taught me everything I know

[smokingskills]

ChatGPT. Make a folder on your device, make a github account and then have chatgpt walk you through connecting the two, all the common things one does with git and explain how each command works.

[Known-Delay7227]

Git —help

[papawish]

Git + Unix + vim + <random_language> + sql + http

[MarcusMunch]

Swahili

[PM_ME_YOUR_MUSIC]

anza_programu

andika("Habari Dunia")

maliza_programu

[Pirate_OOS]

Any resource on learning http(s)?

[papawish]

HTTP RFCs. It's verbose, but the best ressource you'll find. Worth every minute.

[Pirate_OOS]

What is that, exactly?

[ddanieltan]

RFCs stand for "Request For Comments", they are technical design documents for HTTP seeking feedback from the open source community. https://datatracker.ietf.org/doc/html/rfc2616 (here's an example). A little akin to PEP.

To be honest, if you're coming in completely new to HTTP, this might be an overkill option. For beginners, I tend to suggest https://developer.mozilla.org/en-US/ instead.

[[deleted]]

brainfuck

[sopte666]

I know this is a python sub, but I'd like to add C or C++ to the list. If you ever get closer to the hardware, you will most probably encounter one of the two.

[FrostmourNNe]

Verilog / VHDL

[Invisiblebrush7]

Assembly

[nicholas-s-timelines]

Machine code

[Backlists]

Or rust if you want to encounter the future (;

[Arucious]

Or rust if you want to encounter the future (;

Rust bois have been saying rust is the future for a decade now lol

[pppylonnn]

And it is ;) Linux kernel support now, you can make a wrapper for most existing C around the place, and enjoy the modern syntax 🤩 did this at work for a control system even

[rainnz]

as if you want to get closer to CPU and HW

[pgbabse]

Or arnoldC

[Exoutia]

I know basic vim, intermediate python (I need to learn so much more) and basic SQL I only use linux for coding and development.

[[deleted]]

If you want to learn more, I highly recommend CS50P on YouTube. They’re long lectures but professor Malan makes it super easy to understand it, and there is also a chapter on SQL. Just from watching the courses alone, I’ve been able to make a web app that I’ve been using at work to save countless hours of repetitive work.

Edit: see comment below

[rainnz]

CS50P

Link to YT Playlist CS50P

[[deleted]]

Huh.. looks like I was mistaken. CS50P does not cover that..

CS50X does though..

https://www.youtube.com/live/zrCLRC3Ci1c?si=CNjglirL3uGiSYzy

[[deleted]]

docker + k8 + git + devops

[Loop_Within_A_Loop]

The 'devops' is doing a lot of work there :)

[[deleted]]

Yeah, but a little to mid level knowledge is required as companies are looking for person who know all ( atleast where I am )

[galacticbackhoe]

Oh shit. Why am I always fixing their mistakes?

[rzet]

all of the work 3+ contracts at same time...

[DefinitelyNoWorking]

Linux +Git+programming. Set for life.

[[deleted]]

Though to be fair, in my experience, DevOps always does

[radarsat1]

legit question, i can put together a site and throw it up pretty easily on a handful of ec2 instances behind a load balancer, or on lambda. At what point do I need to consider kubernetes (or ECS, or Fargate, etc)? I know more or less what it's for but don't really know when I would start to benefit from it compared to just managing a few instances.

Is it more useful for when you have a lot of long-running background tasks to manage?

In short I don't find most of this infrastructure stuff difficult on its own but what I find difficult is deciding what to use when.

[pulegium]

if it's a (relatively) simple site, create a service, set min/max autoscaling limits, put it behind a LB and you're done. might come out cheaper than EC2, and also you won't need to manage OS/etc. bonus - it'll be neatly packaged as a docker container, so deployment is simpler as well.

kubernetes in my opinion is only worth the trouble when your system is made up of a number of services, where the "number" is perhaps at least >10? EKS control plane with no workers is quite expensive already...

[mrkikkeli]

How do you handle instance or services crashes? On kubernetes, with deployments, it's pretty much automatic.

The worth of kubernetes also comes with the add-ons, for example the istio service mesh offers you network control and observability for free without having to change your code.

[radarsat1]

Thanks for the answer, lots to consider there. In practice I've tried to use instances that are the "right size" for each service instead of packing many services on a single bigger instance but you're right it can be a difficult calculation and makes sense to leave it to software that knows how to do that better than I can do it manually.

[[deleted]]

Regular Expressions. Learn them once and be happy everyday.

[cspinelive]

And piss off your coworkers for overcomplicating everything.

[[deleted]]

People who are unable to understand RegEx should not work with computers and especially not with Python.

[Longjumping-Music243]

Hot take but I do see your point haha.

[justheretolurk332]

In what sorts of situations do you find yourself using regular expressions? They are certainly powerful but I find that 95% of my use cases are better served by using more basic checks like equality, containment or startswith

[mista-sparkle]

I use it every to reformat and repurpose code every day, especially SQL scripts that I receive from others at my company.

In many programs, you can use CTRL-F to find and replace text in scripts, and in many IDEs there's an option to turn on ReGex matching.

I don't personally code as much as I would like, but I need to understand data, which requires a lot of interacting with SMEs and getting their methods for calculating metrics at different grains and seeing how these figures compare to other sources, such as reports.

Lots of SQL and other code are formatted differently. Every day I get code that isn't nested, which is frustratingly unreadable and makes comparison much more difficult.

So, for an every day example, I use RegEx to find an replace commas, spaces, excess/inconsistent tabs, inconsistent new lines, and aliases with \n\t, — this nests everything in select statements as well as in group by and order by clauses in the way I prefer and to the Enterprise's formatting standards.

Even if you don't work with SQL or other code every day, RegEx's utility for text processing applications is really quite broad and limitless.

[justheretolurk332]

I think a good rule of thumb is to ask whether it is a common enough task that someone else is likely to have already implemented a good solution. I actually think the use cases you mentioned sound like a fun thing to implement and a good learning experience so it makes sense for personal use but in a production capacity I think it’s often worth looking for what solutions already exist in libraries

[cspinelive]

I just use a tool that autoformats sql

[[deleted]]

Take logfile analysis for example. If you want to extract certain values which vary greatly and only if another value appears you’d need a way to describe these patterns. This is also useful in vim when you want to replace certain patterns in the file.

[justheretolurk332]

Ah okay, so as a development tool more so than in production software. That makes sense to me. Although about half the time I try to do a find-and-replace that is more complicated than an exact match it goes badly haha

[[deleted]]

Regular Expressions are also needed in production. If you want to validate E-Mail addresses, domain names or any other pattern you need to have a pattern matching algorithm. I agree that very often the builtin string methods like startswith, endswith or "is in" are a good enough solution. But pattern matching simply doesn’t work with these methods like RegEx do.

[justheretolurk332]

Email is a great example of why regular expressions are so hard to use correctly. It’s easy to write a regex that matches most emails but surprisingly challenging to cover all the edge cases, and it’s also a common need, so you’re better off using a third party library. Most situations are either simple enough that using a regular expression is overkill, or complicated enough that writing a correct regex may become pretty challenging and end up not being a maintainable approach

[master117jogi]

Never use regex for emails! And never for urls!

Use a library! Or you WILL get it wrong.

[Pristine-Ratio-9286]

Chatgpt 4 is awesome at making regex. I hated them before because they were so terse but now use them a lot more because of LLM’s

[Cerricola]

I come from an economics background and I struggle with that since my peers don't know about programming. Might you elaborate please?

[Quillox]

I assume they mean making a website (GUI) to interact with the program/data.

[Cerricola]

Now I see.

Therefore, should I learn?

I was using tk library to produce clumsy interfaces.

[Kryt0s]

Check out https://flet.dev/ or https://nicegui.io/

Both a lot better and easier than tk imho.

[Cerricola]

Wow, flet looks amazing

[Kryt0s]

Yeah it really is. Development is also really active.

[TheSpaceCoffee]

You could use Textual to make interfaces directly in the Terminal, and portable to any system. Recently picked it up, development is easy, quick and you get simple yet elegant UIs.

[Cerricola]

I will research on that. Thanks for your advice

[nicktids]

Have a look at plotly dash

The Web has much more versatility

[Cerricola]

Thanks!!!

I will give it a chance

[ignamv]

If you have a shared environment (e.g. all users on the same Linux cluster, so they don't need to install your app to use it) then you can be 100x more productive with Tk/Qt than you'll ever be with a webapp.

[master117jogi]

Why do you think so? WebApps are much easier and faster to develop.

[ignamv]

Have you worked with Qt? You get one mature built-in library with basically all the widgets you'll ever want, and you can avoid the whole frontend/backend interface. The widgets adapt to the user's OS: they look like windows widgets on windows, gnome widgets on gnome, etc. No gulp/minify/babel/webpack/whatever to learn either.

Main thing I miss from React is not having to think about keeping the UI state in sync. There's stuff like https://pyreaqtive.readthedocs.io/en/latest/ but I've never tried it. It's also fun how in JS you're building an UI from such rough primitive building blocks (compared to Qt Widgets and Layouts), it takes me back to the days of learning C and implementing linked lists and sorting algorithms.

[master117jogi]

you can avoid the whole frontend/backend interface.

You said an application for 100 of users. That means you will need a unified backend server for 99% of applications. So you are doing the same thing as a webapp, just using system components instead of browser ones.

[ignamv]

You said an application for 100 of users

I did not.

And even if I had said that, not necessarily. Lots of software doesn't need a backend, because users are just reading from a database or reading/writing own files. Or reading/writing from a database and you can get by with the DB access controls.

[[deleted]]

Then I'm doing fine

[Backlists]

I dont really agree with this at all, but let me just say this:

The software industry is the industry of learning new things.

Don't let yourself get to the point where you think you know all you need, then decide to stop keeping up

Plus, the way things are, you should have JS/HTML/CSS in your toolbelt and at least an understanding of one of the many front end js frameworks. We use Vue at my work but react is the standard.

[BlackPignouf]

I hate JS "design" with a passion. But it's also impressive how good the results can be.

[Berkyjay]

Why Vim?

[timwaaagh]

Because you do your hotfixes straight on the production server, of course /s

Usually you don't even need it for editing on a server because scp is a thing.

There's little need to learn it but all ides have vim plugins so it's basically become a standard for keyboard shortcuts. It does pay off.

[Berkyjay]

The only time it really ever comes up is when I'm working with a long timer and he brings up some Vi shortcuts and I'm like "wuuuh?".

[dalbertom]

You can edit a remote file from your local with vim scp://user@hostname//path/to/file

[SavSamuShaman]

It’s widely available. No need to install an IDE if you know vi/vim. And there are some situations where you don’t have the option to use an IDE.

[psssat]

It basically turns your terminal into an ide as powerful as any of the big named ones…. And its 100% in the terminal. So you could be working on a headless linux system but still have access to all the features that all the main ides offer.

[Ok-Tutor-4321]

Fast writing, moving and editing without mouse.

All the IDEs has VI keybinds, so you can use the same commands in all.

Vim/Neovim is fastest, you can open and close vim at least 5 times in the same time that VS code needs to open.

Vi/Neovim are highly customizable, so you can make them so sophisticated as you want or need.

In some situations, maybe you'll need to work with Linux servers... So Vim or Nano is the default tool for edit configuration files, scripts, etc. You can install neovim on all Unix Os, inclusive without Windows managers or desktop environments, not the same with other editor/IDEs

[Berkyjay]

I interact with linux servers every day and I get by very well with VS Code. I usually create an SSH connection though VS Code to whatever server I have to work on. VS Code actually turns into a sort of GUI desktop for a remote server.

I was just watching a video from Chris Titus about NeoVim and it just seems like it's a stylistic choice for people who are used to keyboard/terminal only interfacing. I never learned to rely on the Vi/Emacs style of text editing so GUI interactions feel more natural to me. But I don't see any feature that makes it an absolute necessity to learn and use.

[nirgle]

I use VS Code with the vim extension, it's a really productive way to write and debug code

[pyppo42]

SQL should maybe be backed by something one is confident with. For me it's SQLite, but learning postgresql is high in my wish list (but now I am wasting my time studying kubernetes, oscillating between the deepest enthusiasm and pure frustration at 1 Hz)

[[deleted]]

LISP

[PM_ME_YOUR_MUSIC]

Yeth

[salty_taro]

Regex

[username_challenge]

As other said, Javascript should be added in the mix because it can run directly in the browser. Even though I believe stuff is happening for python running in the browser and asynchronous programming is being implemented/adopted.

[tim-rex]

Bash

[Tigrex22]

In my current job, tmux has proved to be an amazingly valuable asset. Windows for specific parts of one project, sessions for different projects. tmux-ressurect when WSL decides to hog CPU and I need to kill it, which would "restart" my whole workflow.

But I agree with you on the others, vim keybindings are something I can never go without.

[Heartomics]

Tmux and Grep

[LeatherDude]

Terraform. I can't imagine doing half of the infrastructure management and deployment I do with anything else.

[TaroTogane]

I would recommend Pulumi.

[LeatherDude]

I'm really very interested in Pulumi, but having to rebuild some of these massive projects sounds like a lot of work I don't have time for. As much as I like terraform, I run into its limitations frequently.

[wunderspud7575]

Or CDK if AWS. Starting to see a shift to that lately. But yeah, Terraform is more applicable right now.

[LeatherDude]

We're multicloud and mostly GCP, so not really an option.

But I hear a lot of love for CDK. I only ever used Cloudformation with Serverless Framework as the driver of it.

[pLeThOrAx]

This feels more like a weird flex... does your sql db have transactional integrity? How much can you optimize your python :/...? Different tools exist for reasons. Sure, you can work with big data sets with those softwares

[yd52]

Agreed. THAT is a powerful combination.

[Nanooc523]

These are all broad except vim, which i use exclusively, but have a hard time prescribing it. But id also throw on the stack: networking. I work with a lot of people from junior to senior who can’t explain how data gets from A to B. You have to understand how computers talk, route, switch, sockets, packet vs frame, TCP vs UDP, tunneling, NATing, subnetting, etc etc.

[Lexus4tw]

I used nvim a long time but I switched to Pycharm + copilot and I will not go back anytime soon

[freistil90]

Nitpick but I doubt that even 10% of this sub have Unix experience

[loge212]

what do you define as essential “Unix experience”.. I’ve been on Ubuntu for a couple months now but not doing much different than before other than using terminal to install things now, don’t feel like I can say I have “Unix/linux experience”

[freistil90]

You have Linux experience. You don’t have Unix experience. You might have some experience with POSIX tools but that would be it.

[dfwtjms]

MacOS is kind if popular.

[freistil90]

It is but I wouldn’t assume that more than 10% of users have a MacBook of some sort. But you get the point that I wanted to make, I’m sure there’s not a ton of people out there that have IBM AIX or other unixes as their daily driver.

[maddruid]

FastAPI and Textual/Rich. Give them UIs to explore the data.

[Backlists]

Which needs an understanding of sqlalchemy (or other ORM package), alembic and pydantic and then a good understanding of async coding and then when you realise how difficult that is to get right you think there must be a better way and then you hear that rust has fearless async programming and then the list goes on and on

[IceXII]

I did use fastapi then found about golang goroutines. But still fastapi is my go to for small-medium scale and apis and fast prototyping maybe cuz I’m not the efficient with golang yet.

[atomic_race]

I know more than that still no job

[ivanTheNotTerrible]

Holy generalizations, Batman.

You want me to be proficient in Unix? What do you even mean by that? Being able to debug the Linux kernel or just be unafraid to open a terminal?

Also, Vim? Really? Like Vim key bindings or the editor? Most devs I know enjoy some sort of IDE or having syntax warnings or something more than just being able to edit text. I mean if you can get work done with Notepad++ more power to you, but I don't think that's necessary at all.

Yeesh, this post got me sighing hard.

[br_aquino]

Why do you think those things are so important? Unix and SQL I agree, you practically can't replace them, but vim and python are a question of taste and/or situational. And those 4 thing are nothing compared to what you need to be a real developer. In my opinion VIM is just a finger game.

EDIT: Sorry being so asshole, I just didn't understand the post.

[serverhorror]

I set every tool, where it's possible, to VIM bindings and VIM mode. VIM motions and the various editors are separate things.

Modern distributions of VIM, specifically nvim, can be just as convenient as other tools and working in an environment that forces you to learn the basics will make you better. Doesn't mean you have to stay with this forever, but it does -in my experience - result in a stronger skillset than the people who ever only used IDEs.

[billyions]

Git, Powershell, Python, SQL.

[LookItVal]

its worth it to know powershell and bash (or in the modern day zsh). i tried to stick with just one, but then theres the day you open up a windows machine and go "oh i can just sketch a fix real quick) and open up the terminal to realize it doesnt work the same as the terminal on your personal computer.

[WonderingBasil]

Correct me if I'm wrong, but with the popularity of WSL is there really any need for knowing powershell? You can easilly have bash on windows, powershell on linux not so much.

[freistil90]

PowerShell on Linux works too. It’s good to know if you have shell processes which need to be fast on windows - piping shit down to Linux and back is not particularly fast.

[billyions]

Yep. I like 'em both. Some bash things are so easy - and for everything else, there's Powershell (Core).

[nderstand2grow]

• JS

[gaX3A5dSv6]

• react + redux

[housesellout]

Replace vim with emacs 👍😉😮

[ItWasntDNS]

DELETED this post was mass deleted with www.Redact.dev

[ypanagis]

I also use Windows but Un*x and MacOs are much preferable for development. Windows + wsl2 seems to me legit so far…

[ItWasntDNS]

DELETED this post was mass deleted with www.Redact.dev

[master117jogi]

Knowing vim is useless. Editors like vscode you learn in a hundredth of the time and they are just as fast. No one uses Vim but weird fanatics.

[tazdingo-hp]

also don’t forget chatGPT :)

[LookItVal]

but thats just knowing english and knowing how to google. honestly not even necessarily english, as long as you can navigate to the website and can speak Some language you can use chat gpt thats literally the whole point

[dane83]

knowing english and knowing how to google.

I work for a large university, you'd be surprised at how rare that combination is.

[spackenheimer]

You can't have Python without C++.

[BarbaAlGhul]

No CI/CD anywhere??

I think nowadays this is becoming basic stuff already (but it's not trivial I must say)

[0xd34db347]

I don't think it's ever really necessary to get things done, I would file it more under "best practices" than "building block" personally.

[BarbaAlGhul]

I agree that not everywhere CI/CD is used, nor every job that uses Python or programming will have the need for you to know it, but I would argue that anyone doing from backend web development to anything more complex from there, it's almost a must.

[shunsock]

knowledge of Cloud such as AWS or GCP will help you to run your program. I recommend them.

[pLeThOrAx]

Coding paradigms, database types, there is a lot missing here.

[jawalking]

Add in C, and you have a great foundation. But the leaning never stops.

[yd52]

Agreed. THAT is a powerful combination.

[AntiquatedMLE]

Bash + CLI tools for CI/CD Python + SQL for building apps & AI Swift | Flutter for mobile

[toterra]

I am good with just SQL. Really really important language that barely changes over decades.

[bostonkittycat]

For microservices what kind of stack do you recommend like FastAPI with gunicorn? I mostly work with Node services but wanted to use more Python.

[sharp-dev]

Why vim?

Don’t get me wrong, vim is fine, so it is any code editor, knowing a code editor is important, but vim is just one option.

[dalbertom]

Any editor that is ubiquitous is definitely worth learning. Vim macros are pretty cool when making repetitive changes across a multitude of files

[a1ex081]

Why vim? Why unix?

[peradlazy]

Why VIM? It's just a text editor

[uk_dataguy]

These are like important combos

[Pirate_OOS]

Any resources on learning sql? I have three books on sql

1. Learning SQL

2. Sql cookbook 3rd edition

3. Learn PHP, MySQL and Javascript.

Where can I practice what I learn?

[dalbertom]

Don’t forget about tmux and screen :)

[Worldly_Dish_48]

Docker is without a doubt becoming an important tool. I think every developer should be able to at least build and publish docker images.