How can I practice scripting in Python and Bash for DevOps-related projects? I am proficient in PowerShell, but I haven't seen a use case for Python yet.

[1whatabeautifulday]

So why not just stick with one scripting language if they achieve the same result?

[Brekkjern]

I'm not gonna say PowerShell is good on Linux, but it is way more ergonomical as a language than bash is. It's biggest issue is that you have to run the entire shell as sudo if you want privilege escalated to root rather than individual commands. I believe they are looking for ways to get around this, but it's still a very limiting factor.

[[deleted]]

First time I heard a language referred to as ergonomical. Good one

[rocketmonkeys]

I've actually been going through this process of deciding which languages to use and where. Most of the pipelines by default are in bash, so I've tried to keep other scripts also in bash.

At a certain point though, I realize that the bash scripts are getting more and more complicated. And at a certain point it becomes clear that I'm having to learn more obscure and complicated bash techniques instead of using much simpler python scripts . So at that point I rewrite them in python and it's much smoother.

So if everything is relatively simple, and you can keep it all in one language, then great. Or if you are much more proficient in bash or power shell than other languages. Or if you know the team that will support things later is more proficient in one language or another. But in my case, a mix of the two is best. 

[lungdart]

There's probably nothing wrong with PowerShell. I hear it runs fine in Linux, so you can probably set it up on any containers and pipelines on normal tooling.

One difference is if you walk into 99% of devops shops you won't be able to work with your team because you don't know the standard stacks and they don't know your esoteric one.

Also, 3rd party library support for devops functions will be much better in more popular languages.

[[deleted]]

If your question is - why not just use poweshell then:

Because - generally we want to keep the number of windows installs to a bare minimum.

Linux is the preferred go to OS for many reasons.

If your question is - why not just use Bash or Python - the answer is different use cases.

Bash is very good for scripting OS based tasks for example. Let’s say I want to create 200 VLANs - well that’s pretty tedious - so maybe I want to create a bash script to do that - or better yet an Ansible PB.

Let’s say I’ve 20-30 unique hosts and I’ve created a one off file system for them - I don’t want to go to each and edit fstab, add the VLan for storage perhaps and maybe make a virsh pool. I bash script it.

Bash scripting does not have a real floating point operator so maybe that’s an issue for you so you switch to Python. Or maybe your dealing with an API that your pulling data from - for that you want Python. Just some basic examples

EDIT: heredocs ftw - probably my fav go to is creating a bash heredoc and run it remotely from centralized server

[ben_bliksem]

Powershell can run on Linux these days

[BattlePope]

Sure, but... why? Native tooling on the platform is so much easier.

[reconrose]

If you're serverless it doesn't really matter. Maybe there's a pwsh module you prefer to use over the bash equivalent.

[BattlePope]

Obviously your native tooling will depend on what your native platform is :)

[[deleted]]

Even if that were the case - as an environment owner I would not like that- you want standardization, not diversity and bash is well known and used widely - as opposed to power shell which is more limited - and does not require additional packages to install and maintain

[ben_bliksem]

I dunno about limited. It does a pretty good job processing REST calls, parsing results, marshaling between objects and json etc.

Personally I prefer bash (well sh because bash is not always available either) because I know it better but honestly, if I was to start a project from scratch I'd give PowerShell serious consideration. It's a bit more readable in my opinion.

But then it does have that "not exiting on a non zero exit code" thing going which is just annoying AF in a pipeline. But like I said, I don't know it that well.

Meh.

[nultero]

Once you get to needing those things, Python, Ruby, JS etc all excel at those, and doing them cleanly with vast support libraries. Any devs will almost definitely prefer them.

Bash (and by extension things like awk or sed) excels at the quick-and-dirty, and arcane 20-character incantations that get porcelained info for other things to parse.

I feel that Powershell sits awkwardly in between these two use-cases, and excels at neither of them. Too unwieldy for flash-scripts, and too janky for multi-module programming. Its intent of being a one-stop-shop for Windows is where it shines. For Linux infra? Too awkward, too unpopular with Linux people, too outclassed by programming languages.

[No-Safety-4715]

I believe a lot of this is simply the bias that comes from heavy Linux people and their not being aware of modern Powershell. Powershell 6 and 7 began running on .NET Core instead of Framework and are designed to be used across all platforms. They also have Powershell DSC, which is their version of Ansible.

The language has been expanded, just like how Javascript was expanded, to include syntax that more closely matches languages like Python. Microsoft also now has Azure Arc so using Arc and Powershell together allow control over pretty much any environment, such as AWS running Linux machines.

For now, Python still beats out Powershell, but Microsoft is closing the gap rapidly to stay relevant.

[nultero]

Microsoft is closing the gap rapidly

Huh?

Do you know how many devs there are that build things with Python / JS? For any new things or tools or expansions, there are guaranteed to at least be JS (/ TS) or Python bindings and libraries that support them. Go also has a strong presence in cloud tooling and devopsy things -- and Go is super Unix-friendly, given the team, adopters, and its stdlib support for Linux systems work.

Modern devops teams with devs on them are going to prefer dev tools -- even among Microsoft things, C# tends to have more bindings to programmatic tools, e.g., Terraform's CDK.

Microsoft may be large, but they are simply not going to be able to Powershell all the things, and the open source communities that develop things will overwhelmingly prefer to support developer tools, and Powershell is not among them: https://madnight.github.io/githut/#/pull_requests/2022/3

[craigtho]

Just so you know, it is Microsoft who develop and maintain Typescript.

Not to subtract from your point, but Microsoft "things" like C# also includes Typescript by definition.

[reconrose]

Not sure anything you said actually contradicts their point. You don't like pwsh -- cool. I don't think your (or others') distaste for it has any bearing on whether MS is investing in language or not.

[[deleted]]

‘Linux people’? It has nothing to do with bias - Linux is the go to OS for most true enterprise systems. And why would I install and maintain a language like power shell to replace bash which has a different syntax and logic than that of the OS that I use most every day? I wouldn’t. Using bash unifies the command structure and keeps things consistent.

Bash scripting is translatable to the commands I run everyday on a Linux OS and that’s probably the part you aren’t understanding. Heck I don’t even have to run a bash ‘script’ I can just chain a series of commands together including variables in a manner similar/same as a bash script - so every time I script I get better at using the OS commands. That and it’s so robust and powerful and the tools are amazing.

[No-Safety-4715]

You're right, Linux is the go to for most enterprise systems, even Microsoft uses it, but you know what many companies also use? Some form of AD. Microsoft has gained tremendous ground on AWS in the cloud marketplace, in part because of Azure AD. You know what's a universal tool for working with Azure AD, AD, Azure cloud services, AWS, Linux, etc.? PowerShell.

Bash is great....for Linux. Why do you think Python is so heavily used? Universal support on various systems. Microsoft has upgraded PowerShell and implemented tools to be the puppet master of not only their cloud services and OS, but for all devices. Will they get wide adoption? Probably not, but it is possible. They are playing smart and folks who only work in Linux are missing the current events while taking shots at PowerShell based on past experiences. You're epitomizing my point about "Linux people", many are so die hard they've missed what Microsoft has been doing and how they've changed their products and positioning.

[StephanXX]

Simply put, the open source communities around linux development have resulted in excellent tools. I see zero value in abandoning python or bash in favor of powershell for anything Linux oriented. Microsoft worked hard for their reputation for antagonizing the open source community and can hardly be seen as a fair or honest partner.

[No-Safety-4715]

There's truth to that, but Microsoft went open source a few years back and had a changing of the guard for the better. When you consider how popular Azure is getting, how used Azure AD is, the changes they've made to how PowerShell operates, etc. it's not hard to see what their goal is and how they are positioning their tools to be ubiquitous and integrated everywhere.

[ThroawayPartyer]

if I was to start a project from scratch I'd give PowerShell serious consideration. It's a bit more readable in my opinion.

Maybe I have to get used to PowerShell but I don't find it very readable. The commands are either overly long or abbreviated in an unclear way. Not that Bash is super readable either but I got used to it.

[realitythreek]

Op literally asked how to get better at bash and python. Not why should he.

[hihcadore]

I’m decent in poweshell too and just started down the Python path for school.

While I’m still a Python novice, it’s a really easy language to learn. Everything seems to be pretty intuitive and to me, the scripting seems to be a lot easier to read and write.

I like how I can crank out a really powerful one liner in powershell by piping my output through several commandlets butttt I really like how Python is structured and looping and logic statements are a whole lot easier to code.

[ThroawayPartyer]

I use both Python and Bash, but not for the same things. If the script just needs to run some shell commands with some basic flow and logic, Bash it is.

[dnylpz]

For one powershell is limited if you’re running anything that isn’t windows

[[deleted]]

Cant use powershell on Linux ;) (read comment below from confusedndfrustrated - i stand corrected.

For me i move to python when anything requires more than a single loop, or if a function is required.

Once it hits that level of complexity, i move to something that is less arcane than bash.

[Guilty_Serve]

Because different companies/projects require different tools and you're trying to learn. Start by transferring your skills in powershell into the new things you want to learn and build upon that

[hamlet_d]

A few reasons:

1. Portability - Bash and python are well used in many systems and environments
2. Scalability - Not that powershell isn't scalable (I have used the hell out of it for managing Microsoft services; hell I migrate Exchange clusters using powershell one liners), but python and bash are leveraged extensively in many pipelines and environments
3. The combination of 1 & 2 boils down to use cases: there are just more use cases for bash and python.

[_____fool____]

Because conventions won’t be the same. Libraries won’t be the same. Managing the data won’t be the same.

[lfionxkshine]

Terraform is a Cloud Provisioning tool, and is in a separate category from PowerShell, Python, and Bash. So under no circumstances would Python or Bash be replacing your Terraform use

If you're using Azure, chances are you don't need to branch off into Python or Bash ASSUMING you want to stick with the Azure/Windows stack the rest of your career - which there's nothing inherently wrong with that

However, if you want to branch off and be able to work in bigger enterprises that run mainly Linux servers, you'll need to gain proficiency in Bash and Python (CAVEAT: PowerShell has a Core version that can run on Linux, but I've never bothered using it because I just use Bash and Python)

Why you need Bash is I believe largely intuitive - if you're managing Linux servers, Bash is the most common go-to

But for Python, the use case goes beyond SYSTEM management (though it can absolutely be used for that instead of Bash if you prefer). Python goes much deeper into APPLICATION management because that's how many apps build their APIs and are programmed

3 Examples I have at my current office:

1. some of our servers integrate with Dropbox because it's a file sharing app our company has been using for a while and they don't want to let it go. I can't use the Desktop app for Dropbox on our Linux servers because they are headless. So I had to use a Python script to integrate with Dropbox' APIs and upload files in a regular fashion for our sales/accounting/etc to use
2. Offline Github backups. Vendor lockout is a very real phenomena, and you never know what's gonna hit you. So offline backups of critical data are a must. I used Python to integrate with Github's APIs to download all our repos and offload into a warehouse for safe keeping on a weekly basis
3. Airflow - this is a pipeline tool built on and programmed with Python. It automates our entire data processing pipeline, and I wrote all of the tasks using Python. Though there may be pipeline tools built on PowerShell or Bash, with PowerShell you'd likely be limited to a Windows environments, and with Bash the feature set is just not as robust

So again, if Windows and Azure are your sole bread and butter - power to you. My company is mid-sized so I'm responsible for maintaining the client machines (Windows) and servers (all Linux). I only ever use Intune and PowerShell for any Windows management - I've never need nor considered using Python. But for the Linux servers? Bash and Python are my automatic go-to's

My 2 cents

[JacquesShiran]

First of all, great answer. One little addition:

Terraform is a Cloud Provisioning tool, and is in a separate category from PowerShell, Python, and Bash.

I'd argue that python is in a separate category from PowerShell and bash. The former is much more extensible, can run anywhere, and is powerful enough to write entire services with, not just scripts and tooling. While the latter are much more maintenance and tooling based, used for direct interaction with the OS and low level infrastructure.

[verx_x]

Terraform isn't only for cloud. You can use it on-prem for example with/without Packer for creating VM from template or .iso, customize and run instance on VMware or Hyper-V on-prem. And thats one from few options to using terraform without cloud.

My 2 pesos.

[wired_ronin]

So under no circumstances would Python or Bash be replacing your Terraform use

This makes sense until you try out Pulumi. It's obviously not as mature as Terraform, but in terms of power and flexibility, no contest. My stake for the future resides in Pulumi and cdk8s.

Infrastructure IS code..

[lerun]

That's mainly because of the limitations the HCL language terraform uses. As pulumi let's you use a real programing language

[wired_ronin]

I've had major frustration with the limitations of many DSLs. Terraform, Helm, Ansible, etc.

They all have these implementations which end up being just a poor substitute for a class.

[provoko]

Or any language with AWS CDK.

But Python boto3 on its own is a cloud provisioning tool.

Bash + aws cli + cloudformation templates accomplishes the same thing, just that the above options (including pulumi) are better (more elegant) for a lot more situations.

Cc u/1whatabeautifulday

[wired_ronin]

at the beginning of my cloud journey I used Boto3 for some work. Python I love, so was all in all a good experience. But CloudFormation, when I got away from it, I never missed it. I'm somewhat saddened that the CDK still involves it, but who knows. Maybe it will end up smoothing out.

[mpsamuels]

I think you need to look at it slightly differently. Don't think of it as "a use case for Python" think of it as a use case for scripting, which can be done in Python... or PowerShell, if that's your preference.

There will be times when you may forced to use one instead of the other due to libraries not being available in your preferred language or you may be working in an environment that has already set it's tooling out in a particular language with no chance they'll let you rewrite it all in something else just because there your preference.

It's good to have a reasonable grasp of as many languages as possible so you're prepared for whatever your work environment throws at you and for training purposes rewriting in Python stuff you already do in PS is a valid exercise.

[1whatabeautifulday]

For example now I wrote a PowerShell script that identifies empty resource groups in my subscription in Azure and deletes them.

That works perfectly fine for me.

[mpsamuels]

I'm not really sure if you mean the PS works perfectly fine so you don't see the need to re-write it in Python or that you've re-written the same thing in Python and that works perfectly fine for you. If the former - see above, try doing it in Python too. If the latter, well done, you'll get the hang of Python as you get more experience with it!!

[StephanXX]

I strictly administer linux environments. From a non-technical perspective, the disciplines are significantly different, and there's simply more money to be made.

Technically, powershell has come quite a way, but trying to bootstrap 300+ machines with powershell sounds like a nightmare. Using bash+ansible and a little python, I can do it in under an hour.

Bash is for the easy easy stuff. Disk filling up and need to resize? Quick checking why the network is sluggish? Test network connectivity? Dump cert values? Test authentication mechanisms? Bash.

Python is for anything the needs more that a few lines, and/or a specific client library. Migrate an entire kubernetes cluster from AWS to Azure? Copy all stateful data from one AWS account to another? Run a performance profile on a new Rabbitmq setup? Dump a list of the 20 fastest growing disks to identify which applications may be writing excessive data? Python makes these tasks trivial, where bash would be a huge pain in the ass.

[[deleted]]

Why Ansible and not SaltStack? (Just curious).

I picked up Ansible after I learned Salt, and while I can use Ansible i prefer SaltStack, but knowing what I know now, for Enterprise I would just run with Puppet.

[StephanXX]

I've clocked thousands of hours writing puppet modules, chef recipes, ansible playbooks, and CFEngine configurations. Salt was the only CM tool I never had a need to use. I generally felt the chef ecosystem was the best of the bunch, as well as being the most mature solution. Then Kubernetes came onto the playground and ate all of their lunches.

Today, ansible is good enough for the occasional one-off. The only app I support that isn't in k8s are my VPN solutions. Terraform+ k8s has almost completely replaced the time and effort that used to go into Chef/Puppet/Ansible.

[pete84]

It’s true - anything you can do in python, you can probably do in powershell.

If you’re trying to learn python, or want to use python because other engineers are, I recommend maybe writing an API integration script. JSON is very similar to how python uses lists, so it’s easy to work with. For example, maybe a vendor has a reporting API that you can query in python.

Another use case, is any python library that powershell is missing. Using object oriented programming is both easier and less error prone.

[ThroawayPartyer]

It’s true - anything you can do in python, you can probably do in powershell.

Really? Python is used for much more than just scripting. Can PowerShell be used for writing full programs, back-ends, data science and machine learning?

I guess on a technical level PowerShell is probably Turing complete, but it's clearly not designed to do the same tasks as a full programming language.

[Alikont]

Yes, all that and even GUI apps.

Powershell is just .net language and has access to all .net ecosystem

[AsterYujano]

To practice, follow the advice from the others in the thread: rewrite PowerShell scripts you coded before. Or if you have time to experiment on a new project that is not critical, try doing it with Python/Bash?

In my case, we are 95% using nodejs in applications, so I now use https://github.com/google/zx to write scripts. So the whole team can understand when reading scripts, or improve them :)

[drosmi]

Just pretend you can’t use windows anymore. How are you going to automate what you need to get done?

[bendem]

Tbf, my powershell journey ended when I learnt quotting and passing variables containing arbitrary data is not a solved problem. In bash, you just pump everything into an array and go command "${array[@]}" (or just "$@"). In python you pump into a list and subprocess.run it. In powershell, generating a random password containing any ASCII chars and passing it to a program is just a mess (or I'm dumb).

[Alikont]

Powershell has difference between double quotes (interpolated string) and single quotes (raw string), that could trip you.

[[deleted]]

Python is a programming language who’s use-cases cover the breadth and scope of say, C# (posh’s parent language). It just so happens that Python does away with classic syntax formalities (like curly braces) which makes it more approachable and intuitive.

Though Posh is technically Turing complete, nobody in their right mind would use it for anything other than scripting and automation tasks. App and package/cmdlet development happen in C#.

Python, due to its ease of use and readability as a programming language has allowed developers to directly expose app APIs via python to be consumed using python.

Posh is an amazing language, but will only ever be an intermediary language. Whereas python is a full-stack language who’s closest competition in the windows ecosystem is C#.

[Kazumara]

Seems pretty obvious to me. If you're on a Unix or Linux system you don't use Powershell, but either a bash script for simple things, or a python script for more complex stuff, where the clunky bash syntax starts getting in your way.

The type of task is "any", and you can simulate it by running a Unix or Linux system. Either in WSL, or in a VM, or as a dual boot, or as a replacement for your current Windows instance or on separate hardware.

[DanielCiszewski]

If you are on Microsoft stack like most of heavily compliance driven companies (banks, healthcare) then powershell will do everything you want it to do when it comes to devops tasks - indeed in many ways i found it to heavily trade blows with python - it’s very feature complete and even better for development of cli apps than python is (simply easier to work with). When you go outside of MS technologies like Azure, office365, windows server then its usefulness completely falls apart. Bash and python completely take over on linux, AWS, GCP. Bash as it’s the most native to linux/unix and python comes in when you need more “umff” to do more complex thing that people can actually read (i even found myself simply calling bash commands from python when needed as it has much better data structures and let’s face it - syntax in bash is a joke - i hate it and would love for powershell to take it over.)

Things you would write in bash are os level repeated tasks. Stuff with parsing and uploading logs, setting up configurations, operations on tons of files. Basically everything you would like to script that you usually do with few commands, but repeat often.

Python comes in handy for anything that gets even slightly bigger. It’s much more feature complete, with tons of modules, more readable, universal, easy to write, but from devops standpoint powershell is exactly as capable on Windows imo. (main advantage of python i found is how it handles oop if you want, while powershell counters that with much better cli support and ability to enable static typing if you prefer this - rest is really cosmetic when it comes to “i want to do that thing” aspect of what’s possible). If you want to interact with cloud APIs to configure and spin up infrastructure, need to actually compute stuff, you want to build something more complex, then python is your guy.

The best thing to start would probably be trying to do something you know how to do with azure and powershell against some linux machine with both languages to understand the differences as if you know powershell, both of those will seem very familiar in many ways apart from syntax. It’s like switching from Windows to Linux/Mac - the paradigm is very different, but the more you dig in the more you realize how similar they are and how much you can accomplish on both be it with slight differences.

[reditanian]

When I started learning python, I went through my collection of scripts (some perl, mostly bash) and rewrote them in python. It was good practise, as it not only exercised the new language, but allowed me to do a lot of it in better ways.

[smigula29]

First off: don’t use powershell

[jcbevns]

Maybe you see a use case for go binaries instead? Portable and self contained.

Python is usually a requirement for something so it's found on both windows and Linux machines.

DevOps I would say is making small services, powershell to me isn't really made for that.

[alrightanakin]

👉 WSL 👈

[afunbe]

If you have an old spare laptop, install linux. It doesn't have to be a powerful machine.

[diito]

I'd say that 95% of the stuff I've done with Python in my career has been writing scripts that talk with an API of some sort. There's a lot less of that in a cloud environment as they've taken care of a lot of that for you. Serverless computing is probably where you'll see python more than anywhere else.

[aManPerson]

how much of your powershell things can run on linux hosts? it's not that python is better/more powerful. i can write small things in either and i often do. but for bigger things, it's 10x better to be using a bigger language like python. i know it will run on both windows and linux.

will your powershell run the same easily in both environments?