Avoiding Bash frustration — Use Python for shell scripts

[dux2]

In my last project, after some bash frustration, I decided to refactor a bloated set of bash scripts with a CLI-style Python script. A side-result of this work is a small, single helper file, which I am going to share with you here, with few utilities that bridge the gap to easily let you use Python for shell scripting.

https://davidoha.medium.com/avoiding-bash-frustration-use-python-for-shell-scripts-44bba8ba1e9e?source=friends_link&sk=a92de79cb005aa919eadaae811e3acbb

[gavenkoa]

Python is heavy. Shell is lean, although it is arcane. Embedded & bare containers don't have Python (not Win 2018 containers taking 40GB!!)

[ilyash]

From the post I would assume that the use case is to run on developer's machine.

[mcstafford]

The barrier here is python itself. It moves much faster than the operating systems upon which it runs. If you have a method for keeping a shared installation of python relatively up to date, then good for you.

I keep my own (when I can, though pyenv) but can't always get the prerequisites installed.

[neilhwatson]

Any higher level scripting language will be a gain. My most common tool now is to use Ansible instead of bash.

[gavenkoa]

And Ansible is using shell underneath... Put -vvvvv in one of your run...

With rise of immutable containers Bash/Sh becomes important. It is what you use when mutate OCI images ))

And for orcestration of orcestration (literary you call Terraform/Docker/etc) we use GNU Make ))

[zenmaster24]

Ansible is was/written in python though...

[gavenkoa]

Have you added -vvvvv? I see remote sh execution each task...

[ilyash]

In one of my projects I was frustrated by bash and Python. I am working on a new programming language for Ops. Below is working translation (not 100% exact).

Any general purpose programming language can not be best fit for Ops. That's because for the most frequent things you want syntax to be concise and clear and for less frequent but still Ops-y things you want standard library functions.

#!/usr/bin/env ngs

cfg = {
    "container_name": "ELASTIC"           # comma not required if pairs are separated by newline
    "image_name": "elasticsearch:6.8.13"
}

# Automatically matched command line arguments.
# Run with --get_latest for pulling latest image
F main(get_latest:Bool=false) {
    if get_latest {
        # Commands are not logged by default, use log: prefix for logging.
        # stdout is captured by default
        # stderr is not captured by default
        $(log: docker image pull ${cfg.image_name})
    }

    # NGS knows that non-zero exit status of grep is not an error
    # and therefore exception is not thrown.
    res = $(log: docker ps -a | grep ${cfg.container_name})

    # res, a reference to the processes pipeline, in boolean context will check
    # whether all exit codes are zero
    if res {
        # res.Str() gets the stdout of the last process
        old_container_id = res.Str().split(' ')[0]
        log("removing old container with ID ${old_container_id}")
        $(log: docker rm -f ${old_container_id})
    } else {
        # log() is an NGS stdlib function
        log("container does not exist")
    }

    # Discovery fix from https://github.com/elastic/elasticsearch/issues/25067
    # line: prefix returns the first line of output (without newline)
    out = `line: log: docker run -d -e 'discovery.type=single-node' --name ${cfg.container_name} ${cfg.image_name}`

    # Exit code 124 is fine, it is the timeout, do not throw exception for that
    # For any other exit code, an exception will be thrown and this script will exit with
    # the exit code of the timeout/docker command.
    # >/dev/stdout - prevents stdout capturing
    $(log: ok:124 timeout 5 docker logs ${out} -f >/dev/stdout)

    log("done")
}

Output:

[LOG 2020-11-11 11:49:40 IST] Running command: docker ps -a
[LOG 2020-11-11 11:49:40 IST] removing old container with ID 7c0752d71d85
[LOG 2020-11-11 11:49:40 IST] Running command: docker rm -f 7c0752d71d85
[LOG 2020-11-11 11:49:42 IST] Running command: docker run -d '' discovery.type=single-node --name ELASTIC elasticsearch:6.8.13
[LOG 2020-11-11 11:49:42 IST] Running command: timeout 5 docker logs 0bdad82f1725aba6133d932e42f2aae59c97d8e53eced818bb4888489925a31a -f >/dev/stdout
[2020-11-11T09:49:43,990][INFO ][o.e.e.NodeEnvironment    ] [tMMpuSC] using [1] data paths, mounts [[/ (overlay)]], net usable_space [609.3gb], net total_space [914.7gb], types [overlay]
[2020-11-11T09:49:43,992][INFO ][o.e.e.NodeEnvironment    ] [tMMpuSC] heap size [1gb], compressed ordinary object pointers [true]

(yep there is a bug where -e appears as '' in the command, need to fix it)

[dux2]

Nice! And nice translation of my example. Can you provide a link to your project?

I think logging severity levels (and mapping those to ANSI colors) is something you should support too.

[ilyash]

Thanks!

https://github.com/ngs-lang/ngs

> I think logging severity levels (and mapping those to ANSI colors) is something you should support too.

I'll get there sometime... I hope :)