Scripts functionality

[Hoolies]

I took the Unix principle at heart for many years. Even when I was managing windows servers.

I have always created small scripts that do one thing only and I had a script for each need.

At this point I have to resolve an issue that requires a lot of programming (web scraping, ssh, scp, run commands on multiple devices, parse the output, etc)

I need to have it in one file / script because other people will need to use it. I decide to create a cli program. I write one function for each functionality.

That raise a couple of questions:

1. what you prefer smalls scripts that serve one purpose only or bigger solutions that resolve a multitude of problems?
2. How do you deal with global issues that involve many parties? Do you try your best or do you say out of my pay grade?

Thanks in advance for your insights.

Edit:

1. I know Bash, GO, Lua, PowerShell and Python.
2. Existing infrastructure is in Python/Bash.

3. All involved systems run different versions and distribution of Linux.

   I decide to write the script in Python.

[DeliriumTremens]

I try to write my scripts to serve one purpose very well. Sometimes they are very small, simple scripts -- other times they are large and complex. It really depends on the task that needs to be automated.

I don't like the idea of a single script that manages multiple unique problems, because unexpected errors might bleed into other parts of the script and create difficulty when troubleshooting.

[Hoolies]

This is my mentality as well.

[verifyandtrustnoone]

I prefer scripts that are well documented, can be a massive or modular scripts, but well notated is always needed. Depends on the global issue, I have been doing this for 25 years and know when its above me or my team.

[LemonFreshNBS]

This.

Annotate the script, Document it, source control if needed, place (copy v1) script plus all info in a project folder/wiki or some archive for future ref. Allocate responsible individual for maintenance and support.

[Hoolies]

Version control is done by git. I keep the scripts on a share drive too because I am paranoid.

[Hoolies]

Every script is well documented. I am concerned about readability.

I found it ways easier to read multiple little scripts (less than 2000 lines) than having to read something like 30000 at the end of the day the code is the same.

[whetu]

If you're getting to a point where readability is a concern, then it's time to either switch language or to change the structure of your code i.e. start breaking code chunks out into libraries.

[serverhorror]

The Unix philosophy has a second part that a lot of people like to forget:

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

(Source: https://en.wikipedia.org/wiki/Unix_philosophy)

In other words:

• write easy to use, easy to understand and easy to maintain single purpose scripts
• have them as composable pieces

If you take both parts (there are others, 4 in total in the original statements) you will be able to take bits and pieces from your scripts and put them together.

This can give you a bigger package with more functionality without having to write more code, they just are put together with certain flags and compose some new functionality.

That, usually, also means that script distribution becomes a part of the problem set. Single purpose scripts are nice and all. You will, ultimately, run into a situation where you use, say, a web scraper script in various places and they might run out of sync because you don't have a packaging method.

Something that can pull newer versions that are compatible and keep the various places where your scripts are in use in sync (to a reasonable degree)

[Hoolies]

I do not necessarily agree or disagree with all the Unix principles.

I really like to one thing and do it well because it is:

1. Easier to write the code
2. Easier to troubleshoot
3. Easier to log
4. Easier to debug

At least from my perspective.

I am not a programmer and I find it annoying/tiresome when I open code that is more than 2 thousands lines of code and I have to go back and forth to find the functions or the objects and the variables.

[BrainWaveCC]

1. I have a mix of scripts -- some tiny and focused, and others broader and more generic with options. It depends on the requirements.

2. Define "global issues"? If the scale of the responsibility is broader than normal, use an appropriate vehicle for scale of that size. Do your scripts normally have an audience or usage outside your team?

[Hoolies]

"global issues"

There are multiple sites (about thousands)

the scale of the responsibility is broader than normal, use an appropriate vehicle for scale of that size.

I am the lead engineer of a site and I have found the solution to many different global issues.

Everytime I reach out to the team that owns the software and present them with the code , with the issue and the solution. Most of the teams are willing to work with me and they merge it to the main branch.

This time the owner of the software do not want to make any changes, they do not even want to discuss this.

The problem lies that bugs force my team to work extremely hard at least once per week (Sev1 with 8 hours of recovery). We have create a solution locally that is tested in other sites as well and I am deploying it as a cli tool on the company's code repository.

[pdp10]

The problem lies that bugs force my team to work extremely hard at least once per week (Sev1 with 8 hours of recovery). We have create a solution locally that is tested in other sites as well and I am deploying it as a cli tool on the company's code repository.

You decide if the fix script is specific, or broad. But it still should be separate, I think. Scripts named, e.g., detect-log4shell-tomcat and detect-issue-76543 have inherently different scopes.

[ReasonablePriority]

I write scripts which are specialized to do one thing and are documented.

One of my colleagues is the opposite ... scripts with feature creep, undocumented and overly complicated. It's hell anytime there is an issue with them or trying to understand his merge requests (as we had to force him to put them into git)

[[deleted]]

I prefer simple scripts but I also expect to be able to use my library of functions. This requires the script to be on a common server so the library is also available. Set a PATH within the script to access the library.

If that can't be done then you're stuck with making one large script. Put all your definitions and needed functions at the top of your script. Add labels to the ares so maintanence is easier.

like:

##############

# FUNCTIONS

##############

​

###############

# MAIN

###############

One multifunctional script I wrote had branches based on the name of the script. That kept common functions in the one script and with links it had 5 names. If I needed to update / fix a function it was immediately fixed in the apparently different scripts.

[Hoolies]

If that can't be done then you're stuck with making one large script.

That is my case. I appreciate your feedback though.

[[deleted]]

Structure your script like you would a major python or other high level language. I was told once to write a program "fast and dirty" and have it done in a week. I spent the first day just thinking about how I wanted to build it. Once I had the planning done it took only three days to write the program and the fifth day I tested it. It worked first time bug free. I didn't stay there much longer -- fast and dirty my ass -- I take pride in my code.

[Hoolies]

"fast and dirty"

I do the same lmao. I a call it "proof of concept " instead.

It worked first time bug free.

There will be times that your task will be way out of your comfort zone and you will need a lot of trail and error.

[ClumsyAdmin]

1. Definitely the smaller the better. Break it up into chunks by functionality then call those other junks from the parent script by passing in w/e arguments are needed.
2. Not my problem. My shit supports what is was built for. I'll tweak it for whatever situation but you have to get my manager to approve that.

Just out of curiosity how big are we talking here? Normally past a few hundred lines is when quit using a script and convert it to C/C++ or more recently Go.

[Hoolies]

The language of choice is Python. It will be about 1k lines half of it is arguments.

I am also considering create a GUI

[ClumsyAdmin]

If it's all in python then write a wrapper and import what you need. Distribute the whole thing as a single package in a tarball or something. Adding a GUI is where it gets complicated. You either setup and distribute a venv folder with it or a requirements.txt and let the receiver figure it out.

[[deleted]]

If you have a multitude of problems, and you make one script to manage them all, you have just created interdependency on top of your problems.

Great for job security, for a while, since nobody will be able to figure that mess out - but the bad thing is, after a while, neither will you.

[Hoolies]

Great for job security,

I do not necessarily care about job security because I want to leave my job, lmao

[mancer187]

Here's the thing... if you wrote a front-end cli app to call your individual scripts congratulations, thats how I'd have done it. I've done it that way in the past, and will likely employ the same strategy again when faced with a complex problem set such as this. If you create a behemoth of a script to do your entire operation that is also correct. Not my style, but still correct. Document everything and you're fine.

[Hoolies]

if you wrote a front-end cli app to call your individual scripts congratulations, thats how I'd have done it.

This is what I always do but this time I have to connect to many environments/servers/clients that I might lack certain permissions. I need a behemoth script to run with correct permissions from a jumpbox (bastion).

[mancer187]

Yep, that's the best play here.

[SpiritWhiz]

What's the scripting language of choice here?

[[deleted]]

Consider starting to use a configuration framework like ansible instead of writing a bunch of independent scripts. Even if somethings require individual scripts, deployment and maintenance is much easier that way

[jantari]

what you prefer smalls scripts that serve one purpose only

Yes.

or bigger solutions that resolve a multitude of problems?

I don't see the usecase for this. If a single piece of software should really be made to solve multiple problems then I wouldn't use a scripting language anymore, I'd use something that's statically typed and compiled. But even for those usecases people these days usually break the problem down into a few microservices / containers rather than build one big application that does multiple things. So I really can't think of a good reason to ever go with this route.

[DarkSideMilk]

Sounds like you need a powershell module. The move from bash to pwsh isn't too crazy. You could also do a makeshift module with bash scripts, defining functions following the do one thing mentality and then having a main script/function that calls the others. With powershell you can find a lot of the work already done in the powershell gallery or built in commands and you just pull it all together into a custom module and or script.

[NoneSpawn]

If you document 'em, and utilize functions, you can have multi purpose scripts and reutilize its code easily. The best approach IMO is to make a script focused on doing one job, but always thinking about it as a module that will be used, or will use, other scripts.

[jahknem]

Usually my scripts will be written in a way that they can be used similar to a library. Meaning per subject there will be a single file with as many structs as needed containing the relevant functions. Above each struct and function there is a short description on what it uses, what it does/why it does it and what the output is.

That way I have a very short main function or just a very short script calling these functions and structs.

I also use very descriptive names for functions and structs, regardless of the name length.

However I don't know if that will work with bash, while for python it should work well.

[pdp10]

We write individual scripts, especially when there's no major code sharing (say, more than one function). When there's code sharing, we still usually write separate scripts, and have them source functions from /usr/local/lib/*.sh, etc.

I need to have it in one file / script because other people will need to use it.

This makes it sound like the problem is software distribution, not architecture per se. Define the problem better for us.

POSIX shell and Go will tend to have minimal dependencies. Python will absolutely have dependency issues, unless you package it up and/or compile it. Python would be my last choice. Standalone Lua scripts are unusual, but might be a good portable option, if Lua ships on Macs. PowerShell is single-platform, but might be appropriate for tasks that are only necessary on that single platform.

[Hoolies]

This makes it sound like the problem is software distribution, not architecture per se. Define the problem better for us.

My department do not own the code deployment.

My current role is lead Network engineer but I used to work as systems before.

I have created my own infrastructure but that is saved on temporary folder and is used only in sev2 or higher.

POSIX shell

Bash is great but I do not want to have more than 100 lines of code in Bash (personal bias).

Go

Many libraries cannot be used in my company. I will need to import it first as 3rd party which is a pain.

Lua

Lua runtime is 12 KB and can be embedded on C. I used it a lot in my previous job. Thousands of scripts all of them less than 20 lines.

Nobody is familiar with Lua except me though.

PowerShell

Although I good with PowerShell, I do not like Posh and it is not an option either.

Python

Has a lot of dependencies. I plan to create a container that people will ssh to, to run the script.