What to use instead of Bash / Sh for scripting?

The question you should ask yourself is not what use instead of bash but what is the problem you are trying to solve and what is the best way to solve it.

It's hard for us to help you with you giving us some context.

150-500 loc of bash are not that much, specially when you combine bash with standard Unix utilities.

These scripts you wrote, for what do you use them? Are they executed by humans or machines? On which platform? Linux, Mac, Windows or all?

You mentioned jq. Are you parsing json from HTTP requests or files? And why are you parsing json with bash?

Without a better context, A lot of people will say to use Python and Python is a good choice with you are running you scripts on Linux because you probably already have Python 3 installed and whatever you are writing with 500 loc of bash can be accomplished with zero dependencies with Python standard library.

If relying on the Python runtime is really a problem for you, your next best option is Go. Go is easy to learn and easy to use for building small cli apps with only the standard library, but even if you need some dependency, you will ship a single binary, so no problems with runtime compatibility or dependencies.

Tcl. As with other shells, the scripting model is based on commands and substitutions, but in Tcl substitution is more sane because a substitution is never reparsed and reinterpreted. Tcl was designed to provide a uniform scripting environment that could be extended into any problem domain. In this sense, is specifically intended as a generalization of the concept of a shell beyond interactive use by a logged-in user and into programmatic control of any set of subsystems. Tcl's syntax is more simple and more regular than other shells, and its semantics are designed to make it easy to create a DSL to manage the task at hand. Values are immutable in Tcl so it has the mathematical purity that is attractive in a functional language, but is much easier to get started with. Because every value is a string, there is a fexibility in Tcl that can not be found in other languages. Stringiness is interesting, fun, and productive.

I’ve seen people use Go because it’s a simpler language. It’s what I would probably use if I didn’t want to use bash.

What do you think about editing the title to include “for scripting”? I thought you were asking for recommendations like zsh or fish when I first clicked

Perl (5). It's an incredibly capable scripting language, many things essential for Shell programming are much better integrated than in Python (e.g. regular expressions). It gets a lot of hate these days but it's still a damn fine language and perfectly suited to shell programming (and larger projects).

Next Generation Shell was born exactly out of the pain of using bash and Python. I'm the author and I felt the same way.

> shelling out when needed is not pleasant,

Handled properly because the language was built ground up for Ops tasks. There is even a syntax for "run a command and parse the output". Rationale and examples - https://ngs-lang.org/doc/latest/man/ngswhy.1.html

Currently NGS supports Linux and MacOS.

> smallish binary

Unfortunately there is no static build yet so can't check this checkbox

Next Generation Shell - https://github.com/ngs-lang/ngs

Alternative Shells list by @chubot - https://github.com/oilshell/oil/wiki/Alternative-Shells

Check out Babashka if you enjoy lisps/Clojure.

https://github.com/babashka/babashka

Without knowing what you're trying, I'm often using nim (https://nim-lang.org/) when I want more complexity than a basic bash script, cross-compiled binaries _and_ way smaller binaries than go. It's much like python, easy to pick up, and promising. But not as mature as python, obviously. Go is great, but the binaries are big... not a problem if you've got great bandwidth...

I really like NodeJS for more complex system-scripting. In contrast with Python:

1) There aren't two disjoint versions to juggle

2) Dependency installation is clean and contained

3) Dependencies can easily be installed locally or globally, depending on your preference for isolation vs reuse

4) Lots of packages are available for doing cross-platform (read: Windows compatible) operations

5) JS makes it really easy to do efficient async tasks, for example high-IO scripting involving reading and writing lots of files and making network requests (without any dependencies, I might add)

At my last company I converted all our shell scripts to Node scripts when the pandemic hit, so they'd work predictably on Windows machines, and it was great. These days I'd do it that way to begin with.

Can you be more specific than "fatigue"? That sounds like ennui, in which case maybe consider a job switch instead of a language switch (not being sarcastic or intentionally mean). Bash plus all the Unix tools (awk, grep, sed, tsort, comm, the ones I don't know, etc.) have pretty long legs, in terms of functionality and the syntax is pretty understandable. They provide most of what Perl offers, apart from OO (-ish) and line noise.

If you want strong typing (are you scripting in the large?), then, almost by definition, no scripting language provides that. If you need performance, switch to a compiled language (or a jit-compiled interpreted language) that most of your co-workers understand.

If Python brings too many dependencies, I guess you're trying to do something that is way out of the realm of possibility for bash, so you're not looking for an "alternative" to bash, you're looking for the best language for whatever project you're undertaking. And you may not be able to get away from the dependencies if you're trying something specific/sophisticated.

This won’t work for many people who don’t know the language or its ecosystem, but I’ve started doing my ad-hoc scripting in Rust using rust-script [0]. I’ve got a template with lots of boilerplate for everything I might need and I just copy it and delete the parts I don’t want. Having argument parsing, dotenv init, toml configuration and such ready to go with just a few tweaks, simple user prompting with dialoguer [1] and easy delegation to other tools with cmd_lib [2] means I can often write the tool I’m envisioning in just a couple of minutes.

[0] https://rust-script.org/

[1] https://crates.io/crates/dialoguer

[2] https://crates.io/crates/cmd_lib

For me: I use groovy. Reasoning behind it is that I am a software engineer specializing in JVM. On all the machines I work there is a JVM installed. Plus groovy is Java with extra stuff so it is a flavor of Java nothing new. And it does not require a compiler.

For a quasi embedded project I picked lua. my requirements were small runtime size + fast startup time. I would say that the library availability is tremendously lower than perl,python,node but ample enough for most tasks (openwrt is almost completely built in it, including web views)

However I agree that nim / zig could also be good directions

Is there a language with sane syntax (if, for, comparisons being a syntactic feature not commands requiring ; in weird places) that would still execute every line by default and support pipelines and && and || natively? Sort of bash that looks like a normal language.

Maybe even some library with stuff for parsing outputs of common commands?

Just use PowerShell. You can do so much in it without any dependency (jq is native in it and many other things).

It is cross-platform so you can use your scripts anywhere.

Don't use python or go or ruby or whatever as those are not scripting languages.

IMO Bash is horror and should die already.

A configuration management system may have you write e.g. YAML with Jinja2 so that you don't reinvent the idempotent wheel.

It's really easy to write dangerous shell scripts ("${@}" vs ${@} for example) and also easy to write dangerous Python scripts (cmd="{}; {}").

Sarge is one way to use subprocess in Python. https://sarge.readthedocs.io/en/latest/

If you're doing installation and configuration, the most team-maintainable thing is to avoid custom code and work with a configuration management system test runner.

When you "A shell script will be fine, all I have to do is [...]" and then you realize that you need a portable POSIX shell script and to be merged it must have actual automated tests of things that are supposed to run as root - now in a fresh vm/container for testing - and manual verification of `set +xev` output isn't an automated assertion.

I don't think the breaking point is the number of lines in the script; your pain point is probably one or two specific concepts or features? (or lack of)

Shell is likely to remain the best way to connect together other pieces of software.

The tipping point, in my experience, is folks trying to use arrays or other buffering of data. At that point they're writing in shell, but wishing for a procedural language.

Whereas, shell works excellently (and incredibly efficiently) if you can compose your task strictly into streams of data (a more 'functional' feel)

So the real answer I think is specific to the types of script you are writing!

But also it could be worth revisiting the style in which you're using shell; especially if you can call upon your own helper programs (eg. in Python) where you require buffering or non-streamed access to data.

Whilst there's a lot of documentation of sh/bash functionality, I don't think there's anything like the same momentum behind 'best practices' as other languages like Python or even C.

Perl is nice and ubiquitous.

As a devops engineer who also writes a lot of code, I started using Deno (https://deno.land) since I can write my code in JavaScript and TypeScript. I can also natively use wasm, and have an internal repo for hosting commonly used functions since Deno loads files using url imports.

Can recommend!

If you are ready to invest some time then try `perl`. It is awesome.

I'd say Go might be your best bet - it's easy to generate executables, you get a simple type system, and it's pretty easy for anyone to pick up and modify a program in. However another alternative is to use the language of the project so it's uniform. Most languages are pretty usable for scripting.

I use php(7.4) and python(3.x), works great. Alternatively, perl is also a great choice.

> I don't know what version of Python I'm going to have on the system

You could have assumed Python 2.7 for 10 years or so.

Now you could safely assume Python 3 (find minimum version applicable for your case).

> installing dependencies is not fun,

Pure python dependencies are easy (e.g., pipenv). C extensions are not more difficult than corresponding dependencies for your bash script (likely via system packages).

> shelling out when needed is not pleasant, and the size of program always seemingly doubles.

Have you tried to use shell command for shell one-liners in Python? e.g., directly via `proc = subprocess.run(shell_pipeline, shell=True)` or dedicated libraries such as pyinvoke/fabric?

I've been wondering the same myself... I want: - reliable scripts::in my case this means strong typing. Because for example currently to see the exit status of a program one need to check the exit number 0 or non zero plus the string for the command. - crosplatform:: this eliminates interpreted languages like ruby and python - easy to interact with the environment :: is something that like bash can actually call programs installed on the computer.

As the song says: "But I still... haven't found... what I'm looking for."

With nix you can make anything into a script, there's some people using haskell for example.

I tend to use common lisp but I'm learning guile scheme and will probably move to that instead.

I use Fish as my shell instead of bash precisely because I really like the scripting language for my personal scripts.

It's not the best fit for everyone but it has served me well.

I like to use Python with PyInstaller[1], but it results in fairly large binaries with a relatively slow startup time.

When binary size and startup speed matters, then I use Go instead.

Finally, sometimes I just write in Bash anyway, even though it is such a difficult language to write in.

[1]: https://pyinstaller.readthedocs.io/en/stable/

I recently changed my shell to elvish because of the data structures and extendability. It has a lot of sane defaults that make it very appealing to me. https://elv.sh/

>I don't know what version of Python I'm going to have on the system, installing dependencies is not fun

If your colleagues can be expected to have Docker, you can wrap your Python script in an image that contains specific versions of Python and the dependencies.

#!/usr/bin/newlisp

With GraalVM, you can compile all JVM languages into binary files to run, including Python.

There are many other scripting languages to choose from, such as JavaScript，Ruby, Kotlin and Groovy.

Of course, you can also use Java，Scala, Clojure and so on.

Just use shell or Python3 and stop nitpicking IMHO. There aren’t really any other choices.

Most OSes come with relatively modern versions of either. Pretty much everyone is or should be familiar with either. It’s not like it’s some Lispy thing.

Like others said it depends on each problem you are solving. Consider Java, nodejs, go, rust, ruby, python, c# - the one you know best is the best. Build up some Util classes for stuff you do regularly. NPM has a package for everything eg there is a cron so I like it for that reason.

Consider docker for some situations. Consider kubernetes if the need is more devops/ running a production system.

Consider powershell if on windows.

Travis ci / Jenkins / appveyor etc. for CI

Be lazy - can I get out of writing this code should be in the forefront of your mind (for any code!). The answer might be no but consider it.

No code tools can be considered - Google sheets, Wordpress, plethora of saas to do this. Paas eg aws lambda etc.

So many options. Solve the problem, rather than transliterate bash.

You might want to experiment with Powershell. It runs on Linux.

You might not adopt it, but it can give perspective on the pros and cons that comes with Bash.

I tried using Python with import sh for awhile, but...meh.

  https://amoffat.github.io/sh/sections/faq.html

I used to use Python for this also but now use Ruby and have no desire to ever go back.

Its Perl-ish roots just make it a fantastic tool for the job.

Have you tried other shells like csh or tcsh?

Depends on the program. Sometimes I use shell scripts, and sometimes I use Node.js, and (more rarely) PostScript.

Nim

Honestly it depends on what you're solving, personally I use both.

PHP is great, self-contained solution

NodeJS

TXR Lisp

Python.