DJ Adams

Auto formatting extensionless Bash scripts in Neovim

Here's what I did to make the combination of the Bash language server and shfmt work with Editorconfig settings for Bash script files that don't have extensions.

TL;DR

The Bash language server is easy to install and has support for shfmt which in turn has support for Editorconfig settings.

For shell scripts without extensions, Editorconfig is no good, except if you use shfmt's special support in the form of [[shell]] or [[bash]] section globs.

But that's non-standard, and when invoked via the Bash language server, doesn't work. However, you can get it to work by using a wrapper script that strips off options in the call to shfmt with the result that these special Editorconfig sections are recognised.

Background

In my continuing quest to build out a cleaner and leaner Neovim configuration I turned my attention recently to improving my Bash script editing experience. For Bash there are plenty of facilities that fit into my Personal Development Environment, including shellcheck and shfmt, both of which I have been using for quite a while (see Improving my shell scripting).

I decided to give the Bash language server a try. It's a super easy setup for Neovim 0.11+ and the integration with shfmt was a feature that caught my eye in particular. See Appendix C for the setup configuration details.

Add to this a desire to also more consciously embrace Editorconfig, especially given shfmt's support for that, and I was set for a great experience.

The perfect storm

I am of the opinion that the names of shell scripts that are written to be executed on the command line should not have extensions. This follows a specific guideline in the Google Shell Style Guide. Interpretation of shell script file[type]s is based on the shebang.

This approach, combined with the Bash language server, its support for shfmt, and a custom part of shfmt's support for Editorconfig, turned out to be a small but perfect storm:

Consequently, when editing shell scripts with extensionless filenames using Neovim with the Bash language server configured and shfmt installed1 I got a very odd and confusing experience.

Example of the issue

Here's an example, based on three test files called a, b and fixup, and these (reduced) Editorconfig settings in my .editorconfig file:

root = true

[*]
indent_style = tab

[[bash]]
indent_size = 2
indent_style = space
switch_case_indent = true
binary_next_line = true
space_redirects = true

On the command line, running shfmt2 on each of these three files produced the expected results, in that all were formatted according to the shfmt-specific settings in that [[bash]] section plus of course indented with 2 spaces.

Within Neovim, via the Bash language server3, with shell script files a and b everything seemed to work fine and my Bash scripts were indented appropriately with 2 spaces. When I then started to work on real shell scripts such as the one called fixup, things didn't work the same, and the indentation was done with tabs, not spaces (the horror!).

Yes, you guessed it, this was because in standard Editorconfig procedure, the [bash] part of the [[bash]] section heading is considered a sequence of characters b, a, s and h, matching files of those single-character names; and as fixup was not matched with that sequence, that section (containing indent_style = space) didn't apply, and so indent_style = tab prevailed.

This is unfortunate, but is not really anyone's fault.

Bash language server, shfmt and Editorconfig

I dug in to see what was happening in the Bash language server.

In preparing to call shfmt, properties appropriate for the file in question are determined from the Editorconfig configuration. The editorconfig NPM package is used for this determination, and the custom shfmt extension to allow for [[bash]] and [[shell]] does not exist in that package.

There are generic configuration properties such as indent_style, and shfmt-specific ones such as switch_case_indent. If there are any shfmt-specific ones then the Editorconfig configuration is used. Otherwise, the language server configuration is used. This reflects the approach taken by shfmt itself in using either Editorconfig or its own command line flags, but not both (as we'll see in the next section).

The configuration is then turned into a series of explicit shfmt command line options, in particular (but not exclusively) from the Printer flags group. For example, on surfacing a binary_next_line = true setting in .editorconfig it will generate an equivalent -bn flag to pass in the call to shfmt.

Then the Bash language server invokes shfmt passing these generated options, plus the --filename option to specify the full pathname of the file being formatted.

Example for script called 'fixup'

Here's an example of that for the script called fixup for which the editorconfig package does not match the [[bash]] section. This is what's invoked:

shfmt --filename=/home/dj/test/fixup -i0 -ln=auto -

Example for script called 'a'

Here's another example of what the Bash language server ultimately invokes for the script called a which (by chance) was matched by the [[bash]] section:

shfmt --filename=/home/dj/test/a -i=2 -bn -ci -sr

Between a rock and a hard place

As mentioned in the previous section, the Bash language server's approach takes its cue from the equivalent approach of shfmt itself:

"If any EditorConfig files are found, they will be used to apply formatting options. If any parser or printer flags are given to the tool, no EditorConfig formatting options will be used. A default like -i=0 can be used for this purpose."

In other words, either Editorconfig (only), or no Editorconfig.

But to effect this, Bash language server invokes shfmt with explicit options for both cases (where there's a match, and where there isn't). This can be seen directly in the two example invocations above.

I want to manage my formatting in my .editorconfig file(s), but if I want the Editorconfig settings to be honoured when formatting my shell scripts, I need to give them names that I can match with the Editorconfig section name globs, for example by giving them extensions. I don't want to do that.

I want the Bash language server to invoke shfmt without any parser or printer options so that shfmt itself, with its special [[bash]] processing ability (rather than the Bash language server with the standard editorconfig package) parses the Editorconfig settings. This would effectively bypass any Bash language server logic for shfmt and just let shfmt do its thing, with (or without) an .editorconfig file present.

Forcing the effect

To begin with, I debugged what was going on (see the Useful debugging approaches section) and found that the options for shfmt were being determined and marshalled in server/src/shfmt/index.ts (yay for open source!):

private async runShfmt(
  document: TextDocument,
  formatOptions?: LSP.FormattingOptions | null,
  shfmtConfig?: Record<string, string | boolean> | null,
): Promise<string> {
  const args = await this.getShfmtArguments(document.uri, formatOptions, shfmtConfig)
  ...
  const proc = new Promise((resolve, reject) => {
    const proc = spawn(this.executablePath, [...args, '-'], { cwd: this.cwd })
    ...

Using brute force4 I simply truncated args to [], removing the call to this.getShfmtArguments, and everything worked as planned - shfmt was called without any options at all, with the result that shfmt did the right thing, finding and using the appropriate Editorconfig settings, for all Bash files.

I didn't want to have to make this modification each time I installed the Bash language server, nor do I think it appropriate to suggest as a modification upstream. So I decided to wrap the call to shfmt.

Wrapping shfmt to strip off the options

The Bash language server configuration includes a section specifically for shfmt (see Appendix B). I can use the shfmt.path configuration option to specify an alternate path for the executable, pointing to a simple shell script that removes all the options passed, except for a couple of generic ones.

Here's the script, called shfmt-wrapper, placed in ~/.local/bin/ which is listed first in my PATH environment variable:

#!/usr/bin/env bash

declare -a shfmtargs
for i in "$@"; do
  if [[ $i == --filename=* ]] || [[ $i == "-" ]]; then
    shfmtargs=("${shfmtargs[@]}" "$i")
  fi
done
shfmt "${shfmtargs[@]}"

All it does is take the options passed to it, removes all but the --filename=... and - ones, and then calls the actual shfmt with what's left.

And with the shfmt.path option (see Appendix C) I then told the Bash language server to use this wrapper when invoking shfmt.

Therefore, because no parser or printer flags are ever present in the shfmt option list, shfmt will use the Editorconfig configuration, interpreting it using the custom support for [[bash]] or [[shell]] sections.

The result is that this section of my .editorconfig file is interpreted as I intend:

[[bash]]
indent_size = 2
indent_style = space
switch_case_indent = true
binary_next_line = true
space_redirects = true

and my extensionless Bash script files are formatted appropriately. Not with tabs, and no shfmt-specific configuration, but as I have described in configuration.

Great!

This is of course a bit hacky, but I think it is within the boundaries of "small pieces, loosely joined" and doesn't require any modification to any non-user-serviceable parts.

Useful debugging approaches

In working out what was going on, I used various tools and settings to help me. Here's a quick list of those, mostly to remind my future self.

The editorconfig CLI

The editorconfig package, which I installed globally for these investigations with npm i -g editorconfig, also sports a CLI which can be used to show the "effective configuration", given a filename. Taking the .editorconfig file contents in the example of the issue shown earlier, here's what is output ...

for the shell script file named a:

$ editorconfig a
indent_style=space
indent_size=2
switch_case_indent=true
binary_next_line=true
space_redirects=true
tab_width=2

and for the shell script file named fixup:

$ editorconfig fixup
indent_style=tab
indent_size=tab

Thus I can see first hand what the Bash language server would receive from the editorconfig module, given a filename and some Editorconfig configuration.

Language server logging

Logs are emitted at various points in the Bash language server's operation, as I could see from browsing the server source code.

For example, the shfmt support has many logging lines, including this one:

logger.debug(
  `Shfmt: found .editorconfig properties: ${JSON.stringify(
    editorconfigProperties,
  )}`,
)

In the configuration section I could see that it was probably possible to set my desired logging level with the environment variable BASH_IDE_LOG_LEVEL.

But it turns out that the idea is that settings can be specified in the LSP configuration, and the setting property "path" is what's actually shown in the Mason info, starting with bashIde as the first path section. I didn't find that in the Bash language server README or configuration section, perhaps I should have known that implicitly, or known to look elsewhere for it.

Anyway, once I figured that out, I could set my desired level like this (also shown in Appendix C):

return {
  cmd = { 'bash-language-server', 'start' },
  filetypes = { 'bash', 'sh' },
  settings = {
    bashIde = {
      logLevel = 'debug',
      shfmt = {
        path = vim.fn.expand('$HOME') .. '/dotfiles/scripts/shfmt-wrapper'
      }
    },
  },
}

Where does this log output from the Bash language server show up? Well, now I figured it out, it's obvious - it's all collected in Neovim's general LSP log file at ~/.local/state/nvim/lsp.log.

However, any debug level log output from the Bash language server will only show up there if the general Neovim language server logging level is high enough. Even after adding the logLevel = 'debug' configuration above, nothing showed up ... until I realised I had this in my ~/.config/nvim/init.lua:

-- LSP logging
vim.lsp.set_log_level("WARN")

Setting this to DEBUG, either by changing the value in the init.lua file itself, or simply on the fly with:

:lua vim.lsp.set_log_level("DEBUG")

does the trick, and output like this appears in the lsp.log file:

DEBUG Shfmt: found .editorconfig properties: {"indent_style":"tab","indent_size":"tab"}'
DEBUG Shfmt: no shfmt properties found in .editorconfig - using language server shfmt config"
DEBUG Shfmt: running "/home/dj/.local/bin/shfmt-wrapper --filename=/tmp/formatting/fixup -i=0 -ln=auto"'

Appendix A - Mason info for the Bash language server

Language Filter: press  to apply filter

Installed (1)
  ◍ bash-language-server
    A language server for Bash.

    installed version 5.6.0
    installed purl    pkg:npm/bash-language-server@5.6.0
    homepage          https://github.com/bash-lsp/bash-language-server
    languages         Bash, Csh, Ksh, Sh, Zsh
    categories        LSP
    executables       bash-language-server

    ↓ LSP server configuration schema (press enter to collapse)
      This is a read-only overview of the settings this server accepts.
      Note that some settings might not apply to neovim.

      → bashIde.backgroundAnalysisMaxFiles    default: 500
      → bashIde.enableSourceErrorDiagnostics  default: false
      → bashIde.explainshellEndpoint          default: ""
      → bashIde.globPattern                   default: "**/*@(.sh|.inc|.bash|.command)"
      → bashIde.includeAllWorkspaceSymbols    default: false
      → bashIde.logLevel                      default: "info"
      → bashIde.shellcheckArguments           default: ""
      → bashIde.shellcheckPath                default: "shellcheck"
      → bashIde.shfmt.binaryNextLine          default: false
      → bashIde.shfmt.caseIndent              default: false
      → bashIde.shfmt.funcNextLine            default: false
      → bashIde.shfmt.ignoreEditorconfig      default: false
      → bashIde.shfmt.keepPadding             default: false
      → bashIde.shfmt.languageDialect         default: "auto"
      → bashIde.shfmt.path                    default: "shfmt"
      → bashIde.shfmt.simplifyCode            default: false
      → bashIde.shfmt.spaceRedirects          default: false

Appendix B - Configuration for Bash language server

const rawConfig = {
  backgroundAnalysisMaxFiles: toNumber(process.env.BACKGROUND_ANALYSIS_MAX_FILES),
  enableSourceErrorDiagnostics: toBoolean(process.env.ENABLE_SOURCE_ERROR_DIAGNOSTICS),
  explainshellEndpoint: process.env.EXPLAINSHELL_ENDPOINT,
  globPattern: process.env.GLOB_PATTERN,
  includeAllWorkspaceSymbols: toBoolean(process.env.INCLUDE_ALL_WORKSPACE_SYMBOLS),
  logLevel: process.env[LOG_LEVEL_ENV_VAR],
  shellcheckArguments: process.env.SHELLCHECK_ARGUMENTS,
  shellcheckPath: process.env.SHELLCHECK_PATH,
  shfmt: {
    path: process.env.SHFMT_PATH,
    ignoreEditorconfig: toBoolean(process.env.SHFMT_IGNORE_EDITORCONFIG),
    languageDialect: process.env.SHFMT_LANGUAGE_DIALECT,
    binaryNextLine: toBoolean(process.env.SHFMT_BINARY_NEXT_LINE),
    caseIndent: toBoolean(process.env.SHFMT_CASE_INDENT),
    funcNextLine: toBoolean(process.env.SHFMT_FUNC_NEXT_LINE),
    keepPadding: toBoolean(process.env.SHFMT_KEEP_PADDING),
    simplifyCode: toBoolean(process.env.SHFMT_SIMPLIFY_CODE),
    spaceRedirects: toBoolean(process.env.SHFMT_SPACE_REDIRECTS),
  },
}

Appendix C - Neovim LSP configuration for the Bash language server

To add the Bash language server to my Neovim 0.11+ setup and configure it, all I needed was to create a new configuration file ~/.config/nvim/lsp/bash.lua:

return {
  cmd = { 'bash-language-server', 'start' },
  filetypes = { 'bash', 'sh' },
  settings = {
    bashIde = {
      logLevel = 'debug',
      shfmt = {
        path = vim.fn.expand('$HOME') .. '/dotfiles/scripts/shfmt-wrapper'
      }
    },
  },
}

(The settings will be explained in the course of this post.)

Then it was just a matter of enabling the server in the list in my main ~/config/nvim/init.lua file:

vim.lsp.enable({
  'javascript',
  'cds',
  'markdown',
  'dockerfile',
  'lua',
  'bash', -- here
})

Footnotes

  1. as well as shellcheck which is also supported
  2. version 3.8.0 or higher, as that is when support for Editorconfig sections such as [[shell]] was introduced
  3. configured through a BufWritePre event based call to vim.lsp.buf.format
  4. it was actually in the Mason installed and managed location at ~/.local/share/nvim/mason/packages/bash-language-server/node_modules/bash-language-server/out/shfmt/index.js, i.e. the transpiled JavaScript equivalent file