syntax/sh: Improve heredoc detection

[JoeKar]

• allow spaces between optional '-' and delimiter
• allow additional characters as delimiter

Fixes #3927

[JoeKar]

@Andriamanitra:
I kindly request your review too. :)

[Andriamanitra]

Do people actually use punctuation for defining heredocs? I think it would be beneficial to keep the set of characters more realistic to reduce likelihood of false positives when looking for the end token.

[JoeKar]

I don't really know if they are used, but they are at least allowed. I tried them with the bash.
In case we ignore a valid character there will most probably be a ticket complaining about it. 🤷‍♂️

Edit:
By taking a closer look at [\\s] the [] can be removed.

[Andriamanitra]

I've never seen punctuation used for heredocs but I guess some other editors also allow it (although they don't need to worry about false positives for the end delimiter):

• VS Code: allows [^\"' \\t]+
• Sublime Text: allows [[:alpha:]_][[:alnum:]_.-]*
• neovim: allows [^ \t|>]+

[Andriamanitra]

We should probably handle quotes. GNU Bash reference manual says:

If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded.

I've gathered some examples of heredocs below. This PR has some issues when heredoc is in the middle of a command, and when there are \t characters before the delimiter:

# common usage
cat << EOF
hello ' " world
EOF
cat << 'EOF'
	hello ' " world
EOF
cat <<-"EOF"
	hello ' " world
EOF


# from https://github.com/openwrt/openwrt
	cat <<-EOT | "${CC:-false}" $CFLAGS -o /dev/null -x c - 2>/dev/null
		#include <stdio.h>

		int main(int argc, char **argv) {
			printf("Hello, world!\n");
			return 0;
		}
	EOT

# from https://github.com/goToMain/libosdp/blob/e586d3f700215d16b0e79513880d1c1ade815ba0/configure.sh#L9-L32
cat >&2<<----
	LibOSDP build configure script

	OPTIONS:
	  --packet-trace               Enable raw packet trace for diagnostics
	  --data-trace                 Enable command/reply data buffer tracing
	  --skip-mark                  Don't send the leading mark byte (0xFF)
	  --crypto LIB                 Use methods from LIB (openssl/mbedtls/*tinyaes)
	---


# from https://github.com/mingw-w64/mingw-w64/blob/c2167bc6da600f7fdbd131734767a67ffb9e970e/mingw-w64-headers/configure#L1281
cat <<_ACEOF
'configure' configures mingw-w64-headers 4.0b to adapt to many kinds of systems.

Usage: $0 [OPTION]... [VAR=VALUE]...


Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print 'checking ...' messages
_ACEOF


# from https://github.com/mingw-w64/mingw-w64/blob/c2167bc6da600f7fdbd131734767a67ffb9e970e/mingw-w64-tools/genpeimg/build-aux/compile#L240
    cat <<\EOF
Usage: compile [--help] [--version] PROGRAM [ARGS]

Wrapper for compilers which do not understand '-c -o'.
Remove '-o dest.o' from ARGS, run PROGRAM with the remaining
arguments, and rename the output as expected.
EOF

[JoeKar]

You're right.
But especially the \t(or \s) in front of the closing delimiter is a problem, because this indeed will lead to a false positive delimiter. I still have no good idea how this should be solved. 🤔