10

In this answer to How does "cat << EOF" work in bash? on Stack Overflow, I get the first two points. But I don't get the third point Pass multi-line string to a pipe in Bash

  1. Pass multi-line string to a pipe in Bash

     $ cat <<EOF | grep 'b' | tee b.txt
     foo
     bar
     baz
     EOF
    

Since It have 3 word, 2 pipe character. Then I am not sure how to intrepret it.

2
  • 2
    Drop the leading space in the EOF line. Commented Dec 31, 2021 at 8:58
  • @RudiC I am not sure what does the first pipe ("|") character for? Commented Dec 31, 2021 at 9:38

5 Answers 5

18

From your comment:

I am not sure what does the first pipe ("|") character for?

The first | character connects the output of cat to the input of grep. << redirects the input of cat; it's a totally independent redirection, similar to < in cat <some_file | grep ….

You may prefer <<EOF cat | grep 'b' | tee b.txt (compare this answer) because if you read this from left to right then it will strictly correspond to how the data flows: here documentcatgreptee.

Note all this can be done without cat:

<<EOF grep 'b' | tee b.txt
foo
bar
baz
EOF

(or grep 'b' <<EOF | …).

2
  • However, please note that some programs behave differently if STDIN or STDOUT is a pipe. In such a case, cat < example1 | example2 would work differently than example2 < example1. A real-world example would be ls | cat. Commented Jan 2, 2022 at 11:38
  • @Kamil Maciorowski, Can you help me with this question Commented Jan 5, 2022 at 6:49
7

What you are using with << EOF is a heredoc.

The multi-line string gets written to a temporary file and is then made the stdin of the /bin/cat process. The output of cat, which is the multi-line string, gets piped to grep and its output in turn is piped to tee.

4

As a further example of the heredoc syntax:

cat <<EOF1 | cat /proc/self/fd/3 - 3<<EOF2 | cat /proc/self/fd/3 - 3<<EOF3  
foo
EOF1
bar
EOF2
baz 
EOF3

outputs

baz
bar
foo

One might think at first that after the heredoc redirection one needs to immediately provide the heredoc, but that's not the case. One can continue writing the command, and even provide further heredoc redirections. This means you can have your command in one place without the redirection forcing you to split it to after the possibly long heredocs.

3

You have a pipeline consisting of three simple commands:

  1. A cat command with a here document

    cat <<EOF
      foo
      bar
      baz
    EOF
    
  2. grep 'b'

  3. tee b.txt

The <<EOF is associated with the simple command; the rest of the here document begins on the next physical line. Here's an obfuscated, but equivalent, command:

cat <<EOF | grep 'b' |
foo
bar
baz
EOF
tee b.txt

The first physical line is logically continued, because a pipeline cannot end with a |. However, the rest of the here document takes precedence over the rest of the pipeline; the next command of the pipeline will be parsed after the here-document is terminated.

5
  • 3
    It definitely doesn't help that (as I write this) even the syntax highlighter gets confused about the rest of the line being part of the heredoc. Commented Jan 1, 2022 at 2:18
  • The syntax highlighter is based on regular expressions; shell is decidedly not a regular language. Commented Jan 1, 2022 at 14:21
  • Sure, but that's irrelevant to my point. What I meant is, it might be worth explicitly saying in your answer that the way this code gets highlighted is misleading about what counts as part of the here document. Commented Jan 1, 2022 at 22:13
  • I take it as a given that syntax highlighting should be ignored as a source of truth, since it will vary among editors and formatting libraries. It's fine as a reading aid, but should not be thought of as a syntax checker. Commented Jan 1, 2022 at 23:50
  • 1
    I agree, but most people who read the answer probably will not understand that - and perhaps more importantly, even if you know not to take syntax highlighting as a source of truth, the mere fact that it exists and is incorrect is confusing. Commented Jan 2, 2022 at 0:23
0

cat is redundant in cat | grep chain. It is a sign of newbies copy-pasting other newbies.

grep (or any other tool) could accept the input just fine without cat:

grep PATT <<EOF
...
EOF

Shell grammar allows specification of IO redirection before or after the command:

<<EOF grep OK - <(echo "OK 2")
OK 1
EOF

It prints (searching stdin from HERE-document, specified by -; then searching named pipe from process substitution, specified by <()):

(standard input):OK 1
/dev/fd/63:OK 2

You could have as many HERE-doc as you wish, just forward the data to different file descriptors, with duplicating redirection the latest "wins" (below prints OK3):

<<EOF1 <<EOF2 cat <<EOF3
OK1
EOF1
OK2
EOF2
OK3
EOF3

Example of opening stream 3, writing data to it and reading it later (notice syntax <&N for redirecting descriptor N to descriptor 0, which is stdin):

exec 3<<EOF
> OK
> EOF

cat <&3
OK

There is a possibility to refer to file descriptors of the process via proc file system /proc/self/fd/N (make sure you understand what self is when forks processes!):

<<EOF cat | cat - /proc/self/fd/3 3<<EOF3 | cat - /proc/self/fd/4 4<<EOF4
> OK 1
> EOF
> OK 2
> EOF3
> OK 3
> EOF4
OK 1
OK 2
OK 3
3
  • "It is a sign of newbies copy-pasting other newbies." Hmmm… Could you notice some subtle difference between the output of cat file_a file_b | grep pattern and the one of grep pattern file_a file_b that could invite you to temper the vociferation of your disdain ? Commented Feb 28, 2023 at 15:02
  • cat | CMD is almost always an anti-pattern. grep acts differently when you supply file via CLI vs via stdin (like printing file name as a prefix): the behavior is adjustable via CLI options. Commented Feb 28, 2023 at 15:59
  • BTW I had not asked for whatever demonstration, I had only ventured the suggestion to appear less patronizing in your answer. :-P Commented Feb 28, 2023 at 16:08

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.