19

Is there a way to only execute a command after another is done without a temp file? I have one longer running command and another command that formats the output and sends it to a HTTP server using curl. If i just execute commandA | commandB, commandB will start curl, connect to the server and start sending data. Because commandAtakes so long, the HTTP server will timeout. I can do what I want with commandA > /tmp/file && commandB </tmp/file && rm -f /tmp/file

Out of curiosity I want to know if there is a way to do it without the temp file. I tried mbuffer -m 20M -q -P 100 but the curl process is still started right at the beginning. Mbuffer waits just until commandAis done with actually sending the data. (The data itself is just a few hundred kb at max)

Improve this question
6
  • What about commandA && commandB? Commented Mar 4, 2015 at 17:18
  • 2
    that doesn't transmit the output of commandAto commandB, does it? Commented Mar 4, 2015 at 17:30
  • It starts Command B if Command A completes successfully, which would mean that curl doesn't start early. Commented Mar 4, 2015 at 17:33
  • 2
    @eyoung100 but it doesn't pass stdout from commandA to stdin for commandB, which is what Josef needs! Commented Mar 4, 2015 at 17:34
  • If he wants output passed, he has to use a file. See cuonglm's answer. Commented Mar 4, 2015 at 17:37

5 Answers 5

Reset to default
21

This is similar to a couple of the other answers.  If you have the “moreutils” package, you should have the sponge command.  Try

commandA | sponge | { IFS= read -r x; { printf "%s\n" "$x"; cat; } | commandB; }

The sponge command is basically a pass-through filter (like cat) except that it does not start writing the output until it has read the entire input.  I.e., it “soaks up” the data, and then releases it when you squeeze it (like a sponge).  So, to a certain extent, this is “cheating” – if there’s a non-trivial amount of data, sponge almost certainly uses a temporary file.  But it’s invisible to you; you don’t have to worry about housekeeping things like choosing a unique filename and cleaning up afterwards.

The { IFS= read -r x; { printf "%s\n" "$x"; cat; } | commandB; } reads the first line of output from sponge.  Remember, this doesn’t appear until commandA has finished.  Then it fires up commandB, writes the first line to the pipe, and invokes cat to read the rest of the output and write it to the pipe.

Improve this answer
3
  • thanks! So sponge does that thing I used mbuffer for, but seems to be better suited here. using read is clever here. Definitely will remember this for the future. Commented Mar 5, 2015 at 8:33
  • @Josef: I've never heard of mbuffer before; it might actually be just as good as sponge. I agree that using read is a clever trick. I can't take full credit for it; it pops up in answers throughout Stack Exchange (U&L, Super User, Ask Ubuntu, etc.) from time to time. In fact, roaima's answer to this question is very similar to mine except it doesn't use sponge (or something equivalent), so, as I mentioned in a comment, it doesn't delay starting commandB as much as you need (in my understanding of your problem). Commented Mar 5, 2015 at 18:24
  • github.com/ildar-shaimordanov/perl-utils#sponge has a script version of "sponge" wrapped in a bash function. Commented Jan 10, 2020 at 6:44
7

Commands in pipe line are started concurrently, you need to store commandA output somewhere to use later. You can avoid temp file by using variable:

output=$(command A; echo A)
printf '%s' "${output%%A}" | commandB
Improve this answer
6
  • What is the purpose of the echo A in the process substitution? Commented Mar 4, 2015 at 18:09
  • 3
    @DigitalTrauma: It prevents command substitution from stripping trailing newlines. Commented Mar 4, 2015 at 18:10
  • Ah, I see, yes - good catch of a possible corner case Commented Mar 4, 2015 at 18:13
  • I realized my answer was incorrect, so I deleted it. I think this looks correct instead. +1 Commented Mar 4, 2015 at 18:26
  • 1
    If the assumptions is to solve commandA | commandB with stdout of commandA completely buffered before going to stdin of commandB, then I cannot figure out where A comes from in this example. The command built-in is new to me, so maybe that is all. I do like this concept the best though for my purpose. output="$(for x in $(_get-names); echo -n "$x"; _orderable-output "$(_work-with "$x")"; done;)"; echo -e "$output" | sort -k 2) where anything prefixed with an underscore is a filler for a real process. Commented May 5, 2020 at 14:49
1

I don't know of any standard UNIX utility that can address this issue. One option would be use awk to accumulate commandA output and flush it to commandB at one shot, like so

commandA  | awk '{x = x ORS $0}; END{printf "%s", x | "commandB"}'

Beware that this could be memory intensive since awk is building up a string from its input.

Improve this answer
1
  • 1
    This strips off the last newline. Assuming that the last character is a newline, you need a \n or another ORS at the end. Commented Mar 4, 2015 at 20:32
1

Yet another option is tac;

| tac | tac
Improve this answer
2
  • This will still start commandB immediately; it will only delay sending it the data. Commented Jan 22, 2022 at 23:01
  • wouldn't comething like |tail -9999 do similarly for small files? and whether |tac |tac would handle huge files, exceeding RAM size? Commented Mar 25 at 17:13
0

You can solve the requirement with a little script. This particular variant avoids the temporary file and potential memory hog at the expense of additional processes.

#!/bin/bash
#
IFS= read LINE

if test -n "$LINE"
then
    test -t 2 && echo "Starting $*" >&2
    (
        echo "$LINE"
        cat

    ) | "$@"
else
    exit 0
fi

If you were to call the script waituntil (and make it executable, put it in the PATH, etc), you'd use it like this

commandA... | waituntil commandB...

Example

( sleep 3 ; date ; id ) | waituntil nl
Improve this answer
2
  • 1
    All this does is delay starting commandB until commandA has written its first line of output. That probably isn't good enough. Commented Mar 4, 2015 at 20:35
  • @G-Man but we don't know. I like your sponge. Off to read up on that one now. Commented Mar 4, 2015 at 21:29

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.