Run multiple commands in shell script using `find` and `xargs`

Question

I'm attempting to echo the filename then sqlcmd directly after however I simply cannot figure out how to concatenate the statements in a way that works.

I have tried:

Creating a new function (apparently can't be done like this in shell only bash)
Separating with a semi colon (both escaped and without)
Using the double && as per this example

Any tips please.

find "./src/ALs/Database/SQL" -iname "*.sql" | sort -n | xargs -0 -I % sh -c 'echo % && sqlcmd -S $SQL_HOST -d Database -U $SQL_USER -P $SQL_PWD -i %'

Do not ever use xargs -I % sh -c '...%...'; that way lies serious security bugs. — Charles Duffy
– Charles Duffy, Commented Sep 4, 2020 at 14:54
Consider what happens if you have a file with $(rm -rf ~) in its name; worse, $(rm -rf ~)'$(rm -rf ~)', so it gets expanded whether or not it's in a single-quoted context. — Charles Duffy
– Charles Duffy, Commented Sep 4, 2020 at 14:54

Tom Fenech · Accepted Answer · 2020-09-04 15:42:42Z

2

I think that if you remove -0 from xargs it'll do what you expect. That flag is for null-byte-separated input, whereas you are passing it newline-separated input.

To work with null-byte-separated records throughout the pipeline use find ... -print0, sort -z and xargs -0. This is the most robust way to pass records through a pipeline (it won't break, no matter what your filenames are).

find "./src/AdviserLinks/Database/SQL" -iname "*.sql" -print0 | 
  sort -zn | 
  xargs -0 -n1 sh -c 'echo "$0" && 
    sqlcmd -S "$SQL_HOST" -d WebSupportDatabase -U "$SQL_USER" -P "$SQL_PWD" -i "$0"'

This assumes that the $SQL variables are export-ed to the environment.

I have replaced -I % with -n1, which will process records one at a time. Each filename is passed to sh as $0, which can be used safely; there is no risk that the contents of the record is interpreted as shell syntax, as was the case with -I % in your attempt. Note that this means that a separate child shell is invoked for every file, and it would be more efficient to use a loop as in Charles' answer.

As for using separate statements vs &&, that depends on whether you want the execution of the second command to be conditional on the success of first command.

edited Sep 4, 2020 at 15:42

answered Sep 4, 2020 at 14:19

Tom Fenech

75.1k13 gold badges119 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

tripleee Over a year ago

Definitely the better solution is to keep the -0 and instead add -print0 to find; the bug is you need to have neither, or both, but both is obviously more correct. A much, much better solution still is find -exec.

m.edmondson Over a year ago

Thanks very much, unfortunately I'm getting the following error: ./src/ALs/Database/SQL/10 Schemas/10 [AL].sql Sqlcmd: './src/AL/Database/SQL/10': Invalid filename. There is a space in the filename after the 10

Tom Fenech Over a year ago

@tripleee the reason I didn't propose find -exec was the sort in between, I wasn't aware that could be done as part of find.

tripleee Over a year ago

Yeah, the sort complicates matters if it's strictly necessary.

Charles Duffy Over a year ago

Using -I to substitute a value into the -c argument of sh is inherently unsafe, because it means the values you're substituting in go through the shell's parser. Consider $(rm -rf ~)'$(rm -rf ~)' as an example of a substring that, if put in place of a % inside a sh -c argument, will always execute no matter what the quoting context is (single-quoted, double-quoted, or unquoted).

|

Charles Duffy · Accepted Answer · 2020-09-04 15:31:08Z

Aiming to combine both safety and performance (invoking sh only once per list of sql files that fit on a command line):

find "./src/AdviserLinks/Database/SQL" -iname "*.sql" -print0 | 
  sort -zn | 
  xargs -0 sh -c '
    for arg do
      echo "$arg"
      sqlcmd -S "$SQL_HOST" -d WebSupportDatabase -U "$SQL_USER" -P "$SQL_PWD" -i "$arg"
    done
  ' _

Note:

We aren't using the -I argument to xargs at all. Instead of using a sigil, we let xargs concatenate as many items as possible to the end of the argument list for sh.
Within the sh command, for arg do loops over "$@" by default; thus, it assigns $1, $2, etc. in turn to the variable named arg, so that just one copy of sh can process several SQL files.
We're letting all the expansions of values like SQL_HOST, SQL_USER and SQL_PWD be performed by the child shell, instead of attempting to do them in the parent (note that this does require that these values be exported to the environment, rather than merely set as process-local shell variables). This change means that a SQL password that might have characters meaningful to the shell doesn't risk being parsed as syntax.

Collectives™ on Stack Overflow

Run multiple commands in shell script using `find` and `xargs`

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related