16

When I move a single file with spaces in the filename it works like this:

$ mv "file with spaces.txt" "new_place/file with spaces.txt"

Now I have a list of files which may contain spaces and I want to move them. For example:

$ echo "file with spaces.txt" > file_list.txt
$ for file in $(cat file_list.txt); do mv "$file" "new_place/$file"; done;

mv: cannot stat 'file': No such file or directory
mv: cannot stat 'with': No such file or directory
mv: cannot stat 'spaces.txt': No such file or directory

Why does the first example work, but the second one doese not? How can I make it work?

Improve this question

3 Answers 3

Reset to default
25

Never, ever use for foo in $(cat bar). This is a classic mistake, commonly known as bash pitfall number 1. You should instead use:

while IFS= read -r file; do mv -- "$file" "new_place/$file"; done < file_list.txt

When you run the for loop, bash will apply wordsplitting to what it reads, meaning that a strange blue cloud will be read as a, strange, blue and cloud:

$ cat files 
a strange blue cloud.txt
$ for file in $(cat files); do echo "$file"; done
a
strange
blue
cloud.txt

Compare to:

$ while IFS= read -r file; do echo "$file"; done < files 
a strange blue cloud.txt

Or even, if you insist on the UUoC:

$ cat files | while IFS= read -r file; do echo "$file"; done
a strange blue cloud.txt

So, the while loop will read over its input and use the read to assign each line to a variable. The IFS= sets the input field separator to NULL*, and the -r option of read stops it from interpreting backslash escapes (so that \t is treated as slash + t and not as a tab). The -- after the mv means "treat everything after the -- as an argument and not an option", which lets you deal with file names starting with - correctly.

* This isn't necessary here, strictly speaking, the only benefit in this scenario is that keeps read from removing any leading or trailing whitespace, but it is a good habit to get into for when you need to deal with filenames containing newline characters, or in general, when you need to be able to deal with arbitrary file names.

Improve this answer
7
  • 1
    ok, I was only looking for ways to escape the "$file" part. did not expect the error somewhere else. Commented Sep 15, 2017 at 9:30
  • @MrJingles87 I know. This one gets everyone at some point. It is very unintuitive if you don't know about it. Commented Sep 15, 2017 at 9:35
  • 1
    IFS= won't help with newlines in file names. The format of the file here simply doesn't allow filenames with newlines. IFS= is needed for file names beginning ending in space or tab (or whatever characters other than newline $IFS contained beforehand). Commented Sep 15, 2017 at 12:34
  • 1
    That batch pitfall is more about trying to parse the output of ls or find with command substitutions when there are better, more reliable ways to do it. $(cat file) would be alright when done right. It's about as difficult to "do it right" with a while read loop as it is with that a for + $(...) one. See my answer. Commented Sep 15, 2017 at 12:37
  • 1
    But your while loop still has pitfalls. See in my answer how it has to be written to avoid them, which makes it not more legible than the for approach. Commented Sep 15, 2017 at 12:47
13

That unquoted $(cat file_list.txt) in POSIX shells like bash in list context is the split+glob operator (zsh only does the split part as you'd expect).

It splits on characters of $IFS (by default, SPC, TAB and NL) and does glob unless you turn off globbing altogether.

Here, you want to split on newline only and don't want the glob part, so it should be:

IFS='
' # split on newline only
set -o noglob # disable globbing

for file in $(cat file_list.txt); do # split+glob
  mv -- "$file" "new_place/$file"
done

That also has the advantage (over a while read loop) to discard empty lines, preserve a trailing unterminated line, and preserve mv's stdin (needed in case of prompts for instance).

It does have the disadvantage though that the full content of the file has to be stored in memory (several times with shells like bash and zsh).

With some shells (ksh, zsh and to a lesser extent bash), you can optimise it with $(<file_list.txt) instead of $(cat file_list.txt).

To do the equivalent with a while read loop, you'd need:

while IFS= read <&3 -r file || [ -n "$file" ]; do
  {
    [ -n "$file" ] || mv -- "$file" "new_place/$file"
  } 3<&-
done 3< file_list.txt

Or with bash:

readarray -t files < file_list.txt &&
for file in "${files[@]}"
  [ -n "$file" ] || mv -- "$file" "new_place/$file"
done

Or with zsh:

for file in ${(f)"$(<file_list.txt)"}
  mv -- "$file" "new_place/$file"
done

Or with GNU mv and zsh:

mv -t -- new_place ${(f)"$(<file_list.txt)"}

Or with GNU mv and GNU xargs and ksh/zsh/bash:

xargs -rd '\n' -a <(grep . file_list.txt) mv -t -- new_place

More reading about what it means to leave expansions unquoted at Security implications of forgetting to quote a variable in bash/POSIX shells

Improve this answer
2
  • I think you should acknowledge, though, that $(cat file_list.txt) reads the entire file into memory before iterating, whereas while read ... only reads one line at a time. This could be an issue for large files. Commented Sep 16, 2017 at 12:30
  • @chepner. Good point. Added. Commented Sep 17, 2017 at 5:44
1

Instead of writing a script, you can use find..

 find -type f -iname \*.txt -print0 | xargs -IF -0 mv F /folder/to/destination/

for the case when the files are located in file then you can do the following::

cat file_with_spaces.txt | xargs -IF -0 mv F /folder/to/destination

the second not sure though..

Goodluck

Improve this answer
5
  • 3
    If you're using find, you may as well use find for everything. Why bring xargs into it? find -type f -iname '*.txt' -exec mv {} /folder/to/destination/. Commented Sep 15, 2017 at 9:23
  • xargs simply manipulates the results of find here: -I assigns every results to a variable F while 0 and print0 simply guarantees files with spaces are not escaped. and -0 prevents xargs from escaping the results. not sure how -exec behaves with spaces.. Commented Sep 15, 2017 at 9:31
  • 3
    -exec behaves perfectly with arbitrary filenames (including those that contain spaces, newlines or any other weirdness). I know how xargs works, thanks :) I just don't see the point of calling a separate command when find can already do it for you. Nothing wrong with it, just inefficient. Commented Sep 15, 2017 at 9:32
  • @terdonyou are right.. -exec {} /path/destination works smoothly.. i prefer xargs because of more extra things i am able to do with it. Commented Sep 15, 2017 at 9:45
  • 1
    xargs also has the drawback of clobbering stdin (which mv needs for its prompts). Also a problem with @terdon's solution. With GNU xargs and shells with process substitution, can be alleviated with xargs -0IF -a <(find...) mv F /dest Commented Sep 15, 2017 at 12:41

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.