Combine multiple images using command line tool

Question

I'm trying to recombine multiple images into a single image using a command line tool like ImageMagick.

I have 30000 folders and In each folder there are nearly 50 images. The images are smaller tiles of a larger image that has been broken up into tiles. Each images is prefixed with their xy position

e.g. folder1/01-imagename01-imagename folder1/02-imagename folder1/03-imagename folder1/10-imagename and so on

example here.

00-zzi.....=x0-y0-z2.jpg
01-zzi.....=x0-y1-z2.jpg
02-zzi.....=x0-y2-z2.jpg
03-zzi.....=x0-y3-z2.jpg

each tile image is 512x512 and typically is less than 50kb

I'm trying to figure out if there's any way that the image magick composite capability is the right tool, or any other suggestions.

Thanks!

Debian GNU/Linux 11

identify image output: 
Image:
  Filename: 00-zzi3ZROdz3Lq8hTj7hy2ghoChBAv2D2-9bU_jPT-D4b_jXTraQfK81DEuQ=x0-y0-z2.jpg
  Format: JPEG (Joint Photographic Experts Group JFIF format)
  Mime type: image/jpeg
  Class: DirectClass
  Geometry: 512x512+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: Grayscale
  Base type: Undefined
  Endianness: Undefined
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Pixels: 262144
    Red:
      min: 0  (0)
      max: 19 (0.0745098)
      mean: 3.06709 (0.0120278)
      standard deviation: 0.61571 (0.00241455)
      kurtosis: 28.6716
      skewness: 2.14642
      entropy: 0.251511
    Green:
      min: 0  (0)
      max: 19 (0.0745098)
      mean: 3.06709 (0.0120278)
      standard deviation: 0.61571 (0.00241455)
      kurtosis: 28.6716
      skewness: 2.14642
      entropy: 0.251511
    Blue:
      min: 0  (0)
      max: 19 (0.0745098)
      mean: 3.06709 (0.0120278)
      standard deviation: 0.61571 (0.00241455)
      kurtosis: 28.6716
      skewness: 2.14642
      entropy: 0.251511
  Image statistics:
    Overall:
      min: 0  (0)
      max: 19 (0.0745098)
      mean: 3.06709 (0.0120278)
      standard deviation: 0.61571 (0.00241455)
      kurtosis: 28.6717
      skewness: 2.14643
      entropy: 0.251511
  Colors: 18
  Histogram:
    560: (0,0,0) #000000 black
    2814: (1,1,1) #010101 srgb(1,1,1)
    15055: (2,2,2) #020202 srgb(2,2,2)
    212826: (3,3,3) #030303 grey1
    24467: (4,4,4) #040404 srgb(4,4,4)
    5004: (5,5,5) #050505 grey2
    896: (6,6,6) #060606 srgb(6,6,6)
    237: (7,7,7) #070707 srgb(7,7,7)
    113: (8,8,8) #080808 grey3
    72: (9,9,9) #090909 srgb(9,9,9)
    43: (10,10,10) #0A0A0A grey4
    24: (11,11,11) #0B0B0B srgb(11,11,11)
    14: (12,12,12) #0C0C0C srgb(12,12,12)
    7: (13,13,13) #0D0D0D grey5
    5: (14,14,14) #0E0E0E srgb(14,14,14)
    3: (15,15,15) #0F0F0F grey6
    1: (16,16,16) #101010 srgb(16,16,16)
    3: (19,19,19) #131313 srgb(19,19,19)
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Background color: white
  Border color: srgb(223,223,223)
  Matte color: grey74
  Transparent color: black
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 512x512+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: JPEG
  Quality: 90
  Orientation: Undefined
  Properties:
    date:create: 2022-09-07T00:51:18+00:00
    date:modify: 2022-09-07T00:51:18+00:00
    jpeg:colorspace: 2
    jpeg:sampling-factor: 2x2,1x1,1x1
    signature: ebb4af08227671b45fa62c44887f9b94a8a17d3a7d6c418c26be0e032b766359
  Artifacts:
    filename: 00-zzi3ZROdz3Lq8hTj7hy2ghoChBAv2D2-9bU_jPT-D4b_jXTraQfK81DEuQ=x0-y0-z2.jpg
    verbose: true
  Tainted: False
  Filesize: 3764B
  Number pixels: 262144
  Pixels per second: 67.2793MB
  User time: 0.000u
  Elapsed time: 0:01.003
  Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org

Hi Mark, thank you for your help so far. I have been doing some testing and very nearly there! I had to change the get list of images code to use egrep due to it not finding the files, i have changed to:

row=( $(ls | egrep *-y${y}-z 2> /dev/null) )

The final hurdle, is that when attempting to process a smaller directory of 10 folders as a test of parallel processsing,

find "tiled_images" -type d -print ./processOne {}

It seems to not be printing the folder names after the command and showing:

find: paths must precede expression.

You would have to write a script to collect the images. Then you might be able to use -flatten or montage to build your image. But I see several issues. 1) are the image all the same size. If so, then montage would work. 2) Do you have enough RAM to hold all the input images and the output image? It is not the file size, but the dimensions you have to use to compute the needed file size. What are the image dimensions? — fmw42
– fmw42, Commented Sep 6, 2022 at 19:40
Thanks - i have updated description; each tile image is 512x512 and typically is less than 50kb — mpt38
– mpt38, Commented Sep 6, 2022 at 20:16
It is not the file size you have to consider but the dimensions after decompression control the important file size. So 512x512 is about 250000 pixels times 8 bytes. So one image would be 2 Mbytes. Now you have 30000 x 50 images * 2 Mbytes = 12 Gbytes just for the input. How much RAM do you have? Imagemagick can write to disk if RAM is exhausted. I do not know if Pillow can handle that. — fmw42
– fmw42, Commented Sep 6, 2022 at 23:40
Hi thanks all for the pointers. Just to note, each folder is of a different image. e.g the 50 images in folder1 are all tiles of one image. the 50 images in folder2 are all tiles of a different image, and so on. — mpt38
– mpt38, Commented Sep 7, 2022 at 7:26

Mark Setchell · Accepted Answer · 2022-09-12 14:07:17Z

2

As I see it, there are two aspects to this:

processing 30,000 directories
processing one directory

IMHO, the best way to process 30,000 directories is in parallel, else you'll be there all day. So I would suggest to write the processing as a script that does one directory, passed as a single parameter, and then using a GNU Parallel job that processes all 30,000 directories, keeping all your CPU cores busy till all directories are done.

So, if your directories are under a top-level directory called "tiled_images", and you save the script in the next part of my answer as processOne.sh, you could do this:

find "tiled_images" -type d -print | parallel ./processOne {}

There are many options to GNU Parallel, here are a few of the most useful:

parallel --eta ... will show you the "Estimated Time of Arrival" of job completion
parallel --bar ... will show you a progress bar, and works with zenity
parallel --j 4 ... will run just 4 jobs at a time
parallel --j 50% ... will keep half your CPU cores busy

Now to the processing of a single directory, whose name is passed as parameter:

#!/bin/bash

# Expect one parameter - the directory name
[ $# -ne 2 ] || { >&2 echo "Usage: $0 DIRECTORY"; exit 1; }

d=$1

cd "$d" || { >&2 echo "ERROR: Directory $d does not exist"; exit 1; }

# Assume no more than 100 rows of tiles since fewer than 50 images altogether, and presumably more than 1 image per row
for ((y=0;y<100;y++)) ; do

   # Get list of images in this row
   row=( $(ls *-y${y}-z2.jpg 2> /dev/null) )

   # Break out of loop if no images
   [ -z "$row" ] && break

   # Formulate output filename for this row, being sure that it is zero-padded so the rows collate in correct order
   # Also, write to MPC, or Magick Pixel Cache format, which should be fastest to write and read later
   printf -v out "row-%02d.mpc" $y

   echo "Processing row: ${y}"
   echo "  concatenating: ${row[@]}"
   echo "  into: ${out}"
   magick "${row[@]}" +append "$out"
done

# Concatenate rows into result
magick row-*mpc -append result.jpg

# You should clean up here when it is tested
# rm *.mpc *.cache

You would then save this as processOne.sh and make it executable with:

chmod +x processOne.sh

Then test it on a single directory with:

./processOne SOME_DIRECTORY_CONTAINING_TILES

Note that +append concatenates images side-by-side, whereas -append (different sign) concatenates images above-and-below each other.

Note that you could speed this up by avoiding creation of intermediate files, but that might complicate things and make debugging harder. Just for reference, that would look something very much like this:

...
...

# Assume no more than 100 rows of tiles since fewer than 50 images altogether, and presumable more than 1 image per row
for ((y=0;y<100;y++)) ; do

   # Get list of images in this row
   row=( $(ls *-y${y}-z2.jpg 2> /dev/null) )

   # Break out of loop if no images
   [ -z "$row" ] && break

   # Append images to make a row and pass to outer `magick` command
   magick "${row[@]}" +append miff:-

done | magick miff:- -append result.jpg

edited Sep 12, 2022 at 14:07

answered Sep 12, 2022 at 13:41

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

mpt38 Over a year ago

Wow thank you very much for the comprehensive reply! I'll get testing it asap!

mpt38 Over a year ago

Hi Mark, thank you for your help so far. I have been doing some testing. I had to change the get list of images code to the following due to it not executing, and z seemingly also being variable row=( $(ls | egrep *-y${y}-z 2> /dev/null) ) The final hurdle, is that when attempting to process a smaller directory of 10 folders, "tiled_images" -type d -print ./processOne {} seems to not be printing the folder names after the command and showing: find: paths must precede expression. Thanks!

Mark Setchell Over a year ago

Sorry, I can't tell what the problem is from unformatted code in a comment. Please click edit under your original question and properly format your code and error messages in there.

mpt38 Over a year ago

Sorry Mark. i have updated the bottom of the original question, hope it helps. Thank you - very nearly there!

Collectives™ on Stack Overflow

Combine multiple images using command line tool

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related