Caffe: Empty file exception while training

Question

While training the dataset. I am getting the following error:

I0614 19:07:11.271327 30865 layer_factory.hpp:77] Creating layer data
I0614 19:07:11.271596 30865 net.cpp:84] Creating Layer data
I0614 19:07:11.271848 30865 net.cpp:380] data -> data
I0614 19:07:11.271896 30865 net.cpp:380] data -> label
I0614 19:07:11.271941 30865 data_transformer.cpp:25] Loading mean file from: train_mean
I0614 19:07:11.275465 30865 image_data_layer.cpp:38] Opening file 
F0614 19:07:11.275923 30865 image_data_layer.cpp:49] Check failed: !lines_.empty() File is empty
*** Check failure stack trace: ***
    @     0x7fba518d25cd  google::LogMessage::Fail()
    @     0x7fba518d4433  google::LogMessage::SendToLog()
    @     0x7fba518d215b  google::LogMessage::Flush()
    @     0x7fba518d4e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fba51ce9509  caffe::ImageDataLayer<>::DataLayerSetUp()
    @     0x7fba51d1f62e  caffe::BasePrefetchingDataLayer<>::LayerSetUp()
    @     0x7fba51de7897  caffe::Net<>::Init()
    @     0x7fba51de9fde  caffe::Net<>::Net()
    @     0x7fba51df24e5  caffe::Solver<>::InitTrainNet()
    @     0x7fba51df3925  caffe::Solver<>::Init()
    @     0x7fba51df3c4f  caffe::Solver<>::Solver()
    @     0x7fba51dc8bb1  caffe::Creator_SGDSolver<>()
    @           0x40a4b8  train()
    @           0x406fa0  main
    @     0x7fba50843830  __libc_start_main
    @           0x4077c9  _start
    @              (nil)  (unknown)
Aborted (core dumped)

I used the template from the github repo of Caffe after installing it.

I have created a subdirectory under the Caffe Root directory named playground.

I am attaching the complete folder for reproducibility. GitHub Link

The commands that I successfully executed:

../build/tools/convert_imageset -resize_height 256 -resize_width 256 train_raw_img/ train_files.txt train_lmdb
../build/tools/convert_imageset -resize_height 256 -resize_width 256 test_raw_img/ test_files.txt test_lmdb
 ../build/tools/compute_image_mean train_lmdb train_mean
 ../build/tools/compute_image_mean train_lmdb test_mean

However, when I proceed to train the network I receive the above error:

../build/tools/caffe train --solver=my_solver_val.prototxt

Complete log of error:

I0614 19:32:54.634418 31048 caffe.cpp:211] Use CPU.
I0614 19:32:54.635144 31048 solver.cpp:44] Initializing solver from parameters: 
test_iter: 1000
test_interval: 1000
base_lr: 0.01
display: 20
max_iter: 50000
lr_policy: "step"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 10000
snapshot: 10000
snapshot_prefix: "models/mymodel/caffenet_train"
solver_mode: CPU
net: "my_train_val.prototxt"
train_state {
  level: 0
  stage: ""
}
I0614 19:32:54.639066 31048 solver.cpp:87] Creating training net from net file: my_train_val.prototxt
I0614 19:32:54.640214 31048 net.cpp:294] The NetState phase (0) differed from the phase (1) specified by a rule in layer data
I0614 19:32:54.640645 31048 net.cpp:294] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
I0614 19:32:54.641345 31048 net.cpp:51] Initializing net from parameters: 
name: "CaffeNet"
state {
  phase: TRAIN
  level: 0
  stage: ""
}
layer {
  name: "data"
  type: "ImageData"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    mirror: true
    crop_size: 256
    mean_file: "train_mean"
  }
  data_param {
    source: "train_files.txt"
    batch_size: 2
    backend: LMDB
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm1"
  type: "LRN"
  bottom: "pool1"
  top: "norm1"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "norm1"
  top: "conv2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm2"
  type: "LRN"
  bottom: "pool2"
  top: "norm2"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "norm2"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu4"
  type: "ReLU"
  bottom: "conv4"
  top: "conv4"
}
layer {
  name: "conv5"
  type: "Convolution"
  bottom: "conv4"
  top: "conv5"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu5"
  type: "ReLU"
  bottom: "conv5"
  top: "conv5"
}
layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}
I0614 19:32:54.644022 31048 layer_factory.hpp:77] Creating layer data
I0614 19:32:54.644239 31048 net.cpp:84] Creating Layer data
I0614 19:32:54.644256 31048 net.cpp:380] data -> data
I0614 19:32:54.644280 31048 net.cpp:380] data -> label
I0614 19:32:54.644448 31048 data_transformer.cpp:25] Loading mean file from: train_mean
I0614 19:32:54.646653 31048 image_data_layer.cpp:38] Opening file 
F0614 19:32:54.646975 31048 image_data_layer.cpp:49] Check failed: !lines_.empty() File is empty
*** Check failure stack trace: ***
    @     0x7f83c21c95cd  google::LogMessage::Fail()
    @     0x7f83c21cb433  google::LogMessage::SendToLog()
    @     0x7f83c21c915b  google::LogMessage::Flush()
    @     0x7f83c21cbe1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f83c25e0509  caffe::ImageDataLayer<>::DataLayerSetUp()
    @     0x7f83c261662e  caffe::BasePrefetchingDataLayer<>::LayerSetUp()
    @     0x7f83c26de897  caffe::Net<>::Init()
    @     0x7f83c26e0fde  caffe::Net<>::Net()
    @     0x7f83c26e94e5  caffe::Solver<>::InitTrainNet()
    @     0x7f83c26ea925  caffe::Solver<>::Init()
    @     0x7f83c26eac4f  caffe::Solver<>::Solver()
    @     0x7f83c26bfbb1  caffe::Creator_SGDSolver<>()
    @           0x40a4b8  train()
    @           0x406fa0  main
    @     0x7f83c113a830  __libc_start_main
    @           0x4077c9  _start
    @              (nil)  (unknown)
Aborted (core dumped)

The error you list is merely the crash portion of the problem. Scroll up in the log file to find the more detailed message. There should be one that gives you that fully-qualified name of the empty file. — Prune
– Prune, Commented Jun 14, 2017 at 23:55

Shai · Accepted Answer · 2017-06-15 05:49:41Z

3

You are using "ImageData" input layer. The layer takes a text file (in your case source: "train_files.txt") and expects each line of the file to contain a path to an image file and the classification label for that image.
It seems like this file ('train_files.txt') is empty in your case.
1. Verify that 'train_files.txt' lists image file names.
2. Verify that the listed image files do exist on your machine and you have reading permission for these files.

BTW,
If you already went through all the trouble of creating train_lmdb why not use an input "Data" layer that directly reads the lmdb?

answered Jun 15, 2017 at 5:49

Shai

115k39 gold badges259 silver badges398 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Caffe: Empty file exception while training

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related