caffe – Check failed: proto.SerializeToOstream(&output)

You suddenly get this error when training/saving a model in Caffe or saving a model in pycaffe.

io.cpp:69] Check failed: proto.SerializeToOstream(&output)
*** Check failure stack trace: ***

Here are two possible reasons for this error

  1. The directory the snapshot is trying to write the .caffemodel into does not exist
  2. You are out of disk space

So check to make sure you have enough disk space. If you are sure you do, follow these instructions to make sure the directory where you are trying to save your .caffemodel exists.

If you use a solver.prototxt, make sure that the snapshot_prefix points to a directory that exists.

e.g., in your solver.prototxt, look at this line:
snapshot_prefix: "/local-scratch/jer/models/my_caffe_model

and make sure that the folder /local-scratch/jer/models/ already exists.

If you are using pycaffe and you get the Check failed: proto.SerializeToOstream(&output) error, make sure that the path you specify to save the model exists.

e.g., in Python using pycaffe:

net = caffe.Net(MODEL_FILE, PRETRAINED, caffe.TEST)
save_model_name = '/local-scratch/jer/models/my_caffe_model.caffemodel'
net.save(save_model_name)

make sure that you already created a directory at /local-scratch/jer/models/

[my 20 minute duh moment]
And just in case it helps anyone, if you think the directory exists, but you are still getting the same error, double check that you created the directory on the machine that caffe is running on… perhaps you are working remotely.
[end duh moment]

References:
https://github.com/BVLC/caffe/issues/1394