2019/8/13

caffe train snapshot and resume

resume training (with --snapshot=XXX.solvestate) 之後,在save snapshot 之後,出現 Error:
I0813 15:10:02.564018 12537 solver.cpp:635] Iteration 6000, Testing net (#0)
F0813 15:10:02.564225 12537 net.cpp:1081] Check failed: target_blobs[j]->shape() == source_blob->shape() Cannot share param 0 weights from layer 'conv1a/bn'; shape mismatch.  Source param shape is 1 32 1 1 (32); target param shape is 32 (32)
*** Check failure stack trace: ***
    @     0x7fe99628b0cd  google::LogMessage::Fail()
    @     0x7fe99628cf33  google::LogMessage::SendToLog()
    @     0x7fe99628ac28  google::LogMessage::Flush()
    @     0x7fe99628d999  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fe996e9b28a  caffe::Net::ShareTrainedLayersWith()
    @     0x7fe99729c883  caffe::Solver::TestDetection()
    @     0x7fe99729f91d  caffe::Solver::TestAll()
    @     0x7fe9972a06ac  caffe::Solver::Step()
    @     0x7fe9972a1fd2  caffe::Solver::Solve()
    @     0x555c3341fe4e  train()
    @     0x555c3341cdb1  main
    @     0x7fe9948b9b97  __libc_start_main
    @     0x555c3341db4a  _start
但是如果不用 resume (--snapshot)的話,training 會 success.

沒有留言:

張貼留言