caffe train snapshot and resume

resume training (with --snapshot=XXX.solvestate) 之後，在save snapshot 之後，出現 Error:

I0813 15:10:02.564018 12537 solver.cpp:635] Iteration 6000, Testing net (#0)
F0813 15:10:02.564225 12537 net.cpp:1081] Check failed: target_blobs[j]->shape() == source_blob->shape() Cannot share param 0 weights from layer 'conv1a/bn'; shape mismatch.  Source param shape is 1 32 1 1 (32); target param shape is 32 (32)
*** Check failure stack trace: ***
    @     0x7fe99628b0cd  google::LogMessage::Fail()
    @     0x7fe99628cf33  google::LogMessage::SendToLog()
    @     0x7fe99628ac28  google::LogMessage::Flush()
    @     0x7fe99628d999  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fe996e9b28a  caffe::Net::ShareTrainedLayersWith()
    @     0x7fe99729c883  caffe::Solver::TestDetection()
    @     0x7fe99729f91d  caffe::Solver::TestAll()
    @     0x7fe9972a06ac  caffe::Solver::Step()
    @     0x7fe9972a1fd2  caffe::Solver::Solve()
    @     0x555c3341fe4e  train()
    @     0x555c3341cdb1  main
    @     0x7fe9948b9b97  __libc_start_main
    @     0x555c3341db4a  _start

但是如果不用 resume (--snapshot)的話，training 會 success.

caffe-jacindo-models train_image_object_detection.sh

續上篇，把 training data 準備好後， run 這個 script 就會做 train 了。
這個 script 大概就是..

呼叫 image_object_detection.py ，準備好參數，產生五個 run.sh:

./initial/run.sh
./l1reg/run.sh
./sparse/run.sh
./test/run.sh
./test_quantize/run.sh

然後依序 run..

#run
list_dirs=`command ls -d1 "$folder_name"/*/ | command cut -f5 -d/`
for f in $list_dirs; do "$folder_name"/$f/run.sh; done

所以會依照 folder name 順序 (alphabet) 執行 run.sh

每一個 run.sh 就是 call caffe.bin，並且上一個 stage 的 run.sh 產生的 *.caffemodel 當作 initial weights,
依照自己的 solver.txt 做 taining，產生 *.caffemodel

根據 solver.prototxt , 每 2000 iter snapshot 一次。(caffemodel)

所以在 run train_image_object_detection.sh 中圖中斷的話。
可以看看training/,,,,/ 下，以上各 folder 的內容，知道最後 run 那一個 script。

---

每 200 iter snapshot 一次，所以中斷後，可以用 --snapshot 參數從最新的 snapshoot 繼續 training.
參考：Training and Resuming

但是就要修改 run.sh..
舉例來說，initial stage, iter 16000 開始的話...

--- run.sh 2019-08-12 13:49:23.741507855 +0800
+++ resume.sh 2019-08-13 09:44:19.540395008 +0800
@@ -1,4 +1,4 @@
 /home/checko/caffe-jacinto/build/tools/caffe.bin train \
 --solver="training/voc0712/JDetNet/20190812_13-49_ds_PSP_dsFac_32_hdDS8_1/initial/solver.prototxt" \
---weights="training/imagenet_jacintonet11v2_iter_320000.caffemodel" \
+--snapshot="training/voc0712/JDetNet/20190812_13-49_ds_PSP_dsFac_32_hdDS8_1/initial/voc0712_ssdJacintoNetV2_iter_16000.solverstate" \
 --gpu "0" 2>&1 | tee training/voc0712/JDetNet/20190812_13-49_ds_PSP_dsFac_32_hdDS8_1/initial/run.log

就是原來的 weights 改成 snapshot

---

train 完後，要測試 (infernal)， run 或 import ssd_detect_video.py，會出現找不到 propagate_obj.py 的錯誤。
用 git log 看，在某一版被刪除了。
用

git checkout 6ca88ff12e559a839ae5ee9bc7c25201f0ed9217 scripts/propagate_obj.py

就可以找回來了。

caffe-jacindo-models

先 follow 這一篇 README.md
clone 下來，跟 caffe-jacindo 在同一層：

 
 ./
 |
  -- caffe-jacindo
 |
 |
  -- caffe-jacindo-models

clone 後，checkout 跟 caffe-jacindo 一樣的版本(branch): caffe-0.17

設定環境變數：

export PYTHONPATH=~/caffe-jacinto/python
export CAFFE_ROOT=~/caffe-jacinto

這樣就裝完了。
接下來依照需要做的 example 來 run script..

---

ref:

• Sparse, Quantized CNN training for segmentation
• EBC Exercise 39 setting up tidl on X15

SSD

scripts/train_image_object_detection.sh 就是用 caffe-ssd 的training data 做 SSD training 的 script。
可以知道，他用 caffe-ssd 的 training data:

if [ $dataset = "voc0712" ]
then
  train_data="../../caffe-jacinto/examples/VOC0712/VOC0712_trainval_lmdb"
  test_data="../../caffe-jacinto/examples/VOC0712/VOC0712_test_lmdb"

  name_size_file="../../caffe-jacinto/data/VOC0712/test_name_size.txt"
  label_map_file="../../caffe-jacinto/data/VOC0712/labelmap_voc.prototxt"

所以要先 follow caffe-ssd 的步驟，
download training data 到 ~/data
在 caffe-jacinto 中run create_data.sh, create_list.sh 把 VOC0712 的 data set 準備好。

chmod a+x data/VOC0712/*.sh
./data/VOC0712/create_list.sh
./data/VOC0712/create_data.sh

然後到 caffe-jacinto-models 下 run train_image_object_detection.sh. 根據 train_image_object_detection.sh 的內容，要在 scripts 目錄下執行。
結果：

2019-07-26 11:39:44 (10.2 MB/s) - ‘training/imagenet_jacintonet11v2_iter_320000.caffemodel’ saved [11516054/11516054]

Logging output to training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/train-log_20190726_11-39.txt
training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/initial
Traceback (most recent call last):
  File "./models/image_object_detection.py", line 5, in 
    from models.model_libs import *
ImportError: No module named models.model_libs
training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/l1reg
Traceback (most recent call last):
  File "./models/image_object_detection.py", line 5, in 
    from models.model_libs import *
ImportError: No module named models.model_libs
training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/sparse
Traceback (most recent call last):
  File "./models/image_object_detection.py", line 5, in 
    from models.model_libs import *
ImportError: No module named models.model_libs
training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/test
Traceback (most recent call last):
  File "./models/image_object_detection.py", line 5, in 
    from models.model_libs import *
ImportError: No module named models.model_libs
training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/test_quantize
Traceback (most recent call last):
  File "./models/image_object_detection.py", line 5, in 
    from models.model_libs import *
ImportError: No module named models.model_libs
cat: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/test_quantize/deploy.prototxt: No such file or directory
cat: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/test_quantize/test.prototxt: No such file or directory
./train_image_object_detection.sh: line 382: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/initial/run.sh: No such file or directory
./train_image_object_detection.sh: line 382: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/l1reg/run.sh: No such file or directory
./train_image_object_detection.sh: line 382: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/sparse/run.sh: No such file or directory
./train_image_object_detection.sh: line 382: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/test/run.sh: No such file or directory
./train_image_object_detection.sh: line 382: training/voc0712/JDetNet/20190726_11-39_ds_PSP_dsFac_32_hdDS8_1/test_quantize/run.sh: No such file or directory

好像是 module path 的問題。
image_object_detection.py 在 import 同目錄的 py module時，都加上了models。
把models 刪掉就沒有這個 Error 了。

然後是 Check failed: error == cudaSuccess (10 vs. 0) invalid device ordinal
一樣，把 scripts 下所有*.sh, *.py 中的 gpus= 找出來，都改成 gpus="0" 就可以了。
例如：

diff --git a/scripts/train_image_object_detection.sh b/scripts/train_image_object_detection.sh
index 0e5f99d..f9771f4 100755
--- a/scripts/train_image_object_detection.sh
+++ b/scripts/train_image_object_detection.sh
@@ -5,7 +5,7 @@ DATE_TIME=`date +'%Y%m%d_%H-%M'`
 #-------------------------------------------------------
 
 #------------------------------------------------
-gpus="0,1" #"0,1,2"
+gpus="0" #"0,1,2"


diff --git a/scripts/models/image_object_detection.py b/scripts/models/image_object_detection.py
index 67823a1..f625615 100644
--- a/scripts/models/image_object_detection.py
+++ b/scripts/models/image_object_detection.py
@@ -360,7 +360,7 @@ def main():
     # Which layers to freeze (no backward) during training.
     config_param.freeze_layers = []
     # Defining which GPUs to use.
-    config_param.gpus = "0,1" #gpus = "0"  
+    config_param.gpus = "0" #gpus = "0"  
 
     config_param.batch_size = 32
     config_param.accum_batch_size = 32

我把修改過的放到 https://github.com/checko/caffe-jacinto-models/tree/zoey-wpc
還有要注意 GPU 的 memory。
train_image_object_detection.sh 中的 batch_size 是 16，給 8G 的 GPU card 剛好
如果 GPU memory 只有 4G，要改成 8

caffe-jacinto

make 就出現..

  ^
AR -o .build_release/lib/libcaffe-nv.a
LD -o .build_release/lib/libcaffe-nv.so.0.17.0
/usr/bin/ld: cannot find -lnvidia-ml
collect2: error: ld returned 1 exit status
Makefile:600: recipe for target '.build_release/lib/libcaffe-nv.so.0.17.0' failed
make: *** [.build_release/lib/libcaffe-nv.so.0.17.0] Error 1

ref:ubuntu NVIDIA-SMI couldn't find libnvidia-ml.so library in your system.
< 找一下libnvidia-ml.so 在哪裡...

$ locate libnvidia-ml.so
/usr/lib/nvidia-410/libnvidia-ml.so
/usr/lib/nvidia-410/libnvidia-ml.so.1
/usr/lib/nvidia-410/libnvidia-ml.so.410.104
/usr/lib32/nvidia-410/libnvidia-ml.so
/usr/lib32/nvidia-410/libnvidia-ml.so.1
/usr/lib32/nvidia-410/libnvidia-ml.so.410.104
/usr/local/cuda-10.0/targets/x86_64-linux/lib/stubs/libnvidia-ml.so
/usr/local/cuda-10.1/targets/x86_64-linux/lib/stubs/libnvidia-ml.so

然後依照這一篇檢查 ld.so.conf.d

cuda-10.0.config 的設定是正常的:

$ cat /etc/ld.so.conf.d/cuda-10-0.conf 
/usr/local/cuda-10.0/targets/x86_64-linux/lib
/usr/lib/nvidia-410

有兩個 libnvidia-ml.so 的原因，這一篇 說得很清楚。

結果一樣，還是 fail。用 make -n看一下這個 ld 的完整option..

$ make -n .build_release/lib/libcaffe-nv.so.0.17.0
echo LD -o .build_release/lib/libcaffe-nv.so.0.17.0
g++ -shared -o .build_release/lib/libcaffe-nv.so.0.17.0 .build_release/src/caffe/proto/caffe.pb.o .build_release/src/caffe/net.o
.build_release/src/caffe/layer_factory.o .build_release/src/caffe/quantized_layer.o .build_release/src/caffe/tensor.o 
.build_release/src/caffe/layers/sigmoid_cross_entropy_loss_layer.o .build_release/src/caffe/layers/reshape_layer.o 
.build_release/src/caffe/layers/cudnn_dropout_layer.o .build_release/src/caffe/layers/hdf5_output_layer.o 
.build_release/src/caffe/layers/reduction_layer.o .build_release/src/caffe/layers/tanh_layer.o 
.build_release/src/caffe/layers/sigmoid_layer.o .build_release/src/caffe/layers/smooth_L1_loss_layer.o 
.build_release/src/caffe/layers/cudnn_tanh_layer.o .build_release/src/caffe/layers/silence_layer.o 
.build_release/src/caffe/layers/bias_layer.o .build_release/src/caffe/layers/detection_output_layer.o 
.build_release/src/caffe/layers/normalize_layer.o .build_release/src/caffe/layers/slice_layer.o 
.build_release/src/caffe/layers/pooling_layer.o .build_release/src/caffe/layers/detection_evaluate_layer.o 
.build_release/src/caffe/layers/relu_layer.o .build_release/src/caffe/layers/split_layer.o 
.build_release/src/caffe/layers/batch_reindex_layer.o .build_release/src/caffe/layers/cudnn_sigmoid_layer.o 
.build_release/src/caffe/layers/conv_layer.o .build_release/src/caffe/layers/bnll_layer.o 
.build_release/src/caffe/layers/base_conv_layer.o .build_release/src/caffe/layers/cudnn_softmax_layer.o 
.build_release/src/caffe/layers/crop_layer.o .build_release/src/caffe/layers/multinomial_logistic_loss_layer.o 
.build_release/src/caffe/layers/cudnn_lcn_layer.o .build_release/src/caffe/layers/base_data_layer.o 
.build_release/src/caffe/layers/rnn_layer.o .build_release/src/caffe/layers/flatten_layer.o 
.build_release/src/caffe/layers/permute_layer.o .build_release/src/caffe/layers/cudnn_lrn_layer.o 
.build_release/src/caffe/layers/dropout_layer.o .build_release/src/caffe/layers/cudnn_relu_layer.o 
.build_release/src/caffe/layers/hdf5_data_layer.o .build_release/src/caffe/layers/tile_layer.o 
.build_release/src/caffe/layers/mvn_layer.o 
.build_release/src/caffe/layers/lstm_layer.o .build_release/src/caffe/layers/lrn_layer.o 
.build_release/src/caffe/layers/argmax_layer.o 
.build_release/src/caffe/layers/euclidean_loss_layer.o .build_release/src/caffe/layers/python_layer.o 
.build_release/src/caffe/layers/image_label_data_layer.o .build_release/src/caffe/layers/annotated_data_layer.o 
.build_release/src/caffe/layers/cudnn_deconv_layer.o .build_release/src/caffe/layers/cudnn_batch_norm_layer.o 
.build_release/src/caffe/layers/contrastive_loss_layer.o .build_release/src/caffe/layers/exp_layer.o 
.build_release/src/caffe/layers/memory_data_layer.o .build_release/src/caffe/layers/log_layer.o 
.build_release/src/caffe/layers/accuracy_layer.o .build_release/src/caffe/layers/neuron_layer.o 
.build_release/src/caffe/layers/dummy_data_layer.o .build_release/src/caffe/layers/l1_loss_layer.o 
.build_release/src/caffe/layers/segmentation_accuracy_layer.o .build_release/src/caffe/layers/video_data_layer.o 
.build_release/src/caffe/layers/softmax_layer.o .build_release/src/caffe/layers/image_data_layer.o 
.build_release/src/caffe/layers/absval_layer.o .build_release/src/caffe/layers/deconv_layer.o 
.build_release/src/caffe/layers/detectnet_transform_layer.o .build_release/src/caffe/layers/eltwise_layer.o 
.build_release/src/caffe/layers/loss_layer.o .build_release/src/caffe/layers/input_layer.o 
.build_release/src/caffe/layers/infogain_loss_layer.o .build_release/src/caffe/layers/inner_product_layer.o 
.build_release/src/caffe/layers/hinge_loss_layer.o .build_release/src/caffe/layers/concat_layer.o 
.build_release/src/caffe/layers/multibox_loss_layer.o .build_release/src/caffe/layers/data_layer.o 
.build_release/src/caffe/layers/embed_layer.o .build_release/src/caffe/layers/lstm_unit_layer.o 
.build_release/src/caffe/layers/recurrent_layer.o .build_release/src/caffe/layers/elu_layer.o 
.build_release/src/caffe/layers/scale_layer.o .build_release/src/caffe/layers/cudnn_pooling_layer.o 
.build_release/src/caffe/layers/filter_layer.o .build_release/src/caffe/layers/cudnn_conv_layer.o 
.build_release/src/caffe/layers/im2col_layer.o .build_release/src/caffe/layers/threshold_layer.o 
.build_release/src/caffe/layers/prelu_layer.o .build_release/src/caffe/layers/power_layer.o 
.build_release/src/caffe/layers/spp_layer.o 
.build_release/src/caffe/layers/batch_norm_layer.o .build_release/src/caffe/layers/window_data_layer.o 
.build_release/src/caffe/layers/axpy_layer.o .build_release/src/caffe/layers/softmax_loss_layer.o 
.build_release/src/caffe/layers/prior_box_layer.o .build_release/src/caffe/blob.o .build_release/src/caffe/solver.o 
.build_release/src/caffe/syncedmem.o .build_release/src/caffe/util/gpu_memory.o .build_release/src/caffe/util/io.o 
.build_release/src/caffe/util/im_transforms.o .build_release/src/caffe/util/bbox_util.o .build_release/src/caffe/util/benchmark.o 
.build_release/src/caffe/util/sampler.o .build_release/src/caffe/util/cudnn.o .build_release/src/caffe/util/db_lmdb.o 
.build_release/src/caffe/util/hdf5.o .build_release/src/caffe/util/db.o .build_release/src/caffe/util/upgrade_proto.o 
.build_release/src/caffe/util/detectnet_coverage_rectangular.o .build_release/src/caffe/util/signal_handler.o 
.build_release/src/caffe/util/blocking_queue.o .build_release/src/caffe/util/db_leveldb.o 
.build_release/src/caffe/util/im2col.o 
.build_release/src/caffe/util/math_functions.o .build_release/src/caffe/util/insert_splits.o 
.build_release/src/caffe/solvers/nesterov_solver.o .build_release/src/caffe/solvers/adam_solver.o 
.build_release/src/caffe/solvers/sgd_solver.o .build_release/src/caffe/solvers/rmsprop_solver.o 
.build_release/src/caffe/solvers/adagrad_solver.o .build_release/src/caffe/solvers/adadelta_solver.o 
.build_release/src/caffe/data_transformer.o .build_release/src/caffe/parallel.o .build_release/src/caffe/common.o 
.build_release/src/caffe/batch_transformer.o .build_release/src/caffe/internal_thread.o .build_release/src/caffe/layer.o 
.build_release/src/caffe/data_reader.o .build_release/src/caffe/type.o .build_release/cuda/src/caffe/layers/filter_layer.o 
.build_release/cuda/src/caffe/layers/cudnn_tanh_layer.o .build_release/cuda/src/caffe/layers/silence_layer.o 
.build_release/cuda/src/caffe/layers/softmax_loss_layer.o .build_release/cuda/src/caffe/layers/log_layer.o 
.build_release/cuda/src/caffe/layers/sigmoid_layer.o .build_release/cuda/src/caffe/layers/recurrent_layer.o 
.build_release/cuda/src/caffe/layers/prelu_layer.o .build_release/cuda/src/caffe/layers/elu_layer.o 
.build_release/cuda/src/caffe/layers/accuracy_layer.o .build_release/cuda/src/caffe/layers/deconv_layer.o 
.build_release/cuda/src/caffe/layers/exp_layer.o .build_release/cuda/src/caffe/layers/cudnn_deconv_layer.o 
.build_release/cuda/src/caffe/layers/cudnn_lcn_layer.o .build_release/cuda/src/caffe/layers/cudnn_softmax_layer.o 
.build_release/cuda/src/caffe/layers/sigmoid_cross_entropy_loss_layer.o .build_release/cuda/src/caffe/layers/dropout_layer.o 
.build_release/cuda/src/caffe/layers/crop_layer.o .build_release/cuda/src/caffe/layers/threshold_layer.o 
.build_release/cuda/src/caffe/layers/im2col_layer.o .build_release/cuda/src/caffe/layers/embed_layer.o 
.build_release/cuda/src/caffe/layers/smooth_L1_loss_layer.o .build_release/cuda/src/caffe/layers/lstm_unit_layer.o 
.build_release/cuda/src/caffe/layers/slice_layer.o .build_release/cuda/src/caffe/layers/cudnn_relu_layer.o 
.build_release/cuda/src/caffe/layers/normalize_layer.o .build_release/cuda/src/caffe/layers/conv_layer.o 
.build_release/cuda/src/caffe/layers/contrastive_loss_layer.o .build_release/cuda/src/caffe/layers/inner_product_layer.o 
.build_release/cuda/src/caffe/layers/hdf5_output_layer.o .build_release/cuda/src/caffe/layers/cudnn_conv_layer.o 
.build_release/cuda/src/caffe/layers/bnll_layer.o .build_release/cuda/src/caffe/layers/hdf5_data_layer.o 
.build_release/cuda/src/caffe/layers/relu_layer.o .build_release/cuda/src/caffe/layers/cudnn_sigmoid_layer.o 
.build_release/cuda/src/caffe/layers/power_layer.o .build_release/cuda/src/caffe/layers/lrn_layer.o 
.build_release/cuda/src/caffe/layers/cudnn_batch_norm_layer.o .build_release/cuda/src/caffe/layers/tile_layer.o 
.build_release/cuda/src/caffe/layers/cudnn_pooling_layer.o .build_release/cuda/src/caffe/layers/euclidean_loss_layer.o 
.build_release/cuda/src/caffe/layers/bias_layer.o .build_release/cuda/src/caffe/layers/mvn_layer.o 
.build_release/cuda/src/caffe/layers/tanh_layer.o .build_release/cuda/src/caffe/layers/concat_layer.o 
.build_release/cuda/src/caffe/layers/split_layer.o .build_release/cuda/src/caffe/layers/detectnet_transform_layer.o 
.build_release/cuda/src/caffe/layers/absval_layer.o .build_release/cuda/src/caffe/layers/cudnn_lrn_layer.o 
.build_release/cuda/src/caffe/layers/detection_output_layer.o .build_release/cuda/src/caffe/layers/softmax_layer.o 
.build_release/cuda/src/caffe/layers/permute_layer.o .build_release/cuda/src/caffe/layers/base_data_layer.o 
.build_release/cuda/src/caffe/layers/l1_loss_layer.o .build_release/cuda/src/caffe/layers/reduction_layer.o 
.build_release/cuda/src/caffe/layers/scale_layer.o .build_release/cuda/src/caffe/layers/eltwise_layer.o 
.build_release/cuda/src/caffe/layers/batch_reindex_layer.o .build_release/cuda/src/caffe/layers/pooling_layer.o 
.build_release/cuda/src/caffe/layers/axpy_layer.o .build_release/cuda/src/caffe/layers/batch_norm_layer.o 
.build_release/cuda/src/caffe/layers/cudnn_dropout_layer.o .build_release/cuda/src/caffe/util/math_functions2.o 
.build_release/cuda/src/caffe/util/gpu_amax.o .build_release/cuda/src/caffe/util/gpu_sumsq.o 
.build_release/cuda/src/caffe/util/gpu_asum.o 
.build_release/cuda/src/caffe/util/im2col.o .build_release/cuda/src/caffe/util/bbox_util.o 
.build_release/cuda/src/caffe/util/math_functions.o .build_release/cuda/src/caffe/solvers/nesterov_solver.o 
.build_release/cuda/src/caffe/solvers/adagrad_solver.o .build_release/cuda/src/caffe/solvers/adam_solver.o 
.build_release/cuda/src/caffe/solvers/sgd_solver.o .build_release/cuda/src/caffe/solvers/rmsprop_solver.o 
.build_release/cuda/src/caffe/solvers/adadelta_solver.o .build_release/cuda/src/caffe/quantized_layer.o 
.build_release/cuda/src/caffe/data_transformer.o -Wl,-soname,libcaffe-nv.so.0.17 -Wl,-rpath,\$ORIGIN/../lib -pthread -fPIC -
DCAFFE_VERSION=0.17.0 -std=c++11 -DCUDA_NO_HALF -DNDEBUG -O2 -DUSE_CUDNN -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -DWITH_PYTHON_LAYER -
I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/include/hdf5/serial -
I.build_release/src -I./src -I./include -I./3rdparty -I/usr/include/hdf5/serial -I/usr/local/cuda/include -I/cuda/include -I/include -
I/opt/OpenBLAS/include/ -Wall -Wno-sign-compare -L/usr/lib -L/usr/local/lib -L/usr/lib -L/usr/lib/x86_64-linux-gnu/hdf5/serial -
L/usr/local/cuda/lib64 -L/usr/lib/nvidia-396 -L/usr/lib/nvidia-390 -L/usr/lib/nvidia-387 -L/usr/lib/nvidia-384 -L/usr/lib/nvidia-381 -
L/usr/lib/nvidia-375 -L/usr/lib/nvidia-367 -L/usr/lib/nvidia-361 -L/usr/lib/nvidia-352 -L/usr/local/cuda/lib -L/cuda/lib64 -L/lib64 -
L/usr/lib/x86_64-linux-gnu/hdf5/serial -L/opt/OpenBLAS/lib/ -L.build_release/lib -L/usr/local/lib -lopencv_stitching -lopencv_superres -
lopencv_videostab -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_dnn -lopencv_dpm -lopencv_fuzzy -
lopencv_line_descriptor -lopencv_optflow -lopencv_plot -lopencv_reg -lopencv_saliency -lopencv_stereo -lopencv_structured_light -
lopencv_rgbd -lopencv_surface_matching -lopencv_tracking -lopencv_datasets -lopencv_text -lopencv_face -lopencv_xfeatures2d -lopencv_shape 
-lopencv_video -lopencv_ximgproc -lopencv_calib3d -lopencv_features2d -lopencv_flann -lopencv_xobjdetect -lopencv_objdetect -lopencv_ml -
lopencv_xphoto -lippicv -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_photo -lopencv_imgproc -lopencv_core -lcudart -
lcublas -lcurand -lnvidia-ml -lboost_system -lglog -lgflags -lprotobuf -lboost_filesystem -lm -lturbojpeg -lhdf5_hl -lhdf5 -lleveldb -
lsnappy -llmdb -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lboost_regex -lstdc++ -lcudnn -lboost_python-py27 -
lpython2.7 -lboost_regex -lopenblas 

cd .build_release/lib; rm -f libcaffe-nv.so.0.17; ln -s libcaffe-nv.so.0.17.0 libcaffe-nv.so.0.17

cd .build_release/lib; rm -f libcaffe-nv.so;   ln -s libcaffe-nv.so.0.17 libcaffe-nv.so

所以他沒有吃標準的ld path..
只好手動修改 Makefile..

diff --git a/Makefile b/Makefile
index 858886c..6ce5dca 100644
--- a/Makefile
+++ b/Makefile
@@ -179,7 +179,7 @@ CUDA_LIB_DIR :=
 # add /lib64 only if it exists
 ifneq ("$(wildcard $(CUDA_DIR)/lib64)","")
        CUDA_LIB_DIR += $(CUDA_DIR)/lib64
-       CUDA_LIB_DIR += /usr/lib/nvidia-396 /usr/lib/nvidia-390 /usr/lib/nvidia-387 /usr/lib/nvidia-384 /usr/lib/nvidia-381 /usr/lib/nvidia-375 /usr/lib/nvidia-367 /usr/lib/nvidia-361 /usr/lib/nvidia-352
+       CUDA_LIB_DIR += /usr/lib/nvidia-410 /usr/lib/nvidia-390 /usr/lib/nvidia-387 /usr/lib/nvidia-384 /usr/lib/nvidia-381 /usr/lib/nvidia-375 /usr/lib/nvidia-367 /usr/lib/nvidia-361 /usr/lib/nvidia-352
 endif
 CUDA_LIB_DIR += $(CUDA_DIR)/lib
 

這樣就 build OK 了..

---

make runtest 時有錯：

[==========] 2101 tests from 283 test cases ran. (442196 ms total)
[  PASSED  ] 2097 tests.
[  FAILED  ] 4 tests, listed below:
[  FAILED  ] LayerFactoryTest/0.TestCreateLayer, where TypeParam = caffe::CPUDevice
[  FAILED  ] LayerFactoryTest/1.TestCreateLayer, where TypeParam = caffe::CPUDevice
[  FAILED  ] LayerFactoryTest/2.TestCreateLayer, where TypeParam = caffe::GPUDevice
[  FAILED  ] LayerFactoryTest/3.TestCreateLayer, where TypeParam = caffe::GPUDevice

 4 FAILED TESTS
Makefile:560: recipe for target 'runtest' failed
make: *** [runtest] Error 1

ssd

ref:

• 配置caffe-SSD
• SSD: Single Shot MultiBox Detector的安装配置和运行
• SSD: Single Shot MultiBox Detector 检测单张图
• Caffe学习系列（八）：Caffe-SSD及其轻量化模型
• Caffe上用SSD训练和测试自己的数据

make -j10
make pycaffe

要 make pycaffe，才會有 python/caffe/_caffe.so

完成後，還要設定 環境變數 PYTHONPATH 到 python/caffe，python 才能 import 得到 caffe.

ssd 的修改版本，已經把 ssd 加到 example 中。
examples/ssd/ssd_pascal.py 是產生 network, solve, deploy 的 python script 。
他會依照 hard coded 的 layer 寫出 run 的 script...

/jobs/VGGNet/VOC0712/SSD_300x300$ cat VGG_VOC0712_SSD_300x300.sh 
cd /home/charles-chang/caffessd
./build/tools/caffe train \
--solver="models/VGGNet/VOC0712/SSD_300x300/solver.prototxt" \
--weights="models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel" \
--gpu 0 2>&1 | tee jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.log

和這個script 需要的 prototxt.

ssd_pascal.py 中， run_soon= True 的話，在 create 完需要的net file, shell script 後，
就會直接 call subprocess.call( ) 來 run 這個 script，進行 training.

依照說明，到github 把 .caffemodel下載下來。
放到 create models/VGGNet folder 下。
這是 pretrained 好 converlution layer 的 parameters.

接著依照說明download training data 到 ~/data 下。(因為 ssd_pascal.py hard code path)
run caffe/data/VOC0712/ 下的 create_list.sh, create_data.sh。
他會把 ~/data/ 下的 資料放到 lmdb 中。
create_list.sh, create_data.sh 要照順序做，create_data.sh 最後會把 list 跟 data 合在一起，並且在 example/VOC0712/ 下建立 link 到 ~/data/VOC....
example/VOC0712/ 下的 link 才是 等一下 train 的 reference data path (ref: train.prototxt)

之後，就可以 run ssd_pascal.py

因為 hard coded, gpu 數量，所以會出現Check failed: error == cudaSuccess (10 vs. 0) invalid device ordinal
依照說明改 GPU 可使用數目。
同樣，會出現 out of memory，修改 batch_size, 和 accum_batch_size。

diff --git a/examples/ssd/ssd_pascal.py b/examples/ssd/ssd_pascal.py
index 62129ba..1de3c5b 100644
--- a/examples/ssd/ssd_pascal.py
+++ b/examples/ssd/ssd_pascal.py
@@ -329,13 +329,13 @@ clip = False
 
 # Solver parameters.
 # Defining which GPUs to use.
-gpus = "0,1,2,3"
+gpus = "0"
 gpulist = gpus.split(",")
 num_gpus = len(gpulist)
 
 # Divide the mini-batch to different GPUs.
-batch_size = 32
-accum_batch_size = 32
+batch_size = 16
+accum_batch_size = 16
 iter_size = accum_batch_size / batch_size
 solver_mode = P.Solver.CPU
 device_id = 0

---

create_data.sh 最後的 command 是：

/home/checko/caffessd/build/tools/convert_annoset --anno_type=detection --label_type=xml 
--label_map_file=/home/checko/caffessd/data/VOC0712/../../data/VOC0712/labelmap_voc.prototxt 
--check_label=True --min_dim=0 --max_dim=0 --resize_height=0 --resize_width=0 --backend=lmdb 
--shuffle=False --check_size=False --encode_type=jpg --encoded=True --gray=False 
/home/checko/data/VOCdevkit/ /home/checko/caffessd/data/VOC0712/../../data/VOC0712/test.txt 
/home/checko/data/VOCdevkit/VOC0712/lmdb/VOC0712_test_lmdb

和

/home/checko/caffessd/build/tools/convert_annoset --anno_type=detection --label_type=xml 
--label_map_file=/home/checko/caffessd/data/VOC0712/../../data/VOC0712/labelmap_voc.prototxt 
--check_label=True --min_dim=0 --max_dim=0 --resize_height=0 --resize_width=0 --backend=lmdb 
--shuffle=False --check_size=False --encode_type=jpg --encoded=True --gray=False 
/home/checko/data/VOCdevkit/ /home/checko/caffessd/data/VOC0712/../../data/VOC0712/trainval.txt 
/home/checko/data/VOCdevkit/VOC0712/lmdb/VOC0712_trainval_lmdb

---

train 出現

F0816 17:06:47.531648  3928 math_functions.cpp:250] Check failed: a <= b (0 vs. -1.19209e-07) 
*** Check failure stack trace: ***
    @     0x7fdb6c1a60cd  google::LogMessage::Fail()
    @     0x7fdb6c1a7f33  google::LogMessage::SendToLog()
    @     0x7fdb6c1a5c28  google::LogMessage::Flush()
    @     0x7fdb6c1a8999  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fdb6c8b6087  caffe::caffe_rng_uniform<>()
    @     0x7fdb6c8716a8  caffe::SampleBBox()
    @     0x7fdb6c871a00  caffe::GenerateSamples()
    @     0x7fdb6c871c50  caffe::GenerateBatchSamples()
    @     0x7fdb6c9bcdd2  caffe::AnnotatedDataLayer<>::load_batch()
    @     0x7fdb6c953faa  caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()
    @     0x7fdb6ca9c965  caffe::InternalThread::entry()
    @     0x7fdb60df4bcd  (unknown)
    @     0x7fdb4d0f06db  start_thread
    @     0x7fdb6a8e988f  clone

然後這一篇："caffe 错误 math_functions.cpp Check failed: a <= b" 說改 SampleBBox()..

/etc/profile.d

原來login 之後， shell 會先 run /etc/profile。
然後 /etc/profile 最後有一個 loop，會去 run /etc/profile.d 下所有的 script。
所以，有個別的全域的參數要設定，就可以放在 /etc/profile.d/ 下。

舉例來說，安裝完 anaconda 後，在 $(INSTALLPATH)/etc/profile.d/ 會有 conda.sh
可以建一個 link 到 /etc/profile.d/conda.sh
這樣，所有 user login 後，都會自動執行 anaconda 的環境設定 script

disable touchpad

:~$ xinput list
⎡ Virtual core pointer                     id=2 [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer               id=4 [slave  pointer  (2)]
⎜   ↳ Kensington      Kensington Expert Mouse  id=10 [slave  pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad               id=13 [slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                    id=14 [slave  pointer  (2)]
⎣ Virtual core keyboard                    id=3 [master keyboard (2)]
    ↳ Virtual core XTEST keyboard              id=5 [slave  keyboard (3)]
    ↳ Power Button                             id=6 [slave  keyboard (3)]
    ↳ Video Bus                                id=7 [slave  keyboard (3)]
    ↳ Video Bus                                id=8 [slave  keyboard (3)]
    ↳ Power Button                             id=9 [slave  keyboard (3)]
    ↳ Integrated Camera                        id=11 [slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard             id=12 [slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                   id=15 [slave  keyboard (3)]
:~$ xinput --disable 13

先用xinput 列出所有 input device，然後用 disable 關掉 touchpad

---

在使用wayland 後這個 xinput 方式就失效了，沒有出現我用的 touchpad 名稱。
所以參考這一篇 歷經 11 年後的回答，用gnome setting:

Disable:

gsettings set org.gnome.desktop.peripherals.touchpad send-events disabled

Enable:

gsettings set org.gnome.desktop.peripherals.touchpad send-events enabled