2023/9/27

setup git server by git-daemon

以前做過一次: 但是在 ubuntu 18.04 上就不行了,說沒有 supervise/ok, status, lock.. etc --- 都是 sv 藥用的東西。
因為 git-daemon-run 是用 sv 來啟動的。

實際上 git-daemon 包含在git 的source code ,所以 apt install git-core 就會有 git-daemon 。
git-daemon 提供 git:// 的服務。
所以只要設定對那些 folder 提供服務,允不允許 write (push). 其他就一些 log 之類的..
跟用什麼serive manager 無關。

測試:

手動 adduser gitdaemon,然後登入 gitdaemon, run git-daemon,看看git:// 有沒有作用。

這樣就可以把gitdaemon的 HOME 作為 git repository,create new repo 也只要login 成 gitdaemon, 在 自己目錄下 clone --bare 就可以。

OK 之後再寫到 systemd service.
$ sudo adduser gitdaemon
$ sudo su - gitdaemon
$ mkdir repository
$ cd repository
$ git clone --bare /home/test/test test.git
$ /usr/lib/git-core/git-daemon --export-all --verbose --enable=receive-pack --syslog --base-path=/home/gitdaemon/repository /home/gitdaemon/repository
這樣,其他user 就可以:
git clone git::/127.0.0.1/testgit.git


手動 OK 後,寫 systemd service file:
$ cat /etc/systemd/system/gitdaemon.service 
[Unit]
Description = Git Daemon Service

[Service]
Type=simple
User=gitdaemon
ExecStart=/usr/lib/git-core/git-daemon --export-all --verbose --enable=receive-pack --syslog --base-path=/home/gitdaemon/repository /home/gitdaemon/repository &

[Install]
WantedBy=multi-user.target
新增後,enable & start,看 status:
$ sudo systemctl status gitdaemon.service
● gitdaemon.service - Git Daemon Service
   Loaded: loaded (/etc/systemd/system/gitdaemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2023-09-27 16:29:00 CST; 1min 47s ago
 Main PID: 8231 (git-daemon)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/gitdaemon.service
           └─8231 /usr/lib/git-core/git-daemon --export-all --verbose --enable=receive-pack --syslog --base-path=/home/gitdaemon/repository /home/gitdaemon/repository &

Sep 27 16:29:00 xeontitan systemd[1]: Started Git Daemon Service.
Sep 27 16:29:00 xeontitan git-daemon[8231]: Ready to rumble
Sep 27 16:29:32 xeontitan git-daemon[8252]: Connection from 192.168.147.182:56802
Sep 27 16:29:32 xeontitan git-daemon[8252]: Extended attribute "host": xeontitan
Sep 27 16:29:32 xeontitan git-daemon[8252]: Request upload-pack for '/testgit.git'
Sep 27 16:29:32 xeontitan git-daemon[8231]: [8252] Disconnected
Sep 27 16:29:57 xeontitan git-daemon[8262]: Connection from 192.168.147.182:52568
Sep 27 16:29:57 xeontitan git-daemon[8262]: Extended attribute "host": xeontitan
Sep 27 16:29:57 xeontitan git-daemon[8262]: Request receive-pack for '/testgit.git'
Sep 27 16:29:57 xeontitan git-daemon[8231]: [8262] Disconnected
這樣應該就成功了。



之後多人同時做 android 的 repo sync 時,出現 error
remote: Counting objects: 100% (128934/128934), done.
remote: Compressing objects: 100% (53641/53641), done.
remote: Total 128913 (delta 57825), reused 127304 (delta 56375)
Fetching: 78% (792/1011) 41:10 | 10 jobs | 16:10 platform/frameworks/base @ frameworks/base
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

platform/prebuilts/android-emulator:
fatal: Could not read from remote repository.
查 syslog 有:
Oct  4 13:11:48 rd1-ubuntu git-daemon[1791508]: Too many children, dropping connection
Oct  4 13:11:51 rd1-ubuntu git-daemon[1791508]: message repeated 3 times: [ Too many children, dropping connection]
查git-daemon 的 option 說明有:
     --max-connections=<n>
         Maximum number of concurrent clients, defaults to 32.
         Set it to zero for no limit.
所以修改 git-daemon service:
$ cat /etc/systemd/system/gitdaemon.service
[Unit]
Description = Git Daemon Service

[Service]
Type=simple
User=gitdaemon
ExecStart=/usr/lib/git-core/git-daemon --export-all --verbose --enable=receive-pack --syslog --max-connections=0 --base-path=/home/gitdaemon/repository /home/gitdaemon/repository &

[Install]
WantedBy=multi-user.target
然後做 stop, daemon-reload, start..

2023/9/26

結果是 git-daemon-run
會create gitdaemon user, belongs to nogroup

git-daemon-run 只是幫忙git-daemon 啟動跟設置的一些 script。
使用 sv, 不是用 systemd,所以不太適用了。
直接寫 systemd 的 service 來啟動 git-daemon 比較方便,當然,要手動create git-daemon 的 user 和 group

2023/9/23

about ubuntu 22.04

一些 ubuntu22.04 相關的版本, so 問題,都會記錄在這邊。

因為 用 libssl3,所以一堆用 libssl1 的都會說沒有 supported ssl。
只好裝libssl1
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.0g-2ubuntu4_amd64.deb
sudo dpkg -i libssl1.1_1.1.0g-2ubuntu4_amd64.deb

2023/9/21

Docker : run nvidia cuda ready images

ref: nvidia 有做好一堆有 support cuda 的 docker image : nvidia in docker image

但是docker 要 supoort 這些 cuda ready 的 image,要安裝 nvidia-container-toolkit
sudo apt install nvidia-container-toolkit
之後設定runtime support:
sudo nvidia-ctk runtime configure --runtime=docker
然後重新啟動 docker daemon:
sudo systemctl restart docker
這樣之後,docker command 就會 support --gpus all 這個 option

另外,image 的 cuda 版本不能比 host 的 cuda 版本新,實際用 torch.rand(2000,128,device=torch.device('cuda')) 測試,他會說使用了新的function.


有一個 dockerfile,是從 yolov5s_android" 看到的:
FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu18.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update --fix-missing
RUN apt-get install -y python3 python3-pip
RUN pip3 install --upgrade pip
RUN pip3 install torch==1.7.1+cu110 torchvision==0.8.2+cu110 -f https://download.pytorch.org/whl/cu110/torch_stable.html

# install openvino
RUN apt-get update && apt-get install -y --no-install-recommends \
    wget \
    cpio \
    sudo \
    lsb-release && \
    rm -rf /var/lib/apt/lists/*
# Add a user that UID:GID will be updated by vscode
ARG USERNAME=developer
ARG GROUPNAME=developer
ARG UID=1000
ARG GID=1000
ARG PASSWORD=developer
RUN groupadd -g $GID $GROUPNAME && \
    useradd -m -s /bin/bash -u $UID -g $GID -G sudo $USERNAME && \
    echo $USERNAME:$PASSWORD | chpasswd && \
    echo "$USERNAME   ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER $USERNAME
ENV HOME /home/developer

原文說明是:
git clone --recursive https://github.com/lp6m/yolov5s_android
cd yolov5s_android
docker build ./ -f ./docker/Dockerfile  -t yolov5s_android
docker run -it --gpus all -v `pwd`:/workspace yolov5s_android bash



241122 update

ref

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    
sudo apt update

sudo apt install nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker
測試是否安裝成功,用 nvidia docker 來 run nvidia-smi:
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

RTX3090. python3.6 and pytorch1.10, pytorch1.7

ref: 很麻煩的 torch. cuda.
用新的顯卡(其實也沒多新,就 3090而已),使 sm85,需要某版本以上的 pytorch 才有support.
不然,雖然用 torch.cuda.is_available() 是 True, get_device_name() 也會正確顯示 RTX3090。
但是宣告一個在 GPU 的變數卻會有 Error:
>import torch
>print(torch.__version__)
1.10.2+cu102
>A = torch.rand(2000,128,device=torch.device('cuda'))
NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
說其實 1.7.1 版的 pytorch 就已經 support RTX3090 了,是因為 pip repo 中的 pytorch package 配的 cuda 版本太舊,不 support RTX3090.
所以到 pytorch 網站下載新版 cuda 的 pytorch 來裝就可以了。

我的 機器 cuda 版本是 11.7,所以用:
pip install torch==1.10.0+cu113 torchvision==0.11.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
這樣裝完後,上面的測試command 就不會有 Error 了。

到 cu113 的 torch_stable 去看,支援只有從 torch 1.10.0 開始,所以可以用 ref(2) 的作法,到-f 的位址 cu110/torch_stable.html 去看裡面有沒有 torch-1.7.1
結果真的只有 cu110 有,cu111 之後都是 torch-1.10.0 了。
pip3 install torch==1.7.1+cu110 torchvision==0.8.2+cu110 -f https://download.pytorch.org/whl/cu110/torch_stable.html

2023/9/19

android studio and gradle 8.0

一直出現Error:
compileDebugJavaWithJavac' task (current target is 1.8) and 'kaptGenerateStubsDebugKotlin' task (current target is 17) 
jvm target compatibility should be set to the same Java version.
一堆說是 Gradle 8.0 的問題,結果更新到 8.1 也一樣。
然後說要修改 app 的 build.gradle 指定 targetjvm version.
但是沒有清楚說明。

android studio 的 build system 是 gradle,所以每個版本都會有需要的 gradle 版本。
上面的問題出現在 gradle 8.0,所以使用 gradle 7.X 的最後一版的android studio : Android Studio Electric Eel | 2022.1.1 Patch 2
就沒有問題了。

其實使用gradle 8.0配合新的 build.gradle 內容也沒有問題。

TEST project:
  • Data Binging Basic : Failed
  • New Empty kotline Sample : OK

2023/9/13

build and run SNPE Android Example : image-classifiers

要先 run examples Models 的 inception_v3 的 script 去 download train 好 的 model 和 parameter.
這個 script 要用到 SDK/bin 的 script 和 python 中 tensorflow package 的 py。
所以要設定好 SDK 和 tensorflow 的位置。
另外他用絕對位址download 到 SDK 的 example 目錄,所以 example 的 code 不能 copy 出來。一定要在 SDK 目錄中。

需要用到 snpe 提供的 python module,所以要把 snpe sdk 的 lib/python 加到 PYTHONPATH 中,這個在 bin/envsetup.sh 會做。
envsetup 會設定 SNPE_ROOT, PYTHONPATH, PATH, LD_LIBRARY_PATH

2023/9/11

debugging C++ with VSCode, in ubuntu

ubuntu 跟 windows 使用 VSCode 沒什麼差別,反而是linux 的 c++ compiler 是 opensource 的,還比 windows 方便,不用買 MS 的 build tool 或是裝 msys 來 run g++
所以就是 host 先有 g++ 能 build source code,VSCode 就可以用他來 build.

其他也一樣,VSCode 是以 folder 作為管理 project 的單位。

另外。VSCode 需要的 project setting file : task.json,也是要針對 project folder,放在folder 的 .vscode 中。
以 C++ 來說,第一次 run, debug 的時候,VSCode 發現 project folder 沒有 task.json,就會依照 source language/tool 建一個。

參考: Using C++ on Linux in VS Code

先 create 一個 folder。在 folder 下 run vscode。 (或是在 vscode 中 open folder)。
folder 下就是 C++ source file.
開啟 C++ source file,選 run 或是 debug 就可以了。
就會自動產生 task.json
在source code 中可以設定 break point,之後 run 或是 debug 遇到 break point 都會暫停。

另外。VSCode 第一次開啟 C++ file,會要求安裝 C++ extension。

2023/9/7

install and setup snpe 2.14 in ubuntu 20.04

目前最新的 snpe (2.14) 要用 qpm 來安裝。
所以download 完 snpe sdk後,要先安裝qpm.
qpm 裝完,login 後,會是一個 daemon(service),就可以用 qpm-cli 命令。

用qpm-cli 解開/安裝 snpe..不管在哪裡解開,都會 install 到:
qpm-cli --extract XXOO.qik
SUCCESS: Installed qualcomm_neural_processing_sdk.Core at /opt/qcom/aistack/snpe/2.14.0.230828
裝完可以用:
qpm-cli --info qualcomm_neural_processing_sdk

Product Name           : qualcomm_neural_processing_sdk
Product Classification : Binary
Installed version      : None
Available version(s)   : 2.14.0.230828
                         2.13.4.230831
                         2.13.2.230822
                         2.13.0.230730
                         2.12.0.230626
                         2.11.0.230603
                         2.10.40.4
安裝完後的docs下,有 html 的文件。

follow 這個附的文件,setup,這版 2023/9 月的版本,是用 ubuntu 20.04
提供兩個 script 來檢查/安裝 需要個 package:
  • check-python-dependency
  • check-linux-dependency.sh
其中 check-python-denpency 需要在 virtual env (VENV 或 conda) 下,所以要為 snpe create 一個 python 環境。
文件說明 default 是 3.8,但是如果是要 run tensorflow 1.15.0,就要用 3.6。
先用 3.8 來做..用 conda create snpe3.8

然後 run envsetup.sh
$ source /opt/qcom/aistack/snpe/2.14.0.230828/bin/envsetup.sh 
[INFO] AISW SDK environment set
[INFO] SNPE_ROOT: /opt/qcom/aistack/snpe/2.14.0.230828
check python.. 結果一堆 error:
$ /opt/qcom/aistack/snpe/2.14.0.230828/bin/check-python-dependency 
/opt/qcom/aistack/snpe/2.14.0.230828/bin/check-python-dependency:55: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
WARNING: attrs installed version: 21.4.0 does not match tested version: 22.2.0
WARNING: decorator installed version: 4.4.2 does not match tested version: 5.1.1
WARNING: joblib installed version: 1.1.0 does not match tested version: 1.0.1
WARNING: packaging installed version: 21.3 does not match tested version: 21.0
WARNING: pillow installed version: 9.4.0 does not match tested version: 6.2.1
WARNING: scipy installed version: 1.8.1 does not match tested version: 1.9.1
Python Modules missing: absl-py, invoke, lxml, mako, matplotlib, numpy, opencv-python, pandas, pathlib2, protobuf, pytest, pyyaml, six, tabulate
Installing missing modules using pip3
Installing absl-py version: 0.13.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nbconvert 6.5.0 requires entrypoints>=0.2.2, which is not installed.
Installing invoke version: 2.0.0
Installing lxml version: 4.6.2
Installing mako version: 1.1.0
Installing matplotlib version: 3.3.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
moviepy 1.0.3 requires requests<3.0,>=2.8.1, which is not installed.
Installing numpy version: 1.23.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
moviepy 1.0.3 requires requests<3.0,>=2.8.1, which is not installed.
Installing opencv-python version: 4.5.2.52
Installing pandas version: 1.1.5
Installing pathlib2 version: 2.3.6
Installing protobuf version: 3.19.6
Installing pytest version: 7.0.1
Installing pyyaml version: 3.10
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
naked 0.1.31 requires requests, which is not installed.
Installing six version: 1.16.0
Installing tabulate version: 0.8.5
用 pip 分別安裝 error 的 package,應該過了,但是 version 不一定一樣。

文件說明 android ndk 經過 snpe 認證的是 r19c.
用 android-studio 的 sdk-manager 來看,19版 只有 19.2
裝完看 CHANGELOG.md,剛好是 r19c,所以 path 就是 Android/Sdk/ndk/19.2.5345600

所以設完變數,run check...
$export ANDROID_NDK_ROOT=/home/charles-chang/Android/Sdk/ndk/19.2.5345600
$export PATH=${ANDROID_NDK_ROOT}:${PATH}
$/opt/qcom/aistack/snpe/2.14.0.230828/bin/envcheck -n
Checking Android NDK Environment
--------------------------------------------------------------
[INFO] Found ndk-build at /home/charles-chang/Android/Sdk/ndk/19.2.5345600/ndk-build and ANDROID_NDK_ROOT is also set.
--------------------------------------------------------------
$ /opt/qcom/aistack/snpe/2.14.0.230828/bin/envcheck -c
Checking Clang-9 Environment
--------------------------------------------------------------
[INFO] Found clang++-9 at /usr/bin/clang++-9
--------------------------------------------------------------
另外,pip install torch==1.8.1 onnx==1.11.0 tensorflow==2.10.1 tflite==2.3.0

2023/9/6

/proc/meminfo MemAvaileble

在 /proc/meminfo.c:
    available = si_mem_available();
    ...

    show_val_kb(m, "MemTotal:       ", i.totalram);
    show_val_kb(m, "MemFree:        ", i.freeram);
    show_val_kb(m, "MemAvailable:   ", available);
    ...
在 mm/page_alloc.c
long si_mem_available(void)
{
    long available;
    unsigned long pagecache;
    unsigned long wmark_low = 0;
    unsigned long pages[NR_LRU_LISTS];
    unsigned long reclaimable;
    struct zone *zone;
    int lru;

    for (lru = LRU_BASE; lru < NR_LRU_LISTS; lru++)
        pages[lru] = global_node_page_state(NR_LRU_BASE + lru);

    for_each_zone(zone)
        wmark_low += low_wmark_pages(zone);

    /*
     * Estimate the amount of memory available for userspace allocations,
     * without causing swapping.
     */
    available = global_zone_page_state(NR_FREE_PAGES) - totalreserve_pages;

    /*
     * Not all the page cache can be freed, otherwise the system will
     * start swapping. Assume at least half of the page cache, or the
     * low watermark worth of cache, needs to stay.
     */
    pagecache = pages[LRU_ACTIVE_FILE] + pages[LRU_INACTIVE_FILE];
    pagecache -= min(pagecache / 2, wmark_low);
    available += pagecache;

    /*
     * Part of the reclaimable slab and other kernel memory consists of
     * items that are in use, and cannot be freed. Cap this estimate at the
     * low watermark.
     */
    reclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE) +
            global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE);
    available += reclaimable - min(reclaimable / 2, wmark_low);

    if (available < 0)
        available = 0;
    return available;
}
所以 MemAvailable 是 free + cache + reclaimable slab (buffer?)

2023/9/4

windows10 : pyopengl , glutInit( ) error : OpenGL.error.NullFunctionError

ref: windows7 下run 的好好的,升級到windows10 之後就fail 了,
用 example code 來測試:
from OpenGL.GL import *
from OpenGL.GLU import *
from OpenGL.GLUT import *

def Draw():
        glClear(GL_COLOR_BUFFER_BIT)
        glutWireTeapot(0.5)
        glFlush()

glutInit()
glutInitDisplayMode(GLUT_SINGLE | GLUT_RGBA)
glutInitWindowSize(300,300)
glutCreateWindow(b"teapot")
glutDisplayFunc(Draw)
glutIdleFunc(Draw)
glutMainLoop()

if __name__ == '__main__':
        Draw()
出現 Error:
Traceback (most recent call last):
  File "D:\teapot.py", line 11, in &;t;module>
    glutInit()
  File "D:\Python311\Lib\site-packages\OpenGL\GLUT\special.py", line 333, in glutInit
    _base_glutInit( ctypes.byref(count), holder )
  File "D:\Python311\Lib\site-packages\OpenGL\platform\baseplatform.py", line 423, in __call__
     raise error.NullFunctionError(
OpenGL.error.NullFunctionError: Attempt to call an undefined function glutInit, check for bool(glutInit) before calling  
結果google 說是 pypl official 的 opengl package 沒有包到兩個 dlll...
pipewire 0.3.78-1 (https://pipewire-debian.github.io)
 

  Debian Package - 

	- enable modemmanager

  Pipewire -

    - For more : https://gitlab.freedesktop.org/pipewire/pipewire/-/releases
  .

Troubleshooting - 

  - Have any package regarding issue? report on github :
    https://github.com/pipewire-debian/pipewire-debian/issues/new/choose

  - Upstream recommends to use 'WirePlumber' instead 'pipewire-media-session'      
    as session manager, to get it add another PPA,      
    'sudo add-apt-repository ppa:pipewire-debian/wireplumber-upstream'      
    For more instruction read : https://pipewire-debian.github.io