Gaussian Splatting con Nerfstudio e Opensplat

Sto interessando a gaussian splatting ed ho iniziato provando OpenSplat

Prima di usare OpenSplat si devono preparare i dati con NerfStudio

Ho creato un virtualenv con Python e quindi ho installato Nerfstudio ed alcune dipendenze

pip install nerfstudio

apt install ffmpeg

apt install colmap

A questo punto ho preso le immagini di esempio del Lego ed ho lanciato il processing (no gpu)

 

ns-process-data images --data ./data/img/lego/train/ --output-dir ./data/img/elabora --no-gpu

E' giunto il momento di passare ad OpenSplat.

Ho creato il docker clonando il git del progetto e lanciando (compilazione abbastanza veloce)

docker build -t opensplat .

a questo punto si lancia il docker puntando a dove sono state salvate le elaborazioni di Nerfstudio

docker run -it -v /home/luca/OpenSplat/NerfStudio/data/img:/data --device=/dev/kfd --device=/dev/dri opensplat:latest bash

e lanciando la elaborazione

root@c52b8089c5f6:/code/build# ./opensplat ../../data/elabora/ -n 2000

il risultato dopo 2000 steps e' il file splat.ply

https://playcanvas.com/viewer

Yolo 8 in Google Colab

Ho provato a vedere di sfruttare le risorse gratuite di Google Colab per fare il retraining di Yolo del precedente post

Per prima cosa si copiano i files su Google Drive e si concede la lettura a Colab

import os, sys

from google.colab import drive

drive.mount('/content/drive')

nb_path = '/content/notebooks'

os.symlink('/content/drive/My Drive/Colab Notebooks', nb_path)

sys.path.insert(0,nb_path)

In seguito si installa Yolo 

%pip install ultralytics

import ultralytics

ultralytics.checks()

Ho usato per semplicita' la modalita'a linea nei comando (basta mettere un punto interrogativo prima del comando) effettuando i puntamenti sui folder GDrive

!yolo task=detect mode=train model=yolov8m.pt imgsz=640 data=/content/drive/MyDrive/ocean_trash/data.yaml epochs=60 batch=32 name=yolov8m_e60_b32_trash

Rispetto alla prova sul portatile ho potuto utilizzare la YOLO M al posto di YOLO N impostando le epoch a 60 e la batch a 32 (con queste impostazioni siamo quasi al limite dei 15 Gb di memoria disponibili sulla GPU T4 disponibile nel profilo gratuito)

Rispetto alla rete addestrata sul portatile con YOLO N, 10 epoch e 4 batch c'e' un significativo miglioramento a parita'di set di immagini di addestramento anche se la classe Rope risulta permanere come mal classificata

Retrain Yolo8 con rifiuti su spiaggia

Ho provato a fare il retraining di Yolo 8 con rifiuti spiaggiati usando il dataset al link sottostante

 https://universe.roboflow.com/baeulang/ocean_trash

 

 Il dataset e' composto da 1133 immagini

Il train e' stato eseguito sull' NVidia Geforce 940M su mio portatile Debian. Per questo motivo ho dovuto usare il modello nano e ridurre le batch da 16 a 8 altrimenti saturavo completamente il Gb di ram della scheda video

yolo task=detect mode=train model=yolov8n.pt imgsz=640 data=/home/luca/yolo8/trash/data.yaml epochs=10 batch=4 name=yolov8n_trash

 

yolo detect val model='/home/luca/yolo8/runs/detect/yolov8n_trash4/weights/best.pt'  imgsz=640 data=/home/luca/yolo8/trash/data.yaml

Matrice di confusione

a questo punto ho usato il file addestrato per riconoscere immagini che non sono state inserite nella fase di train

yolo predict model='/home/luca/yolo8/runs/detect/yolov8n_trash4/weights/best.pt' source='/home/luca/yolo8/trash/test/images/CD_9581_jpg.rf.ac853dd458749924fe0891cd4c5f9a21.jpg' imgsz=640

 

Questo e' un esempio di riconoscimento

Il risultato non e' eccezionale ma c'e' considerare che e' stato utilizzato come base il modello nano

SV305 raw image

Sto provando a prendere il controllo della SVBony SV305 senza passare da AstroDMX

La camera puo' essere controllata tramite l'SDK che si trova al link https://www.svbony.com/Support/SoftWare-Driver  (il driver e' proprietario e la libreria e' distribuita compilata)

Per prima cosa su Linux si copia il file 90-ckusb.rules in /etc/udev/rules.d e si riavvia udev con udevadm control --reload-rules

La camera puo' essere controllata sia mediante l'esempio in C sia mediante Python con la libreria https://pypi.org/project/pysvb/

Il salvataggio e' in ogni caso in formato raw con spazio colore GRBG, cio' vuol dire che il filtro di Bayes riporta nella prima riga valore RGRGRGRGRG mentre nella seconda GBGBGBGB

Nella demo viene selezionata una ROI di 800x600 (la massima risoluzione e' 1920x1080x16bit) con un file di uscita di dimensione di 3840000 byte.  Considerando che 800x600 sono 480000 pixels ci sono 8 byte per pixel (il calcolo sarebbe 800x600x4x2) 

oltre al formato RAW16 sono supportati i formati RAW8,RAW10, RAW12,RAW14,RAW16,RGB24,RGB32,Y8, Y10,Y12,Y14,Y16

http://www.suffolksky.com/2022/04/21/indi-allsky-sv305-gain-and-color-setting/

Scan camera number: 1

Friendly name: SVBONY SV305
Port type: USB2.0
SN: 01232E4B8B52E627EE20081200a4
Device ID: 0x1201
Camera ID: 1
sn: 01232E4B8B52E627EE20081200a4
camera maximum width 1920
camera maximum height 1080
camera color space: color
camera bayer pattern: 2
support bin: 1
support bin: 2
support bin: 0
support img type: 0
support img type: 4
support img type: 5
support img type: 10
support img type: -1
Max depth: 16
Is trigger camera: YES
=================================
control type: 1
control name: Exposure
control Description: Exposure
Maximum value: 1999999991
minimum value: 29
default value: 30000
is auto supported: YES
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 1 value 989
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 0
control name: Gain
control Description: Gain
Maximum value: 720
minimum value: 0
default value: 10
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 0 value 0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 9
control name: Contrast
control Description: Contrast
Maximum value: 100
minimum value: 0
default value: 50
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 9 value 50
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 10
control name: Sharpness
control Description: Sharpness
Maximum value: 100
minimum value: 0
default value: 0
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 10 value 0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 11
control name: Saturation
control Description: Saturation
Maximum value: 255
minimum value: 0
default value: 128
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 11 value 128
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 4
control name: WB_R
control Description: WB Red
Maximum value: 511
minimum value: 0
default value: 128
is auto supported: YES
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 4 value 231
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 5
control name: WB_G
control Description: WB Green
Maximum value: 511
minimum value: 0
default value: 128
is auto supported: YES
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 5 value 128
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 6
control name: WB_B
control Description: WB Blue
Maximum value: 511
minimum value: 0
default value: 128
is auto supported: YES
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 6 value 253
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 2
control name: Gamma
control Description: Gamma
Maximum value: 1000
minimum value: 0
default value: 100
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 2 value 100
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 8
control name: Frame speed
control Description: Frame speed
Maximum value: 2
minimum value: 0
default value: 1
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 8 value 2
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 13
control name: Offset
control Description: Offset
Maximum value: 255
minimum value: 0
default value: 0
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 13 value 0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 12
control name: Auto exposure target
control Description: Auto exposure target
Maximum value: 160
minimum value: 60
default value: 100
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 12 value 100
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 7
control name: Flip
control Description: Flip
Maximum value: 3
minimum value: 0
default value: 0
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 7 value 0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 18
control name: Bad pixel correction
control Description: Bad pixel correction
Maximum value: 1
minimum value: 0
default value: 1
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 18 value 1
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
=================================
control type: 19
control name: Bad pixel correction threshold
control Description: Bad pixel correction threshold
Maximum value: 200
minimum value: 10
default value: 60
is auto supported: NO
is writable: YES
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
control type 19 value 60
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
camera ROI, startX 0, startY 0, width 1920, height 1080, bin 1
camera ROI, startX 0, startY 0, width 800, height 600, bin 1
cameraMode: 0
pixel size 2.900000
write image file SVB_image_0.raw
drop frames: 0
write image file SVB_image_1.raw
drop frames: 0
write image file SVB_image_2.raw
drop frames: 0
write image file SVB_image_3.raw
drop frames: 0
write image file SVB_image_4.raw
drop frames: 0
write image file SVB_image_5.raw
drop frames: 0
write image file SVB_image_6.raw
drop frames: 0
write image file SVB_image_7.raw
drop frames: 0
write image file SVB_image_8.raw
drop frames: 0
write image file SVB_image_9.raw
drop frames: 0
stop camera capture
close camera
 

OpenCV e camera UVC

Ho scoperto per caso che le camere UVC sono facilmente controllabili da OpenCV  in particolare per quanto riguardo il parametro dell'auto esposizione

Per le caratteristiche della camera si usa 

v4l2-ctl --list-formats-ext -d 0

con il parametro d che e' il numero del device

 

===============================================

import cv2

cap = cv2.VideoCapture(0)

# The control range can be viewed through v4l2-ctl -L
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
cap.set(cv2.CAP_PROP_BRIGHTNESS, 64)
cap.set(cv2.CAP_PROP_CONTRAST, 0)
cap.set(cv2.CAP_PROP_EXPOSURE, 100) #in Linux l'esposizione e' 1/n
#cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1) #controlla autoesposizione 1=true

while(True):
    ret, frame = cap.read()
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
         break

cap.release()
cv2.destroyAllWindows()

Aruco tag e image stacking

Per cercare di migliorare il riconoscimento dei tag (Aruco, April) ho provato a fare image stacking per ridurre il rumore

Ho trovato una soluzione gia' pronta in questo repository

https://github.com/maitek/image_stacking

La definizione dei bordi e' nettamente migliorata

Image stacking

 

Immagine singola

FTP Bash script

 Uno script rubato su stack overflow per fare l'upload automatico su FTP senza criptazione

 

#!/bin/sh
HOST='ftp.xxxxx.altervista.org'
USER='xxxxxxx'
PASSWD='xxxxxxxxxxxxxxxxxxx'
FILE='luca.jpg'

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
binary
put $FILE
quit
END_SCRIPT
exit 0