638 lines
7.0 KiB
Markdown
638 lines
7.0 KiB
Markdown
# Aplikasi Penerjemah Bahasa Isyarat Indonesia Menggunakan MediaPipe Hands, Face Mesh, dan Temporal Convolutional Network (TCN)
|
|
|
|
## Deskripsi
|
|
|
|
Project ini merupakan backend machine learning untuk aplikasi penerjemah Bahasa Isyarat Indonesia (SIBI dan BISINDO) menggunakan MediaPipe Hands, MediaPipe Face Mesh, dan Temporal Convolutional Network (TCN).
|
|
|
|
Model dibagi menjadi empat kategori agar sesuai dengan alur aplikasi frontend.
|
|
|
|
- Abjad Bisindo
|
|
- Abjad SIbi
|
|
- Angka
|
|
- Kata
|
|
|
|
Setiap kategori memiliki model TCN tersendiri sehingga proses inferensi lebih cepat dan akurasi lebih baik.
|
|
|
|
---
|
|
|
|
# Pipeline Project
|
|
|
|
```
|
|
Raw Video
|
|
│
|
|
▼
|
|
Extract Landmark
|
|
│
|
|
▼
|
|
Normalize Landmark
|
|
│
|
|
▼
|
|
Split Dataset
|
|
│
|
|
├── Train 80%
|
|
└── Test 20%
|
|
│
|
|
▼
|
|
Training TCN
|
|
│
|
|
▼
|
|
Testing Model
|
|
│
|
|
▼
|
|
Realtime Camera
|
|
│
|
|
▼
|
|
Frontend ReactJS
|
|
```
|
|
|
|
---
|
|
|
|
# Struktur Folder
|
|
|
|
```
|
|
backend/
|
|
|
|
│
|
|
├── dataset/
|
|
│
|
|
│ ├── raw/
|
|
│ │
|
|
│ │ ├── abjad_bisindo/
|
|
│ │ │
|
|
│ │ ├── abjad_sibi/
|
|
│ │ │
|
|
│ │ ├── angka/
|
|
│ │ │
|
|
│ │ └── kata/
|
|
│ │
|
|
│ ├── landmark/
|
|
│ │
|
|
│ ├── normalized/
|
|
│ │
|
|
│ ├── train/
|
|
│ │
|
|
│ └── test/
|
|
│
|
|
│
|
|
├── models/
|
|
│
|
|
│ ├── abjad_bisindo/
|
|
│ │
|
|
│ ├── abjad_sibi/
|
|
│ │
|
|
│ ├── angka/
|
|
│ │
|
|
│ └── kata/
|
|
│
|
|
│
|
|
├── outputs/
|
|
│
|
|
│ ├── confusion_matrix/
|
|
│ │
|
|
│ ├── classification_report/
|
|
│ │
|
|
│ ├── accuracy/
|
|
│ │
|
|
│ └── loss/
|
|
│
|
|
│
|
|
├── realtime/
|
|
│
|
|
│ ├── realtime_abjad_bisindo.py
|
|
│ │
|
|
│ ├── realtime_abjad_sibi.py
|
|
│ │
|
|
│ ├── realtime_angka.py
|
|
│ │
|
|
│ └── realtime_kata.py
|
|
│
|
|
│
|
|
├── scripts/
|
|
│
|
|
│ ├── 01_extract_landmark.py
|
|
│ │
|
|
│ ├── 02_normalize_landmark.py
|
|
│ │
|
|
│ ├── 03_split_dataset.py
|
|
│ │
|
|
│ ├── 04_train_abjad_kata.py
|
|
│ │
|
|
│ ├── 05_test_abjad_kata.py
|
|
│ │
|
|
│ ├── 06_train_abjad_bisindo.py
|
|
│ │
|
|
│ ├── 07_test_abjad_bisindo.py
|
|
│ │
|
|
│ ├── 08_train_sibi.py
|
|
│ │
|
|
│ ├── 09_test_sibi.py
|
|
│ │
|
|
│ ├── 10_train_angka.py
|
|
│ │
|
|
│ └── 11_test_angka.py
|
|
│
|
|
│
|
|
├── utils/
|
|
│
|
|
│ ├── mediapipe_detector.py
|
|
│ │
|
|
│ ├── normalization.py
|
|
│ │
|
|
│ ├── dataset.py
|
|
│ │
|
|
│ ├── tcn.py
|
|
│ │
|
|
│ ├── inference.py
|
|
│ │
|
|
│ └── config.py
|
|
│
|
|
│
|
|
├── requirements.txt
|
|
│
|
|
└── README.md
|
|
```
|
|
|
|
---
|
|
|
|
# Struktur Dataset
|
|
|
|
Semua tahapan preprocessing akan mempertahankan struktur folder yang sama seperti folder `raw`.
|
|
|
|
```
|
|
dataset/
|
|
|
|
├── raw/
|
|
├── landmark/
|
|
├── normalized/
|
|
├── train/
|
|
└── test/
|
|
```
|
|
|
|
Artinya setiap folder memiliki struktur yang identik sehingga proses preprocessing menjadi lebih mudah, konsisten, dan mudah dikembangkan apabila terdapat penambahan dataset baru.
|
|
|
|
---
|
|
|
|
# Struktur Dataset Raw
|
|
|
|
```
|
|
dataset/raw/
|
|
|
|
abjad_bisindo/
|
|
|
|
a/
|
|
a_s1_light_on.mp4
|
|
a_s2_light_off.mp4
|
|
...
|
|
|
|
b/
|
|
c/
|
|
...
|
|
|
|
abjad_sibi/
|
|
|
|
a/
|
|
b/
|
|
...
|
|
|
|
angka/
|
|
|
|
0/
|
|
1/
|
|
...
|
|
|
|
kata/
|
|
|
|
halo/
|
|
terimakasih/
|
|
...
|
|
```
|
|
|
|
**Format video yang didukung**
|
|
|
|
- `.mp4`
|
|
|
|
---
|
|
|
|
# Tahap 1
|
|
|
|
## Extract Landmark
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/01_extract_landmark.py
|
|
```
|
|
|
|
Input
|
|
|
|
```
|
|
dataset/raw/
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
dataset/landmark/
|
|
```
|
|
|
|
Proses
|
|
|
|
- Membaca seluruh video
|
|
- Membaca frame menggunakan OpenCV
|
|
- Deteksi MediaPipe Hands 21 landmark tangan kiri, 21 landmark tangan kanan
|
|
- Deteksi MediaPipe Face Mesh 60 landmark dari dagu, mulut, bibir, hidung, mata, pipi
|
|
- Mengambil landmark
|
|
- Membentuk Sliding Window 30 Frame dan 5 Stride
|
|
- Menyimpan file .npy
|
|
|
|
Output contoh
|
|
|
|
```
|
|
dataset/
|
|
|
|
landmark/
|
|
|
|
abjad_bisindo/
|
|
|
|
A/
|
|
|
|
sample_1.npy
|
|
sample_2.npy
|
|
...
|
|
```
|
|
|
|
Shape
|
|
|
|
```
|
|
(30, feature_size)
|
|
```
|
|
|
|
> **Catatan:** Struktur folder pada `landmark/` dibuat **sama persis** dengan struktur folder `raw/`.
|
|
|
|
---
|
|
|
|
# Tahap 2
|
|
|
|
## Normalize Landmark
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/02_normalize_landmark.py
|
|
```
|
|
|
|
Input
|
|
|
|
```
|
|
dataset/landmark/
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
dataset/normalized/
|
|
```
|
|
|
|
Proses
|
|
|
|
- Membaca seluruh landmark
|
|
- Menggeser koordinat ke titik referensi
|
|
- Scaling
|
|
- Menyimpan landmark hasil normalisasi
|
|
|
|
> **Catatan:** Struktur folder pada `normalized/` akan dibuat **identik** dengan folder `landmark/` dan `raw/`.
|
|
|
|
---
|
|
|
|
# Tahap 3
|
|
|
|
## Split Dataset
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/03_split_dataset.py
|
|
```
|
|
|
|
Input
|
|
|
|
```
|
|
dataset/normalized/
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
dataset/train/
|
|
|
|
dataset/test/
|
|
```
|
|
|
|
Persentase
|
|
|
|
```
|
|
Train
|
|
|
|
80%
|
|
|
|
Test
|
|
|
|
20%
|
|
```
|
|
|
|
Pembagian dilakukan **untuk setiap kelas**, sehingga semua kelas tetap memiliki data train dan data test.
|
|
|
|
Seluruh struktur folder pada `train/` dan `test/` akan dibuat **sama persis** dengan struktur folder `normalized/`, `landmark/`, dan `raw/`.
|
|
|
|
---
|
|
|
|
# Tahap 4
|
|
|
|
## Training Kata
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/04_train_kata.py
|
|
```
|
|
|
|
Input
|
|
|
|
```
|
|
dataset/train/kata/
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
models/kata/model_kata.pt
|
|
```
|
|
|
|
Output tambahan
|
|
|
|
```
|
|
Accuracy
|
|
|
|
Loss
|
|
|
|
Training History
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 5
|
|
|
|
## Testing Kata
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/05_test_kata.py
|
|
```
|
|
|
|
Input
|
|
|
|
```
|
|
dataset/test/kata/
|
|
```
|
|
|
|
Model
|
|
|
|
```
|
|
model_kata.pt
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
Confusion Matrix
|
|
|
|
Classification Report
|
|
|
|
Accuracy
|
|
|
|
Precision
|
|
|
|
Recall
|
|
|
|
F1 Score
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 6
|
|
|
|
## Training Abjad BISINDO
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/06_train_abjad_bisindo.py
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
model_abjad_bisindo.pt
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 7
|
|
|
|
## Testing Abjad BISINDO
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/07_test_abjad_bisindo.py
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 8
|
|
|
|
## Training Abjad SIBI
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/08_train_abjad_sibi.py
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
model_abjad_sibi.pt
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 9
|
|
|
|
## Testing Abjad SIBI
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/09_test_abjad_sibi.py
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 10
|
|
|
|
## Training Angka
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/10_train_angka.py
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
model_angka.pt
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 11
|
|
|
|
## Testing Angka
|
|
|
|
Script
|
|
|
|
```
|
|
scripts/11_test_angka.py
|
|
```
|
|
|
|
Output
|
|
|
|
```
|
|
Accuracy
|
|
|
|
Precision
|
|
|
|
Recall
|
|
|
|
F1 Score
|
|
|
|
Confusion Matrix
|
|
```
|
|
|
|
---
|
|
|
|
# Tahap 12
|
|
|
|
## Realtime Detection
|
|
|
|
Folder
|
|
|
|
```
|
|
realtime/
|
|
```
|
|
|
|
Script
|
|
|
|
```
|
|
realtime_abjad_bisindo.py
|
|
|
|
realtime_abjad_sibi.py
|
|
|
|
realtime_angka.py
|
|
|
|
realtime_kata.py
|
|
```
|
|
|
|
Proses
|
|
|
|
```
|
|
Webcam
|
|
|
|
↓
|
|
|
|
MediaPipe
|
|
|
|
↓
|
|
|
|
Landmark
|
|
|
|
↓
|
|
|
|
Normalisasi
|
|
|
|
↓
|
|
|
|
Window 30 Frame
|
|
|
|
↓
|
|
|
|
Model TCN
|
|
|
|
↓
|
|
|
|
Prediksi
|
|
|
|
↓
|
|
|
|
Tampilkan hasil
|
|
```
|
|
|
|
---
|
|
|
|
# Fungsi Folder Utils
|
|
|
|
## mediapipe_detector.py
|
|
|
|
Mengambil landmark tangan dan wajah menggunakan MediaPipe.
|
|
|
|
---
|
|
|
|
## normalization.py
|
|
|
|
Melakukan normalisasi koordinat landmark.
|
|
|
|
---
|
|
|
|
## dataset.py
|
|
|
|
Membaca dataset .npy menjadi Dataset PyTorch.
|
|
|
|
---
|
|
|
|
## tcn.py
|
|
|
|
Berisi implementasi model Temporal Convolutional Network.
|
|
|
|
---
|
|
|
|
## inference.py
|
|
|
|
Melakukan prediksi menggunakan model TCN.
|
|
|
|
---
|
|
|
|
## config.py
|
|
|
|
Berisi konfigurasi project.
|
|
|
|
---
|
|
|
|
# Model
|
|
|
|
Project menghasilkan empat model.
|
|
|
|
```
|
|
model_abjad_bisindo.pt
|
|
|
|
model_abjad_sibi.pt
|
|
|
|
model_angka.pt
|
|
|
|
model_kata.pt
|
|
```
|
|
|
|
---
|