TIF_E4253362/README.md

# Aplikasi Penerjemah Bahasa Isyarat Indonesia

Aplikasi penerjemah bahasa isyarat Indonesia (BISINDO & SIBI) berbasis machine learning menggunakan **MediaPipe Hands**, **MediaPipe Face Mesh**, dan **Temporal Convolutional Network (TCN)**.

---

## 📋 Daftar Isi

- [Deskripsi](#deskripsi)
- [Fitur](#fitur)
- [Arsitektur Sistem](#arsitektur-sistem)
- [Struktur Direktori](#struktur-direktori)
- [Instalasi](#instalasi)
- [Library dan Dependensi](#library-dan-dependensi)
- [Pipeline Project](#pipeline-project)
- [Penggunaan](#penggunaan)
  - [1. Ekstraksi Landmark](#1-ekstraksi-landmark)
  - [2. Normalisasi Landmark](#2-normalisasi-landmark)
  - [3. Split Dataset](#3-split-dataset)
  - [4. Training Model](#4-training-model)
  - [5. Testing Model](#5-testing-model)
  - [6. Deteksi Realtime](#6-deteksi-realtime)
  - [7. Menjalankan API Backend](#7-menjalankan-api-backend)
  - [8. Menjalankan Frontend](#8-menjalankan-frontend)
- [API Reference](#api-reference)
- [Struktur Dataset](#struktur-dataset)
- [Model TCN](#model-tcn)
- [Konfigurasi](#konfigurasi)

---

## 📝 Deskripsi

Project ini merupakan aplikasi penerjemah bahasa isyarat Indonesia yang mendukung:
- **BISINDO** (Bahasa Isyarat Indonesia)
- **SIBI** (Sistem Isyarat Bahasa Indonesia)
- **Angka**
- **Kata/Kosakata**

### Teknologi yang Digunakan

| Teknologi | Fungsi |
|-----------|--------|
| **MediaPipe Hands** | Deteksi 21 landmark tangan kiri + 21 landmark tangan kanan |
| **MediaPipe Face Mesh** | Deteksi 60 landmark wajah (dagu, mulut, bibir, hidung, mata, pipi) |
| **Temporal Convolutional Network (TCN)** | Klasifikasi sequence gesture |
| **Flask** | Backend API server |
| **React.js** | Frontend web application |

---

## ✨ Fitur

- 🎥 Deteksi realtime gestur bahasa isyarat via webcam
- 🤖 Model ML untuk 4 kategori (Abjad BISINDO, Abjad SIBI, Angka, Kata)
- 📊 Visualisasi landmark tangan dan wajah
- 📖 Kamus isyarat dengan video contoh
- 🔄 Temporal smoothing untuk prediksi yang stabil
- 📡 REST API untuk komunikasi frontend-backend

---

## 🏗️ Arsitektur Sistem

```
┌─────────────────────────────────────────────────────────────────┐
│                        FRONTEND (React.js)                       │
│                     http://localhost:3000                        │
└─────────────────────────────────────────────────────────────────┘
                                │
                                │ HTTP/JSON (Base64 Frame)
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                    BACKEND (Flask API)                          │
│                     http://localhost:5000                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐  │
│  │ /init        │  │ /detect      │  │ /                   │  │
│  │ POST         │  │ POST         │  │ GET                 │  │
│  └──────────────┘  └──────────────┘  └──────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        ▼                       ▼                       ▼
┌──────────────┐  ┌──────────────────┐  ┌──────────────────────┐
│ MediaPipe    │  │ Normalizer       │  │ TCN Model Inference  │
│ Detector     │  │                  │  │                      │
│ - Hands      │  │ - Translation    │  │ - 30 Frame Window    │
│ - Face Mesh  │  │ - Scaling        │  │ - Temporal Smoothing│
└──────────────┘  └──────────────────┘  └──────────────────────┘
```

### Alur Data Processing

```
Video/ webcam
    │
    ▼
Extract Landmark (MediaPipe)
    │
    ├── 21 landmark tangan kiri
    ├── 21 landmark tangan kanan
    └── 60 landmark wajah
    │
    ▼
Normalisasi Landmark
    │
    ├── Translation ke wrist/nose
    └── Scaling
    │
    ▼
Sliding Window (30 frame, stride 5)
    │
    ▼
Model TCN Inference
    │
    ▼
Prediksi Label + Confidence
```

---

## 📁 Struktur Direktori

```
APP BAHASA ISYARAT/
│
├── backend/
│   │
│   ├── app.py                      # Flask API server
│   ├── requirements.txt             # Dependensi Python
│   │
│   ├── dataset/
│   │   ├── raw/                    # Dataset video mentah
│   │   │   ├── abjad_bisindo/
│   │   │   ├── abjad_sibi/
│   │   │   ├── angka/
│   │   │   └── kata/
│   │   ├── landmark/               # Landmark hasil ekstraksi
│   │   ├── normalized/             # Landmark hasil normalisasi
│   │   ├── train/                  # Data training (80%)
│   │   └── test/                   # Data testing (20%)
│   │
│   ├── models/                     # Model TCN terlatih
│   │   ├── abjad_bisindo/
│   │   │   ├── model_abjad_bisindo.pt
│   │   │   └── label_encoder.json
│   │   ├── abjad_sibi/
│   │   ├── angka/
│   │   └── kata/
│   │
│   ├── outputs/                    # Output training/testing
│   │   ├── confusion_matrix/
│   │   ├── classification_report/
│   │   ├── accuracy/
│   │   └── loss/
│   │
│   ├── realtime/                   # Script deteksi realtime
│   │   ├── realtime_abjad_bisindo.py
│   │   ├── realtime_abjad_sibi.py
│   │   ├── realtime_angka.py
│   │   └── realtime_kata.py
│   │
│   ├── scripts/                    # Pipeline preprocessing & training
│   │   ├── 01_extract_landmark.py
│   │   ├── 02_normalize_landmark.py
│   │   ├── 03_split_dataset.py
│   │   ├── 04_train_kata.py
│   │   ├── 05_test_kata.py
│   │   ├── 06_train_abjad_bisindo.py
│   │   ├── 07_test_abjad_bisindo.py
│   │   ├── 08_train_abjad_sibi.py
│   │   ├── 09_test_abjad_sibi.py
│   │   ├── 10_train_angka.py
│   │   └── 11_test_angka.py
│   │
│   └── utils/                      # Utility modules
│       ├── mediapipe_detector.py   # MediaPipe detector
│       ├── normalization.py        # Normalisasi landmark
│       ├── dataset.py              # PyTorch Dataset
│       ├── tcn.py                  # Arsitektur TCN
│       ├── inference.py            # Inference utilities
│       └── config.py               # Konfigurasi project
│
└── frontend/
    ├── package.json                # Dependensi Node.js
    ├── public/
    └── src/
        ├── App.js                  # Router utama
        ├── index.js                # Entry point
        └── pages/                  # Halaman React
            ├── HomePage.jsx
            ├── CategoryPage.jsx
            ├── DetectionPage.jsx   # Halaman deteksi webcam
            ├── DictionaryPage.jsx  # Kamus isyarat
            ├── AngkaPage.jsx
            ├── SibiPage.jsx
            ├── BisindoPage.jsx
            └── KataPage.jsx
```

---

## 🔧 Instalasi

### Prasyarat

- Python 3.8+
- Node.js 16+
- npm atau yarn
- Webcam (untuk deteksi realtime)

### 1. Setup Backend

```bash
# Masuk ke direktori backend
cd backend

# Buat virtual environment (opsional tapi disarankan)
python -m venv venv
source venv/bin/activate  # Linux/Mac
# atau
.\venv\Scripts\activate   # Windows

# Install dependensi
pip install -r requirements.txt
```

### 2. Setup Frontend

```bash
# Masuk ke direktori frontend
cd frontend

# Install dependensi
npm install
```

---

## 📚 Library dan Dependensi

### Backend (Python)

| Library | Versi Minimal | Fungsi |
|---------|---------------|--------|
| `numpy` | >=1.21.0 | Manipulasi array dan operasi matematika |
| `torch` | >=1.10.0 | Framework deep learning |
| `torchvision` | >=0.11.0 | Transformasi data PyTorch |
| `mediapipe` | >=0.10.0 | Deteksi landmark tangan & wajah |
| `opencv-python` | >=4.8.0 | Processing video |
| `opencv-python-headless` | >=4.8.0 | OpenCV tanpa GUI |
| `scikit-learn` | >=0.24.0 | Evaluasi model (confusion matrix, dll) |
| `matplotlib` | >=3.7.0 | Visualisasi data |
| `seaborn` | >=0.12.0 | Visualisasi statistik |
| `tqdm` | >=4.65.0 | Progress bar |
| `Flask` | >=2.3.0 | Web framework |
| `flask-cors` | >=4.0.0 | CORS support |
| `ml_dtypes` | >=0.3.0 | Dependensi MediaPipe |

### Frontend (JavaScript/React)

| Library | Fungsi |
|---------|--------|
| `react` | UI framework |
| `react-dom` | React DOM rendering |
| `react-router-dom` | Routing |
| `react-scripts` | Build tool (Create React App) |
| `@testing-library/*` | Testing utilities |
| `web-vitals` | Performance metrics |

---

## 🔄 Pipeline Project

```
┌─────────────────────────────────────────────────────────────────┐
│                     PIPELINE MACHINE LEARNING                    │
└─────────────────────────────────────────────────────────────────┘

Raw Video (.mp4)
    │
    ▼
┌─────────────────────────────────────────┐
│ Tahap 1: Extract Landmark               │
│ Script: 01_extract_landmark.py          │
│ Output: dataset/landmark/              │
│ Input shape: (30, 306) per window       │
└─────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────┐
│ Tahap 2: Normalize Landmark             │
│ Script: 02_normalize_landmark.py        │
│ Output: dataset/normalized/             │
│ Proses: Translation + Scaling           │
└─────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────┐
│ Tahap 3: Split Dataset                  │
│ Script: 03_split_dataset.py             │
│ Output: dataset/train/ & dataset/test/  │
│ Rasio: 80% train, 20% test              │
└─────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────┐
│ Tahap 4: Training TCN                   │
│ Scripts: 04-11_train_*.py               │
│ Output: models/{category}/model_*.pt    │
│ Output: models/{category}/label_encoder │
└─────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────┐
│ Tahap 5: Testing Model                  │
│ Scripts: 05-11_test_*.py                │
│ Output: Confusion Matrix, Classification│
│         Report, Accuracy, F1-Score      │
└─────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────┐
│ Tahap 6: Realtime Detection            │
│ Scripts: realtime/*.py                  │
│ Input: Webcam                           │
│ Output: Prediksi real-time              │
└─────────────────────────────────────────┘
```

---

## 📖 Penggunaan

### 1. Ekstraksi Landmark

Ekstrak landmark dari video menggunakan MediaPipe.

```bash
cd backend

# Ekstrak semua kategori
python scripts/01_extract_landmark.py

# Ekstrak kategori tertentu
python scripts/01_extract_landmark.py --categories abjad_bisindo angka

# Custom parameter
python scripts/01_extract_landmark.py --window-size 30 --stride 5
```

**Parameter:**
| Parameter | Default | Deskripsi |
|-----------|---------|-----------|
| `--input` | `dataset/raw/` | Direktori video mentah |
| `--output` | `dataset/landmark/` | Direktori output landmark |
| `--categories` | Semua | Kategori yang diproses |
| `--window-size` | 30 | Jumlah frame per window |
| `--stride` | 5 | Step size sliding window |

---

### 2. Normalisasi Landmark

Normalisasi koordinat landmark untuk konsistensi.

```bash
cd backend

# Normalisasi semua kategori
python scripts/02_normalize_landmark.py

# Custom input/output
python scripts/02_normalize_landmark.py --input dataset/landmark --output dataset/normalized
```

---

### 3. Split Dataset

Bagi dataset menjadi training (80%) dan testing (20%).

```bash
cd backend

# Split semua kategori
python scripts/03_split_dataset.py

# Custom rasio
python scripts/03_split_dataset.py --train-ratio 0.8

# Split kategori tertentu
python scripts/03_split_dataset.py --categories abjad_bisindo kata
```

---

### 4. Training Model

Latih model TCN untuk setiap kategori.

```bash
cd backend

# Training model kata
python scripts/04_train_kata.py

# Training model abjad BISINDO
python scripts/06_train_abjad_bisindo.py

# Training model abjad SIBI
python scripts/08_train_abjad_sibi.py

# Training model angka
python scripts/10_train_angka.py

# Custom parameter training
python scripts/04_train_kata.py --epochs 100 --batch-size 32 --lr 0.001
```

**Parameter Training:**
| Parameter | Default | Deskripsi |
|-----------|---------|-----------|
| `--epochs` | 100 | Jumlah epoch |
| `--batch-size` | 32 | Batch size |
| `--lr` | 0.001 | Learning rate |
| `--data-dir` | `dataset/` | Direktori dataset |
| `--model-dir` | `models/` | Direktori output model |

**Output Training:**
```
models/{category}/
├── model_{category}.pt      # Model PyTorch
├── label_encoder.json       # Mapping label ke index
└── history.json             # History training
```

---

### 5. Testing Model

Evaluasi model menggunakan data test.

```bash
cd backend

# Test model kata
python scripts/05_test_kata.py

# Test model abjad BISINDO
python scripts/07_test_abjad_bisindo.py

# Test model abjad SIBI
python scripts/09_test_abjad_sibi.py

# Test model angka
python scripts/11_test_angka.py
```

**Output Testing:**
```
outputs/
├── confusion_matrix/
│   └── {category}_confusion_matrix.png
├── classification_report/
│   └── {category}_classification_report.txt
└── {category}_test_results.json
```

**Metrik Evaluasi:**
- Accuracy
- Precision
- Recall
- F1-Score
- Confusion Matrix

---

### 6. Deteksi Realtime

Jalankan deteksi realtime menggunakan webcam (tanpa frontend).

```bash
cd backend

# Deteksi kata
python realtime/realtime_kata.py

# Deteksi abjad BISINDO
python realtime/realtime_abjad_bisindo.py

# Deteksi abjad SIBI
python realtime/realtime_abjad_sibi.py

# Deteksi angka
python realtime/realtime_angka.py

# Custom camera index
python realtime/realtime_kata.py --camera 0
```

**Kontrol Keyboard:**
| Key | Fungsi |
|-----|--------|
| `q` | Quit |
| `r` | Reset buffer |
| `l` | Toggle landmark visibility |

---

### 7. Menjalankan API Backend

Jalankan Flask API server untuk komunikasi dengan frontend.

```bash
cd backend

# Jalankan server
python app.py
```

Server akan berjalan di `http://localhost:5000`

**Endpoints:**

| Endpoint | Method | Deskripsi |
|----------|--------|-----------|
| `/` | GET | Status server |
| `/init` | POST | Inisialisasi session model |
| `/detect` | POST | Proses frame dan prediksi |

---

### 8. Menjalankan Frontend

Jalankan aplikasi React.

```bash
cd frontend

# Development mode
npm start

# Production build
npm run build
```

Frontend akan berjalan di `http://localhost:3000`

---

## 🔌 API Reference

### GET /

Cek status server.

**Response:**
```json
{
  "status": "online",
  "message": "Backend Penerjemah Bahasa Isyarat Indonesia (TCN) Siap!",
  "categories": ["abjad_bisindo", "abjad_sibi", "angka", "kata"],
  "endpoints": {
    "/init": "POST - Inisialisasi session model",
    "/detect": "POST - Kirim frame base64 dan dapatkan prediksi"
  }
}
```

### POST /init

Inisialisasi session inferensi untuk kategori tertentu.

**Request Body:**
```json
{
  "category": "kata"
}
```

**Response:**
```json
{
  "session_id": "uuid-string",
  "category": "kata",
  "status": "ready"
}
```

**Kategori Valid:**
- `abjad_bisindo`
- `abjad_sibi`
- `angka`
- `kata`

### POST /detect

Kirim frame webcam dan dapatkan prediksi.

**Request Body:**
```json
{
  "session_id": "uuid-string",
  "frame": "data:image/jpeg;base64,..."
}
```

**Response:**
```json
{
  "label": "halo",
  "confidence": 0.95,
  "buffer_size": 30,
  "window_size": 30
}
```

---

## 📂 Struktur Dataset

### Format Dataset Raw

```
dataset/raw/
├── abjad_bisindo/
│   ├── A/
│   │   ├── video1.mp4
│   │   └── video2.mp4
│   ├── B/
│   │   └── ...
│   └── ...
├── abjad_sibi/
│   └── ...
├── angka/
│   ├── 0/
│   ├── 1/
│   └── ...
└── kata/
    ├── halo/
    ├── terima_kasih/
    └── ...
```

### Format Landmark

```
dataset/landmark/
├── abjad_bisindo/
│   ├── A/
│   │   ├── video1_w0.npy   # Window 0
│   │   ├── video1_w1.npy   # Window 1
│   │   └── ...
│   └── ...
└── ...
```

### Shape Data

| Tahap | Shape | Deskripsi |
|-------|-------|-----------|
| Raw Video | (N, H, W, 3) | N frame |
| Landmark | (window, 306) | 30 frame × 306 fitur |
| Normalized | (window, 306) | Setelah normalisasi |

**Detail Feature Size (306):**
- Tangan kiri: 21 × 3 = 63
- Tangan kanan: 21 × 3 = 63
- Wajah: 60 × 3 = 180
- **Total: 306**

---

## 🧠 Model TCN

### Arsitektur

```
TCNClassifier(
  ├── TemporalConvNet
  │   ├── TemporalBlock (dilation 1)
  │   ├── TemporalBlock (dilation 2)
  │   └── TemporalBlock (dilation 4)
  │
  └── FC Layers
      ├── Linear(256, 512)
      ├── ReLU()
      ├── Dropout(0.2)
      └── Linear(512, num_classes)
)
```

### Hyperparameter Default

| Parameter | Nilai |
|-----------|-------|
| Input Size | 306 |
| Hidden Channels | [64, 128, 256] |
| Kernel Size | 3 |
| Dropout | 0.2 |
| FC Hidden Size | 512 |
| Window Size | 30 |

---

## ⚙️ Konfigurasi

Konfigurasi utama ada di `backend/utils/config.py`:

```python
# Paths
BASE_DIR = Path(__file__).parent.parent
DATASET_DIR = BASE_DIR / "dataset"
MODEL_DIR = BASE_DIR / "models"

# Categories
CATEGORIES = ["abjad_bisindo", "abjad_sibi", "angka", "kata"]

# Sliding Window
WINDOW_SIZE = 30
STRIDE = 5

# Feature Configuration
NUM_HAND_LANDMARKS = 21
NUM_FACE_LANDMARKS = 60
FEATURE_SIZE = 306  # 63 + 63 + 180

# Training
BATCH_SIZE = 32
EPOCHS = 100
LEARNING_RATE = 0.001
TRAIN_RATIO = 0.8

# Session
SESSION_TIMEOUT_SECONDS = 600  # 10 menit
```

---

## 📌 Catatan Penting

### Persyaratan Video
- Format: `.mp4`
- Durasi minimum: 30 frame (untuk window size 30)
- Disarankan: 60-90 frame per video

### GPU Acceleration
Model akan otomatis menggunakan GPU jika tersedia:
```python
device = 'cuda' if torch.cuda.is_available() else 'cpu'
```

### Session Management
- Session akan expire setelah 10 menit tidak aktif
- Frontend akanauto-reconnect saat session expired

---

## 🐛 Troubleshooting

### Error: "ModuleNotFoundError"
```bash
pip install -r requirements.txt
```

### Error: "Camera not found"
```python
# Ganti camera index di realtime script
cap = cv2.VideoCapture(0)  # Coba 0, 1, 2...
```

### Error: "Model not found"
Pastikan model sudah dilatih:
```bash
python scripts/04_train_kata.py
```

### Frontend tidak connect ke Backend
1. Pastikan backend running di port 5000
2. Cek CORS settings di `app.py`
3. Pastikan URL di frontend sesuai

---

## 📄 Lisensi

Project ini adalah bagian dari Tugas Akhir (TA) - Aplikasi Penerjemah Bahasa Isyarat Indonesia.

---

## 👨‍💻 Kontribusi

Project ini dikembangkan menggunakan:
- MediaPipe (Google)
- PyTorch
- Flask
- React.js