first commit
This commit is contained in:
Alfnnr
commit
7866b7d185
|
|
@ -0,0 +1,175 @@
|
||||||
|
# GitHub Copilot Instructions — Prediksi Harga Cabai
|
||||||
|
|
||||||
|
## Project
|
||||||
|
This repository is the FastAPI deployment backend for the D4 thesis:
|
||||||
|
"Analisis Model Long Short-Term Memory dalam Sistem Prediksi Harga Cabai dengan Visualisasi Peta Digital Interaktif di Kabupaten Jember"
|
||||||
|
|
||||||
|
The API serves one-day-ahead LSTM predictions for cabai rawit at:
|
||||||
|
- Pasar_Tanjung
|
||||||
|
- Pasar_Kreongan
|
||||||
|
- Pasar_Tegal_Besar
|
||||||
|
- Pasar_Mangli
|
||||||
|
- Pasar_Wirolegi
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
Use the existing Anaconda environment:
|
||||||
|
- Conda: tf_gpu39
|
||||||
|
- Python: 3.9.23
|
||||||
|
- Interpreter: C:\Users\alfia\anaconda3\envs\tf_gpu39\python.exe
|
||||||
|
|
||||||
|
Do not create another Conda environment unless explicitly requested.
|
||||||
|
|
||||||
|
## ML artifacts
|
||||||
|
The models are already trained and deployed. Do NOT retrain them in the API.
|
||||||
|
|
||||||
|
Each market has:
|
||||||
|
- model.h5
|
||||||
|
- scaler.pkl
|
||||||
|
- config.json
|
||||||
|
|
||||||
|
Final deployment configuration:
|
||||||
|
- neurons: 64
|
||||||
|
- activation: tanh
|
||||||
|
- learning_rate: 0.05
|
||||||
|
- batch_size: 32
|
||||||
|
- epochs: 50
|
||||||
|
- optimizer: Adam
|
||||||
|
- time_step: 7
|
||||||
|
- seed: 42
|
||||||
|
- shuffle: true
|
||||||
|
|
||||||
|
These values are reference metadata. Do not hard-code predictions.
|
||||||
|
|
||||||
|
## Inference rules
|
||||||
|
For each request:
|
||||||
|
1. Accept exactly 7 recent prices.
|
||||||
|
2. Convert to numeric shape (7, 1).
|
||||||
|
3. Use the saved market-specific scaler.
|
||||||
|
4. Transform the 7 values.
|
||||||
|
5. Reshape to (1, 7, 1).
|
||||||
|
6. Run the saved .h5 model once.
|
||||||
|
7. Inverse-transform the output.
|
||||||
|
8. Return the predicted next price.
|
||||||
|
|
||||||
|
The model unit is thousand rupiah:
|
||||||
|
- 22.0 means approximately Rp22,000.
|
||||||
|
|
||||||
|
Do not fit a new scaler on request data.
|
||||||
|
|
||||||
|
## Model loading
|
||||||
|
Load all five model/scaler/config artifacts once during FastAPI startup/lifespan and keep them in memory.
|
||||||
|
|
||||||
|
Do NOT:
|
||||||
|
- load models on every request
|
||||||
|
- call model.fit()
|
||||||
|
- retrain on startup
|
||||||
|
- recreate a different LSTM architecture when the saved model can be loaded directly
|
||||||
|
|
||||||
|
For inference, loading Keras .h5 with compile=False is preferred.
|
||||||
|
|
||||||
|
## Suggested project structure
|
||||||
|
C:\PrediksiHargaCabe\
|
||||||
|
├── api/
|
||||||
|
│ ├── main.py
|
||||||
|
│ ├── requirements.txt
|
||||||
|
│ └── models/
|
||||||
|
│ └── rawit/
|
||||||
|
│ ├── pasar_tanjung/
|
||||||
|
│ │ ├── model.h5
|
||||||
|
│ │ ├── scaler.pkl
|
||||||
|
│ │ └── config.json
|
||||||
|
│ ├── pasar_kreongan/
|
||||||
|
│ ├── pasar_tegal_besar/
|
||||||
|
│ ├── pasar_mangli/
|
||||||
|
│ └── pasar_wirolegi/
|
||||||
|
├── website/
|
||||||
|
└── .github/
|
||||||
|
└── copilot-instructions.md
|
||||||
|
|
||||||
|
Inspect the workspace first and reuse existing files instead of creating duplicates.
|
||||||
|
|
||||||
|
## Required API
|
||||||
|
Implement at least:
|
||||||
|
|
||||||
|
GET /
|
||||||
|
Basic status response.
|
||||||
|
|
||||||
|
GET /api/markets
|
||||||
|
Return Rawit and the five supported markets.
|
||||||
|
|
||||||
|
GET /api/health
|
||||||
|
Report API status and whether all five model artifacts loaded successfully.
|
||||||
|
|
||||||
|
POST /api/prediction/rawit/{market}
|
||||||
|
Request example:
|
||||||
|
{
|
||||||
|
"prices": [18, 16, 19, 19, 22, 22, 22]
|
||||||
|
}
|
||||||
|
|
||||||
|
Response should include:
|
||||||
|
- chili
|
||||||
|
- market
|
||||||
|
- time_step
|
||||||
|
- input_prices
|
||||||
|
- predicted_price_thousand
|
||||||
|
- predicted_price_rupiah
|
||||||
|
|
||||||
|
Use Pydantic validation.
|
||||||
|
|
||||||
|
Reject:
|
||||||
|
- unknown markets
|
||||||
|
- fewer/more than 7 prices
|
||||||
|
- non-numeric values
|
||||||
|
- NaN/infinite values
|
||||||
|
- negative prices
|
||||||
|
|
||||||
|
## CORS
|
||||||
|
Prepare CORS so a React/React Native frontend can call the API during development. Keep allowed origins configurable and avoid recommending unrestricted wildcard CORS for production.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
Before declaring the API complete:
|
||||||
|
1. Start from the `api` directory with:
|
||||||
|
python -m uvicorn main:app --reload
|
||||||
|
2. Verify /.
|
||||||
|
3. Verify /api/markets.
|
||||||
|
4. Verify /api/health.
|
||||||
|
5. Verify /docs.
|
||||||
|
6. Test prediction for all five markets.
|
||||||
|
7. Compare at least one known API prediction with the validated notebook inference, allowing normal floating-point differences.
|
||||||
|
|
||||||
|
## Thesis safety
|
||||||
|
Do not silently:
|
||||||
|
- change hyperparameters
|
||||||
|
- change time_step
|
||||||
|
- change the scaler method
|
||||||
|
- retrain models
|
||||||
|
- introduce another ML model
|
||||||
|
- modify the validated ML methodology
|
||||||
|
|
||||||
|
Never call the final configuration a "global optimum". Use wording such as:
|
||||||
|
"konfigurasi terbaik yang diperoleh dari skema eksperimen yang diterapkan pada dataset penelitian."
|
||||||
|
|
||||||
|
## Agent workflow
|
||||||
|
Before editing:
|
||||||
|
1. Inspect the workspace.
|
||||||
|
2. Inspect existing files and model paths.
|
||||||
|
3. Reuse correct code.
|
||||||
|
4. Avoid duplicate entry points or model directories.
|
||||||
|
|
||||||
|
When editing:
|
||||||
|
1. Make small coherent changes.
|
||||||
|
2. Preserve validated ML inference logic.
|
||||||
|
3. Use clear Python/FastAPI structure.
|
||||||
|
4. Run a practical validation after each important change.
|
||||||
|
5. If a command fails, inspect and fix the actual error; do not fake success.
|
||||||
|
|
||||||
|
The immediate priority is Rawit only:
|
||||||
|
1. Build the FastAPI backend.
|
||||||
|
2. Load the five saved artifacts.
|
||||||
|
3. Verify startup.
|
||||||
|
4. Implement /api/markets, /api/health, and /api/prediction/rawit/{market}.
|
||||||
|
5. Test the API.
|
||||||
|
6. Prepare it for connection to the thesis frontend.
|
||||||
|
|
||||||
|
Communication preference:
|
||||||
|
Explain important decisions briefly in Indonesian, give exact commands, state which files are changed, and avoid unrelated technologies.
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
{
|
||||||
|
"python-envs.defaultEnvManager": "ms-python.python:conda",
|
||||||
|
"python-envs.defaultPackageManager": "ms-python.python:conda"
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,651 @@
|
||||||
|
# Referensi Lengkap Aplikasi Panen Harga
|
||||||
|
|
||||||
|
Dokumen ini menjelaskan aplikasi **Panen Harga**, yaitu sistem prediksi harga cabai
|
||||||
|
di Kabupaten Jember dengan visualisasi peta digital interaktif. Dokumen ini dapat
|
||||||
|
digunakan sebagai referensi untuk menjelaskan tujuan aplikasi, arsitektur, fitur,
|
||||||
|
alur data, teknologi, API, database, model LSTM, pengelolaan scraping, serta
|
||||||
|
batasan sistem.
|
||||||
|
|
||||||
|
## 1. Identitas dan tujuan aplikasi
|
||||||
|
|
||||||
|
### Nama aplikasi
|
||||||
|
|
||||||
|
**Panen Harga**
|
||||||
|
|
||||||
|
### Tujuan
|
||||||
|
|
||||||
|
Aplikasi digunakan untuk:
|
||||||
|
|
||||||
|
- menampilkan harga cabai berdasarkan pasar di Kabupaten Jember;
|
||||||
|
- menampilkan lokasi pasar melalui peta digital;
|
||||||
|
- menghasilkan prediksi harga menggunakan model LSTM yang sudah dilatih;
|
||||||
|
- membantu pengguna membandingkan harga aktual dan hasil forecast;
|
||||||
|
- menyediakan panel admin untuk memperbarui data secara manual;
|
||||||
|
- menyediakan preprocessing terbatas untuk menangani gap data;
|
||||||
|
- mencatat riwayat scraping dan proses pengolahan data.
|
||||||
|
|
||||||
|
Aplikasi ini bukan marketplace dan tidak melakukan transaksi jual beli. Fokusnya
|
||||||
|
adalah informasi harga, visualisasi lokasi, dan prediksi harga.
|
||||||
|
|
||||||
|
### Komoditas
|
||||||
|
|
||||||
|
Sistem mendukung tiga komoditas:
|
||||||
|
|
||||||
|
| Slug API | Nama |
|
||||||
|
|---|---|
|
||||||
|
| `rawit` | Cabai Rawit |
|
||||||
|
| `merah` | Cabai Merah Besar |
|
||||||
|
| `keriting` | Cabai Merah Keriting |
|
||||||
|
|
||||||
|
### Pasar
|
||||||
|
|
||||||
|
Pasar yang terdaftar:
|
||||||
|
|
||||||
|
- Pasar Tanjung
|
||||||
|
- Pasar Kreongan
|
||||||
|
- Pasar Tegal Besar
|
||||||
|
- Pasar Mangli
|
||||||
|
- Pasar Wirolegi
|
||||||
|
|
||||||
|
Ketersediaan model dapat berbeda untuk setiap komoditas. Saat ini model cabai
|
||||||
|
merah keriting hanya tersedia untuk pasar yang memiliki artifact lengkap.
|
||||||
|
Aplikasi tidak membuat prediksi palsu untuk pasar tanpa model atau data.
|
||||||
|
|
||||||
|
## 2. Teknologi dan tools
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
|
||||||
|
- **Bahasa:** Python 3.9
|
||||||
|
- **Framework:** Flask
|
||||||
|
- **Server development:** Werkzeug melalui `app.run`
|
||||||
|
- **Database driver:** PyMySQL
|
||||||
|
- **Data processing:** pandas dan NumPy
|
||||||
|
- **Machine learning runtime:** TensorFlow 2.10
|
||||||
|
- **Preprocessing dan scaler:** scikit-learn
|
||||||
|
- **Pemuat scaler:** joblib atau pickle
|
||||||
|
- **HTTP scraping:** requests
|
||||||
|
- **Parsing tabel HTML:** pandas dan lxml
|
||||||
|
- **Pengujian:** pytest
|
||||||
|
- **CORS:** flask-cors
|
||||||
|
|
||||||
|
Entry point backend adalah [api/main.py](api/main.py). Aplikasi Flask dibuat
|
||||||
|
melalui application factory `create_app()`.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
|
||||||
|
- **Bahasa:** JavaScript modern dengan JSX
|
||||||
|
- **Framework:** React 19
|
||||||
|
- **Build tool:** Vite
|
||||||
|
- **Lint:** Oxlint
|
||||||
|
- **Peta:** Leaflet dan React Leaflet
|
||||||
|
- **Grafik:** Recharts
|
||||||
|
- **Ikon:** Lucide React
|
||||||
|
- **Styling:** CSS biasa melalui `website/src/Public.css`
|
||||||
|
|
||||||
|
### Database
|
||||||
|
|
||||||
|
- **Database:** MySQL/MariaDB
|
||||||
|
- **Pengelolaan lokal:** phpMyAdmin, XAMPP, Laragon, atau WAMP
|
||||||
|
- **Nama database default:** `prediksicabe`
|
||||||
|
|
||||||
|
### Model dan artifact
|
||||||
|
|
||||||
|
Setiap kombinasi komoditas/pasar dapat memiliki:
|
||||||
|
|
||||||
|
- `model.h5`: model LSTM yang sudah dilatih;
|
||||||
|
- `scaler.pkl`: scaler yang digunakan saat training;
|
||||||
|
- `config.json`: metadata model dan konfigurasi training.
|
||||||
|
|
||||||
|
API memuat artifact satu kali saat startup. API tidak menjalankan training ulang
|
||||||
|
ketika aplikasi berjalan.
|
||||||
|
|
||||||
|
## 3. Struktur proyek
|
||||||
|
|
||||||
|
Struktur utama:
|
||||||
|
|
||||||
|
```text
|
||||||
|
C:\PrediksiHargaCabe\
|
||||||
|
├── api\
|
||||||
|
│ ├── main.py
|
||||||
|
│ ├── requirements.txt
|
||||||
|
│ ├── restart_backend.ps1
|
||||||
|
│ ├── README.md
|
||||||
|
│ ├── app\
|
||||||
|
│ │ ├── __init__.py
|
||||||
|
│ │ ├── config.py
|
||||||
|
│ │ ├── errors.py
|
||||||
|
│ │ ├── api\
|
||||||
|
│ │ │ └── routes.py
|
||||||
|
│ │ ├── data\
|
||||||
|
│ │ │ ├── markets.json
|
||||||
|
│ │ │ └── repository.py
|
||||||
|
│ │ └── services\
|
||||||
|
│ │ ├── forecaster.py
|
||||||
|
│ │ ├── model_registry.py
|
||||||
|
│ │ ├── prediction_service.py
|
||||||
|
│ │ └── siskaperbapo_scraper.py
|
||||||
|
│ ├── models\
|
||||||
|
│ │ ├── rawit\
|
||||||
|
│ │ ├── merah\
|
||||||
|
│ │ └── keriting\
|
||||||
|
│ └── tests\
|
||||||
|
├── database\
|
||||||
|
│ └── panen_harga.sql
|
||||||
|
├── website\
|
||||||
|
│ ├── package.json
|
||||||
|
│ └── src\
|
||||||
|
│ ├── App.jsx
|
||||||
|
│ ├── Public.css
|
||||||
|
│ └── services\
|
||||||
|
│ └── api.js
|
||||||
|
├── RINGKASAN_FITUR.md
|
||||||
|
└── REFERENSI_APLIKASI.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Arsitektur aplikasi
|
||||||
|
|
||||||
|
```text
|
||||||
|
Pengguna
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
React/Vite frontend :5173
|
||||||
|
│ HTTP/JSON + session cookie
|
||||||
|
▼
|
||||||
|
Flask REST API :8000
|
||||||
|
├── PredictionService
|
||||||
|
├── ModelRegistry
|
||||||
|
├── PriceRepository
|
||||||
|
└── SiskaperbapoScraper
|
||||||
|
│
|
||||||
|
├── Artifact model LSTM
|
||||||
|
└── MySQL prediksicabe
|
||||||
|
```
|
||||||
|
|
||||||
|
Frontend tidak mengakses database secara langsung. Semua data diambil melalui
|
||||||
|
service layer pada [website/src/services/api.js](website/src/services/api.js).
|
||||||
|
Backend bertanggung jawab atas validasi, akses database, scraping, preprocessing,
|
||||||
|
dan inference model.
|
||||||
|
|
||||||
|
## 5. Alur data utama
|
||||||
|
|
||||||
|
### 5.1 Data penelitian awal
|
||||||
|
|
||||||
|
Data CSV penelitian dapat dimasukkan satu kali ke MySQL menggunakan script import.
|
||||||
|
Harga CSV dikonversi ke Rupiah penuh sebelum disimpan.
|
||||||
|
|
||||||
|
```text
|
||||||
|
CSV penelitian
|
||||||
|
↓
|
||||||
|
import_csv_to_mysql.py
|
||||||
|
↓
|
||||||
|
price_history
|
||||||
|
↓
|
||||||
|
PriceRepository
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 Data scraping manual
|
||||||
|
|
||||||
|
Scraping tidak berjalan otomatis dan tidak memiliki scheduler.
|
||||||
|
|
||||||
|
```text
|
||||||
|
Admin memilih komoditas dan tanggal
|
||||||
|
↓
|
||||||
|
POST /api/v1/admin/scraping/jobs
|
||||||
|
↓
|
||||||
|
Background thread
|
||||||
|
↓
|
||||||
|
SISKAPERBAPO
|
||||||
|
↓
|
||||||
|
Normalisasi harga
|
||||||
|
↓
|
||||||
|
price_history
|
||||||
|
↓
|
||||||
|
scraping_logs
|
||||||
|
```
|
||||||
|
|
||||||
|
Data baru dicatat menggunakan `scraping_log_id`, sehingga satu riwayat scraping
|
||||||
|
dapat ditelusuri dan dihapus bersama data yang dihasilkannya.
|
||||||
|
|
||||||
|
### 5.3 Preprocessing
|
||||||
|
|
||||||
|
Preprocessing manual dapat dijalankan terhadap satu log scraping yang dipilih.
|
||||||
|
Metode yang digunakan adalah interpolasi linear untuk gap internal yang memenuhi
|
||||||
|
batas sistem. Gap di awal/akhir atau gap terlalu panjang tidak diisi secara
|
||||||
|
sembarangan dan dilaporkan sebagai warning.
|
||||||
|
|
||||||
|
```text
|
||||||
|
Pilih riwayat scraping
|
||||||
|
↓
|
||||||
|
POST /api/v1/admin/preprocessing/jobs
|
||||||
|
↓
|
||||||
|
Pemeriksaan gap
|
||||||
|
↓
|
||||||
|
Interpolasi linear yang diizinkan
|
||||||
|
↓
|
||||||
|
price_history + preprocessing_runs
|
||||||
|
```
|
||||||
|
|
||||||
|
Repository juga melakukan interpolasi internal terbatas ketika membaca series
|
||||||
|
untuk inference. Perilaku ini berfungsi sebagai validasi/fallback agar gap kecil
|
||||||
|
tidak langsung membuat prediksi gagal.
|
||||||
|
|
||||||
|
### 5.4 Inference prediksi
|
||||||
|
|
||||||
|
Alur inference:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Data historis terbaru dari database
|
||||||
|
↓
|
||||||
|
Pilih data valid terakhir sesuai time_step
|
||||||
|
↓
|
||||||
|
Konversi input ke satuan ribuan untuk model
|
||||||
|
↓
|
||||||
|
Transform dengan scaler.pkl
|
||||||
|
↓
|
||||||
|
Model H5 menghasilkan prediksi
|
||||||
|
↓
|
||||||
|
Inverse transform
|
||||||
|
↓
|
||||||
|
Konversi kembali ke Rupiah penuh
|
||||||
|
↓
|
||||||
|
Recursive forecast 1, 3, atau 7 hari
|
||||||
|
↓
|
||||||
|
Response API
|
||||||
|
```
|
||||||
|
|
||||||
|
Data scraping baru tidak melatih ulang model. Data tersebut hanya menjadi data
|
||||||
|
aktual terbaru yang digunakan sebagai input inference.
|
||||||
|
|
||||||
|
## 6. Satuan harga dan presisi
|
||||||
|
|
||||||
|
Aturan penyimpanan database:
|
||||||
|
|
||||||
|
> Semua harga positif di `price_history` disimpan dalam Rupiah penuh.
|
||||||
|
|
||||||
|
Contoh:
|
||||||
|
|
||||||
|
```text
|
||||||
|
8234 → Rp8.234
|
||||||
|
8500 → Rp8.500
|
||||||
|
9275 → Rp9.275
|
||||||
|
```
|
||||||
|
|
||||||
|
Model dapat bekerja dalam satuan ribuan Rupiah sesuai artifact training. API
|
||||||
|
melakukan penyesuaian satuan ketika inference dan mengembalikan hasil dalam
|
||||||
|
Rupiah penuh.
|
||||||
|
|
||||||
|
Harga prediksi tidak dibulatkan ke ribuan. Frontend menampilkan maksimal dua
|
||||||
|
angka desimal jika hasil model memiliki pecahan.
|
||||||
|
|
||||||
|
Nilai `0`, kosong, dan nilai tidak valid diperlakukan sebagai data kosong, bukan
|
||||||
|
harga aktual yang valid.
|
||||||
|
|
||||||
|
## 7. Fitur dashboard publik
|
||||||
|
|
||||||
|
Dashboard publik berada pada route utama website.
|
||||||
|
|
||||||
|
### 7.1 Pemilihan filter
|
||||||
|
|
||||||
|
Pengguna dapat memilih:
|
||||||
|
|
||||||
|
- jenis cabai;
|
||||||
|
- pasar;
|
||||||
|
- horizon prediksi 1, 3, atau 7 hari.
|
||||||
|
|
||||||
|
Prediksi tidak dijalankan otomatis ketika halaman pertama kali dibuka. Pengguna
|
||||||
|
harus menekan tombol **Prediksi**.
|
||||||
|
|
||||||
|
### 7.2 Data terakhir
|
||||||
|
|
||||||
|
Bagian Data terakhir menampilkan tanggal data terbaru untuk setiap komoditas.
|
||||||
|
Sistem mencari pasar aktif yang memiliki ketersediaan komoditas tersebut, sehingga
|
||||||
|
komoditas tidak bergantung pada pasar yang kosong.
|
||||||
|
|
||||||
|
### 7.3 Peta interaktif
|
||||||
|
|
||||||
|
Peta menggunakan Leaflet dan OpenStreetMap tile layer. Fitur peta:
|
||||||
|
|
||||||
|
- marker lokasi pasar;
|
||||||
|
- marker terpilih;
|
||||||
|
- tooltip nama pasar dan harga;
|
||||||
|
- popup harga aktual, forecast, perubahan, dan tren;
|
||||||
|
- pemusatan peta ke pasar yang dipilih;
|
||||||
|
- mode **Aktual** dan **Forecast**;
|
||||||
|
- legenda status harga;
|
||||||
|
- peringkat pasar berdasarkan harga aktual atau forecast.
|
||||||
|
|
||||||
|
Pasar yang tidak memiliki model atau data tidak dipaksakan menjadi hasil prediksi.
|
||||||
|
|
||||||
|
### 7.4 Grafik dan tabel
|
||||||
|
|
||||||
|
Dashboard menampilkan:
|
||||||
|
|
||||||
|
- riwayat harga aktual;
|
||||||
|
- grafik aktual dan forecast;
|
||||||
|
- tabel forecast per tanggal;
|
||||||
|
- perubahan harga dalam Rupiah;
|
||||||
|
- perubahan persentase;
|
||||||
|
- indikator tren;
|
||||||
|
- indikator harga di bagian bawah dashboard.
|
||||||
|
|
||||||
|
### 7.5 Indikator harga
|
||||||
|
|
||||||
|
Indikator harga mengambil data dari response prediksi, bukan membuat prediksi
|
||||||
|
sendiri di frontend. Indikator menampilkan:
|
||||||
|
|
||||||
|
- Naik;
|
||||||
|
- Turun;
|
||||||
|
- Stabil;
|
||||||
|
- perubahan Rupiah pada langkah terakhir;
|
||||||
|
- perubahan persentase pada langkah terakhir.
|
||||||
|
|
||||||
|
## 8. Fitur panel admin
|
||||||
|
|
||||||
|
Panel admin menggunakan route `#/admin`.
|
||||||
|
|
||||||
|
### 8.1 Autentikasi
|
||||||
|
|
||||||
|
Fitur autentikasi:
|
||||||
|
|
||||||
|
- register;
|
||||||
|
- login;
|
||||||
|
- logout;
|
||||||
|
- session cookie;
|
||||||
|
- role `user` dan `admin`.
|
||||||
|
|
||||||
|
Endpoint admin memerlukan role admin.
|
||||||
|
|
||||||
|
### 8.2 Ringkasan
|
||||||
|
|
||||||
|
Ringkasan menampilkan informasi operasional:
|
||||||
|
|
||||||
|
- jumlah data harga;
|
||||||
|
- jumlah komoditas;
|
||||||
|
- jumlah pasar;
|
||||||
|
- status pembaruan terakhir;
|
||||||
|
- jumlah data terbaru dari scraping terakhir.
|
||||||
|
|
||||||
|
### 8.3 Dataset harga
|
||||||
|
|
||||||
|
Halaman Dataset harga menyediakan:
|
||||||
|
|
||||||
|
- tabel data harga;
|
||||||
|
- filter komoditas;
|
||||||
|
- filter pasar;
|
||||||
|
- pagination;
|
||||||
|
- harga dan tanggal data.
|
||||||
|
|
||||||
|
### 8.4 Perbarui data
|
||||||
|
|
||||||
|
Fitur Perbarui data menyediakan:
|
||||||
|
|
||||||
|
- pilihan komoditas scraping;
|
||||||
|
- tanggal mulai dan tanggal akhir;
|
||||||
|
- progress bar scraping;
|
||||||
|
- log riwayat scraping;
|
||||||
|
- jenis cabai pada setiap log;
|
||||||
|
- status scraping;
|
||||||
|
- jumlah data baru;
|
||||||
|
- detail data yang masuk;
|
||||||
|
- filter detail berdasarkan komoditas dan pasar;
|
||||||
|
- pagination;
|
||||||
|
- tombol hapus log;
|
||||||
|
- modal konfirmasi penghapusan.
|
||||||
|
|
||||||
|
### 8.5 Preprocessing
|
||||||
|
|
||||||
|
Section preprocessing berada di bagian bawah halaman Perbarui data. Admin dapat:
|
||||||
|
|
||||||
|
1. memilih satu riwayat scraping;
|
||||||
|
2. melihat jenis cabai dan jumlah data pada dropdown;
|
||||||
|
3. menjalankan preprocessing;
|
||||||
|
4. melihat progress;
|
||||||
|
5. melihat jumlah baris yang diisi;
|
||||||
|
6. melihat warning gap yang tidak dapat diisi.
|
||||||
|
|
||||||
|
## 9. Endpoint API
|
||||||
|
|
||||||
|
Prefix API:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/api/v1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Endpoint publik
|
||||||
|
|
||||||
|
| Method | Endpoint | Keterangan |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET` | `/` | Status API dasar |
|
||||||
|
| `GET` | `/api/v1/health` | Status API dan artifact model |
|
||||||
|
| `GET` | `/api/v1/commodities` | Daftar komoditas tersedia |
|
||||||
|
| `GET` | `/api/v1/markets` | Daftar pasar dan ketersediaan komoditas |
|
||||||
|
| `GET` | `/api/v1/history` | Riwayat harga |
|
||||||
|
| `GET` | `/api/v1/predict` | Prediksi satu pasar |
|
||||||
|
| `GET` | `/api/v1/predict/map` | Prediksi ringkas seluruh pasar |
|
||||||
|
|
||||||
|
Contoh:
|
||||||
|
|
||||||
|
```text
|
||||||
|
GET /api/v1/history?commodity=rawit&market=pasar_tanjung&days=30
|
||||||
|
GET /api/v1/predict?commodity=rawit&market=pasar_tanjung&horizon=7
|
||||||
|
GET /api/v1/predict/map?commodity=rawit&horizon=7
|
||||||
|
```
|
||||||
|
|
||||||
|
### Endpoint autentikasi
|
||||||
|
|
||||||
|
| Method | Endpoint | Keterangan |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET` | `/api/v1/auth/me` | User session saat ini |
|
||||||
|
| `POST` | `/api/v1/auth/register` | Registrasi user |
|
||||||
|
| `POST` | `/api/v1/auth/login` | Login |
|
||||||
|
| `POST` | `/api/v1/auth/logout` | Logout |
|
||||||
|
|
||||||
|
### Endpoint admin
|
||||||
|
|
||||||
|
| Method | Endpoint | Keterangan |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET` | `/api/v1/admin/overview` | Ringkasan operasional |
|
||||||
|
| `GET` | `/api/v1/admin/models` | Metadata/status model |
|
||||||
|
| `GET` | `/api/v1/admin/scraping/logs` | Riwayat scraping |
|
||||||
|
| `GET` | `/api/v1/admin/scraping/logs/<id>/records` | Detail data dari log |
|
||||||
|
| `POST` | `/api/v1/admin/scraping/jobs` | Memulai scraping |
|
||||||
|
| `GET` | `/api/v1/admin/scraping/jobs/<id>` | Status scraping |
|
||||||
|
| `DELETE` | `/api/v1/admin/scraping/logs/<id>` | Hapus log dan data terkait |
|
||||||
|
| `DELETE` | `/api/v1/admin/scraping/history` | Hapus history scraping |
|
||||||
|
| `POST` | `/api/v1/admin/preprocessing/jobs` | Memulai preprocessing |
|
||||||
|
| `GET` | `/api/v1/admin/preprocessing/jobs/<id>` | Status preprocessing |
|
||||||
|
| `GET` | `/api/v1/prices` | Dataset harga dengan pagination |
|
||||||
|
|
||||||
|
Format error API:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": {
|
||||||
|
"code": "ERROR_CODE",
|
||||||
|
"message": "Pesan untuk pengguna",
|
||||||
|
"details": {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Struktur database
|
||||||
|
|
||||||
|
Schema lengkap tersedia pada [database/panen_harga.sql](database/panen_harga.sql).
|
||||||
|
|
||||||
|
### Tabel utama
|
||||||
|
|
||||||
|
| Tabel | Fungsi |
|
||||||
|
|---|---|
|
||||||
|
| `commodities` | Master jenis cabai |
|
||||||
|
| `markets` | Master pasar dan koordinat |
|
||||||
|
| `price_history` | Harga aktual per komoditas, pasar, dan tanggal |
|
||||||
|
| `scraping_logs` | Riwayat proses scraping |
|
||||||
|
| `preprocessing_runs` | Riwayat proses preprocessing |
|
||||||
|
| `users` | Akun dan role |
|
||||||
|
| `training_runs` | Metadata training jika dikelola di luar API |
|
||||||
|
| `model_versions` | Metadata versi model |
|
||||||
|
| `predictions` | Struktur penyimpanan forecast jika digunakan |
|
||||||
|
|
||||||
|
Relasi penting:
|
||||||
|
|
||||||
|
```text
|
||||||
|
commodities 1 ─── * price_history
|
||||||
|
markets 1 ─── * price_history
|
||||||
|
scraping_logs 1 ─── * price_history
|
||||||
|
commodities 1 ─── * training_runs
|
||||||
|
commodities 1 ─── * model_versions
|
||||||
|
```
|
||||||
|
|
||||||
|
Kunci unik `price_history` adalah kombinasi:
|
||||||
|
|
||||||
|
```text
|
||||||
|
commodity_id + market_id + price_date
|
||||||
|
```
|
||||||
|
|
||||||
|
Karena itu scraping tanggal yang sama akan memperbarui data yang sudah ada,
|
||||||
|
bukan membuat duplikat baru.
|
||||||
|
|
||||||
|
## 11. Konfigurasi environment
|
||||||
|
|
||||||
|
Variabel penting backend:
|
||||||
|
|
||||||
|
| Variabel | Default | Fungsi |
|
||||||
|
|---|---|---|
|
||||||
|
| `DATA_SOURCE` | `mysql` | Sumber data aktif |
|
||||||
|
| `MYSQL_HOST` | `127.0.0.1` | Host MySQL |
|
||||||
|
| `MYSQL_PORT` | `3306` | Port MySQL |
|
||||||
|
| `MYSQL_DATABASE` | `prediksicabe` | Nama database |
|
||||||
|
| `MYSQL_USER` | `root` | User database |
|
||||||
|
| `MYSQL_PASSWORD` | kosong | Password database |
|
||||||
|
| `MODELS_DIR` | `api/models` | Folder artifact |
|
||||||
|
| `DATA_DIR` | folder api | Folder CSV |
|
||||||
|
| `PRICE_UNIT_SCALE` | `1000` | Skala CSV lama ke Rupiah |
|
||||||
|
| `ALLOWED_HORIZONS` | `1,3,7` | Horizon yang diizinkan |
|
||||||
|
| `MAX_HORIZON` | `7` | Batas horizon |
|
||||||
|
| `CORS_ORIGINS` | localhost:5173 | Origin frontend |
|
||||||
|
| `STALE_DATA_DAYS` | `30` | Ambang data lama |
|
||||||
|
| `TREND_THRESHOLD_PCT` | `1` | Ambang status tren |
|
||||||
|
| `SISKAPERBAPO_TIMEOUT` | `30` | Timeout request scraper |
|
||||||
|
|
||||||
|
Untuk produksi, ganti `SECRET_KEY` default dan batasi `CORS_ORIGINS` hanya pada
|
||||||
|
origin frontend yang dipercaya.
|
||||||
|
|
||||||
|
## 12. Cara menjalankan aplikasi
|
||||||
|
|
||||||
|
### 12.1 Menjalankan backend
|
||||||
|
|
||||||
|
Gunakan environment Conda yang sudah tersedia:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
C:\Users\alfia\anaconda3\envs\tf_gpu39\python.exe main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Atau gunakan script restart:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
powershell -ExecutionPolicy Bypass -File .\restart_backend.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
Backend berjalan di:
|
||||||
|
|
||||||
|
```text
|
||||||
|
http://127.0.0.1:8000
|
||||||
|
```
|
||||||
|
|
||||||
|
### 12.2 Menjalankan frontend
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\website
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Frontend Vite biasanya berjalan di:
|
||||||
|
|
||||||
|
```text
|
||||||
|
http://localhost:5173
|
||||||
|
```
|
||||||
|
|
||||||
|
### 12.3 Build frontend
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\website
|
||||||
|
npm run lint
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
### 12.4 Pengujian backend
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
C:\Users\alfia\anaconda3\envs\tf_gpu39\python.exe -m pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
## 13. Keamanan dan batasan operasional
|
||||||
|
|
||||||
|
- Endpoint admin dilindungi autentikasi dan role admin.
|
||||||
|
- Password disimpan dalam bentuk hash.
|
||||||
|
- Frontend tidak terhubung langsung ke database.
|
||||||
|
- Scheduler scraping otomatis belum diaktifkan.
|
||||||
|
- Training ulang otomatis belum diaktifkan.
|
||||||
|
- API tidak menjalankan `model.fit()`.
|
||||||
|
- Data input scraping tidak boleh dianggap sebagai hasil evaluasi model baru.
|
||||||
|
- Data terbaru hasil scraping adalah data operasional untuk inference.
|
||||||
|
- Jangan mengklaim model sebagai “global optimum”. Gunakan istilah:
|
||||||
|
“konfigurasi terbaik yang diperoleh dari skema eksperimen yang diterapkan pada
|
||||||
|
dataset penelitian.”
|
||||||
|
|
||||||
|
## 14. Batasan model dan interpretasi hasil
|
||||||
|
|
||||||
|
### Recursive forecasting
|
||||||
|
|
||||||
|
Prediksi lebih dari satu hari menggunakan hasil prediksi sebelumnya sebagai input
|
||||||
|
berikutnya. Akibatnya, error dapat terakumulasi pada horizon yang lebih panjang.
|
||||||
|
|
||||||
|
### Ketersediaan artifact
|
||||||
|
|
||||||
|
Tidak semua komoditas harus tersedia pada semua pasar. Kombinasi tanpa artifact
|
||||||
|
lengkap ditampilkan sebagai tidak tersedia.
|
||||||
|
|
||||||
|
### Data lama
|
||||||
|
|
||||||
|
API dapat memberikan warning `data_stale` apabila tanggal data terakhir melewati
|
||||||
|
ambang `STALE_DATA_DAYS`.
|
||||||
|
|
||||||
|
### Model deployment tidak sama dengan training ulang
|
||||||
|
|
||||||
|
Scraping baru tidak mengubah bobot model. Jika penelitian memerlukan model baru,
|
||||||
|
training harus dilakukan melalui proses eksperimen terpisah, divalidasi, lalu
|
||||||
|
artifact baru dideploy secara sadar.
|
||||||
|
|
||||||
|
## 15. Cara menjelaskan aplikasi saat presentasi
|
||||||
|
|
||||||
|
Urutan penjelasan yang disarankan:
|
||||||
|
|
||||||
|
1. Jelaskan masalah: harga cabai berbeda antar pasar dan berubah dari waktu ke
|
||||||
|
waktu.
|
||||||
|
2. Jelaskan sumber data: dataset penelitian dan data SISKAPERBAPO.
|
||||||
|
3. Jelaskan tiga komoditas dan pasar yang diamati.
|
||||||
|
4. Tunjukkan peta digital pasar.
|
||||||
|
5. Pilih komoditas dan pasar.
|
||||||
|
6. Pilih horizon prediksi.
|
||||||
|
7. Tekan tombol Prediksi.
|
||||||
|
8. Jelaskan harga aktual terakhir, forecast, grafik, tabel, dan indikator tren.
|
||||||
|
9. Jelaskan bahwa model menggunakan artifact LSTM yang sudah divalidasi.
|
||||||
|
10. Jelaskan bahwa scraping memperbarui data aktual, bukan melatih ulang model.
|
||||||
|
11. Tunjukkan panel admin untuk log scraping dan preprocessing.
|
||||||
|
12. Jelaskan keterbatasan model dan pasar yang artifact-nya belum tersedia.
|
||||||
|
|
||||||
|
## 16. Ringkasan satu paragraf
|
||||||
|
|
||||||
|
Panen Harga adalah aplikasi web berbasis React dan Flask yang menggunakan model
|
||||||
|
LSTM tersimpan untuk memprediksi harga tiga komoditas cabai di pasar-pasar
|
||||||
|
Kabupaten Jember. Data harga disimpan dalam MySQL menggunakan Rupiah penuh,
|
||||||
|
sedangkan model dan scaler dimuat dari artifact saat backend dimulai. Pengguna
|
||||||
|
dapat memilih komoditas, pasar, dan horizon prediksi, kemudian melihat hasilnya
|
||||||
|
melalui peta Leaflet, ranking pasar, grafik riwayat, grafik forecast, tabel
|
||||||
|
prediksi, dan indikator tren. Admin dapat melakukan scraping manual dari
|
||||||
|
SISKAPERBAPO, memantau progress, melihat log, menghapus hasil scraping, dan
|
||||||
|
menjalankan preprocessing terbatas. Sistem tidak melakukan scheduler maupun
|
||||||
|
training ulang otomatis sehingga artifact model penelitian tetap terkendali.
|
||||||
|
|
@ -0,0 +1,344 @@
|
||||||
|
# Ringkasan Fitur Prediksi Harga Cabai Jember
|
||||||
|
|
||||||
|
Dokumen ini merangkum fitur yang terdeteksi pada seluruh proyek `PrediksiHargaCabe`, terutama folder `api` dan `website`.
|
||||||
|
|
||||||
|
## 1. Gambaran Umum
|
||||||
|
|
||||||
|
Proyek terdiri dari dua bagian utama:
|
||||||
|
|
||||||
|
- **Backend `api`**: REST API berbasis Flask untuk membaca data harga, memuat model LSTM tersimpan, menghasilkan prediksi, serta menyediakan data untuk peta.
|
||||||
|
- **Frontend `website`**: aplikasi React + Vite untuk menampilkan peta pasar, harga aktual, prediksi, riwayat harga, dan indikator tren.
|
||||||
|
|
||||||
|
Walaupun beberapa instruksi atau dokumentasi lama menyebut FastAPI, implementasi backend yang aktif saat ini menggunakan **Flask**. Entry point-nya adalah `api/main.py` dan aplikasi dibuat melalui `app.create_app()`.
|
||||||
|
|
||||||
|
## 2. Fitur Backend
|
||||||
|
|
||||||
|
### 2.1 Status API
|
||||||
|
|
||||||
|
- `GET /`
|
||||||
|
- Mengecek bahwa API aktif.
|
||||||
|
- Mengembalikan pesan status dalam bahasa Indonesia.
|
||||||
|
- `GET /api/v1/health`
|
||||||
|
- Menampilkan status kesehatan API.
|
||||||
|
- Menampilkan jumlah model yang berhasil dimuat.
|
||||||
|
- Menampilkan daftar model yang berhasil dimuat per komoditas.
|
||||||
|
- Menampilkan daftar artefak model yang dilewati beserta alasannya.
|
||||||
|
|
||||||
|
### 2.2 Informasi Komoditas dan Pasar
|
||||||
|
|
||||||
|
- `GET /api/v1/commodities`
|
||||||
|
- Mengembalikan komoditas yang memiliki model tersedia.
|
||||||
|
- Menyertakan nama komoditas dan jumlah pasar yang tersedia.
|
||||||
|
- `GET /api/v1/markets`
|
||||||
|
- Mengembalikan lima pasar di Kabupaten Jember beserta koordinatnya.
|
||||||
|
- Dapat difilter dengan query `commodity`.
|
||||||
|
- Menandai komoditas yang tersedia untuk setiap pasar.
|
||||||
|
|
||||||
|
Komoditas yang didefinisikan backend:
|
||||||
|
|
||||||
|
| Slug | Nama |
|
||||||
|
|---|---|
|
||||||
|
| `rawit` | Cabai Rawit |
|
||||||
|
| `merah` | Cabai Merah Besar |
|
||||||
|
| `keriting` | Cabai Merah Keriting |
|
||||||
|
|
||||||
|
Pasar yang didefinisikan backend:
|
||||||
|
|
||||||
|
- Pasar Tanjung
|
||||||
|
- Pasar Kreongan
|
||||||
|
- Pasar Tegal Besar
|
||||||
|
- Pasar Mangli
|
||||||
|
- Pasar Wirolegi
|
||||||
|
|
||||||
|
### 2.3 Riwayat Harga
|
||||||
|
|
||||||
|
- `GET /api/v1/history?commodity={komoditas}&market={pasar}&days={jumlah_hari}`
|
||||||
|
- Mengambil riwayat harga dari CSV.
|
||||||
|
- Nilai `days` default-nya 90 hari dan dibatasi antara 1 sampai 365 hari.
|
||||||
|
- Data tanggal diurutkan dan data tanggal duplikat menggunakan nilai terakhir.
|
||||||
|
- Nilai kosong atau tidak valid diproses sebagai data yang tidak tersedia.
|
||||||
|
- Nilai nol diperlakukan sebagai data kosong, lalu nilai kosong di tengah rangkaian dapat diinterpolasi secara linear hingga dua titik.
|
||||||
|
- Nilai harga dikonversi menggunakan `PRICE_UNIT_SCALE`, default `1000`.
|
||||||
|
|
||||||
|
### 2.4 Prediksi Multi-Langkah
|
||||||
|
|
||||||
|
- `GET /api/v1/predict?commodity={komoditas}&market={pasar}&horizon={hari}`
|
||||||
|
- Horizon yang tersedia secara default: 1, 7, 24, dan 30 hari.
|
||||||
|
- Prediksi menggunakan pendekatan **recursive multi-step forecasting**.
|
||||||
|
- Hasil prediksi mencakup:
|
||||||
|
- harga aktual terakhir;
|
||||||
|
- tanggal prediksi;
|
||||||
|
- harga setiap langkah;
|
||||||
|
- perubahan harga dalam rupiah;
|
||||||
|
- perubahan persentase;
|
||||||
|
- harga minimum, maksimum, dan rata-rata;
|
||||||
|
- tren `up`, `down`, atau `flat`;
|
||||||
|
- umur data;
|
||||||
|
- peringatan data lama atau prediksi di luar rentang training;
|
||||||
|
- metadata metode dan `time_step`.
|
||||||
|
- Hasil prediksi disimpan dalam cache berdasarkan komoditas, pasar, dan tanggal data terakhir.
|
||||||
|
|
||||||
|
### 2.5 Prediksi untuk Peta
|
||||||
|
|
||||||
|
- `GET /api/v1/predict/map?commodity={komoditas}&horizon={hari}`
|
||||||
|
- Menghasilkan ringkasan prediksi untuk seluruh pasar.
|
||||||
|
- Setiap lokasi dapat berisi:
|
||||||
|
- koordinat pasar;
|
||||||
|
- ketersediaan model;
|
||||||
|
- harga aktual terakhir;
|
||||||
|
- harga prediksi;
|
||||||
|
- perubahan harga;
|
||||||
|
- persentase perubahan;
|
||||||
|
- arah tren.
|
||||||
|
- Jika model atau data suatu pasar tidak tersedia, lokasi tetap dikembalikan dengan status tidak tersedia dan alasan kegagalan.
|
||||||
|
|
||||||
|
### 2.6 Pemuatan Model dan Scaler
|
||||||
|
|
||||||
|
- Model dan scaler dimuat ketika aplikasi dibuat melalui `ModelRegistry`.
|
||||||
|
- Model Keras dimuat menggunakan `compile=False`.
|
||||||
|
- Scaler dimuat menggunakan `joblib`, dengan fallback ke `pickle`.
|
||||||
|
- Setiap konfigurasi model dapat menentukan nama file model dan scaler melalui `config.json`.
|
||||||
|
- Path artefak divalidasi agar tidak keluar dari folder pasar.
|
||||||
|
- Model yang tidak lengkap atau gagal dimuat dilewati dan dicatat pada informasi health.
|
||||||
|
- API tidak melakukan training atau `model.fit()` ketika melayani prediksi.
|
||||||
|
|
||||||
|
Artefak yang tersedia:
|
||||||
|
|
||||||
|
- **Cabai Rawit**: model untuk lima pasar.
|
||||||
|
- **Cabai Merah Besar**: model untuk lima pasar.
|
||||||
|
- **Cabai Merah Keriting**: model untuk Pasar Kreongan, Pasar Mangli, dan Pasar Wirolegi.
|
||||||
|
- Setiap kombinasi model menggunakan file model Keras `.h5`, scaler `.pkl`, dan konfigurasi `.json` jika lengkap.
|
||||||
|
|
||||||
|
### 2.7 Validasi dan Error Handling
|
||||||
|
|
||||||
|
Backend memiliki format error terstruktur:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": {
|
||||||
|
"code": "ERROR_CODE",
|
||||||
|
"message": "Pesan kesalahan",
|
||||||
|
"details": {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Validasi yang tersedia meliputi:
|
||||||
|
|
||||||
|
- komoditas tidak dikenal;
|
||||||
|
- pasar tidak dikenal;
|
||||||
|
- model tidak tersedia;
|
||||||
|
- parameter query wajib;
|
||||||
|
- horizon tidak valid;
|
||||||
|
- jumlah data historis tidak mencukupi;
|
||||||
|
- file CSV tidak ditemukan;
|
||||||
|
- kolom tanggal atau kolom pasar tidak ditemukan;
|
||||||
|
- kesalahan internal server.
|
||||||
|
|
||||||
|
### 2.8 CORS dan Konfigurasi
|
||||||
|
|
||||||
|
Konfigurasi backend dapat diubah melalui environment variable:
|
||||||
|
|
||||||
|
- `MODELS_DIR`: lokasi folder model;
|
||||||
|
- `DATA_DIR`: lokasi CSV;
|
||||||
|
- `PRICE_UNIT_SCALE`: skala harga, default `1000`;
|
||||||
|
- `ALLOWED_HORIZONS`: daftar horizon yang dipisahkan koma, default `1,3,7`;
|
||||||
|
- `MAX_HORIZON`: batas horizon maksimum, default `7`;
|
||||||
|
- `CORS_ORIGINS`: origin frontend yang diizinkan;
|
||||||
|
- `STALE_DATA_DAYS`: ambang data dianggap lama;
|
||||||
|
- `TREND_THRESHOLD_PCT`: ambang perubahan untuk menentukan tren.
|
||||||
|
|
||||||
|
Nilai default `CORS_ORIGINS` saat ini adalah `*` untuk kemudahan pengembangan. Untuk deployment produksi, sebaiknya diisi dengan daftar origin frontend yang spesifik.
|
||||||
|
|
||||||
|
## 3. Fitur Frontend
|
||||||
|
|
||||||
|
### 3.1 Dashboard Publik
|
||||||
|
|
||||||
|
Frontend menampilkan dashboard bernama **Panen Harga** dengan fokus pada harga cabai Kabupaten Jember.
|
||||||
|
|
||||||
|
Fitur dashboard:
|
||||||
|
|
||||||
|
- indikator koneksi API;
|
||||||
|
- pilihan komoditas;
|
||||||
|
- pilihan horizon 1, 7, 24, atau 30 hari;
|
||||||
|
- pencarian pasar;
|
||||||
|
- notifikasi kesalahan dan tombol coba lagi;
|
||||||
|
- tampilan tanggal data terakhir;
|
||||||
|
- tata letak responsif untuk layar desktop dan mobile.
|
||||||
|
|
||||||
|
### 3.2 Peta Interaktif
|
||||||
|
|
||||||
|
- Menggunakan React Leaflet dan OpenStreetMap.
|
||||||
|
- Menampilkan marker untuk lokasi pasar.
|
||||||
|
- Marker dapat dipilih untuk membuka detail lokasi.
|
||||||
|
- Peta otomatis memusatkan tampilan ke pasar yang dipilih.
|
||||||
|
- Marker dibedakan berdasarkan status terpilih dan ketersediaan model/data.
|
||||||
|
- Popup marker menampilkan nama pasar dan prediksi harga.
|
||||||
|
- Daftar pasar di samping peta dapat digunakan sebagai alternatif pemilihan lokasi.
|
||||||
|
|
||||||
|
### 3.3 Panel Detail Pasar
|
||||||
|
|
||||||
|
Untuk pasar yang dipilih, frontend menampilkan:
|
||||||
|
|
||||||
|
- nama pasar;
|
||||||
|
- koordinat;
|
||||||
|
- komoditas;
|
||||||
|
- indikator tren naik, turun, atau datar;
|
||||||
|
- prediksi pada horizon yang dipilih;
|
||||||
|
- harga aktual terakhir;
|
||||||
|
- tanggal data terakhir;
|
||||||
|
- tabel prediksi per langkah;
|
||||||
|
- perubahan harga per langkah;
|
||||||
|
- peringatan jika data sudah lama atau prediksi berada di luar rentang training.
|
||||||
|
|
||||||
|
Untuk horizon lebih dari tujuh hari, tabel visual menampilkan tujuh langkah pertama dan mengarahkan pengguna ke endpoint API untuk data lengkap.
|
||||||
|
|
||||||
|
### 3.4 Riwayat Harga
|
||||||
|
|
||||||
|
- Menampilkan ringkasan harga aktual terbaru.
|
||||||
|
- Menampilkan riwayat 30 hari untuk pasar yang dipilih.
|
||||||
|
- Menggunakan visualisasi sparkline sederhana berbasis elemen HTML/CSS.
|
||||||
|
- Menampilkan tanggal awal dan tanggal akhir riwayat.
|
||||||
|
|
||||||
|
### 3.5 State dan Loading UI
|
||||||
|
|
||||||
|
Frontend memiliki state untuk:
|
||||||
|
|
||||||
|
- daftar komoditas;
|
||||||
|
- daftar pasar;
|
||||||
|
- prediksi peta;
|
||||||
|
- pasar terpilih;
|
||||||
|
- horizon;
|
||||||
|
- pencarian pasar;
|
||||||
|
- detail prediksi;
|
||||||
|
- riwayat harga;
|
||||||
|
- status loading;
|
||||||
|
- status error;
|
||||||
|
- filter mobile.
|
||||||
|
|
||||||
|
Tersedia tampilan loading saat menghubungkan ke API dan tampilan kosong ketika data API belum tersedia.
|
||||||
|
|
||||||
|
## 4. Alur Integrasi Frontend dan Backend
|
||||||
|
|
||||||
|
1. Frontend memanggil `/commodities` dan `/markets` ketika halaman dibuka.
|
||||||
|
2. Frontend memanggil `/predict/map` ketika komoditas atau horizon berubah.
|
||||||
|
3. Frontend memanggil `/predict` dan `/history` ketika pasar, komoditas, atau horizon berubah.
|
||||||
|
4. Respons API dipetakan ke marker, daftar pasar, panel detail, dan riwayat harga.
|
||||||
|
5. Error dari backend dibaca dari format `error.message` dan ditampilkan kepada pengguna.
|
||||||
|
|
||||||
|
Base URL frontend dapat diubah dengan:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:VITE_API_BASE_URL="http://127.0.0.1:8000/api/v1"
|
||||||
|
```
|
||||||
|
|
||||||
|
Jika environment variable tersebut tidak diatur, frontend menggunakan default:
|
||||||
|
|
||||||
|
```text
|
||||||
|
http://127.0.0.1:8000/api/v1
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Data dan Model
|
||||||
|
|
||||||
|
Data input yang ditemukan di backend:
|
||||||
|
|
||||||
|
- CSV Cabai Rawit;
|
||||||
|
- CSV Cabai Merah Keriting;
|
||||||
|
- CSV Cabai Merah Besar.
|
||||||
|
|
||||||
|
Data tersebut digunakan oleh `CsvPriceRepository`, bukan database relasional.
|
||||||
|
|
||||||
|
Folder model tersimpan di `api/models` dan dikelompokkan berdasarkan komoditas lalu pasar. Model di-load dari file yang dirujuk oleh `config.json`; nama file tidak ditebak secara bebas ketika konfigurasi tersedia.
|
||||||
|
|
||||||
|
## 6. Pengujian
|
||||||
|
|
||||||
|
Backend memiliki pengujian untuk:
|
||||||
|
|
||||||
|
- proses recursive forecasting;
|
||||||
|
- perlindungan agar hasil prediksi tidak negatif;
|
||||||
|
- pemuatan registry dengan artefak tidak lengkap;
|
||||||
|
- kontrak endpoint health;
|
||||||
|
- validasi komoditas;
|
||||||
|
- validasi horizon;
|
||||||
|
- format error API.
|
||||||
|
|
||||||
|
Perintah pengujian:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd api
|
||||||
|
pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
Perintah menjalankan backend:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd api
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Atau dengan Flask CLI:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd api
|
||||||
|
flask --app main:app run --host 127.0.0.1 --port 8000
|
||||||
|
```
|
||||||
|
|
||||||
|
Perintah menjalankan frontend:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd website
|
||||||
|
npm install
|
||||||
|
npm run dev -- --host 127.0.0.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Perintah build dan lint frontend:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd website
|
||||||
|
npm run build
|
||||||
|
npm run lint
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Struktur File Penting
|
||||||
|
|
||||||
|
```text
|
||||||
|
api/
|
||||||
|
├── main.py
|
||||||
|
├── requirements.txt
|
||||||
|
├── app/
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── config.py
|
||||||
|
│ ├── errors.py
|
||||||
|
│ ├── api/
|
||||||
|
│ │ ├── routes.py
|
||||||
|
│ │ └── schemas.py
|
||||||
|
│ ├── data/
|
||||||
|
│ │ ├── markets.json
|
||||||
|
│ │ └── repository.py
|
||||||
|
│ └── services/
|
||||||
|
│ ├── forecaster.py
|
||||||
|
│ ├── model_registry.py
|
||||||
|
│ └── prediction_service.py
|
||||||
|
├── models/
|
||||||
|
└── tests/
|
||||||
|
|
||||||
|
website/
|
||||||
|
├── package.json
|
||||||
|
├── vite.config.js
|
||||||
|
├── index.html
|
||||||
|
└── src/
|
||||||
|
├── App.jsx
|
||||||
|
├── main.jsx
|
||||||
|
├── services/api.js
|
||||||
|
├── data/mockData.js
|
||||||
|
├── Public.css
|
||||||
|
└── assets/
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Catatan Implementasi Saat Ini
|
||||||
|
|
||||||
|
- Backend aktual adalah Flask dengan prefix API `/api/v1`.
|
||||||
|
- Dokumentasi `website/README.md` masih menyebut fase mock data dan integrasi backend belum dilakukan, tetapi kode `website/src/services/api.js` dan `website/src/App.jsx` saat ini sudah memanggil API backend secara langsung.
|
||||||
|
- `website/src/data/mockData.js` masih tersimpan sebagai data contoh/warisan, tetapi bukan sumber data utama pada alur dashboard saat ini.
|
||||||
|
- Folder `node_modules`, cache Python, dan `website/dist` merupakan dependensi atau hasil build; folder tersebut tidak dianggap sebagai fitur aplikasi inti.
|
||||||
|
- Halaman publik yang aktif pada implementasi saat ini adalah satu dashboard React. Rute hash seperti `#/home`, `#/map`, `#/about`, dan `#/admin` yang tertulis di README belum terlihat di komponen `App.jsx`.
|
||||||
|
|
@ -0,0 +1,489 @@
|
||||||
|
DOKUMEN SPESIFIKASI UI/UX
|
||||||
|
Sistem Prediksi Harga Cabai
|
||||||
|
dengan Visualisasi Peta Digital Interaktif di Kabupaten Jember
|
||||||
|
Frontend target: React.js | Backend existing: FastAPI | Focus: UI/UX Phase 1
|
||||||
|
Basis dokumen: Dokumen ini disusun berdasarkan proposal TA "Analisis Model Long Short-Term Memory dalam Sistem Prediksi Harga Cabai dengan Visualisasi Peta Digital Interaktif di Kabupaten Jember", khususnya bagian tujuan sistem, sumber/struktur data, hasil prediksi, peta digital interaktif, rekomendasi, dan pengujian UAT. Proposal menjelaskan bahwa hasil prediksi akan ditampilkan melalui peta wilayah Kabupaten Jember dengan titik pasar dan informasi harga aktual serta hasil prediksi. (Proposal TA, Bab 1 dan Bab 3, terutama pp. 3 - 4 dan 26 - 29).1. Tujuan Dokumen
|
||||||
|
Dokumen ini menjadi blueprint visual dan UX yang dapat langsung dipakai sebagai konteks kerja GitHub Copilot/Copilot Chat saat membangun frontend React. Fokus tahap pertama adalah menghasilkan UI/UX yang rapi, konsisten, responsif, dan mudah diuji menggunakan data mock. Integrasi endpoint FastAPI dilakukan setelah struktur antarmuka dan alur pengguna stabil.Menerjemahkan tujuan TA menjadi struktur halaman dan komponen frontend.
|
||||||
|
Menetapkan design system: warna, tipografi, spacing, card, button, tabel, chart, dan map.
|
||||||
|
Mendefinisikan perilaku pengguna untuk melihat harga aktual, membandingkan pasar, dan melihat prediksi.
|
||||||
|
Menghindari UI yang terlalu teknis pada halaman publik; detail parameter/model ditampilkan di halaman Analisis Model.
|
||||||
|
Menyediakan spesifikasi yang cukup eksplisit untuk digenerate secara bertahap oleh Copilot.
|
||||||
|
2. Posisi Produk dan Prinsip UX
|
||||||
|
Produk diposisikan sebagai dashboard informasi harga cabai berbasis lokasi, bukan marketplace dan bukan dashboard machine learning yang penuh konfigurasi teknis. Keberhasilan UX diukur dari seberapa cepat pengguna menemukan tiga informasi utama: harga aktual, lokasi pasar, dan hasil prediksi.Prinsip
|
||||||
|
Penerapan
|
||||||
|
Map-first
|
||||||
|
Peta Kabupaten Jember adalah elemen visual utama karena peta merupakan cara penyajian inti yang disebut dalam proposal.
|
||||||
|
Data-first
|
||||||
|
Angka harga, perubahan, tanggal pembaruan, dan prediksi harus terlihat lebih dahulu daripada penjelasan teknis.
|
||||||
|
Simple for public users
|
||||||
|
Bahasa UI sederhana dan familiar; hindari istilah machine learning pada dashboard utama.
|
||||||
|
Evidence over decoration
|
||||||
|
Gunakan visual, chart, dan marker untuk memperjelas data, bukan sekadar dekorasi.
|
||||||
|
Consistent
|
||||||
|
Komponen yang sama harus mempertahankan pola warna, ukuran, ikon, spacing, dan interaksi.
|
||||||
|
Responsive
|
||||||
|
Desktop menjadi fokus utama, tetapi layout wajib dapat beradaptasi ke tablet dan mobile.
|
||||||
|
3. Target Pengguna dan Kebutuhan Informasi
|
||||||
|
Kelompok pengguna
|
||||||
|
Kebutuhan utama
|
||||||
|
Informasi yang ditonjolkan
|
||||||
|
Masyarakat/konsumen
|
||||||
|
Mengetahui harga cabai terkini dan kondisi harga antar pasar.
|
||||||
|
Harga aktual, pasar, perubahan harga, peta.
|
||||||
|
Pedagang/pelaku pasar
|
||||||
|
Membandingkan harga pasar dan melihat kecenderungan harga.
|
||||||
|
Harga aktual, prediksi, grafik tren, detail pasar.
|
||||||
|
Pemerintah/instansi
|
||||||
|
Melihat gambaran spasial dan tren harga untuk pemantauan.
|
||||||
|
Peta, sebaran harga, grafik, ringkasan prediksi.
|
||||||
|
4. Information Architecture
|
||||||
|
Struktur awal dibuat ringkas. Fitur administrasi/data pipeline tidak dijadikan menu publik utama karena proposal berfokus pada penyajian hasil prediksi dan peta interaktif. Jika nantinya dibutuhkan area admin, area tersebut dapat dibuat terpisah tanpa mengubah IA publik.SISTEM PREDIKSI HARGA CABAI JEMBER
|
||||||
|
│
|
||||||
|
├── Dashboard
|
||||||
|
│ ├── Ringkasan harga
|
||||||
|
│ ├── Peta interaktif
|
||||||
|
│ ├── Prediksi terpilih
|
||||||
|
│ ├── Grafik tren
|
||||||
|
│ └── Insight
|
||||||
|
│
|
||||||
|
├── Harga Pasar
|
||||||
|
│ ├── Filter jenis cabai
|
||||||
|
│ ├── Filter tanggal
|
||||||
|
│ ├── Tabel harga
|
||||||
|
│ └── Detail pasar
|
||||||
|
│
|
||||||
|
├── Prediksi
|
||||||
|
│ ├── Pilih jenis cabai
|
||||||
|
│ ├── Pilih pasar
|
||||||
|
│ ├── Ringkasan prediksi
|
||||||
|
│ └── Grafik actual vs prediction
|
||||||
|
│
|
||||||
|
├── Analisis Model
|
||||||
|
│ ├── RMSE
|
||||||
|
│ ├── MAE
|
||||||
|
│ ├── Konfigurasi model final
|
||||||
|
│ └── Visualisasi performa
|
||||||
|
│
|
||||||
|
└── Tentang Sistem
|
||||||
|
├── Tujuan
|
||||||
|
├── Sumber data
|
||||||
|
└── Penjelasan singkat LSTM
|
||||||
|
Catatan: Nama menu dapat berubah saat implementasi, tetapi urutan prioritas informasi sebaiknya tetap: Dashboard --> Harga --> Prediksi --> Analisis Model --> Tentang.5. Navigasi dan Routing
|
||||||
|
Route
|
||||||
|
Halaman
|
||||||
|
Tujuan UX
|
||||||
|
/
|
||||||
|
Dashboard
|
||||||
|
Halaman utama untuk memahami kondisi harga secara cepat.
|
||||||
|
/harga
|
||||||
|
Harga Pasar
|
||||||
|
Melihat dan memfilter data harga antar pasar.
|
||||||
|
/prediksi
|
||||||
|
Prediksi
|
||||||
|
Melihat hasil prediksi berdasarkan jenis cabai dan pasar.
|
||||||
|
/analisis
|
||||||
|
Analisis Model
|
||||||
|
Melihat performa model dan konfigurasi eksperimen final.
|
||||||
|
/tentang
|
||||||
|
Tentang Sistem
|
||||||
|
Memberi konteks penelitian dan sumber data.
|
||||||
|
6. Design System
|
||||||
|
6.1 Warna
|
||||||
|
Token
|
||||||
|
Hex
|
||||||
|
Peran
|
||||||
|
Penggunaan
|
||||||
|
Primary
|
||||||
|
#1F6B45
|
||||||
|
Hijau pertanian
|
||||||
|
Navbar, primary button, judul/aksen utama
|
||||||
|
Primary Dark
|
||||||
|
#14532D
|
||||||
|
Hijau gelap
|
||||||
|
Hover, heading penting, teks brand
|
||||||
|
Accent Chili
|
||||||
|
#D94A3D
|
||||||
|
Merah cabai
|
||||||
|
Aksen jenis cabai, kenaikan/perhatian
|
||||||
|
Accent Gold
|
||||||
|
#F2B84B
|
||||||
|
Kuning hangat
|
||||||
|
Highlight ringan, badge informasi
|
||||||
|
Background
|
||||||
|
#F7F8F5
|
||||||
|
Off-white
|
||||||
|
Background halaman
|
||||||
|
Surface
|
||||||
|
#FFFFFF
|
||||||
|
Putih
|
||||||
|
Card, modal, panel
|
||||||
|
Text
|
||||||
|
#1F2937
|
||||||
|
Teks utama
|
||||||
|
Body dan label
|
||||||
|
Muted
|
||||||
|
#6B7280
|
||||||
|
Teks sekunder
|
||||||
|
Metadata, hint, caption
|
||||||
|
Border
|
||||||
|
#E5E7EB
|
||||||
|
Garis pembatas
|
||||||
|
Card/table/filter
|
||||||
|
Positive
|
||||||
|
#2E7D32
|
||||||
|
Perubahan naik/turun yang dipetakan secara semantik
|
||||||
|
Gunakan bersama ikon/teks, bukan warna saja
|
||||||
|
Negative
|
||||||
|
#B42318
|
||||||
|
Perubahan negatif/alert
|
||||||
|
Gunakan bersama ikon/teks, bukan warna saja
|
||||||
|
Aturan warna status: Jangan memakai hijau/merah sebagai satu-satunya penanda makna. Setiap perubahan harga harus disertai angka dan/atau ikon panah agar informasi tetap terbaca tanpa bergantung pada warna.6.2 Tipografi
|
||||||
|
Elemen
|
||||||
|
Ukuran
|
||||||
|
Bobot
|
||||||
|
Catatan
|
||||||
|
Hero title
|
||||||
|
32 - 40 px
|
||||||
|
700
|
||||||
|
Desktop; responsif menjadi 28 - 32 px
|
||||||
|
Section title
|
||||||
|
22 - 26 px
|
||||||
|
700
|
||||||
|
Judul bagian
|
||||||
|
Card number
|
||||||
|
24 - 32 px
|
||||||
|
700
|
||||||
|
Harga/prediksi
|
||||||
|
Body
|
||||||
|
14 - 16 px
|
||||||
|
400
|
||||||
|
Teks utama
|
||||||
|
Caption
|
||||||
|
12 - 13 px
|
||||||
|
400
|
||||||
|
Tanggal, sumber, metadata
|
||||||
|
Font yang disarankan: Inter atau Plus Jakarta Sans. Gunakan satu keluarga font secara konsisten.6.3 Spacing, radius, dan elevation
|
||||||
|
Spacing system berbasis 4 px; gunakan 8, 12, 16, 24, 32 px sebagai ukuran utama.
|
||||||
|
Card radius 12 - 16 px; input/button 10 - 12 px; pill badge maksimal 999 px.
|
||||||
|
Gunakan shadow tipis; jangan gunakan efek glassmorphism atau gradient berlebihan.
|
||||||
|
Lebar konten desktop disarankan sekitar 1200 - 1280 px dengan ruang kosong yang cukup.
|
||||||
|
7. Spesifikasi Halaman: Dashboard
|
||||||
|
Dashboard adalah halaman prioritas tertinggi. Tujuan utamanya adalah memberi orientasi dalam satu layar: komoditas yang dipilih, kondisi harga saat ini, posisi pasar pada peta, hasil prediksi, dan tren harga.┌──────────────────────────────────────────────────────────────┐
|
||||||
|
│ LOGO / NAMA SISTEM Dashboard Harga Prediksi Analisis │
|
||||||
|
├──────────────────────────────────────────────────────────────┤
|
||||||
|
│ Prediksi Harga Cabai Kabupaten Jember │
|
||||||
|
│ Pantau harga aktual dan prediksi berdasarkan pasar. │
|
||||||
|
│ │
|
||||||
|
│ [Rawit ▼] [Tanggal 📅] [Cari pasar 🔍] │
|
||||||
|
├──────────────────────────────────────────────────────────────┤
|
||||||
|
│ [Rawit Rp42k] [Merah Besar Rp37k] [Keriting Rp40k] │
|
||||||
|
├───────────────────────────────────┬──────────────────────────┤
|
||||||
|
│ │ DETAIL TERPILIH │
|
||||||
|
│ PETA KABUPATEN JEMBER │ Pasar Tanjung │
|
||||||
|
│ ● ● │ Aktual Rp42.000/kg │
|
||||||
|
│ ● ● │ Prediksi Rp44.300/kg │
|
||||||
|
│ ● │ Perubahan +5,48% │
|
||||||
|
│ │ [Lihat detail] │
|
||||||
|
├───────────────────────────────────┴──────────────────────────┤
|
||||||
|
│ Grafik Historis vs Prediksi │
|
||||||
|
│ │
|
||||||
|
│ Actual ──────────────── │
|
||||||
|
│ Predict · · · · · · · · │
|
||||||
|
├──────────────────────────────────────────────────────────────┤
|
||||||
|
│ Insight Harga │
|
||||||
|
└──────────────────────────────────────────────────────────────┘
|
||||||
|
Komponen
|
||||||
|
Spesifikasi UX
|
||||||
|
Header/Navbar
|
||||||
|
Logo/nama sistem, Dashboard, Harga, Prediksi, Analisis, Tentang. Sticky di desktop.
|
||||||
|
Hero
|
||||||
|
Headline singkat + deskripsi + filter utama.
|
||||||
|
Summary cards
|
||||||
|
Satu card per jenis cabai: harga terkini, perubahan, tanggal pembaruan.
|
||||||
|
Interactive map
|
||||||
|
Peta Jember dengan market markers; klik marker membuka detail.
|
||||||
|
Selected market panel
|
||||||
|
Detail marker aktif: pasar, jenis cabai, aktual, prediksi, perubahan.
|
||||||
|
Trend chart
|
||||||
|
Grafik historis dan prediksi untuk konteks tren.
|
||||||
|
Insight card
|
||||||
|
Ringkasan informatif berbasis perbandingan aktual vs prediksi.
|
||||||
|
8. Spesifikasi UX Peta Interaktif
|
||||||
|
Peta merupakan elemen inti sesuai proposal. Pengguna harus dapat memilih jenis cabai, menemukan pasar, melihat harga aktual, melihat prediksi, lalu berpindah ke detail pasar.Gunakan basemap light/muted agar marker data lebih menonjol.
|
||||||
|
Marker harus menampilkan status yang ringkas; detail lengkap muncul ketika marker dipilih.
|
||||||
|
Marker minimal memiliki label/tooltip: nama pasar + harga aktual.
|
||||||
|
Popup/side panel menampilkan aktual, prediksi, persentase perubahan, tanggal data, dan tombol "Lihat Detail".
|
||||||
|
Sediakan kontrol zoom, pencarian pasar, dan filter jenis cabai.
|
||||||
|
Pada mobile, panel detail berubah menjadi bottom sheet atau drawer sehingga peta tetap terlihat.
|
||||||
|
Marker states
|
||||||
|
1. Default: marker standar.
|
||||||
|
2. Hover: marker membesar sedikit + tooltip nama pasar.
|
||||||
|
3. Selected: marker aktif + panel detail terbuka.
|
||||||
|
4. Loading: skeleton/spinner pada panel detail.
|
||||||
|
5. Empty: pesan "Tidak ada data untuk filter ini".
|
||||||
|
6. Error: pesan ramah + tombol coba lagi.
|
||||||
|
9. Spesifikasi Halaman Harga Pasar
|
||||||
|
Halaman ini berorientasi pada perbandingan. Tabel harus mudah dipindai, bukan menjadi dump seluruh dataset.Harga Pasar Kabupaten Jember
|
||||||
|
|
||||||
|
[ Jenis Cabai ▼ ] [ Tanggal ▼ ] [ Cari Pasar __________________ ]
|
||||||
|
|
||||||
|
┌────────────────┬──────────────┬──────────────┬────────────┐
|
||||||
|
│ Pasar │ Aktual │ Sebelumnya │ Perubahan │
|
||||||
|
├────────────────┼──────────────┼──────────────┼────────────┤
|
||||||
|
│ Pasar Tanjung │ Rp42.000/kg │ Rp40.000 │ ↑ +5,0% │
|
||||||
|
│ Pasar Mangli │ Rp41.000/kg │ Rp41.000 │ -- 0,0% │
|
||||||
|
│ Pasar Wirolegi │ Rp39.000/kg │ Rp40.000 │ ↓ -2,5% │
|
||||||
|
└────────────────┴──────────────┴──────────────┴────────────┘
|
||||||
|
Desktop: tabel penuh; mobile: ubah menjadi stacked cards atau tabel horizontal yang dapat di-scroll.
|
||||||
|
Klik baris pasar untuk menuju /prediksi?market=... atau halaman detail pasar.
|
||||||
|
Gunakan pagination atau virtualisasi jika jumlah baris besar.
|
||||||
|
Tetapkan format harga konsisten "Rp42.000/kg"; jangan campur format Rp42k dan Rp42.000 dalam satu tabel.
|
||||||
|
10. Spesifikasi Halaman Prediksi
|
||||||
|
Halaman prediksi merupakan titik temu antara output model LSTM dan kebutuhan pengguna. Angka prediksi harus terlihat jelas, tetapi selalu disertai konteks tanggal dan pasar.Prediksi Harga Cabai
|
||||||
|
|
||||||
|
[Cabai Rawit ▼] [Pasar Tanjung ▼]
|
||||||
|
|
||||||
|
┌─────────────────────────┐
|
||||||
|
│ Harga aktual │
|
||||||
|
│ Rp42.000 / kg │
|
||||||
|
└─────────────────────────┘
|
||||||
|
|
||||||
|
┌─────────────────────────┐
|
||||||
|
│ Prediksi │
|
||||||
|
│ Rp44.300 / kg │
|
||||||
|
│ Perubahan +5,48% │
|
||||||
|
└─────────────────────────┘
|
||||||
|
|
||||||
|
[ Grafik Actual vs Prediction ]
|
||||||
|
|
||||||
|
Prediksi ini merupakan hasil model berdasarkan data historis.
|
||||||
|
|
||||||
|
Microcopy: Hindari kata-kata yang menyatakan kepastian seperti "harga pasti besok". Gunakan "hasil prediksi", "diprediksi", atau "perkiraan berdasarkan model".11. Spesifikasi Halaman Analisis Model
|
||||||
|
Halaman ini ditujukan terutama untuk kebutuhan penelitian, demonstrasi sidang, dan transparansi model. Halaman publik tidak perlu memuat seluruh hyperparameter.Analisis Model LSTM
|
||||||
|
|
||||||
|
┌──────────────┐ ┌──────────────┐
|
||||||
|
│ RMSE │ │ MAE │
|
||||||
|
│ xxxx │ │ xxxx │
|
||||||
|
└──────────────┘ └──────────────┘
|
||||||
|
|
||||||
|
Konfigurasi model final
|
||||||
|
Epoch xxx
|
||||||
|
Batch Size xx
|
||||||
|
Learning Rate 0.xxx
|
||||||
|
Neuron xx
|
||||||
|
Activation xxxx
|
||||||
|
Optimizer Adam
|
||||||
|
|
||||||
|
[ Grafik Actual vs Prediction ]
|
||||||
|
[ Tabel ringkas evaluasi ]
|
||||||
|
Nilai RMSE dan MAE menggunakan hasil eksperimen final dari notebook/model yang sudah ditetapkan.
|
||||||
|
Jangan hard-code hasil penelitian yang belum final; gunakan mock values pada fase UI.
|
||||||
|
Sediakan area untuk grafik evaluasi tanpa mengubah layout ketika nilai berubah panjang.
|
||||||
|
12. Spesifikasi Halaman Tentang Sistem
|
||||||
|
Ringkasan penelitian dan tujuan sistem.
|
||||||
|
Penjelasan singkat LSTM menggunakan bahasa non-teknis.
|
||||||
|
Sumber data: SISKAPERBAPO Jawa Timur.
|
||||||
|
Cakupan komoditas: cabai rawit merah, cabai merah keriting, dan cabai merah besar.
|
||||||
|
Penjelasan bahwa hasil ditampilkan per pasar di Kabupaten Jember.
|
||||||
|
13. Component Inventory React
|
||||||
|
Komponen
|
||||||
|
Props/data penting
|
||||||
|
Perilaku
|
||||||
|
Navbar
|
||||||
|
activeRoute, navigationItems
|
||||||
|
Sticky; mobile menjadi hamburger menu.
|
||||||
|
PageHeader
|
||||||
|
title, description, breadcrumbs
|
||||||
|
Konsisten di setiap halaman.
|
||||||
|
CommoditySelector
|
||||||
|
selectedCommodity, options, onChange
|
||||||
|
Dropdown/select dengan 3 jenis cabai.
|
||||||
|
DateFilter
|
||||||
|
date, minDate, maxDate, onChange
|
||||||
|
Validasi tanggal dan state loading.
|
||||||
|
PriceSummaryCard
|
||||||
|
commodity, price, change, updatedAt
|
||||||
|
Card angka besar + status perubahan.
|
||||||
|
MarketMap
|
||||||
|
markets, selectedMarket, onMarkerClick
|
||||||
|
Leaflet map + markers + selected state.
|
||||||
|
MarketPopup
|
||||||
|
market, actualPrice, predictedPrice, change
|
||||||
|
Popup ringkas; tombol detail.
|
||||||
|
PredictionCard
|
||||||
|
actual, predicted, change, date
|
||||||
|
Menonjolkan hasil prediksi.
|
||||||
|
PriceChart
|
||||||
|
actualSeries, predictionSeries, labels
|
||||||
|
Actual vs prediction dengan legend.
|
||||||
|
MarketTable
|
||||||
|
rows, sorting, pagination
|
||||||
|
Table responsive.
|
||||||
|
InsightCard
|
||||||
|
message, metric, severity
|
||||||
|
Informasi ringkas berdasarkan data.
|
||||||
|
Skeleton
|
||||||
|
variant
|
||||||
|
Dipakai saat loading.
|
||||||
|
EmptyState
|
||||||
|
title, description, action
|
||||||
|
Untuk filter tanpa data.
|
||||||
|
ErrorState
|
||||||
|
message, retry
|
||||||
|
Untuk kegagalan fetch API.
|
||||||
|
14. UX States yang Wajib Disiapkan
|
||||||
|
Loading -- Skeleton pada card dan panel; jangan menampilkan angka sementara yang menyesatkan.Success -- Tampilkan data aktual/prediksi lengkap dengan timestamp.Empty -- "Belum ada data untuk filter yang dipilih."Error -- "Data belum dapat dimuat. Coba lagi." + tombol Retry.Offline/slow -- Tampilkan state jaringan secara ringkas, bukan error teknis mentah.Selected map -- Marker aktif, popup/panel terbuka, map tetap menjaga konteks lokasi.15. Responsive Layout
|
||||||
|
Breakpoint
|
||||||
|
Navbar
|
||||||
|
Map + panel
|
||||||
|
Chart/table
|
||||||
|
Desktop >= 1200 px
|
||||||
|
Full navbar
|
||||||
|
2 kolom; map 60 - 65%, panel 35 - 40%
|
||||||
|
Chart penuh; tabel penuh
|
||||||
|
Tablet 768 - 1199 px
|
||||||
|
Navbar ringkas
|
||||||
|
2 kolom dengan panel lebih sempit
|
||||||
|
Chart penuh; tabel dapat scroll
|
||||||
|
Mobile < 768 px
|
||||||
|
Hamburger/drawer
|
||||||
|
Map full width + bottom sheet
|
||||||
|
Chart tinggi 240 - 280 px; cards stacked
|
||||||
|
16. Urutan Implementasi UI/UX Tahap 1
|
||||||
|
Setup project React (Vite) + Tailwind CSS atau styling system yang sudah dipakai project.
|
||||||
|
Buat AppShell, Navbar, routing, typography, color tokens, button, card, input, badge.
|
||||||
|
Buat Dashboard menggunakan mock data statis; belum perlu memanggil FastAPI.
|
||||||
|
Implementasikan map + marker + popup menggunakan mock coordinates pasar.
|
||||||
|
Implementasikan tabel harga dan filter.
|
||||||
|
Implementasikan halaman prediksi + chart actual vs prediction menggunakan mock series.
|
||||||
|
Implementasikan halaman Analisis Model dengan mock RMSE/MAE/hyperparameters.
|
||||||
|
Implementasikan responsive/mobile states.
|
||||||
|
Lakukan visual QA, perbaiki spacing, alignment, overflow, dan consistency.
|
||||||
|
Setelah UI stabil, ganti mock data dengan service layer yang memanggil FastAPI.
|
||||||
|
Pemisahan tanggung jawab: Frontend UI/UX tidak boleh bergantung langsung pada struktur database. Buat service/API adapter di layer terpisah agar perubahan endpoint FastAPI tidak memaksa perubahan komponen UI.17. Kontrak Frontend terhadap FastAPI
|
||||||
|
Karena backend FastAPI sudah tersedia, UI/UX fase pertama harus dibuat dengan contract-friendly architecture. Endpoint nyata dapat dihubungkan setelah komponen visual selesai. Jangan membuat komponen memanggil fetch/axios langsung di banyak tempat.src/
|
||||||
|
├── components/
|
||||||
|
│ ├── common/
|
||||||
|
│ ├── dashboard/
|
||||||
|
│ ├── map/
|
||||||
|
│ ├── market/
|
||||||
|
│ ├── prediction/
|
||||||
|
│ └── analysis/
|
||||||
|
├── pages/
|
||||||
|
├── layouts/
|
||||||
|
├── services/
|
||||||
|
│ └── api/
|
||||||
|
├── hooks/
|
||||||
|
├── data/
|
||||||
|
│ └── mock/
|
||||||
|
├── types/
|
||||||
|
├── utils/
|
||||||
|
└── App.jsx
|
||||||
|
Gunakan mock data JSON yang bentuknya menyerupai response FastAPI.
|
||||||
|
Komponen menerima data melalui props/hooks, bukan mengakses endpoint langsung.
|
||||||
|
Satu service layer menjadi pintu masuk API setelah backend disambungkan.
|
||||||
|
Jangan mengubah backend hanya untuk menyesuaikan UI pada fase pertama.
|
||||||
|
18. Contoh Mock Data untuk UI
|
||||||
|
{
|
||||||
|
"commodity": "cabai_rawit_merah",
|
||||||
|
"market": "Pasar Tanjung",
|
||||||
|
"date": "2026-09-22",
|
||||||
|
"actual_price": 42000,
|
||||||
|
"previous_price": 40000,
|
||||||
|
"predicted_price": 44300,
|
||||||
|
"change_percent": 5.48,
|
||||||
|
"location": {"lat": -8.17, "lng": 113.70}
|
||||||
|
}
|
||||||
|
Nama field di atas hanyalah contoh kontrak UI. Sesuaikan dengan response FastAPI yang sudah kamu miliki tanpa mengubah struktur visual komponen.19. MASTER PROMPT UNTUK GITHUB COPILOT
|
||||||
|
Bagian ini dapat langsung disalin ke Copilot Chat / Agent sebagai brief pembuatan frontend tahap pertama. Prompt sengaja menahan integrasi backend agar Copilot fokus pada UI/UX terlebih dahulu.Anda adalah senior frontend engineer sekaligus UI/UX designer. Bangun frontend React.js untuk tugas akhir berjudul "Analisis Model Long Short-Term Memory dalam Sistem Prediksi Harga Cabai dengan Visualisasi Peta Digital Interaktif di Kabupaten Jember".
|
||||||
|
|
||||||
|
KONTEKS PRODUK
|
||||||
|
- Sistem menampilkan informasi harga cabai berdasarkan pasar di Kabupaten Jember.
|
||||||
|
- Terdapat 3 komoditas: cabai rawit merah, cabai merah besar, dan cabai merah keriting.
|
||||||
|
- Fitur inti: harga aktual, perubahan harga, peta pasar interaktif, hasil prediksi harga, grafik actual vs prediction, dan insight informatif.
|
||||||
|
- Model machine learning adalah LSTM. Detail teknis model tidak boleh mendominasi dashboard publik.
|
||||||
|
- Backend sudah tersedia menggunakan FastAPI, tetapi TAHAP SEKARANG HANYA UI/UX. Jangan ubah backend dan jangan bergantung pada endpoint nyata. Gunakan mock data.
|
||||||
|
|
||||||
|
ARAH DESAIN
|
||||||
|
- Tema: modern agricultural data dashboard.
|
||||||
|
- Bukan marketplace, bukan website pemerintah yang kaku, dan bukan dashboard AI yang penuh istilah teknis.
|
||||||
|
- Map-first, data-first, clean, minimal, professional, responsive.
|
||||||
|
- Warna utama: #1F6B45, #14532D, aksen cabai #D94A3D, aksen #F2B84B, background #F7F8F5, surface #FFFFFF, text #1F2937, muted #6B7280, border #E5E7EB.
|
||||||
|
- Gunakan Inter atau Plus Jakarta Sans.
|
||||||
|
- Card radius 12 - 16px; shadow tipis; spacing system berbasis 4px.
|
||||||
|
|
||||||
|
HALAMAN
|
||||||
|
1. / -- Dashboard
|
||||||
|
- Navbar
|
||||||
|
- Hero singkat
|
||||||
|
- Filter jenis cabai, tanggal, pencarian pasar
|
||||||
|
- 3 summary price cards
|
||||||
|
- Interactive Jember map
|
||||||
|
- Market detail panel ketika marker dipilih
|
||||||
|
- Actual vs Prediction chart
|
||||||
|
- Insight card
|
||||||
|
2. /harga -- Harga Pasar
|
||||||
|
- filter
|
||||||
|
- table harga per pasar
|
||||||
|
- responsive mobile card/scroll
|
||||||
|
3. /prediksi -- Prediksi
|
||||||
|
- commodity selector
|
||||||
|
- market selector
|
||||||
|
- actual price
|
||||||
|
- predicted price
|
||||||
|
- change percent
|
||||||
|
- chart actual vs prediction
|
||||||
|
4. /analisis -- Analisis Model
|
||||||
|
- RMSE
|
||||||
|
- MAE
|
||||||
|
- konfigurasi model final
|
||||||
|
- evaluasi/graph
|
||||||
|
5. /tentang -- Tentang Sistem
|
||||||
|
- tujuan
|
||||||
|
- sumber data SISKAPERBAPO
|
||||||
|
- penjelasan singkat sistem dan LSTM
|
||||||
|
|
||||||
|
KOMPONEN WAJIB
|
||||||
|
Navbar, PageHeader, CommoditySelector, DateFilter, SearchMarket, PriceSummaryCard, MarketMap, MarketPopup, PredictionCard, PriceChart, MarketTable, InsightCard, Skeleton, EmptyState, ErrorState.
|
||||||
|
|
||||||
|
UX MAP
|
||||||
|
- Gunakan basemap light/muted.
|
||||||
|
- Marker default, hover, selected, loading, empty, error.
|
||||||
|
- Klik marker membuka detail: nama pasar, aktual, prediksi, perubahan, tanggal.
|
||||||
|
- Desktop: map + detail panel 60/40.
|
||||||
|
- Mobile: map full width + bottom sheet.
|
||||||
|
|
||||||
|
UX RULES
|
||||||
|
- Jangan menampilkan angka palsu yang terlihat final; gunakan mock data yang jelas bersifat demo.
|
||||||
|
- Setiap status kenaikan/penurunan harus memakai ikon/teks selain warna.
|
||||||
|
- Gunakan "diprediksi" atau "perkiraan berdasarkan model", bukan bahasa yang menjamin hasil.
|
||||||
|
- Semua halaman harus punya loading, empty, dan error state.
|
||||||
|
- Jangan hard-code style per komponen jika token/style system bisa dipakai bersama.
|
||||||
|
- Hindari gradient berlebihan, glassmorphism, animasi berlebihan, shadow besar, dan layout yang padat.
|
||||||
|
|
||||||
|
STRUKTUR KODE
|
||||||
|
Gunakan pemisahan components/pages/layouts/services/data/mock/types/hooks/utils. Komponen UI tidak boleh memanggil fetch langsung. Siapkan service layer tetapi isikan mock data terlebih dahulu.
|
||||||
|
|
||||||
|
HASIL YANG DIHARAPKAN
|
||||||
|
- Semua route bekerja.
|
||||||
|
- Dashboard terasa seperti produk data sungguhan.
|
||||||
|
- UI konsisten dan responsif.
|
||||||
|
- Map dan chart menggunakan mock data.
|
||||||
|
- Fokus pada visual quality dan usability terlebih dahulu.
|
||||||
|
- Setelah UI selesai, berhenti dan tunggu instruksi berikutnya untuk integrasi FastAPI.
|
||||||
|
|
||||||
|
20. Acceptance Checklist UI/UX Phase 1
|
||||||
|
☐ Dashboard dapat dipahami dalam <=10 detik tanpa membaca dokumentasi.
|
||||||
|
☐ Pengguna dapat mengganti jenis cabai tanpa reload manual.
|
||||||
|
☐ Pengguna dapat menemukan pasar melalui map dan/atau pencarian.
|
||||||
|
☐ Marker terpilih selalu menampilkan aktual + prediksi dengan konteks tanggal.
|
||||||
|
☐ Tabel harga dapat dipindai dan tidak pecah pada mobile.
|
||||||
|
☐ Chart actual vs prediction memiliki legend dan label yang jelas.
|
||||||
|
☐ RMSE dan MAE berada pada halaman Analisis Model, bukan dipaksakan ke hero dashboard.
|
||||||
|
☐ Loading/empty/error state tersedia untuk komponen data.
|
||||||
|
☐ Tidak ada overflow horizontal pada mobile kecuali area tabel yang sengaja scrollable.
|
||||||
|
☐ Warna, typography, radius, spacing, dan button konsisten di seluruh halaman.
|
||||||
|
☐ Tidak ada ketergantungan komponen UI langsung ke database atau endpoint tertentu.
|
||||||
|
☐ Frontend fase 1 dapat berjalan penuh menggunakan mock data.
|
||||||
|
21. Catatan Kesesuaian dengan Proposal TA
|
||||||
|
Dokumen ini mempertahankan fokus proposal pada prediksi harga cabai di Kabupaten Jember, tiga jenis cabai, data per pasar, hasil prediksi model LSTM, peta digital interaktif, serta rekomendasi/insight. Proposal juga menyebut pengujian User Acceptance Testing (UAT), sehingga komponen dan alur di atas sengaja dibuat sederhana dan dapat diuji dari perspektif pengguna. (Proposal TA, Bab 3.3.4 dan 3.3.5).Catatan teknologi: Proposal yang diunggah mencantumkan Python dan Flask pada bagian software, sedangkan implementasi backend yang sedang kamu gunakan adalah FastAPI. Dokumen UI/UX ini sengaja dibuat tidak bergantung pada framework backend; frontend hanya menyiapkan service layer agar dapat mengonsumsi REST API FastAPI ketika tahap integrasi dimulai.
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
DATA_SOURCE=mysql
|
||||||
|
MYSQL_HOST=127.0.0.1
|
||||||
|
MYSQL_PORT=3306
|
||||||
|
MYSQL_DATABASE=prediksicabe
|
||||||
|
MYSQL_USER=root
|
||||||
|
MYSQL_PASSWORD=
|
||||||
|
MYSQL_CONNECT_TIMEOUT=5
|
||||||
|
CORS_ORIGINS=http://127.0.0.1:5173,http://localhost:5173
|
||||||
|
SISKAPERBAPO_URL=https://siskaperbapo.jatimprov.go.id/harga-komoditas
|
||||||
|
SISKAPERBAPO_COMMODITY_PARAM=komoditas
|
||||||
|
SISKAPERBAPO_TIMEOUT=30
|
||||||
|
SISKAPERBAPO_COMMODITIES=rawit
|
||||||
|
|
||||||
|
# Gunakan DATA_SOURCE=csv hanya jika ingin kembali memakai file CSV.
|
||||||
|
# DATA_DIR=C:\PrediksiHargaCabe\api
|
||||||
|
|
@ -0,0 +1,137 @@
|
||||||
|
# API Prediksi Harga Cabai
|
||||||
|
|
||||||
|
API ini menggunakan Flask, application factory, dan Blueprint pada prefix `/api/v1`. Model LSTM tidak dilatih ulang dan artifact di `models/` hanya dimuat satu kali saat startup.
|
||||||
|
|
||||||
|
## Menjalankan di Windows
|
||||||
|
|
||||||
|
Dari folder `api` dengan environment Python proyek:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
pip install -r requirements.txt
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Untuk menghentikan proses backend lama dan menjalankannya kembali dengan auto-reloader:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
powershell -ExecutionPolicy Bypass -File .\restart_backend.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
Skrip tersebut menghentikan proses backend proyek pada port `8000`, lalu menjalankan Flask dengan
|
||||||
|
`FLASK_DEBUG=1`. Perubahan file Python akan dimuat otomatis selama proses tetap berjalan.
|
||||||
|
|
||||||
|
Environment yang tersedia:
|
||||||
|
|
||||||
|
- `MODELS_DIR`: lokasi folder model, default `api/models`.
|
||||||
|
- `DATA_DIR`: lokasi data CSV historis, default folder `api` karena file aktual berada di sana.
|
||||||
|
- `DATA_SOURCE`: sumber data aktif, `mysql` (default) atau `csv`.
|
||||||
|
- `MYSQL_HOST`, `MYSQL_PORT`, `MYSQL_DATABASE`, `MYSQL_USER`, `MYSQL_PASSWORD`: koneksi database MySQL melalui PyMySQL.
|
||||||
|
- `MYSQL_CONNECT_TIMEOUT`: batas waktu koneksi database dalam detik.
|
||||||
|
- `PRICE_UNIT_SCALE`: pengali nilai CSV ke Rupiah penuh, default `1000`; sumber CSV penelitian menyimpan `12.000` sebagai 12 ribu Rupiah.
|
||||||
|
- `CORS_ORIGINS`: daftar origin dipisahkan koma; default `http://127.0.0.1:5173,http://localhost:5173` untuk development dengan session login.
|
||||||
|
- `ALLOWED_HORIZONS`: default `1,3,7`.
|
||||||
|
- `MAX_HORIZON`: default `7`.
|
||||||
|
|
||||||
|
## Endpoint
|
||||||
|
|
||||||
|
- `GET /api/v1/health`
|
||||||
|
- `GET /api/v1/commodities`
|
||||||
|
- `GET /api/v1/markets?commodity=rawit`
|
||||||
|
- `GET /api/v1/history?commodity=rawit&market=pasar_tanjung&days=90`
|
||||||
|
- `GET /api/v1/predict?commodity=rawit&market=pasar_tanjung&horizon=7`
|
||||||
|
- `GET /api/v1/predict/map?commodity=rawit&horizon=7`
|
||||||
|
|
||||||
|
Semua error memakai format `{ "error": { "code": "...", "message": "...", "details": {} } }`.
|
||||||
|
|
||||||
|
## Format data historis
|
||||||
|
|
||||||
|
Dengan `DATA_SOURCE=mysql`, loader membaca tabel `price_history` dan melakukan join ke `commodities` serta `markets` menggunakan `slug`. Dengan `DATA_SOURCE=csv`, loader membaca file aktual `harga_cabai_rawit_*.csv`, `harga_cabe_merah_besar_*.csv`, dan `harga_cabai_merah_keriting_*.csv` dari `DATA_DIR`. Pada mode CSV, kolom tanggal dapat bernama `Tanggal`, dan kolom pasar memakai nama `Pasar_Tanjung`, `Pasar_Kreongan`, `Pasar_Tegal_Besar`, `Pasar_Mangli`, serta `Pasar_Wirolegi`. Harga bernilai `0`, `N/A`, atau kosong diperlakukan sebagai `NaN`; hanya gap internal sampai dua titik yang diinterpolasi linear.
|
||||||
|
|
||||||
|
## Menghubungkan ke MySQL phpMyAdmin
|
||||||
|
|
||||||
|
1. Pastikan service MySQL pada XAMPP/Laragon/WAMP sedang berjalan.
|
||||||
|
2. Pastikan database yang dipakai adalah `prediksicabe`.
|
||||||
|
3. Pastikan tabel `commodities`, `markets`, dan `price_history` ada.
|
||||||
|
4. Isi data harga ke `price_history`; seed SQL hanya mengisi master komoditas dan pasar.
|
||||||
|
5. Set environment variable pada terminal; file `.env.example` hanya template dan belum dibaca otomatis oleh aplikasi.
|
||||||
|
6. Isi `MYSQL_USER` dan `MYSQL_PASSWORD` sesuai akun MySQL lokal.
|
||||||
|
7. Jalankan API dari folder `api`.
|
||||||
|
|
||||||
|
Untuk membuat akun admin, daftar dahulu melalui endpoint register atau halaman register website, lalu ubah role akun tersebut melalui phpMyAdmin:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
UPDATE users
|
||||||
|
SET role = 'admin'
|
||||||
|
WHERE email = 'email-admin-anda@example.com';
|
||||||
|
```
|
||||||
|
|
||||||
|
Password tetap disimpan dalam bentuk hash oleh API; jangan memasukkan password ke query SQL.
|
||||||
|
|
||||||
|
Endpoint admin yang sudah tersedia:
|
||||||
|
|
||||||
|
- `GET /api/v1/admin/overview`
|
||||||
|
- `GET /api/v1/admin/models`
|
||||||
|
- `GET /api/v1/admin/scraping/logs`
|
||||||
|
- `GET /api/v1/admin/scraping/logs/<log_id>/records`
|
||||||
|
- `POST /api/v1/admin/scraping/jobs`
|
||||||
|
- `GET /api/v1/admin/scraping/jobs/<job_id>`
|
||||||
|
- `POST /api/v1/admin/preprocessing/jobs` dengan `{"scraping_log_id": 123}` untuk memproses satu riwayat scraping yang dipilih.
|
||||||
|
- `GET /api/v1/admin/preprocessing/jobs/<job_id>`
|
||||||
|
- `DELETE /api/v1/admin/scraping/history`
|
||||||
|
- `DELETE /api/v1/admin/scraping/logs/<log_id>`
|
||||||
|
- `POST /api/v1/admin/scraping/refresh`
|
||||||
|
- `GET /api/v1/prices?page=1&limit=25` (admin)
|
||||||
|
|
||||||
|
Panel admin sekarang memiliki menu dataset dan pembaruan data manual. Endpoint refresh mengambil tabel SISKAPERBAPO, menyimpan observasi baru atau perubahan ke `price_history`, dan mencatat proses ke `scraping_logs`. Endpoint records menampilkan tanggal, komoditas, pasar, dan harga dari hasil scraping terakhir. Data baru tidak melatih ulang model; model deployment yang sudah aktif tetap digunakan untuk inference. Scheduler, preprocessing aktif, dan training aktif tetap tidak dijalankan.
|
||||||
|
|
||||||
|
Endpoint jobs menerima opsi `commodities` (array slug), `start_date`, dan `end_date`. Jika rentang tanggal dipilih, jumlah observasi dihitung otomatis sebagai hasil tanggal yang tersedia dikalikan pasar yang berhasil dibaca. Proses berjalan di background dan dapat dipantau melalui endpoint status job; panel admin menampilkan progress bar. Penghapusan history menghapus log scraping dan baris `price_history` dengan `source = 'siskaperbapo'` saja, sehingga data hasil import CSV tetap aman.
|
||||||
|
Setiap observasi scraping memiliki `scraping_log_id`, sehingga tombol trash pada baris log menghapus hanya log tersebut dan data yang dihasilkan proses itu. Jika database lama digunakan, application factory otomatis menambahkan kolom dan mengaitkan observasi SISKAPERBAPO lama ke log terakhir.
|
||||||
|
|
||||||
|
Konfigurasi scraper:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:SISKAPERBAPO_URL="https://siskaperbapo.jatimprov.go.id/harga-komoditas"
|
||||||
|
$env:SISKAPERBAPO_COMMODITY_PARAM="komoditas"
|
||||||
|
$env:SISKAPERBAPO_TIMEOUT="30"
|
||||||
|
$env:SISKAPERBAPO_COMMODITIES="rawit"
|
||||||
|
```
|
||||||
|
|
||||||
|
Tahap awal scraper dibatasi ke `rawit` agar tidak mengisi komoditas lain dengan data yang salah apabila halaman sumber mengabaikan parameter komoditas. Tambahkan slug lain setelah URL/parameter sumber sudah diverifikasi. Jika struktur halaman SISKAPERBAPO berubah atau parameter komoditas berbeda, endpoint akan mengembalikan error dan mencatat detailnya sebagai scraping gagal; sistem tidak membuat data sukses palsu.
|
||||||
|
|
||||||
|
Untuk mengisi tabel `price_history` dari CSV penelitian secara manual satu kali:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
C:/Users/alfia/anaconda3/envs/tf_gpu39/python.exe scripts/import_csv_to_mysql.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Script ini hanya melakukan import manual dan **tidak membuat scheduler atau auto-scraping**. Jika dijalankan lagi, constraint unik tanggal/komoditas/pasar mencegah duplikat dan nilai akan diperbarui.
|
||||||
|
|
||||||
|
Contoh PowerShell:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
$env:DATA_SOURCE="mysql"
|
||||||
|
$env:MYSQL_DATABASE="prediksicabe"
|
||||||
|
$env:MYSQL_USER="root"
|
||||||
|
$env:MYSQL_PASSWORD=""
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Uji koneksi melalui endpoint:
|
||||||
|
|
||||||
|
```text
|
||||||
|
http://127.0.0.1:8000/api/v1/history?commodity=rawit&market=pasar_tanjung&days=30
|
||||||
|
```
|
||||||
|
|
||||||
|
Jika response menyatakan tidak ada data, koneksi berhasil tetapi baris harga belum ada untuk kombinasi tersebut. Jika muncul `Gagal membaca database MySQL`, periksa service MySQL, port, username, password, dan nama database.
|
||||||
|
|
||||||
|
## Asumsi dan keterbatasan
|
||||||
|
|
||||||
|
- Koordinat pada `app/data/markets.json` berasal dari spesifikasi dan perlu diverifikasi manual sebelum dipakai sebagai titik final peta.
|
||||||
|
- `keriting` memakai `time_step=14`. Config dan H5 yang tersedia untuk Kreongan, Mangli, dan Wirolegi sekarang sudah memiliki input 14 langkah dan dapat dipakai oleh registry.
|
||||||
|
- Folder model Keriting untuk Tanjung dan Tegal Besar tidak lengkap dan dilaporkan pada `/api/v1/health`.
|
||||||
|
- Evaluasi model yang tersedia adalah satu hari ke depan. Prediksi 7, 24, dan 30 hari memakai recursive forecasting, sehingga error dapat menumpuk dan tidak boleh dianggap memiliki akurasi satu hari.
|
||||||
|
- Harga API adalah Rupiah penuh. Scaler/model yang dilatih dalam ribuan Rupiah harus disesuaikan saat artifact inference divalidasi; CSV historis juga harus mengikuti unit yang sama dengan proses training.
|
||||||
|
- Semua nilai positif yang disimpan ke `price_history` menggunakan Rupiah penuh. Scraper menormalkan nilai sumber yang masih dalam ribuan sebelum menyimpannya; nilai `0` diperlakukan sebagai data kosong.
|
||||||
|
|
@ -0,0 +1,458 @@
|
||||||
|
# Dokumentasi Scraping Harga Cabai
|
||||||
|
|
||||||
|
Dokumen ini menjelaskan proses pembaruan data harga melalui panel admin, mulai dari tombol **Perbarui data** di frontend sampai data hasil scraping disimpan ke MySQL.
|
||||||
|
|
||||||
|
## Ringkasan
|
||||||
|
|
||||||
|
Fitur scraping mengambil data dari SISKAPERBAPO menggunakan:
|
||||||
|
|
||||||
|
- `requests` untuk mengirim HTTP `GET` dan `POST`;
|
||||||
|
- `pandas.read_html()` untuk membaca tabel HTML;
|
||||||
|
- `lxml` sebagai parser HTML;
|
||||||
|
- MySQL untuk menyimpan hasil ke tabel `price_history`.
|
||||||
|
|
||||||
|
Implementasi saat ini **tidak menggunakan Selenium, WebDriver, Playwright, atau browser automation**.
|
||||||
|
|
||||||
|
Model LSTM tidak dilatih ulang setelah scraping. Data baru hanya disimpan sebagai data historis yang dapat digunakan oleh API saat melakukan inference.
|
||||||
|
|
||||||
|
## Komponen yang terlibat
|
||||||
|
|
||||||
|
| Komponen | Lokasi | Fungsi |
|
||||||
|
|---|---|---|
|
||||||
|
| Tombol dan panel admin | `../website/src/App.jsx` | Memulai scraping, menampilkan progress, dan memuat ulang data |
|
||||||
|
| Client API frontend | `../website/src/services/api.js` | Mengirim request ke backend |
|
||||||
|
| Route scraping | `app/api/routes.py` | Validasi admin, membuat job, membaca status, dan menyimpan data |
|
||||||
|
| Scraper SISKAPERBAPO | `app/services/siskaperbapo_scraper.py` | Mengambil dan membersihkan tabel harga |
|
||||||
|
| Konfigurasi aplikasi | `app/config.py` | Menentukan URL sumber, komoditas, timeout, dan sumber data |
|
||||||
|
| Data lokasi pasar | `app/data/markets.json` | Menyimpan slug, nama, dan koordinat pasar |
|
||||||
|
| Database | MySQL | Menyimpan `price_history` dan `scraping_logs` |
|
||||||
|
|
||||||
|
## Alur tombol "Perbarui data"
|
||||||
|
|
||||||
|
Alur normalnya adalah:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Admin klik "Perbarui data"
|
||||||
|
↓
|
||||||
|
Frontend mengirim POST /api/v1/admin/scraping/jobs
|
||||||
|
↓
|
||||||
|
Backend memvalidasi role admin dan opsi scraping
|
||||||
|
↓
|
||||||
|
Backend membuat job_id dan menjalankan thread background
|
||||||
|
↓
|
||||||
|
Scraper mengambil tabel SISKAPERBAPO
|
||||||
|
↓
|
||||||
|
Harga dibersihkan dan dikonversi ke Rupiah
|
||||||
|
↓
|
||||||
|
Data di-insert atau di-update ke price_history
|
||||||
|
↓
|
||||||
|
Status scraping_logs diperbarui
|
||||||
|
↓
|
||||||
|
Frontend mengecek status job setiap 700 ms
|
||||||
|
↓
|
||||||
|
Dashboard admin memuat ulang data
|
||||||
|
```
|
||||||
|
|
||||||
|
Proses ini berbeda dari pemanggilan prediksi. Scraping hanya memperbarui data historis; artifact model di `models/` tetap digunakan seperti sebelumnya.
|
||||||
|
|
||||||
|
## Proses di frontend
|
||||||
|
|
||||||
|
Komponen admin berada di `../website/src/App.jsx`.
|
||||||
|
|
||||||
|
Fungsi `refreshPrices()`:
|
||||||
|
|
||||||
|
1. Mengaktifkan status `refreshing`;
|
||||||
|
2. Memanggil `dashboardService.startScraping(options)`;
|
||||||
|
3. Menerima `job_id`;
|
||||||
|
4. Meminta status job setiap 700 ms;
|
||||||
|
5. Menampilkan progress dan pesan proses;
|
||||||
|
6. Menampilkan error jika status berubah menjadi `failed`;
|
||||||
|
7. Memanggil `loadAdminData()` setelah berhasil.
|
||||||
|
|
||||||
|
Request frontend didefinisikan di `../website/src/services/api.js`:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
startScraping: (options) =>
|
||||||
|
request('/admin/scraping/jobs', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify(options)
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
Contoh payload:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"commodities": ["rawit"],
|
||||||
|
"start_date": "",
|
||||||
|
"end_date": ""
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Contoh untuk rentang tanggal:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"commodities": ["rawit"],
|
||||||
|
"start_date": "2025-10-01",
|
||||||
|
"end_date": "2025-10-07"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Jika tanggal tidak diisi, scraper mengambil tabel terbaru melalui metode `scrape()`. Jika tanggal awal dan akhir diisi, scraper menggunakan metode `scrape_range()`.
|
||||||
|
|
||||||
|
## Endpoint scraping
|
||||||
|
|
||||||
|
Semua endpoint berikut memiliki prefix `/api/v1` dan membutuhkan session admin.
|
||||||
|
|
||||||
|
### Memulai scraping
|
||||||
|
|
||||||
|
```text
|
||||||
|
POST /api/v1/admin/scraping/jobs
|
||||||
|
```
|
||||||
|
|
||||||
|
Payload:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"commodities": ["rawit"],
|
||||||
|
"start_date": "2025-10-01",
|
||||||
|
"end_date": "2025-10-07",
|
||||||
|
"max_records": 1000
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Field:
|
||||||
|
|
||||||
|
| Field | Wajib | Keterangan |
|
||||||
|
|---|---:|---|
|
||||||
|
| `commodities` | Tidak | Array slug: `rawit`, `merah`, atau `keriting`. Default mengikuti `SISKAPERBAPO_COMMODITIES` |
|
||||||
|
| `start_date` | Tidak | Tanggal awal format `YYYY-MM-DD` |
|
||||||
|
| `end_date` | Tidak | Tanggal akhir format `YYYY-MM-DD` |
|
||||||
|
| `max_records` | Tidak | Batas hasil antara 1 sampai 10.000 |
|
||||||
|
|
||||||
|
Response awal menggunakan HTTP `202` karena proses berjalan di background:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"id": "job-id",
|
||||||
|
"status": "running",
|
||||||
|
"progress": 0,
|
||||||
|
"message": "Mengantri scraping",
|
||||||
|
"records_inserted": 0,
|
||||||
|
"records_updated": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Melihat status scraping
|
||||||
|
|
||||||
|
```text
|
||||||
|
GET /api/v1/admin/scraping/jobs/{job_id}
|
||||||
|
```
|
||||||
|
|
||||||
|
Status yang mungkin:
|
||||||
|
|
||||||
|
- `running`: proses masih berlangsung;
|
||||||
|
- `success`: proses selesai;
|
||||||
|
- `failed`: proses gagal.
|
||||||
|
|
||||||
|
Field progress menunjukkan persentase proses. Field `message` menjelaskan tahap yang sedang berjalan.
|
||||||
|
|
||||||
|
### Melihat daftar riwayat
|
||||||
|
|
||||||
|
```text
|
||||||
|
GET /api/v1/admin/scraping/logs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Melihat isi satu riwayat
|
||||||
|
|
||||||
|
```text
|
||||||
|
GET /api/v1/admin/scraping/logs/{log_id}/records
|
||||||
|
```
|
||||||
|
|
||||||
|
### Menghapus satu riwayat
|
||||||
|
|
||||||
|
```text
|
||||||
|
DELETE /api/v1/admin/scraping/logs/{log_id}
|
||||||
|
```
|
||||||
|
|
||||||
|
Endpoint ini menghapus log yang dipilih dan record `price_history` yang memiliki `scraping_log_id` tersebut.
|
||||||
|
|
||||||
|
### Menghapus seluruh riwayat scraping
|
||||||
|
|
||||||
|
```text
|
||||||
|
DELETE /api/v1/admin/scraping/history
|
||||||
|
```
|
||||||
|
|
||||||
|
Penghapusan dibatasi pada data yang bersumber dari `siskaperbapo`; data hasil import CSV tidak ikut dihapus.
|
||||||
|
|
||||||
|
## Proses backend
|
||||||
|
|
||||||
|
Route `POST /admin/scraping/jobs` berada di `app/api/routes.py`. Route tersebut:
|
||||||
|
|
||||||
|
1. Memastikan user memiliki role `admin`;
|
||||||
|
2. Membaca payload JSON;
|
||||||
|
3. Memvalidasi komoditas, tanggal, dan batas record;
|
||||||
|
4. Menolak proses baru jika ada scraping atau preprocessing yang masih `running`;
|
||||||
|
5. Membuat ID job;
|
||||||
|
6. Memulai `_run_scraping_job()` di thread background.
|
||||||
|
|
||||||
|
Fungsi `_run_scraping_job()` melakukan langkah berikut:
|
||||||
|
|
||||||
|
1. Membuat row baru pada `scraping_logs` dengan status `running`;
|
||||||
|
2. Membuka koneksi MySQL;
|
||||||
|
3. Memanggil `scrape()` atau `scrape_range()`;
|
||||||
|
4. Mencocokkan slug komoditas dan pasar dengan tabel master;
|
||||||
|
5. Menyimpan observasi ke `price_history`;
|
||||||
|
6. Menghitung jumlah record baru dan record yang diperbarui;
|
||||||
|
7. Mengubah status log menjadi `success`;
|
||||||
|
8. Melakukan rollback dan mencatat error jika proses gagal.
|
||||||
|
|
||||||
|
## Cara scraper mengambil data
|
||||||
|
|
||||||
|
Implementasinya berada di `app/services/siskaperbapo_scraper.py`.
|
||||||
|
|
||||||
|
### Data terbaru dengan `scrape()`
|
||||||
|
|
||||||
|
Untuk setiap komoditas yang dipilih, scraper mengirim:
|
||||||
|
|
||||||
|
```python
|
||||||
|
requests.get(
|
||||||
|
self.url,
|
||||||
|
params={self.commodity_param: commodity},
|
||||||
|
timeout=self.timeout,
|
||||||
|
headers={"User-Agent": "PanenHarga/1.0"},
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Response HTML dibaca dengan:
|
||||||
|
|
||||||
|
```python
|
||||||
|
pd.read_html(StringIO(response.text))
|
||||||
|
```
|
||||||
|
|
||||||
|
Scraper kemudian mencari:
|
||||||
|
|
||||||
|
- kolom lokasi atau pasar;
|
||||||
|
- kolom yang dapat dibaca sebagai tanggal;
|
||||||
|
- baris lima pasar target;
|
||||||
|
- harga pada setiap tanggal.
|
||||||
|
|
||||||
|
### Data historis dengan `scrape_range()`
|
||||||
|
|
||||||
|
Untuk rentang tanggal, scraper mengakses endpoint:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/harga/tabel.nodesign
|
||||||
|
```
|
||||||
|
|
||||||
|
melalui HTTP `POST`, dengan data:
|
||||||
|
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
"tanggal": "YYYY-MM-DD",
|
||||||
|
"kabkota": "jemberkab",
|
||||||
|
"pasar": "market_id"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Proses dilakukan untuk setiap kombinasi:
|
||||||
|
|
||||||
|
```text
|
||||||
|
tanggal × pasar
|
||||||
|
```
|
||||||
|
|
||||||
|
ID pasar yang digunakan:
|
||||||
|
|
||||||
|
| Slug | Nama | ID sumber |
|
||||||
|
|---|---|---:|
|
||||||
|
| `pasar_tanjung` | Pasar Tanjung | `11` |
|
||||||
|
| `pasar_kreongan` | Pasar Kreongan | `12` |
|
||||||
|
| `pasar_tegal_besar` | Pasar Tegal Besar | `13` |
|
||||||
|
| `pasar_mangli` | Pasar Mangli | `148` |
|
||||||
|
| `pasar_wirolegi` | Pasar Wirolegi | `149` |
|
||||||
|
|
||||||
|
## Normalisasi harga
|
||||||
|
|
||||||
|
Scraper memakai `_parse_price()` untuk membersihkan nilai seperti:
|
||||||
|
|
||||||
|
- `Rp22`;
|
||||||
|
- `22`;
|
||||||
|
- `22,5`;
|
||||||
|
- `22.000`.
|
||||||
|
|
||||||
|
Kemudian `_price_in_rupiah()` menerapkan aturan:
|
||||||
|
|
||||||
|
```python
|
||||||
|
price * 1000 if 0 < price < 1000 else price
|
||||||
|
```
|
||||||
|
|
||||||
|
Dengan demikian:
|
||||||
|
|
||||||
|
| Nilai sumber | Nilai yang disimpan |
|
||||||
|
|---:|---:|
|
||||||
|
| `22` | `22000` |
|
||||||
|
| `22.5` | `22500` |
|
||||||
|
| `22000` | `22000` |
|
||||||
|
|
||||||
|
Harga dalam `price_history` disimpan sebagai Rupiah penuh.
|
||||||
|
|
||||||
|
## Penyimpanan dan duplikasi data
|
||||||
|
|
||||||
|
Setiap observasi memiliki bentuk:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"commodity": "rawit",
|
||||||
|
"market": "pasar_tanjung",
|
||||||
|
"date": "2025-10-01",
|
||||||
|
"price": 22000,
|
||||||
|
"source": "siskaperbapo"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Record disimpan ke `price_history` menggunakan pola:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO price_history (...)
|
||||||
|
VALUES (...)
|
||||||
|
ON DUPLICATE KEY UPDATE
|
||||||
|
price = VALUES(price),
|
||||||
|
source = VALUES(source),
|
||||||
|
scraping_log_id = VALUES(scraping_log_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
Artinya:
|
||||||
|
|
||||||
|
- kombinasi yang belum ada menjadi record baru;
|
||||||
|
- kombinasi yang sudah ada diperbarui;
|
||||||
|
- harga tidak disimpan sebagai duplikat jika unique key database telah dikonfigurasi.
|
||||||
|
|
||||||
|
Kolom `scraping_log_id` menghubungkan setiap harga dengan riwayat proses scraping.
|
||||||
|
|
||||||
|
## `scraping_logs`
|
||||||
|
|
||||||
|
Tabel `scraping_logs` dipakai untuk mencatat proses. Informasi yang disimpan meliputi:
|
||||||
|
|
||||||
|
- waktu mulai;
|
||||||
|
- waktu selesai;
|
||||||
|
- sumber data;
|
||||||
|
- status;
|
||||||
|
- jumlah record baru;
|
||||||
|
- jumlah record diperbarui;
|
||||||
|
- pesan error atau warning.
|
||||||
|
|
||||||
|
Status awal adalah `running`. Setelah proses selesai, status menjadi `success` atau `failed`.
|
||||||
|
|
||||||
|
## Scraping dan preprocessing berbeda
|
||||||
|
|
||||||
|
Scraping:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Mengambil data dari SISKAPERBAPO
|
||||||
|
```
|
||||||
|
|
||||||
|
Preprocessing:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Memeriksa dan mengisi data kosong dari riwayat scraping
|
||||||
|
```
|
||||||
|
|
||||||
|
Preprocessing dijalankan melalui:
|
||||||
|
|
||||||
|
```text
|
||||||
|
POST /api/v1/admin/preprocessing/jobs
|
||||||
|
```
|
||||||
|
|
||||||
|
Proses ini menggunakan interpolasi linear dan bukan bagian dari pengambilan data dari website.
|
||||||
|
|
||||||
|
## Konfigurasi
|
||||||
|
|
||||||
|
Konfigurasi berada di `app/config.py` dan dapat diatur melalui environment variable:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:SISKAPERBAPO_URL="https://siskaperbapo.jatimprov.go.id/harga-komoditas"
|
||||||
|
$env:SISKAPERBAPO_COMMODITY_PARAM="komoditas"
|
||||||
|
$env:SISKAPERBAPO_TIMEOUT="30"
|
||||||
|
$env:SISKAPERBAPO_COMMODITIES="rawit"
|
||||||
|
```
|
||||||
|
|
||||||
|
Nilai default:
|
||||||
|
|
||||||
|
- URL: `https://siskaperbapo.jatimprov.go.id/harga-komoditas`;
|
||||||
|
- parameter komoditas: `komoditas`;
|
||||||
|
- timeout: 30 detik;
|
||||||
|
- komoditas default: `rawit`.
|
||||||
|
|
||||||
|
Tahap awal menggunakan `rawit` untuk mengurangi risiko sumber website mengabaikan parameter komoditas. Komoditas lain sebaiknya diaktifkan setelah struktur response sumber diverifikasi.
|
||||||
|
|
||||||
|
## Cara menjalankan dan memeriksa proses
|
||||||
|
|
||||||
|
Jalankan backend dari folder `api`:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cd C:\PrediksiHargaCabe\api
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Kemudian:
|
||||||
|
|
||||||
|
1. Buka website frontend;
|
||||||
|
2. Login sebagai admin;
|
||||||
|
3. Buka menu **Perbarui data**;
|
||||||
|
4. Pilih komoditas;
|
||||||
|
5. Isi rentang tanggal jika ingin mengambil data historis tertentu;
|
||||||
|
6. Klik **Mulai scraping** atau tombol **Perbarui data** di bagian atas;
|
||||||
|
7. Amati progress;
|
||||||
|
8. Periksa bagian **Riwayat pembaruan**;
|
||||||
|
9. Buka detail record untuk memeriksa hasil yang disimpan.
|
||||||
|
|
||||||
|
Jika ingin melihat proses langsung dari database, periksa:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT *
|
||||||
|
FROM scraping_logs
|
||||||
|
ORDER BY id DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
dan:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT *
|
||||||
|
FROM price_history
|
||||||
|
WHERE source = 'siskaperbapo'
|
||||||
|
ORDER BY price_date DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Penanganan error
|
||||||
|
|
||||||
|
Jika request ke sumber gagal, scraper melempar `ScrapingError`. Job akan berstatus `failed`, transaksi database di-rollback, dan pesan error dicatat di `scraping_logs`.
|
||||||
|
|
||||||
|
Penyebab yang umum:
|
||||||
|
|
||||||
|
- website SISKAPERBAPO tidak dapat diakses;
|
||||||
|
- struktur tabel HTML berubah;
|
||||||
|
- parameter komoditas tidak sesuai;
|
||||||
|
- koneksi MySQL gagal;
|
||||||
|
- tabel master komoditas atau pasar belum tersedia;
|
||||||
|
- response tidak berisi data yang cocok.
|
||||||
|
|
||||||
|
Jika proses gagal, periksa:
|
||||||
|
|
||||||
|
1. status dan pesan pada panel admin;
|
||||||
|
2. kolom error di `scraping_logs`;
|
||||||
|
3. URL dan konfigurasi SISKAPERBAPO;
|
||||||
|
4. koneksi MySQL;
|
||||||
|
5. struktur HTML response sumber.
|
||||||
|
|
||||||
|
## Catatan penting
|
||||||
|
|
||||||
|
- Scraping hanya dapat dijalankan oleh admin.
|
||||||
|
- Hanya satu scraping atau preprocessing yang boleh berjalan pada satu waktu di proses aplikasi.
|
||||||
|
- Data baru tidak otomatis melatih ulang model.
|
||||||
|
- Tidak ada Selenium pada implementasi saat ini.
|
||||||
|
- Perubahan struktur HTML SISKAPERBAPO dapat membuat parser gagal.
|
||||||
|
- Data kosong yang dihasilkan scraping ditangani pada tahap preprocessing terpisah.
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,95 @@
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from flask import Flask, jsonify, request
|
||||||
|
|
||||||
|
from app.api.routes import api
|
||||||
|
from app.config import Config
|
||||||
|
from app.data.repository import CsvPriceRepository, MySQLPriceRepository
|
||||||
|
from app.errors import register_error_handlers
|
||||||
|
from app.services.model_registry import ModelRegistry
|
||||||
|
from app.services.prediction_service import PredictionService
|
||||||
|
from app.services.siskaperbapo_scraper import SiskaperbapoScraper
|
||||||
|
|
||||||
|
|
||||||
|
def create_app(config_class=Config) -> Flask:
|
||||||
|
app = Flask(__name__)
|
||||||
|
app.config.from_object(config_class)
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
origins = app.config["CORS_ORIGINS"]
|
||||||
|
allowed_origins = [value.strip() for value in origins.split(",") if value.strip()]
|
||||||
|
|
||||||
|
@app.after_request
|
||||||
|
def add_cors_credentials(response):
|
||||||
|
origin = request.headers.get("Origin")
|
||||||
|
if origin and (origins == "*" or origin in allowed_origins):
|
||||||
|
response.headers["Access-Control-Allow-Origin"] = origin
|
||||||
|
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||||
|
response.headers["Access-Control-Allow-Headers"] = "Content-Type"
|
||||||
|
response.headers["Access-Control-Allow-Methods"] = "GET,POST,DELETE,OPTIONS"
|
||||||
|
response.headers["Vary"] = "Origin"
|
||||||
|
return response
|
||||||
|
|
||||||
|
model_registry = ModelRegistry(app.config["MODELS_DIR"])
|
||||||
|
if app.config["DATA_SOURCE"] == "mysql":
|
||||||
|
repository = MySQLPriceRepository(app.config)
|
||||||
|
elif app.config["DATA_SOURCE"] == "csv":
|
||||||
|
repository = CsvPriceRepository(app.config["DATA_DIR"], app.config["PRICE_UNIT_SCALE"])
|
||||||
|
else:
|
||||||
|
raise ValueError("DATA_SOURCE harus bernilai 'mysql' atau 'csv'.")
|
||||||
|
app.extensions["model_registry"] = model_registry
|
||||||
|
app.extensions["prediction_service"] = PredictionService(model_registry, repository, app.config["STALE_DATA_DAYS"], app.config["TREND_THRESHOLD_PCT"])
|
||||||
|
if hasattr(repository, "_connect"):
|
||||||
|
connection = repository._connect()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) FROM information_schema.columns
|
||||||
|
WHERE table_schema = DATABASE()
|
||||||
|
AND table_name = 'price_history'
|
||||||
|
AND column_name = 'scraping_log_id'
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
if cursor.fetchone()[0] == 0:
|
||||||
|
cursor.execute(
|
||||||
|
"ALTER TABLE price_history ADD COLUMN scraping_log_id BIGINT UNSIGNED NULL AFTER source"
|
||||||
|
)
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
ALTER TABLE price_history
|
||||||
|
ADD KEY idx_price_history_scraping_log (scraping_log_id),
|
||||||
|
ADD CONSTRAINT fk_price_history_scraping_log
|
||||||
|
FOREIGN KEY (scraping_log_id) REFERENCES scraping_logs(id)
|
||||||
|
ON UPDATE CASCADE ON DELETE CASCADE
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
UPDATE price_history ph
|
||||||
|
JOIN (
|
||||||
|
SELECT MAX(id) AS id FROM scraping_logs WHERE source = 'siskaperbapo'
|
||||||
|
) latest ON 1 = 1
|
||||||
|
SET ph.scraping_log_id = latest.id
|
||||||
|
WHERE ph.source = 'siskaperbapo' AND ph.scraping_log_id IS NULL
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
app.extensions["siskaperbapo_scraper"] = SiskaperbapoScraper(
|
||||||
|
app.config["SISKAPERBAPO_URL"],
|
||||||
|
app.config["SISKAPERBAPO_COMMODITY_PARAM"],
|
||||||
|
app.config["SISKAPERBAPO_TIMEOUT"],
|
||||||
|
)
|
||||||
|
markets_path = Path(__file__).parent / "data" / "markets.json"
|
||||||
|
app.extensions["markets_config"] = json.loads(markets_path.read_text(encoding="utf-8"))
|
||||||
|
app.register_blueprint(api)
|
||||||
|
register_error_handlers(app)
|
||||||
|
|
||||||
|
@app.get("/")
|
||||||
|
def root():
|
||||||
|
return jsonify({"message": "API Prediksi Harga Cabai aktif", "status": "ok"})
|
||||||
|
|
||||||
|
return app
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,830 @@
|
||||||
|
from datetime import date, datetime, timedelta
|
||||||
|
from functools import wraps
|
||||||
|
from threading import Lock, Thread
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
from flask import Blueprint, current_app, jsonify, request, session
|
||||||
|
from pymysql import MySQLError
|
||||||
|
from werkzeug.security import check_password_hash, generate_password_hash
|
||||||
|
|
||||||
|
from app.api.schemas import parse_days, parse_horizon, required_query
|
||||||
|
from app.errors import ApiError
|
||||||
|
from app.services.model_registry import COMMODITIES, MARKETS
|
||||||
|
from app.services.siskaperbapo_scraper import ScrapingError
|
||||||
|
|
||||||
|
api = Blueprint("api", __name__, url_prefix="/api/v1")
|
||||||
|
_scraping_jobs = {}
|
||||||
|
_scraping_jobs_lock = Lock()
|
||||||
|
_preprocessing_jobs = {}
|
||||||
|
|
||||||
|
|
||||||
|
def service():
|
||||||
|
return current_app.extensions["prediction_service"]
|
||||||
|
|
||||||
|
|
||||||
|
def registry():
|
||||||
|
return current_app.extensions["model_registry"]
|
||||||
|
|
||||||
|
|
||||||
|
def markets_config():
|
||||||
|
return current_app.extensions["markets_config"]
|
||||||
|
|
||||||
|
|
||||||
|
def _db():
|
||||||
|
repository = current_app.extensions["prediction_service"].repository
|
||||||
|
if not hasattr(repository, "_connect"):
|
||||||
|
raise ApiError("DATABASE_NOT_CONFIGURED", "Database MySQL belum dikonfigurasi.", 503)
|
||||||
|
return repository._connect()
|
||||||
|
|
||||||
|
|
||||||
|
def admin_required(handler):
|
||||||
|
@wraps(handler)
|
||||||
|
def wrapped(*args, **kwargs):
|
||||||
|
if session.get("user_role") != "admin":
|
||||||
|
raise ApiError("ADMIN_REQUIRED", "Akses admin diperlukan.", 403)
|
||||||
|
return handler(*args, **kwargs)
|
||||||
|
return wrapped
|
||||||
|
|
||||||
|
|
||||||
|
def _set_scraping_job(job_id, **values):
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
_scraping_jobs.setdefault(job_id, {}).update(values)
|
||||||
|
|
||||||
|
|
||||||
|
def _scraping_job_payload(job):
|
||||||
|
return {
|
||||||
|
"id": job["id"],
|
||||||
|
"status": job["status"],
|
||||||
|
"progress": job["progress"],
|
||||||
|
"message": job["message"],
|
||||||
|
"records_inserted": job.get("records_inserted", 0),
|
||||||
|
"records_updated": job.get("records_updated", 0),
|
||||||
|
"log_id": job.get("log_id"),
|
||||||
|
"warning": job.get("warning"),
|
||||||
|
"error": job.get("error"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _preprocessing_job_payload(job):
|
||||||
|
return {
|
||||||
|
"id": job["id"],
|
||||||
|
"status": job["status"],
|
||||||
|
"progress": job["progress"],
|
||||||
|
"message": job["message"],
|
||||||
|
"rows_filled": job.get("rows_filled", 0),
|
||||||
|
"warning": job.get("warning"),
|
||||||
|
"run_id": job.get("run_id"),
|
||||||
|
"error": job.get("error"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _run_preprocessing_job(app, job_id, scraping_log_id):
|
||||||
|
with app.app_context():
|
||||||
|
connection = None
|
||||||
|
run_id = None
|
||||||
|
try:
|
||||||
|
repository = app.extensions["prediction_service"].repository
|
||||||
|
if not hasattr(repository, "_connect"):
|
||||||
|
raise ApiError("DATABASE_NOT_CONFIGURED", "Database MySQL belum dikonfigurasi.", 503)
|
||||||
|
connection = repository._connect()
|
||||||
|
started_at = datetime.utcnow()
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO preprocessing_runs
|
||||||
|
(dataset_source, started_at, status, parameters_json)
|
||||||
|
VALUES ('mysql.price_history', %s, 'running', %s)
|
||||||
|
""",
|
||||||
|
(started_at, '{"method":"linear_interpolation","scope":"selected_scraping_log"}'),
|
||||||
|
)
|
||||||
|
run_id = cursor.lastrowid
|
||||||
|
connection.commit()
|
||||||
|
_set_preprocessing_job(job_id, run_id=run_id, progress=5, message="Memeriksa data kosong")
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT c.id, m.id, c.slug, m.slug, ph.price_date, ph.price
|
||||||
|
FROM price_history ph
|
||||||
|
JOIN commodities c ON c.id = ph.commodity_id
|
||||||
|
JOIN markets m ON m.id = ph.market_id
|
||||||
|
WHERE ph.scraping_log_id = %s
|
||||||
|
ORDER BY c.slug, m.slug, ph.price_date
|
||||||
|
""",
|
||||||
|
(scraping_log_id,),
|
||||||
|
)
|
||||||
|
rows = cursor.fetchall()
|
||||||
|
groups = {}
|
||||||
|
for commodity_id, market_id, commodity, market, price_date, price in rows:
|
||||||
|
groups.setdefault((commodity_id, market_id, commodity, market), []).append((price_date, float(price)))
|
||||||
|
filled = 0
|
||||||
|
long_gaps = []
|
||||||
|
total_groups = max(len(groups), 1)
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
for index, ((commodity_id, market_id, commodity, market), observations) in enumerate(groups.items(), start=1):
|
||||||
|
values = {item[0]: item[1] for item in observations}
|
||||||
|
first_date, last_date = observations[0][0], observations[-1][0]
|
||||||
|
missing = []
|
||||||
|
current = first_date
|
||||||
|
while current <= last_date:
|
||||||
|
if current not in values:
|
||||||
|
missing.append(current)
|
||||||
|
current += timedelta(days=1)
|
||||||
|
missing_runs = []
|
||||||
|
for missing_date in missing:
|
||||||
|
if not missing_runs or (missing_date - missing_runs[-1][-1]).days > 1:
|
||||||
|
missing_runs.append([missing_date])
|
||||||
|
else:
|
||||||
|
missing_runs[-1].append(missing_date)
|
||||||
|
for missing_run in missing_runs:
|
||||||
|
if len(missing_run) > 2:
|
||||||
|
long_gaps.append(f"{commodity}/{market}: {len(missing_run)} hari")
|
||||||
|
continue
|
||||||
|
for missing_date in missing_run:
|
||||||
|
previous = max((day for day in values if day < missing_date), default=None)
|
||||||
|
following = min((day for day in values if day > missing_date), default=None)
|
||||||
|
if previous is None or following is None:
|
||||||
|
continue
|
||||||
|
ratio = (missing_date - previous).days / (following - previous).days
|
||||||
|
interpolated = values[previous] + (values[following] - values[previous]) * ratio
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO price_history
|
||||||
|
(commodity_id, market_id, price_date, price, source, scraping_log_id)
|
||||||
|
VALUES (%s, %s, %s, %s, 'preprocessing_interpolation', %s)
|
||||||
|
""",
|
||||||
|
(commodity_id, market_id, missing_date, round(interpolated, 2), scraping_log_id),
|
||||||
|
)
|
||||||
|
filled += 1
|
||||||
|
progress = 5 + int(index / total_groups * 85)
|
||||||
|
_set_preprocessing_job(job_id, progress=progress, message=f"Memproses {commodity} / {market}")
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT COUNT(*) FROM price_history"
|
||||||
|
)
|
||||||
|
rows_after = cursor.fetchone()[0]
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE preprocessing_runs SET finished_at = %s, status = 'success', rows_before = %s, rows_after = %s WHERE id = %s",
|
||||||
|
(datetime.utcnow(), len(rows), rows_after, run_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
warning = f"Gap lebih dari 2 hari tidak diisi: {', '.join(long_gaps[:5])}" if long_gaps else None
|
||||||
|
_set_preprocessing_job(job_id, status="success", progress=100, message="Preprocessing selesai", rows_filled=filled, warning=warning)
|
||||||
|
except Exception as error:
|
||||||
|
if connection and run_id:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE preprocessing_runs SET finished_at = %s, status = 'failed', error_message = %s WHERE id = %s",
|
||||||
|
(datetime.utcnow(), str(error), run_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
_set_preprocessing_job(job_id, status="failed", progress=100, message="Preprocessing gagal", error=str(error))
|
||||||
|
finally:
|
||||||
|
if connection:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _set_preprocessing_job(job_id, **values):
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
_preprocessing_jobs.setdefault(job_id, {}).update(values)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_scraping_job(app, job_id, selected_commodities, start_date, end_date, max_records):
|
||||||
|
with app.app_context():
|
||||||
|
connection = None
|
||||||
|
log_id = None
|
||||||
|
try:
|
||||||
|
_set_scraping_job(job_id, progress=5, message="Menyiapkan proses scraping")
|
||||||
|
repository = app.extensions["prediction_service"].repository
|
||||||
|
if not hasattr(repository, "_connect"):
|
||||||
|
raise ScrapingError("Database MySQL belum dikonfigurasi.")
|
||||||
|
connection = repository._connect()
|
||||||
|
started_at = datetime.utcnow()
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"INSERT INTO scraping_logs (started_at, status, source) VALUES (%s, 'running', 'siskaperbapo')",
|
||||||
|
(started_at,),
|
||||||
|
)
|
||||||
|
log_id = cursor.lastrowid
|
||||||
|
connection.commit()
|
||||||
|
_set_scraping_job(job_id, log_id=log_id, progress=8, message="Mengambil tabel SISKAPERBAPO")
|
||||||
|
|
||||||
|
def report(progress, message):
|
||||||
|
_set_scraping_job(job_id, progress=progress, message=message)
|
||||||
|
|
||||||
|
if start_date and end_date:
|
||||||
|
observations = app.extensions["siskaperbapo_scraper"].scrape_range(
|
||||||
|
selected_commodities, start_date, end_date, on_progress=report
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
observations = app.extensions["siskaperbapo_scraper"].scrape(
|
||||||
|
selected_commodities, on_progress=report
|
||||||
|
)
|
||||||
|
filtered = [
|
||||||
|
item for item in observations
|
||||||
|
if (not start_date or item["date"] >= start_date)
|
||||||
|
and (not end_date or item["date"] <= end_date)
|
||||||
|
]
|
||||||
|
filtered.sort(key=lambda item: (item["date"], item["commodity"], item["market"]))
|
||||||
|
if max_records:
|
||||||
|
filtered = filtered[:max_records]
|
||||||
|
if not filtered:
|
||||||
|
raise ScrapingError("Tidak ada data yang sesuai dengan filter scraping.")
|
||||||
|
warning = None
|
||||||
|
if start_date and end_date:
|
||||||
|
expected = {
|
||||||
|
(day, commodity, market)
|
||||||
|
for day_offset in range((end_date - start_date).days + 1)
|
||||||
|
for day in [start_date + timedelta(days=day_offset)]
|
||||||
|
for commodity in selected_commodities
|
||||||
|
for market in [item["slug"] for item in markets_config()]
|
||||||
|
}
|
||||||
|
observed = {(item["date"], item["commodity"], item["market"]) for item in filtered}
|
||||||
|
missing_count = len(expected - observed)
|
||||||
|
if missing_count:
|
||||||
|
warning = f"{missing_count} data kosong/tidak tersedia dilewati dari rentang scraping."
|
||||||
|
|
||||||
|
inserted = 0
|
||||||
|
updated = 0
|
||||||
|
total = len(filtered)
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
for index, observation in enumerate(filtered, start=1):
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT c.id, m.id FROM commodities c CROSS JOIN markets m WHERE c.slug = %s AND m.slug = %s",
|
||||||
|
(observation["commodity"], observation["market"]),
|
||||||
|
)
|
||||||
|
ids = cursor.fetchone()
|
||||||
|
if not ids:
|
||||||
|
continue
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO price_history
|
||||||
|
(commodity_id, market_id, price_date, price, source, scraping_log_id)
|
||||||
|
VALUES (%s, %s, %s, %s, %s, %s)
|
||||||
|
ON DUPLICATE KEY UPDATE price = VALUES(price), source = VALUES(source),
|
||||||
|
scraping_log_id = VALUES(scraping_log_id)
|
||||||
|
""",
|
||||||
|
(*ids, observation["date"], observation["price"], observation["source"], log_id),
|
||||||
|
)
|
||||||
|
if cursor.rowcount == 1:
|
||||||
|
inserted += 1
|
||||||
|
elif cursor.rowcount == 2:
|
||||||
|
updated += 1
|
||||||
|
report(65 + int(index / total * 30), f"Menyimpan data {index} dari {total}")
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
UPDATE scraping_logs
|
||||||
|
SET status = 'success', finished_at = UTC_TIMESTAMP(),
|
||||||
|
records_inserted = %s, records_updated = %s, error_message = %s
|
||||||
|
WHERE id = %s
|
||||||
|
""",
|
||||||
|
(inserted, updated, warning, log_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
_set_scraping_job(
|
||||||
|
job_id,
|
||||||
|
status="success",
|
||||||
|
progress=100,
|
||||||
|
message="Scraping selesai",
|
||||||
|
records_inserted=inserted,
|
||||||
|
records_updated=updated,
|
||||||
|
warning=warning,
|
||||||
|
)
|
||||||
|
except (ScrapingError, MySQLError) as error:
|
||||||
|
if connection:
|
||||||
|
connection.rollback()
|
||||||
|
if log_id:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE scraping_logs SET status = 'failed', finished_at = UTC_TIMESTAMP(), error_message = %s WHERE id = %s",
|
||||||
|
(str(error), log_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
_set_scraping_job(job_id, status="failed", progress=100, message="Scraping gagal", error=str(error))
|
||||||
|
except Exception as error:
|
||||||
|
app.logger.exception("Unhandled scraping job error: %s", error)
|
||||||
|
if connection:
|
||||||
|
connection.rollback()
|
||||||
|
_set_scraping_job(
|
||||||
|
job_id,
|
||||||
|
status="failed",
|
||||||
|
progress=100,
|
||||||
|
message="Scraping gagal karena kesalahan internal",
|
||||||
|
error="Terjadi kesalahan internal pada proses scraping.",
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
if connection:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
|
||||||
|
@api.post("/auth/register")
|
||||||
|
def register():
|
||||||
|
payload = request.get_json(silent=True) or {}
|
||||||
|
name = str(payload.get("name", "")).strip()
|
||||||
|
email = str(payload.get("email", "")).strip().lower()
|
||||||
|
password = str(payload.get("password", ""))
|
||||||
|
if not name or not email or len(password) < 8:
|
||||||
|
raise ApiError("INVALID_AUTH_INPUT", "Nama, email, dan password minimal 8 karakter wajib diisi.", 400)
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute("SELECT id FROM users WHERE email = %s", (email,))
|
||||||
|
if cursor.fetchone():
|
||||||
|
raise ApiError("EMAIL_EXISTS", "Email sudah terdaftar.", 409)
|
||||||
|
cursor.execute(
|
||||||
|
"INSERT INTO users (name, email, password_hash, role) VALUES (%s, %s, %s, 'user')",
|
||||||
|
(name, email, generate_password_hash(password)),
|
||||||
|
)
|
||||||
|
user_id = cursor.lastrowid
|
||||||
|
connection.commit()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
session.update(user_id=user_id, user_name=name, user_email=email, user_role="user")
|
||||||
|
return jsonify({"data": {"id": user_id, "name": name, "email": email, "role": "user"}}), 201
|
||||||
|
|
||||||
|
|
||||||
|
@api.post("/auth/login")
|
||||||
|
def login():
|
||||||
|
payload = request.get_json(silent=True) or {}
|
||||||
|
email = str(payload.get("email", "")).strip().lower()
|
||||||
|
password = str(payload.get("password", ""))
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute("SELECT id, name, email, password_hash, role FROM users WHERE email = %s", (email,))
|
||||||
|
user = cursor.fetchone()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
if not user or not check_password_hash(user[3], password):
|
||||||
|
raise ApiError("INVALID_CREDENTIALS", "Email atau password salah.", 401)
|
||||||
|
session.update(user_id=user[0], user_name=user[1], user_email=user[2], user_role=user[4])
|
||||||
|
return jsonify({"data": {"id": user[0], "name": user[1], "email": user[2], "role": user[4]}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.post("/auth/logout")
|
||||||
|
def logout():
|
||||||
|
session.clear()
|
||||||
|
return jsonify({"data": {"logged_out": True}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/auth/me")
|
||||||
|
def me():
|
||||||
|
if not session.get("user_id"):
|
||||||
|
return jsonify({"data": None})
|
||||||
|
return jsonify({"data": {"id": session["user_id"], "name": session["user_name"], "email": session["user_email"], "role": session["user_role"]}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/prices")
|
||||||
|
@admin_required
|
||||||
|
def prices():
|
||||||
|
repository = current_app.extensions["prediction_service"].repository
|
||||||
|
if not hasattr(repository, "_connect"):
|
||||||
|
raise ApiError("DATABASE_NOT_CONFIGURED", "Endpoint raw data membutuhkan MySQL.", 503)
|
||||||
|
page = max(int(request.args.get("page", 1)), 1)
|
||||||
|
limit = min(max(int(request.args.get("limit", 25)), 1), 20000)
|
||||||
|
offset = (page - 1) * limit
|
||||||
|
connection = repository._connect()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
f"""
|
||||||
|
SELECT ph.price_date, c.name, m.name, ph.price
|
||||||
|
FROM price_history ph
|
||||||
|
JOIN commodities c ON c.id = ph.commodity_id
|
||||||
|
JOIN markets m ON m.id = ph.market_id
|
||||||
|
WHERE (%s = '' OR c.slug = %s) AND (%s = '' OR m.slug = %s)
|
||||||
|
ORDER BY ph.price_date DESC, c.name, m.name
|
||||||
|
LIMIT %s OFFSET %s
|
||||||
|
""",
|
||||||
|
(request.args.get("commodity", ""), request.args.get("commodity", ""), request.args.get("market", ""), request.args.get("market", ""), limit, offset),
|
||||||
|
)
|
||||||
|
rows = cursor.fetchall()
|
||||||
|
cursor.execute(
|
||||||
|
f"""
|
||||||
|
SELECT COUNT(*)
|
||||||
|
FROM price_history ph
|
||||||
|
JOIN commodities c ON c.id = ph.commodity_id
|
||||||
|
JOIN markets m ON m.id = ph.market_id
|
||||||
|
WHERE (%s = '' OR c.slug = %s) AND (%s = '' OR m.slug = %s)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
request.args.get("commodity", ""),
|
||||||
|
request.args.get("commodity", ""),
|
||||||
|
request.args.get("market", ""),
|
||||||
|
request.args.get("market", ""),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
total = cursor.fetchone()[0]
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": [{"date": row[0].isoformat(), "commodity": row[1], "market": row[2], "price": float(row[3])} for row in rows], "meta": {"page": page, "limit": limit, "total": total}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/admin/overview")
|
||||||
|
@admin_required
|
||||||
|
def admin_overview():
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute("SELECT COUNT(*) FROM price_history")
|
||||||
|
total_prices = cursor.fetchone()[0]
|
||||||
|
cursor.execute("SELECT COUNT(*) FROM commodities")
|
||||||
|
total_commodities = cursor.fetchone()[0]
|
||||||
|
cursor.execute("SELECT COUNT(*) FROM markets")
|
||||||
|
total_markets = cursor.fetchone()[0]
|
||||||
|
cursor.execute("SELECT COUNT(*) FROM model_versions WHERE is_active = 1")
|
||||||
|
active_models = cursor.fetchone()[0]
|
||||||
|
cursor.execute("SELECT COUNT(*) FROM training_runs")
|
||||||
|
training_runs = cursor.fetchone()[0]
|
||||||
|
cursor.execute("SELECT MAX(created_at) FROM preprocessing_runs")
|
||||||
|
last_preprocessing = cursor.fetchone()[0]
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT id, status, started_at, finished_at, records_inserted,
|
||||||
|
records_updated, error_message, source,
|
||||||
|
(SELECT GROUP_CONCAT(DISTINCT c.name ORDER BY c.name SEPARATOR ', ')
|
||||||
|
FROM price_history ph JOIN commodities c ON c.id = ph.commodity_id
|
||||||
|
WHERE ph.scraping_log_id = scraping_logs.id) AS commodities
|
||||||
|
FROM scraping_logs ORDER BY id DESC LIMIT 1
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
last_scraping = cursor.fetchone()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": {"total_prices": total_prices, "total_commodities": total_commodities, "total_markets": total_markets, "active_models": active_models, "training_runs": training_runs, "last_preprocessing": last_preprocessing.isoformat() if last_preprocessing else None, "scraping": {"enabled": True, "message": "Pembaruan manual tersedia.", "last_run": _scraping_log_payload(last_scraping)}}})
|
||||||
|
|
||||||
|
|
||||||
|
def _scraping_log_payload(row):
|
||||||
|
if not row:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"id": row[0],
|
||||||
|
"status": row[1],
|
||||||
|
"started_at": row[2].isoformat() if row[2] else None,
|
||||||
|
"finished_at": row[3].isoformat() if row[3] else None,
|
||||||
|
"records_inserted": row[4],
|
||||||
|
"records_updated": row[5],
|
||||||
|
"error_message": row[6],
|
||||||
|
"source": row[7],
|
||||||
|
"commodities": row[8] if len(row) > 8 and row[8] else "-",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/admin/scraping/logs")
|
||||||
|
@admin_required
|
||||||
|
def scraping_logs():
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT id, status, started_at, finished_at, records_inserted,
|
||||||
|
records_updated, error_message, source,
|
||||||
|
(SELECT GROUP_CONCAT(DISTINCT c.name ORDER BY c.name SEPARATOR ', ')
|
||||||
|
FROM price_history ph JOIN commodities c ON c.id = ph.commodity_id
|
||||||
|
WHERE ph.scraping_log_id = scraping_logs.id) AS commodities
|
||||||
|
FROM scraping_logs ORDER BY id DESC LIMIT 20
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
rows = cursor.fetchall()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": [_scraping_log_payload(row) for row in rows]})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/admin/scraping/logs/<int:log_id>/records")
|
||||||
|
@admin_required
|
||||||
|
def scraping_log_records(log_id):
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT status, source FROM scraping_logs WHERE id = %s",
|
||||||
|
(log_id,),
|
||||||
|
)
|
||||||
|
log = cursor.fetchone()
|
||||||
|
if not log:
|
||||||
|
raise ApiError("SCRAPING_LOG_NOT_FOUND", "Log scraping tidak ditemukan.", 404)
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT ph.price_date, c.name, m.name, ph.price
|
||||||
|
FROM price_history ph
|
||||||
|
JOIN commodities c ON c.id = ph.commodity_id
|
||||||
|
JOIN markets m ON m.id = ph.market_id
|
||||||
|
WHERE ph.scraping_log_id = %s
|
||||||
|
ORDER BY ph.price_date DESC, c.name, m.name
|
||||||
|
""",
|
||||||
|
(log_id,),
|
||||||
|
)
|
||||||
|
rows = cursor.fetchall()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify(
|
||||||
|
{
|
||||||
|
"data": [
|
||||||
|
{
|
||||||
|
"date": row[0].isoformat(),
|
||||||
|
"commodity": row[1],
|
||||||
|
"market": row[2],
|
||||||
|
"price": float(row[3]),
|
||||||
|
}
|
||||||
|
for row in rows
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@api.post("/admin/scraping/refresh")
|
||||||
|
@admin_required
|
||||||
|
def refresh_scraping():
|
||||||
|
scraper = current_app.extensions["siskaperbapo_scraper"]
|
||||||
|
connection = _db()
|
||||||
|
started_at = datetime.utcnow()
|
||||||
|
log_id = None
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"INSERT INTO scraping_logs (started_at, status, source) VALUES (%s, 'running', 'siskaperbapo')",
|
||||||
|
(started_at,),
|
||||||
|
)
|
||||||
|
log_id = cursor.lastrowid
|
||||||
|
connection.commit()
|
||||||
|
selected_commodities = {
|
||||||
|
slug: COMMODITIES[slug]
|
||||||
|
for slug in current_app.config["SISKAPERBAPO_COMMODITIES"]
|
||||||
|
if slug in COMMODITIES
|
||||||
|
}
|
||||||
|
if not selected_commodities:
|
||||||
|
raise ScrapingError("Tidak ada komoditas scraper yang dikonfigurasi.")
|
||||||
|
observations = scraper.scrape(selected_commodities)
|
||||||
|
inserted = 0
|
||||||
|
updated = 0
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
for observation in observations:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT c.id, m.id FROM commodities c CROSS JOIN markets m WHERE c.slug = %s AND m.slug = %s",
|
||||||
|
(observation["commodity"], observation["market"]),
|
||||||
|
)
|
||||||
|
ids = cursor.fetchone()
|
||||||
|
if not ids:
|
||||||
|
continue
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO price_history
|
||||||
|
(commodity_id, market_id, price_date, price, source, scraping_log_id)
|
||||||
|
VALUES (%s, %s, %s, %s, %s, %s)
|
||||||
|
ON DUPLICATE KEY UPDATE price = VALUES(price), source = VALUES(source),
|
||||||
|
scraping_log_id = VALUES(scraping_log_id)
|
||||||
|
""",
|
||||||
|
(*ids, observation["date"], observation["price"], observation["source"], log_id),
|
||||||
|
)
|
||||||
|
if cursor.rowcount == 1:
|
||||||
|
inserted += 1
|
||||||
|
elif cursor.rowcount in (2,):
|
||||||
|
updated += 1
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
UPDATE scraping_logs
|
||||||
|
SET status = 'success', finished_at = UTC_TIMESTAMP(),
|
||||||
|
records_inserted = %s, records_updated = %s
|
||||||
|
WHERE id = %s
|
||||||
|
""",
|
||||||
|
(inserted, updated, log_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
except ScrapingError as error:
|
||||||
|
connection.rollback()
|
||||||
|
if log_id:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE scraping_logs SET status = 'failed', finished_at = UTC_TIMESTAMP(), error_message = %s WHERE id = %s",
|
||||||
|
(str(error), log_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
raise ApiError("SCRAPING_FAILED", str(error), 502)
|
||||||
|
except MySQLError as error:
|
||||||
|
connection.rollback()
|
||||||
|
if log_id:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE scraping_logs SET status = 'failed', finished_at = UTC_TIMESTAMP(), error_message = %s WHERE id = %s",
|
||||||
|
(str(error), log_id),
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
raise ApiError("SCRAPING_DATABASE_FAILED", "Gagal menyimpan hasil pembaruan ke database.", 503) from error
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": {"status": "success", "records_inserted": inserted, "records_updated": updated, "log_id": log_id}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.delete("/admin/scraping/logs/<int:log_id>")
|
||||||
|
@admin_required
|
||||||
|
def delete_scraping_log(log_id):
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT id, source FROM scraping_logs WHERE id = %s",
|
||||||
|
(log_id,),
|
||||||
|
)
|
||||||
|
log = cursor.fetchone()
|
||||||
|
if not log:
|
||||||
|
raise ApiError("SCRAPING_LOG_NOT_FOUND", "Log scraping tidak ditemukan.", 404)
|
||||||
|
if log[1] != "siskaperbapo":
|
||||||
|
raise ApiError("SCRAPING_LOG_NOT_ALLOWED", "Sumber log ini tidak dapat dihapus dari panel.", 400)
|
||||||
|
cursor.execute(
|
||||||
|
"DELETE FROM price_history WHERE scraping_log_id = %s",
|
||||||
|
(log_id,),
|
||||||
|
)
|
||||||
|
deleted_records = cursor.rowcount
|
||||||
|
cursor.execute("DELETE FROM scraping_logs WHERE id = %s", (log_id,))
|
||||||
|
connection.commit()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": {"log_id": log_id, "deleted_records": deleted_records}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.post("/admin/scraping/jobs")
|
||||||
|
@admin_required
|
||||||
|
def start_scraping_job():
|
||||||
|
payload = request.get_json(silent=True) or {}
|
||||||
|
selected_slugs = payload.get("commodities") or list(current_app.config["SISKAPERBAPO_COMMODITIES"])
|
||||||
|
if not isinstance(selected_slugs, list) or not selected_slugs:
|
||||||
|
raise ApiError("INVALID_SCRAPING_OPTIONS", "Pilih minimal satu jenis cabai.", 400)
|
||||||
|
selected_commodities = {
|
||||||
|
slug: COMMODITIES[slug] for slug in selected_slugs if slug in COMMODITIES
|
||||||
|
}
|
||||||
|
if not selected_commodities:
|
||||||
|
raise ApiError("INVALID_SCRAPING_OPTIONS", "Jenis cabai tidak dikenali.", 400)
|
||||||
|
try:
|
||||||
|
start_date = date.fromisoformat(str(payload["start_date"])) if payload.get("start_date") else None
|
||||||
|
end_date = date.fromisoformat(str(payload["end_date"])) if payload.get("end_date") else None
|
||||||
|
max_records = int(payload["max_records"]) if payload.get("max_records") else None
|
||||||
|
except (TypeError, ValueError) as error:
|
||||||
|
raise ApiError("INVALID_SCRAPING_OPTIONS", "Tanggal atau jumlah data tidak valid.", 400) from error
|
||||||
|
if start_date and end_date and start_date > end_date:
|
||||||
|
raise ApiError("INVALID_SCRAPING_OPTIONS", "Tanggal mulai tidak boleh setelah tanggal akhir.", 400)
|
||||||
|
if max_records is not None and not 1 <= max_records <= 10000:
|
||||||
|
raise ApiError("INVALID_SCRAPING_OPTIONS", "Jumlah data harus antara 1 dan 10.000.", 400)
|
||||||
|
job_id = uuid4().hex
|
||||||
|
job = {"id": job_id, "status": "running", "progress": 0, "message": "Mengantri scraping"}
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
if any(item["status"] == "running" for item in _scraping_jobs.values()) or any(item["status"] == "running" for item in _preprocessing_jobs.values()):
|
||||||
|
raise ApiError("PROCESS_IN_PROGRESS", "Proses scraping atau preprocessing lain masih berjalan.", 409)
|
||||||
|
_scraping_jobs[job_id] = job
|
||||||
|
thread = Thread(
|
||||||
|
target=_run_scraping_job,
|
||||||
|
args=(current_app._get_current_object(), job_id, selected_commodities, start_date, end_date, max_records),
|
||||||
|
daemon=True,
|
||||||
|
)
|
||||||
|
thread.start()
|
||||||
|
return jsonify({"data": _scraping_job_payload(job)}), 202
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/admin/scraping/jobs/<job_id>")
|
||||||
|
@admin_required
|
||||||
|
def scraping_job_status(job_id):
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
job = _scraping_jobs.get(job_id)
|
||||||
|
if not job:
|
||||||
|
raise ApiError("SCRAPING_JOB_NOT_FOUND", "Proses scraping tidak ditemukan.", 404)
|
||||||
|
return jsonify({"data": _scraping_job_payload(job)})
|
||||||
|
|
||||||
|
|
||||||
|
@api.post("/admin/preprocessing/jobs")
|
||||||
|
@admin_required
|
||||||
|
def start_preprocessing_job():
|
||||||
|
payload = request.get_json(silent=True) or {}
|
||||||
|
try:
|
||||||
|
scraping_log_id = int(payload.get("scraping_log_id"))
|
||||||
|
except (TypeError, ValueError) as error:
|
||||||
|
raise ApiError("INVALID_PREPROCESSING_OPTIONS", "Pilih riwayat scraping yang akan diproses.", 400) from error
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT id, status, records_inserted FROM scraping_logs WHERE id = %s AND source = 'siskaperbapo'",
|
||||||
|
(scraping_log_id,),
|
||||||
|
)
|
||||||
|
log = cursor.fetchone()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
if not log:
|
||||||
|
raise ApiError("SCRAPING_LOG_NOT_FOUND", "Riwayat scraping yang dipilih tidak ditemukan.", 404)
|
||||||
|
if log[1] != "success" or not log[2]:
|
||||||
|
raise ApiError("INVALID_PREPROCESSING_OPTIONS", "Riwayat scraping tidak memiliki data valid untuk diproses.", 400)
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
if any(item["status"] == "running" for item in _scraping_jobs.values()) or any(item["status"] == "running" for item in _preprocessing_jobs.values()):
|
||||||
|
raise ApiError("PROCESS_IN_PROGRESS", "Proses scraping atau preprocessing lain masih berjalan.", 409)
|
||||||
|
job_id = uuid4().hex
|
||||||
|
job = {"id": job_id, "status": "running", "progress": 0, "message": "Mengantri preprocessing", "rows_filled": 0}
|
||||||
|
_preprocessing_jobs[job_id] = job
|
||||||
|
thread = Thread(
|
||||||
|
target=_run_preprocessing_job,
|
||||||
|
args=(current_app._get_current_object(), job_id, scraping_log_id),
|
||||||
|
daemon=True,
|
||||||
|
)
|
||||||
|
thread.start()
|
||||||
|
return jsonify({"data": _preprocessing_job_payload(job)}), 202
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/admin/preprocessing/jobs/<job_id>")
|
||||||
|
@admin_required
|
||||||
|
def preprocessing_job_status(job_id):
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
job = _preprocessing_jobs.get(job_id)
|
||||||
|
if not job:
|
||||||
|
raise ApiError("PREPROCESSING_JOB_NOT_FOUND", "Proses preprocessing tidak ditemukan.", 404)
|
||||||
|
return jsonify({"data": _preprocessing_job_payload(job)})
|
||||||
|
|
||||||
|
|
||||||
|
@api.delete("/admin/scraping/history")
|
||||||
|
@admin_required
|
||||||
|
def delete_scraping_history():
|
||||||
|
with _scraping_jobs_lock:
|
||||||
|
if any(item["status"] == "running" for item in _scraping_jobs.values()):
|
||||||
|
raise ApiError("SCRAPING_IN_PROGRESS", "Tunggu scraping selesai sebelum menghapus riwayat.", 409)
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute("DELETE FROM price_history WHERE source = 'siskaperbapo'")
|
||||||
|
deleted_records = cursor.rowcount
|
||||||
|
cursor.execute("DELETE FROM scraping_logs WHERE source = 'siskaperbapo'")
|
||||||
|
deleted_logs = cursor.rowcount
|
||||||
|
connection.commit()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": {"deleted_records": deleted_records, "deleted_logs": deleted_logs}})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/admin/models")
|
||||||
|
@admin_required
|
||||||
|
def admin_models():
|
||||||
|
connection = _db()
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT mv.id, c.name, mv.version, mv.metrics_json, mv.is_active, mv.created_at
|
||||||
|
FROM model_versions mv JOIN commodities c ON c.id = mv.commodity_id
|
||||||
|
ORDER BY mv.created_at DESC
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
rows = cursor.fetchall()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return jsonify({"data": [{"id": row[0], "commodity": row[1], "version": row[2], "metrics": row[3], "is_active": bool(row[4]), "created_at": row[5].isoformat()} for row in rows]})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/health")
|
||||||
|
def health():
|
||||||
|
model_registry = registry()
|
||||||
|
loaded = sum(len(markets) for markets in model_registry.artifacts.values())
|
||||||
|
return jsonify({"status": "healthy" if loaded else "degraded", "models_loaded": loaded, "loaded": {commodity: sorted(markets) for commodity, markets in model_registry.artifacts.items()}, "skipped": model_registry.skipped})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/commodities")
|
||||||
|
def commodities():
|
||||||
|
model_registry = registry()
|
||||||
|
data = [{"slug": slug, "name": name, "market_count": len(model_registry.available_markets(slug))} for slug, name in COMMODITIES.items() if model_registry.available_markets(slug)]
|
||||||
|
return jsonify({"data": data})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/markets")
|
||||||
|
def markets():
|
||||||
|
commodity = request.args.get("commodity", "").strip()
|
||||||
|
if commodity and commodity not in COMMODITIES:
|
||||||
|
raise ApiError("UNKNOWN_COMMODITY", "Komoditas tidak dikenal.", 404)
|
||||||
|
model_registry = registry()
|
||||||
|
data = []
|
||||||
|
for market in markets_config():
|
||||||
|
available = [slug for slug in COMMODITIES if market["slug"] in model_registry.available_markets(slug)]
|
||||||
|
if commodity:
|
||||||
|
if market["slug"] not in model_registry.available_markets(commodity):
|
||||||
|
continue
|
||||||
|
data.append({**market, "available": True})
|
||||||
|
else:
|
||||||
|
data.append({**market, "available_commodities": available})
|
||||||
|
return jsonify({"data": data})
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/history")
|
||||||
|
def history():
|
||||||
|
commodity = required_query("commodity")
|
||||||
|
market = required_query("market")
|
||||||
|
return jsonify(service().history(commodity, market, parse_days()))
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/predict")
|
||||||
|
def predict():
|
||||||
|
commodity = required_query("commodity")
|
||||||
|
market = required_query("market")
|
||||||
|
return jsonify(service().predict(commodity, market, parse_horizon(current_app.config["ALLOWED_HORIZONS"], current_app.config["MAX_HORIZON"])))
|
||||||
|
|
||||||
|
|
||||||
|
@api.get("/predict/map")
|
||||||
|
def predict_map():
|
||||||
|
commodity = required_query("commodity")
|
||||||
|
horizon = parse_horizon(current_app.config["ALLOWED_HORIZONS"], current_app.config["MAX_HORIZON"])
|
||||||
|
if commodity not in COMMODITIES:
|
||||||
|
raise ApiError("UNKNOWN_COMMODITY", "Komoditas tidak dikenal.", 404)
|
||||||
|
return jsonify(service().predict_map(commodity, horizon, markets_config()))
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
from flask import request
|
||||||
|
|
||||||
|
from app.errors import ApiError
|
||||||
|
|
||||||
|
|
||||||
|
def required_query(name: str) -> str:
|
||||||
|
value = request.args.get(name, "").strip()
|
||||||
|
if not value:
|
||||||
|
raise ApiError("INVALID_PARAMETER", f"Parameter {name} wajib diisi.", 400)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def parse_horizon(allowed: Iterable[int], maximum: int) -> int:
|
||||||
|
raw = request.args.get("horizon", "1")
|
||||||
|
try:
|
||||||
|
horizon = int(raw)
|
||||||
|
except ValueError:
|
||||||
|
raise ApiError("INVALID_HORIZON", "Horizon harus berupa angka.", 400, {"allowed": list(allowed)})
|
||||||
|
allowed_values = list(allowed)
|
||||||
|
if horizon not in allowed_values or horizon > maximum:
|
||||||
|
raise ApiError("INVALID_HORIZON", f"Horizon harus salah satu dari {', '.join(map(str, allowed_values))}.", 400, {"allowed": allowed_values})
|
||||||
|
return horizon
|
||||||
|
|
||||||
|
|
||||||
|
def parse_days() -> int:
|
||||||
|
raw = request.args.get("days", "90")
|
||||||
|
try:
|
||||||
|
days = int(raw)
|
||||||
|
except ValueError:
|
||||||
|
raise ApiError("INVALID_PARAMETER", "Parameter days harus berupa angka.", 400)
|
||||||
|
if days < 1 or days > 365:
|
||||||
|
raise ApiError("INVALID_PARAMETER", "Parameter days harus berada antara 1 dan 365.", 400, {"maximum": 365})
|
||||||
|
return days
|
||||||
|
|
@ -0,0 +1,38 @@
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
BASE_DIR = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
|
|
||||||
|
class Config:
|
||||||
|
SECRET_KEY = os.getenv("SECRET_KEY", "dev-only-change-this-secret")
|
||||||
|
MODELS_DIR = Path(os.getenv("MODELS_DIR", str(BASE_DIR / "models"))).resolve()
|
||||||
|
DATA_DIR = Path(os.getenv("DATA_DIR", str(BASE_DIR))).resolve()
|
||||||
|
DATA_SOURCE = os.getenv("DATA_SOURCE", "mysql").strip().lower()
|
||||||
|
MYSQL_HOST = os.getenv("MYSQL_HOST", "127.0.0.1")
|
||||||
|
MYSQL_PORT = int(os.getenv("MYSQL_PORT", "3306"))
|
||||||
|
MYSQL_DATABASE = os.getenv("MYSQL_DATABASE", "prediksicabe")
|
||||||
|
MYSQL_USER = os.getenv("MYSQL_USER", "root")
|
||||||
|
MYSQL_PASSWORD = os.getenv("MYSQL_PASSWORD", "")
|
||||||
|
MYSQL_CONNECT_TIMEOUT = int(os.getenv("MYSQL_CONNECT_TIMEOUT", "5"))
|
||||||
|
PRICE_UNIT_SCALE = float(os.getenv("PRICE_UNIT_SCALE", "1000"))
|
||||||
|
ALLOWED_HORIZONS = (1, 3, 7, 14, 30)
|
||||||
|
MAX_HORIZON = 30
|
||||||
|
CORS_ORIGINS = os.getenv(
|
||||||
|
"CORS_ORIGINS",
|
||||||
|
"http://127.0.0.1:5173,http://localhost:5173",
|
||||||
|
)
|
||||||
|
STALE_DATA_DAYS = int(os.getenv("STALE_DATA_DAYS", "30"))
|
||||||
|
TREND_THRESHOLD_PCT = float(os.getenv("TREND_THRESHOLD_PCT", "1"))
|
||||||
|
SISKAPERBAPO_URL = os.getenv(
|
||||||
|
"SISKAPERBAPO_URL",
|
||||||
|
"https://siskaperbapo.jatimprov.go.id/harga-komoditas",
|
||||||
|
)
|
||||||
|
SISKAPERBAPO_COMMODITY_PARAM = os.getenv("SISKAPERBAPO_COMMODITY_PARAM", "komoditas")
|
||||||
|
SISKAPERBAPO_TIMEOUT = int(os.getenv("SISKAPERBAPO_TIMEOUT", "30"))
|
||||||
|
SISKAPERBAPO_COMMODITIES = tuple(
|
||||||
|
value.strip()
|
||||||
|
for value in os.getenv("SISKAPERBAPO_COMMODITIES", "rawit").split(",")
|
||||||
|
if value.strip()
|
||||||
|
)
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,32 @@
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"slug": "pasar_tanjung",
|
||||||
|
"name": "Pasar Tanjung",
|
||||||
|
"latitude": -8.1738152,
|
||||||
|
"longitude": 113.6954052
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"slug": "pasar_kreongan",
|
||||||
|
"name": "Pasar Kreongan",
|
||||||
|
"latitude": -8.1556968,
|
||||||
|
"longitude": 113.7008754
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"slug": "pasar_tegal_besar",
|
||||||
|
"name": "Pasar Tegal Besar",
|
||||||
|
"latitude": -8.1926152,
|
||||||
|
"longitude": 113.6987226
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"slug": "pasar_mangli",
|
||||||
|
"name": "Pasar Mangli",
|
||||||
|
"latitude": -8.1902978,
|
||||||
|
"longitude": 113.6493539
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"slug": "pasar_wirolegi",
|
||||||
|
"name": "Pasar Wirolegi",
|
||||||
|
"latitude": -8.1872926,
|
||||||
|
"longitude": 113.7431422
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
@ -0,0 +1,158 @@
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from datetime import date
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pymysql
|
||||||
|
from pymysql import MySQLError
|
||||||
|
|
||||||
|
|
||||||
|
class DataUnavailableError(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class PriceRepository(ABC):
|
||||||
|
@abstractmethod
|
||||||
|
def get_series(self, commodity: str, market: str) -> pd.Series:
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def last_data_date(self, commodity: str, market: str) -> date:
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
|
||||||
|
class CsvPriceRepository(PriceRepository):
|
||||||
|
FILE_PATTERNS = {
|
||||||
|
"rawit": ("*rawit*.csv",),
|
||||||
|
"merah": ("*cabe_merah_besar*.csv", "*merah_besar*.csv"),
|
||||||
|
"keriting": ("*merah_keriting*.csv",),
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(self, data_dir: Path, unit_scale: float = 1.0) -> None:
|
||||||
|
self.data_dir = data_dir
|
||||||
|
self.unit_scale = unit_scale
|
||||||
|
self._frames: Dict[str, pd.DataFrame] = {}
|
||||||
|
|
||||||
|
def _find_file(self, commodity: str) -> Path:
|
||||||
|
candidates = [self.data_dir / f"{commodity}.csv", self.data_dir / f"data_{commodity}.csv"]
|
||||||
|
for pattern in self.FILE_PATTERNS.get(commodity, (f"*{commodity}*.csv",)):
|
||||||
|
candidates.extend(sorted(self.data_dir.rglob(pattern))) if self.data_dir.exists() else None
|
||||||
|
for candidate in candidates:
|
||||||
|
if candidate.is_file():
|
||||||
|
return candidate
|
||||||
|
raise DataUnavailableError(
|
||||||
|
f"File CSV untuk komoditas {commodity} tidak ditemukan di {self.data_dir}."
|
||||||
|
)
|
||||||
|
|
||||||
|
def _load(self, commodity: str) -> pd.DataFrame:
|
||||||
|
if commodity in self._frames:
|
||||||
|
return self._frames[commodity]
|
||||||
|
path = self._find_file(commodity)
|
||||||
|
frame = pd.read_csv(path)
|
||||||
|
date_column = next(
|
||||||
|
(column for column in frame.columns if str(column).strip().lower() in {"date", "tanggal", "datetime"}),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if date_column is None:
|
||||||
|
raise DataUnavailableError(
|
||||||
|
f"CSV {path.name} harus memiliki kolom date atau tanggal."
|
||||||
|
)
|
||||||
|
frame[date_column] = pd.to_datetime(frame[date_column], errors="coerce")
|
||||||
|
frame = frame.dropna(subset=[date_column]).set_index(date_column).sort_index()
|
||||||
|
if not frame.empty:
|
||||||
|
frame = frame[~frame.index.duplicated(keep="last")]
|
||||||
|
frame = frame.reindex(pd.date_range(frame.index.min(), frame.index.max(), freq="D"))
|
||||||
|
self._frames[commodity] = frame
|
||||||
|
return frame
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _market_column(frame: pd.DataFrame, market: str) -> str:
|
||||||
|
aliases = {market, market.replace("pasar_", "Pasar_")}
|
||||||
|
for column in frame.columns:
|
||||||
|
normalized = str(column).strip().lower().replace(" ", "_")
|
||||||
|
if normalized in {alias.lower() for alias in aliases}:
|
||||||
|
return column
|
||||||
|
raise DataUnavailableError(f"Kolom harga untuk pasar {market} tidak ditemukan.")
|
||||||
|
|
||||||
|
def get_series(self, commodity: str, market: str) -> pd.Series:
|
||||||
|
frame = self._load(commodity)
|
||||||
|
column = self._market_column(frame, market)
|
||||||
|
series = pd.to_numeric(frame[column], errors="coerce").replace(0, np.nan)
|
||||||
|
series = series * self.unit_scale
|
||||||
|
series = series.interpolate(method="linear", limit=2, limit_area="inside")
|
||||||
|
return series.astype(float)
|
||||||
|
|
||||||
|
def last_data_date(self, commodity: str, market: str) -> date:
|
||||||
|
series = self.get_series(commodity, market).dropna()
|
||||||
|
if series.empty:
|
||||||
|
raise DataUnavailableError(
|
||||||
|
f"Tidak ada data valid untuk {commodity}/{market}."
|
||||||
|
)
|
||||||
|
return series.index[-1].date()
|
||||||
|
|
||||||
|
|
||||||
|
class MySQLPriceRepository(PriceRepository):
|
||||||
|
def __init__(self, config) -> None:
|
||||||
|
self.config = config
|
||||||
|
|
||||||
|
def _connect(self):
|
||||||
|
return pymysql.connect(
|
||||||
|
host=self.config["MYSQL_HOST"],
|
||||||
|
port=self.config["MYSQL_PORT"],
|
||||||
|
database=self.config["MYSQL_DATABASE"],
|
||||||
|
user=self.config["MYSQL_USER"],
|
||||||
|
password=self.config["MYSQL_PASSWORD"],
|
||||||
|
connect_timeout=self.config["MYSQL_CONNECT_TIMEOUT"],
|
||||||
|
)
|
||||||
|
|
||||||
|
def _load(self, commodity: str, market: str) -> pd.Series:
|
||||||
|
connection = None
|
||||||
|
cursor = None
|
||||||
|
try:
|
||||||
|
connection = self._connect()
|
||||||
|
cursor = connection.cursor()
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
SELECT ph.price_date, ph.price
|
||||||
|
FROM price_history AS ph
|
||||||
|
INNER JOIN commodities AS c ON c.id = ph.commodity_id
|
||||||
|
INNER JOIN markets AS m ON m.id = ph.market_id
|
||||||
|
WHERE c.slug = %s AND m.slug = %s
|
||||||
|
ORDER BY ph.price_date ASC
|
||||||
|
""",
|
||||||
|
(commodity, market),
|
||||||
|
)
|
||||||
|
rows = cursor.fetchall()
|
||||||
|
except MySQLError as error:
|
||||||
|
raise DataUnavailableError(f"Gagal membaca database MySQL: {error}") from error
|
||||||
|
finally:
|
||||||
|
if cursor is not None:
|
||||||
|
cursor.close()
|
||||||
|
if connection is not None:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
if not rows:
|
||||||
|
raise DataUnavailableError(
|
||||||
|
f"Tidak ada data harga untuk {commodity}/{market} di database."
|
||||||
|
)
|
||||||
|
frame = pd.DataFrame(rows, columns=["date", "price"])
|
||||||
|
frame["date"] = pd.to_datetime(frame["date"], errors="coerce")
|
||||||
|
frame["price"] = pd.to_numeric(frame["price"], errors="coerce")
|
||||||
|
frame = frame.dropna(subset=["date"]).set_index("date").sort_index()
|
||||||
|
frame = frame[~frame.index.duplicated(keep="last")]
|
||||||
|
frame["price"] = frame["price"].replace(0, np.nan)
|
||||||
|
frame["price"] = frame["price"].interpolate(method="linear", limit=2, limit_area="inside")
|
||||||
|
return frame["price"].astype(float)
|
||||||
|
|
||||||
|
def get_series(self, commodity: str, market: str) -> pd.Series:
|
||||||
|
return self._load(commodity, market)
|
||||||
|
|
||||||
|
def last_data_date(self, commodity: str, market: str) -> date:
|
||||||
|
series = self.get_series(commodity, market).dropna()
|
||||||
|
if series.empty:
|
||||||
|
raise DataUnavailableError(
|
||||||
|
f"Tidak ada data valid untuk {commodity}/{market} di database."
|
||||||
|
)
|
||||||
|
return series.index[-1].date()
|
||||||
|
|
@ -0,0 +1,44 @@
|
||||||
|
from typing import Any, Dict, Optional
|
||||||
|
|
||||||
|
from flask import jsonify
|
||||||
|
|
||||||
|
|
||||||
|
class ApiError(Exception):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
code: str,
|
||||||
|
message: str,
|
||||||
|
status_code: int,
|
||||||
|
details: Optional[Dict[str, Any]] = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(message)
|
||||||
|
self.code = code
|
||||||
|
self.message = message
|
||||||
|
self.status_code = status_code
|
||||||
|
self.details = details or {}
|
||||||
|
|
||||||
|
def to_response(self):
|
||||||
|
return jsonify(
|
||||||
|
{
|
||||||
|
"error": {
|
||||||
|
"code": self.code,
|
||||||
|
"message": self.message,
|
||||||
|
"details": self.details,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
), self.status_code
|
||||||
|
|
||||||
|
|
||||||
|
def register_error_handlers(app) -> None:
|
||||||
|
@app.errorhandler(ApiError)
|
||||||
|
def handle_api_error(error: ApiError):
|
||||||
|
return error.to_response()
|
||||||
|
|
||||||
|
@app.errorhandler(Exception)
|
||||||
|
def handle_unexpected_error(error: Exception):
|
||||||
|
app.logger.exception("Unhandled API error: %s", error)
|
||||||
|
return ApiError(
|
||||||
|
"INTERNAL_ERROR",
|
||||||
|
"Terjadi kesalahan internal pada server.",
|
||||||
|
500,
|
||||||
|
).to_response()
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,30 @@
|
||||||
|
from typing import Any, List, Tuple
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
def forecast_recursive(
|
||||||
|
model: Any,
|
||||||
|
scaler: Any,
|
||||||
|
window: np.ndarray,
|
||||||
|
horizon: int,
|
||||||
|
) -> Tuple[List[float], bool]:
|
||||||
|
values = np.asarray(window, dtype=float).reshape(-1, 1)
|
||||||
|
scaled_window = scaler.transform(values)
|
||||||
|
scaled_predictions = []
|
||||||
|
|
||||||
|
for _ in range(horizon):
|
||||||
|
model_input = scaled_window.reshape(1, len(scaled_window), 1)
|
||||||
|
prediction = model.predict(model_input, verbose=0)
|
||||||
|
prediction_value = float(np.asarray(prediction).reshape(-1)[0])
|
||||||
|
scaled_predictions.append(prediction_value)
|
||||||
|
scaled_window = np.vstack([scaled_window[1:], [[prediction_value]]])
|
||||||
|
|
||||||
|
predicted_values = scaler.inverse_transform(
|
||||||
|
np.asarray(scaled_predictions, dtype=float).reshape(-1, 1)
|
||||||
|
).reshape(-1)
|
||||||
|
data_min = float(np.asarray(scaler.data_min_).reshape(-1)[0])
|
||||||
|
data_max = float(np.asarray(scaler.data_max_).reshape(-1)[0])
|
||||||
|
out_of_range = bool(np.any((predicted_values < data_min) | (predicted_values > data_max)))
|
||||||
|
prices = [max(0.0, float(np.round(value, 6))) for value in predicted_values]
|
||||||
|
return prices, out_of_range
|
||||||
|
|
@ -0,0 +1,100 @@
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import pickle
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
import joblib
|
||||||
|
from tensorflow.keras.models import load_model
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
COMMODITIES = {
|
||||||
|
"rawit": "Cabai Rawit",
|
||||||
|
"merah": "Cabai Merah Besar",
|
||||||
|
"keriting": "Cabai Merah Keriting",
|
||||||
|
}
|
||||||
|
MARKETS = {
|
||||||
|
"pasar_tanjung": "Pasar Tanjung",
|
||||||
|
"pasar_kreongan": "Pasar Kreongan",
|
||||||
|
"pasar_tegal_besar": "Pasar Tegal Besar",
|
||||||
|
"pasar_mangli": "Pasar Mangli",
|
||||||
|
"pasar_wirolegi": "Pasar Wirolegi",
|
||||||
|
}
|
||||||
|
DEFAULT_TIME_STEPS = {"rawit": 7, "merah": 7, "keriting": 14}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ModelArtifact:
|
||||||
|
commodity: str
|
||||||
|
market: str
|
||||||
|
model: Any
|
||||||
|
scaler: Any
|
||||||
|
config: Dict[str, Any]
|
||||||
|
time_step: int
|
||||||
|
model_time_step: int
|
||||||
|
|
||||||
|
|
||||||
|
class ModelRegistry:
|
||||||
|
def __init__(self, models_dir: Path) -> None:
|
||||||
|
self.models_dir = models_dir
|
||||||
|
self.artifacts: Dict[str, Dict[str, ModelArtifact]] = {}
|
||||||
|
self.skipped: List[Dict[str, str]] = []
|
||||||
|
self.scan()
|
||||||
|
|
||||||
|
def scan(self) -> None:
|
||||||
|
self.artifacts.clear()
|
||||||
|
self.skipped.clear()
|
||||||
|
for commodity, commodity_name in COMMODITIES.items():
|
||||||
|
commodity_dir = self.models_dir / commodity
|
||||||
|
for market in MARKETS:
|
||||||
|
market_dir = commodity_dir / market
|
||||||
|
config_path = market_dir / "config.json"
|
||||||
|
if not config_path.is_file():
|
||||||
|
self._skip(commodity, market, "config.json tidak ditemukan")
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
config = json.loads(config_path.read_text(encoding="utf-8"))
|
||||||
|
files = config.get("files", {})
|
||||||
|
model_path = self._trusted_path(market_dir, files.get("model"))
|
||||||
|
scaler_path = self._trusted_path(market_dir, files.get("scaler"))
|
||||||
|
if not model_path.is_file() or not scaler_path.is_file():
|
||||||
|
raise FileNotFoundError("model atau scaler tidak lengkap")
|
||||||
|
model = load_model(model_path, compile=False)
|
||||||
|
try:
|
||||||
|
scaler = joblib.load(scaler_path)
|
||||||
|
except Exception:
|
||||||
|
with scaler_path.open("rb") as scaler_file:
|
||||||
|
scaler = pickle.load(scaler_file)
|
||||||
|
if not hasattr(scaler, "data_min_") or not hasattr(scaler, "data_max_"):
|
||||||
|
raise ValueError("scaler tidak memiliki data_min_ dan data_max_")
|
||||||
|
model_time_step = int(model.input_shape[1])
|
||||||
|
configured_step = int(config.get("model", {}).get("time_step", DEFAULT_TIME_STEPS[commodity]))
|
||||||
|
time_step = DEFAULT_TIME_STEPS[commodity]
|
||||||
|
if commodity == "keriting" and configured_step != time_step:
|
||||||
|
logger.warning("Menggunakan time_step API 14 untuk keriting; config model menyatakan %s", configured_step)
|
||||||
|
self.artifacts.setdefault(commodity, {})[market] = ModelArtifact(
|
||||||
|
commodity, market, model, scaler, config, time_step, model_time_step
|
||||||
|
)
|
||||||
|
except Exception as error:
|
||||||
|
self._skip(commodity, market, f"{type(error).__name__}: {error}")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _trusted_path(directory: Path, filename: Optional[str]) -> Path:
|
||||||
|
if not filename:
|
||||||
|
raise FileNotFoundError("nama file artifact tidak ada di config.json")
|
||||||
|
path = (directory / filename).resolve()
|
||||||
|
if path.parent != directory.resolve():
|
||||||
|
raise ValueError("path artifact keluar dari folder market")
|
||||||
|
return path
|
||||||
|
|
||||||
|
def _skip(self, commodity: str, market: str, reason: str) -> None:
|
||||||
|
logger.warning("Melewati model %s/%s: %s", commodity, market, reason)
|
||||||
|
self.skipped.append({"commodity": commodity, "market": market, "reason": reason})
|
||||||
|
|
||||||
|
def get(self, commodity: str, market: str) -> Optional[ModelArtifact]:
|
||||||
|
return self.artifacts.get(commodity, {}).get(market)
|
||||||
|
|
||||||
|
def available_markets(self, commodity: str) -> List[str]:
|
||||||
|
return sorted(self.artifacts.get(commodity, {}))
|
||||||
|
|
@ -0,0 +1,112 @@
|
||||||
|
from datetime import date, datetime, timedelta, timezone
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from app.errors import ApiError
|
||||||
|
from app.services.forecaster import forecast_recursive
|
||||||
|
from app.services.model_registry import COMMODITIES, MARKETS, ModelRegistry
|
||||||
|
from app.data.repository import DataUnavailableError, PriceRepository
|
||||||
|
|
||||||
|
|
||||||
|
NOTES = [
|
||||||
|
"Model menghasilkan prediksi rekursif hingga 30 hari; error dapat menumpuk pada horizon panjang.",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class PredictionService:
|
||||||
|
def __init__(self, registry: ModelRegistry, repository: PriceRepository, stale_days: int, trend_threshold_pct: float) -> None:
|
||||||
|
self.registry = registry
|
||||||
|
self.repository = repository
|
||||||
|
self.stale_days = stale_days
|
||||||
|
self.trend_threshold_pct = trend_threshold_pct
|
||||||
|
self._cache: Dict[Any, Dict[str, Any]] = {}
|
||||||
|
|
||||||
|
def _validate(self, commodity: str, market: str) -> Any:
|
||||||
|
if commodity not in COMMODITIES:
|
||||||
|
raise ApiError("UNKNOWN_COMMODITY", "Komoditas tidak dikenal.", 404)
|
||||||
|
if market not in MARKETS:
|
||||||
|
raise ApiError("UNKNOWN_MARKET", "Pasar tidak dikenal.", 404)
|
||||||
|
artifact = self.registry.get(commodity, market)
|
||||||
|
if artifact is None:
|
||||||
|
raise ApiError("MODEL_NOT_AVAILABLE", "Model untuk kombinasi ini tidak tersedia.", 404)
|
||||||
|
return artifact
|
||||||
|
|
||||||
|
def _series(self, commodity: str, market: str):
|
||||||
|
try:
|
||||||
|
series = self.repository.get_series(commodity, market).dropna()
|
||||||
|
except DataUnavailableError as error:
|
||||||
|
raise ApiError("INSUFFICIENT_DATA", str(error), 422)
|
||||||
|
if series.empty:
|
||||||
|
raise ApiError("INSUFFICIENT_DATA", "Data historis valid tidak tersedia.", 422)
|
||||||
|
return series
|
||||||
|
|
||||||
|
def predict(self, commodity: str, market: str, horizon: int) -> Dict[str, Any]:
|
||||||
|
artifact = self._validate(commodity, market)
|
||||||
|
series = self._series(commodity, market)
|
||||||
|
if len(series) < artifact.time_step:
|
||||||
|
raise ApiError("INSUFFICIENT_DATA", "Jumlah data valid belum mencukupi window model.", 422, {"required": artifact.time_step, "available": len(series)})
|
||||||
|
last_date = series.index[-1].date()
|
||||||
|
window_key = tuple(np.round(series.iloc[-artifact.time_step:].to_numpy(dtype=float), 6))
|
||||||
|
cache_key = (commodity, market, last_date, window_key)
|
||||||
|
cached = self._cache.get(cache_key)
|
||||||
|
out_of_range = False
|
||||||
|
if cached is None:
|
||||||
|
window = series.iloc[-artifact.time_step:].to_numpy(dtype=float) / 1000.0
|
||||||
|
if artifact.model_time_step != artifact.time_step:
|
||||||
|
raise ApiError("MODEL_NOT_AVAILABLE", "Artifact model keriting masih berformat 7 langkah, sedangkan API memerlukan 14 langkah.", 404, {"api_time_step": artifact.time_step, "model_time_step": artifact.model_time_step})
|
||||||
|
prices_in_thousand, out_of_range = forecast_recursive(artifact.model, artifact.scaler, window, 30)
|
||||||
|
prices = [price * 1000 for price in prices_in_thousand]
|
||||||
|
cached = {"prices": prices, "out_of_range": out_of_range}
|
||||||
|
self._cache[cache_key] = cached
|
||||||
|
prices = cached["prices"][:horizon]
|
||||||
|
last_actual = float(series.iloc[-1])
|
||||||
|
warnings: List[str] = []
|
||||||
|
data_age_days = (date.today() - last_date).days
|
||||||
|
if data_age_days > self.stale_days:
|
||||||
|
warnings.append("data_stale")
|
||||||
|
if cached["out_of_range"]:
|
||||||
|
warnings.append("out_of_training_range")
|
||||||
|
predictions = []
|
||||||
|
previous = last_actual
|
||||||
|
for step, price in enumerate(prices, 1):
|
||||||
|
change = price - previous
|
||||||
|
change_pct = round(change / previous * 100, 2) if previous else None
|
||||||
|
predictions.append({"step": step, "date": (last_date + timedelta(days=step)).isoformat(), "price": price, "change_rp": change, "change_pct": change_pct})
|
||||||
|
previous = price
|
||||||
|
first_price = prices[0]
|
||||||
|
last_price = prices[-1]
|
||||||
|
total_pct = ((last_price - last_actual) / last_actual * 100) if last_actual else 0
|
||||||
|
trend = "up" if total_pct > self.trend_threshold_pct else "down" if total_pct < -self.trend_threshold_pct else "flat"
|
||||||
|
return {"data": {"commodity": {"slug": commodity, "name": COMMODITIES[commodity]}, "market": {"slug": market, "name": MARKETS[market]}, "horizon": horizon, "last_data_date": last_date.isoformat(), "last_actual_price": last_actual, "predictions": predictions, "summary": {"first_price": first_price, "last_price": last_price, "min_price": min(prices), "max_price": max(prices), "average_price": sum(prices) / len(prices), "trend": trend}}, "meta": {"method": "recursive_multi_step", "time_step": artifact.time_step, "data_age_days": data_age_days, "generated_at": datetime.now(timezone(timedelta(hours=7))).isoformat(), "warnings": warnings, "notes": NOTES}}
|
||||||
|
|
||||||
|
def history(self, commodity: str, market: str, days: int) -> Dict[str, Any]:
|
||||||
|
self._validate(commodity, market)
|
||||||
|
try:
|
||||||
|
series = self.repository.get_series(commodity, market)
|
||||||
|
except DataUnavailableError as error:
|
||||||
|
raise ApiError("INSUFFICIENT_DATA", str(error), 422)
|
||||||
|
if series.dropna().empty:
|
||||||
|
raise ApiError("INSUFFICIENT_DATA", "Data historis valid tidak tersedia.", 422)
|
||||||
|
selected = series.tail(days)
|
||||||
|
return {"data": [{"date": index.date().isoformat(), "price": None if np.isnan(value) else float(value)} for index, value in selected.items()]}
|
||||||
|
|
||||||
|
def predict_map(self, commodity: str, horizon: int, markets: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||||
|
if commodity not in COMMODITIES:
|
||||||
|
raise ApiError("UNKNOWN_COMMODITY", "Komoditas tidak dikenal.", 404)
|
||||||
|
data = []
|
||||||
|
for market_info in markets:
|
||||||
|
market = market_info["slug"]
|
||||||
|
artifact = self.registry.get(commodity, market)
|
||||||
|
item = {**market_info, "market": market, "available": artifact is not None}
|
||||||
|
if artifact is None:
|
||||||
|
data.append(item)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
result = self.predict(commodity, market, horizon)["data"]
|
||||||
|
prediction = result["predictions"][-1]
|
||||||
|
item.update({"last_actual_price": result["last_actual_price"], "predicted_price": prediction["price"], "change_rp": prediction["change_rp"], "change_pct": prediction["change_pct"], "trend": result["summary"]["trend"]})
|
||||||
|
except ApiError as error:
|
||||||
|
item.update({"available": False, "reason": error.code})
|
||||||
|
data.append(item)
|
||||||
|
return {"data": data, "meta": {"commodity": {"slug": commodity, "name": COMMODITIES[commodity]}, "horizon": horizon}}
|
||||||
|
|
@ -0,0 +1,193 @@
|
||||||
|
from datetime import date, timedelta
|
||||||
|
from io import StringIO
|
||||||
|
from typing import Dict, List
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
class ScrapingError(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class SiskaperbapoScraper:
|
||||||
|
MARKET_ALIASES = {
|
||||||
|
"pasar_tanjung": "Pasar Tanjung",
|
||||||
|
"pasar_kreongan": "Pasar Kreongan",
|
||||||
|
"pasar_tegal_besar": "Pasar Tegal Besar",
|
||||||
|
"pasar_mangli": "Pasar Mangli",
|
||||||
|
"pasar_wirolegi": "Pasar Wirolegi",
|
||||||
|
}
|
||||||
|
MARKET_IDS = {
|
||||||
|
"pasar_tanjung": "11",
|
||||||
|
"pasar_kreongan": "12",
|
||||||
|
"pasar_tegal_besar": "13",
|
||||||
|
"pasar_mangli": "148",
|
||||||
|
"pasar_wirolegi": "149",
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(self, url: str, commodity_param: str, timeout: int = 30) -> None:
|
||||||
|
self.url = url
|
||||||
|
self.commodity_param = commodity_param
|
||||||
|
self.timeout = timeout
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _parse_price(value) -> float:
|
||||||
|
if isinstance(value, (int, float)) and not pd.isna(value):
|
||||||
|
return float(value)
|
||||||
|
text = str(value).strip().replace("Rp", "").replace(" ", "")
|
||||||
|
if not text or text.lower() in {"nan", "none", "-", "0"}:
|
||||||
|
raise ValueError("harga kosong")
|
||||||
|
text = text.replace(".", "").replace(",", ".")
|
||||||
|
return float(text)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _price_in_rupiah(value) -> float:
|
||||||
|
price = SiskaperbapoScraper._parse_price(value)
|
||||||
|
return price * 1000 if 0 < price < 1000 else price
|
||||||
|
|
||||||
|
def _read_tables(self, commodity: str) -> List[pd.DataFrame]:
|
||||||
|
try:
|
||||||
|
response = requests.get(
|
||||||
|
self.url,
|
||||||
|
params={self.commodity_param: commodity},
|
||||||
|
timeout=self.timeout,
|
||||||
|
headers={"User-Agent": "PanenHarga/1.0"},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return pd.read_html(StringIO(response.text))
|
||||||
|
except (requests.RequestException, ValueError) as error:
|
||||||
|
raise ScrapingError(f"Gagal membaca halaman SISKAPERBAPO: {error}") from error
|
||||||
|
|
||||||
|
def _read_historical_table(self, price_date: date, market_id: str) -> List[pd.DataFrame]:
|
||||||
|
try:
|
||||||
|
response = requests.post(
|
||||||
|
self.url.replace("/harga-komoditas", "/harga/tabel.nodesign"),
|
||||||
|
data={"tanggal": price_date.isoformat(), "kabkota": "jemberkab", "pasar": market_id},
|
||||||
|
timeout=self.timeout,
|
||||||
|
headers={"User-Agent": "PanenHarga/1.0"},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return pd.read_html(StringIO(response.text))
|
||||||
|
except (requests.RequestException, ValueError) as error:
|
||||||
|
raise ScrapingError(f"Gagal membaca histori SISKAPERBAPO: {error}") from error
|
||||||
|
|
||||||
|
def scrape_range(self, commodities: Dict[str, str], start_date: date, end_date: date, on_progress=None) -> List[dict]:
|
||||||
|
observations = []
|
||||||
|
dates = [
|
||||||
|
start_date + timedelta(days=offset)
|
||||||
|
for offset in range((end_date - start_date).days + 1)
|
||||||
|
]
|
||||||
|
total = len(dates) * len(self.MARKET_IDS)
|
||||||
|
completed = 0
|
||||||
|
row_patterns = {
|
||||||
|
"rawit": "cabe rawit merah",
|
||||||
|
"merah": "cabe merah besar",
|
||||||
|
"keriting": "cabe merah keriting",
|
||||||
|
}
|
||||||
|
for price_date in dates:
|
||||||
|
for market_slug, market_id in self.MARKET_IDS.items():
|
||||||
|
tables = self._read_historical_table(price_date, market_id)
|
||||||
|
for table in tables:
|
||||||
|
if table.empty:
|
||||||
|
continue
|
||||||
|
for _, row in table.iterrows():
|
||||||
|
text = " ".join(str(value).strip() for value in row.tolist())
|
||||||
|
normalized = text.casefold().replace("-", " ")
|
||||||
|
commodity_slug = next(
|
||||||
|
(
|
||||||
|
slug for slug in commodities
|
||||||
|
if row_patterns.get(slug, "") in normalized
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if not commodity_slug:
|
||||||
|
continue
|
||||||
|
values = [
|
||||||
|
value for value in row.tolist()
|
||||||
|
if str(value).strip().replace(".", "").replace(",", "").isdigit()
|
||||||
|
]
|
||||||
|
if len(values) < 2:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
price = self._price_in_rupiah(values[1])
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
observations.append(
|
||||||
|
{
|
||||||
|
"commodity": commodity_slug,
|
||||||
|
"market": market_slug,
|
||||||
|
"date": price_date,
|
||||||
|
"price": price,
|
||||||
|
"source": "siskaperbapo",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
break
|
||||||
|
completed += 1
|
||||||
|
if on_progress:
|
||||||
|
on_progress(
|
||||||
|
10 + int(completed / max(total, 1) * 55),
|
||||||
|
f"Membaca {price_date.isoformat()} · {self.MARKET_ALIASES[market_slug]}",
|
||||||
|
)
|
||||||
|
if not observations:
|
||||||
|
raise ScrapingError(
|
||||||
|
f"Tidak ada data SISKAPERBAPO untuk periode {start_date.isoformat()} sampai {end_date.isoformat()}."
|
||||||
|
)
|
||||||
|
return observations
|
||||||
|
|
||||||
|
def scrape(self, commodities: Dict[str, str], on_progress=None) -> List[dict]:
|
||||||
|
observations = []
|
||||||
|
total_commodities = len(commodities)
|
||||||
|
for commodity_index, commodity_slug in enumerate(commodities, start=1):
|
||||||
|
if on_progress:
|
||||||
|
on_progress(
|
||||||
|
10 + int((commodity_index - 1) / max(total_commodities, 1) * 55),
|
||||||
|
f"Membaca data {commodity_slug}",
|
||||||
|
)
|
||||||
|
tables = self._read_tables(commodity_slug)
|
||||||
|
for table in tables:
|
||||||
|
if table.empty:
|
||||||
|
continue
|
||||||
|
location_column = next(
|
||||||
|
(column for column in table.columns if "lokasi" in str(column).lower() or "pasar" in str(column).lower()),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if location_column is None:
|
||||||
|
continue
|
||||||
|
date_columns = []
|
||||||
|
for column in table.columns:
|
||||||
|
try:
|
||||||
|
parsed = pd.to_datetime(str(column), errors="raise")
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
if not pd.isna(parsed):
|
||||||
|
date_columns.append((column, parsed.date()))
|
||||||
|
if not date_columns:
|
||||||
|
continue
|
||||||
|
for market_slug, market_name in self.MARKET_ALIASES.items():
|
||||||
|
matches = table[table[location_column].astype(str).str.strip().str.casefold() == market_name.casefold()]
|
||||||
|
if matches.empty:
|
||||||
|
continue
|
||||||
|
row = matches.iloc[0]
|
||||||
|
for column, price_date in date_columns:
|
||||||
|
try:
|
||||||
|
price = self._price_in_rupiah(row[column])
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
observations.append(
|
||||||
|
{
|
||||||
|
"commodity": commodity_slug,
|
||||||
|
"market": market_slug,
|
||||||
|
"date": price_date,
|
||||||
|
"price": price,
|
||||||
|
"source": "siskaperbapo",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if on_progress:
|
||||||
|
on_progress(
|
||||||
|
10 + int(commodity_index / max(total_commodities, 1) * 55),
|
||||||
|
f"Data {commodity_slug} selesai dibaca",
|
||||||
|
)
|
||||||
|
if not observations:
|
||||||
|
raise ScrapingError("Tidak ada data pasar Jember yang cocok dari SISKAPERBAPO.")
|
||||||
|
return observations
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,13 @@
|
||||||
|
import os
|
||||||
|
from app import create_app
|
||||||
|
|
||||||
|
|
||||||
|
app = create_app()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
app.run(
|
||||||
|
host="0.0.0.0",
|
||||||
|
port=int(os.getenv("PORT", "8000")),
|
||||||
|
debug=os.getenv("FLASK_DEBUG", "0") == "1",
|
||||||
|
)
|
||||||
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Keriting",
|
||||||
|
"market": "Pasar_Kreongan",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "relu",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 100,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 14
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 587,
|
||||||
|
"jumlah_sequence": 573,
|
||||||
|
"tanggal_awal": "2024-03-25",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.043749384582042694,
|
||||||
|
"loss_akhir": 0.003062221920117736
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_keriting_pasar_kreongan.h5",
|
||||||
|
"scaler": "scaler_keriting_pasar_kreongan.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-10-07T08:39:14.713085"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Keriting",
|
||||||
|
"market": "Pasar_Mangli",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "relu",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 100,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 14
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 144,
|
||||||
|
"jumlah_sequence": 130,
|
||||||
|
"tanggal_awal": "2025-06-10",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.12295424938201904,
|
||||||
|
"loss_akhir": 0.006369924172759056
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_keriting_pasar_mangli.h5",
|
||||||
|
"scaler": "scaler_keriting_pasar_mangli.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-10-07T08:39:14.744300"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Keriting",
|
||||||
|
"market": "Pasar_Wirolegi",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "relu",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 100,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 14
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 75,
|
||||||
|
"jumlah_sequence": 61,
|
||||||
|
"tanggal_awal": "2022-10-03",
|
||||||
|
"tanggal_akhir": "2022-12-16",
|
||||||
|
"loss_awal": 0.035635966807603836,
|
||||||
|
"loss_akhir": 0.001526627573184669
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_keriting_pasar_wirolegi.h5",
|
||||||
|
"scaler": "scaler_keriting_pasar_wirolegi.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-10-07T08:39:14.777439"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Merah",
|
||||||
|
"market": "Pasar_Kreongan",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "linear",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 24,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1462,
|
||||||
|
"jumlah_sequence": 1455,
|
||||||
|
"tanggal_awal": "2021-11-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.011226602829992771,
|
||||||
|
"loss_akhir": 0.001685838564299047
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_merah_pasar_kreongan.h5",
|
||||||
|
"scaler": "scaler_merah_pasar_kreongan.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-20T23:58:40.791301"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Merah",
|
||||||
|
"market": "Pasar_Mangli",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "linear",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 24,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1126,
|
||||||
|
"jumlah_sequence": 1119,
|
||||||
|
"tanggal_awal": "2022-10-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.009805780835449696,
|
||||||
|
"loss_akhir": 0.0020311949774622917
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_merah_pasar_mangli.h5",
|
||||||
|
"scaler": "scaler_merah_pasar_mangli.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-20T23:58:40.870281"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Merah",
|
||||||
|
"market": "Pasar_Tanjung",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "linear",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 24,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1461,
|
||||||
|
"jumlah_sequence": 1454,
|
||||||
|
"tanggal_awal": "2021-11-02",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.008021093904972076,
|
||||||
|
"loss_akhir": 0.0015390263870358467
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_merah_pasar_tanjung.h5",
|
||||||
|
"scaler": "scaler_merah_pasar_tanjung.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-20T23:58:40.751940"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Merah",
|
||||||
|
"market": "Pasar_Tegal_Besar",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "linear",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 24,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1462,
|
||||||
|
"jumlah_sequence": 1455,
|
||||||
|
"tanggal_awal": "2021-11-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.010425683110952377,
|
||||||
|
"loss_akhir": 0.0022090484853833914
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_merah_pasar_tegal_besar.h5",
|
||||||
|
"scaler": "scaler_merah_pasar_tegal_besar.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-20T23:58:40.836691"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Merah",
|
||||||
|
"market": "Pasar_Wirolegi",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "linear",
|
||||||
|
"learning_rate": 0.01,
|
||||||
|
"batch_size": 24,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1124,
|
||||||
|
"jumlah_sequence": 1117,
|
||||||
|
"tanggal_awal": "2022-10-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.007541376631706953,
|
||||||
|
"loss_akhir": 0.0022759095299988985
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_merah_pasar_wirolegi.h5",
|
||||||
|
"scaler": "scaler_merah_pasar_wirolegi.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-20T23:58:40.909924"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Rawit",
|
||||||
|
"market": "Pasar_Kreongan",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "tanh",
|
||||||
|
"learning_rate": 0.05,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1462,
|
||||||
|
"jumlah_sequence": 1455,
|
||||||
|
"tanggal_awal": "2021-11-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.03841866925358772,
|
||||||
|
"loss_akhir": 0.002082317601889372
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_rawit_pasar_kreongan.h5",
|
||||||
|
"scaler": "scaler_rawit_pasar_kreongan.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-19T12:25:26.940616"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Rawit",
|
||||||
|
"market": "Pasar_Mangli",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "tanh",
|
||||||
|
"learning_rate": 0.05,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1126,
|
||||||
|
"jumlah_sequence": 1119,
|
||||||
|
"tanggal_awal": "2022-10-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.07843399047851562,
|
||||||
|
"loss_akhir": 0.002254883525893092
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_rawit_pasar_mangli.h5",
|
||||||
|
"scaler": "scaler_rawit_pasar_mangli.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-19T12:25:26.985791"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Rawit",
|
||||||
|
"market": "Pasar_Tanjung",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "tanh",
|
||||||
|
"learning_rate": 0.05,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1462,
|
||||||
|
"jumlah_sequence": 1455,
|
||||||
|
"tanggal_awal": "2021-11-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.06235548108816147,
|
||||||
|
"loss_akhir": 0.0018674645107239485
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_rawit_pasar_tanjung.h5",
|
||||||
|
"scaler": "scaler_rawit_pasar_tanjung.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-19T12:25:26.921200"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Rawit",
|
||||||
|
"market": "Pasar_Tegal_Besar",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "tanh",
|
||||||
|
"learning_rate": 0.05,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1462,
|
||||||
|
"jumlah_sequence": 1455,
|
||||||
|
"tanggal_awal": "2021-11-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.10447828471660614,
|
||||||
|
"loss_akhir": 0.0019537468906491995
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_rawit_pasar_tegal_besar.h5",
|
||||||
|
"scaler": "scaler_rawit_pasar_tegal_besar.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-19T12:25:26.962971"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"chili": "Cabai Rawit",
|
||||||
|
"market": "Pasar_Wirolegi",
|
||||||
|
"model": {
|
||||||
|
"neurons": 64,
|
||||||
|
"activation": "tanh",
|
||||||
|
"learning_rate": 0.05,
|
||||||
|
"batch_size": 32,
|
||||||
|
"epochs": 50,
|
||||||
|
"optimizer": "Adam",
|
||||||
|
"time_step": 7
|
||||||
|
},
|
||||||
|
"training": {
|
||||||
|
"seed": 42,
|
||||||
|
"shuffle": true,
|
||||||
|
"jumlah_data": 1123,
|
||||||
|
"jumlah_sequence": 1116,
|
||||||
|
"tanggal_awal": "2022-10-01",
|
||||||
|
"tanggal_akhir": "2025-11-01",
|
||||||
|
"loss_awal": 0.10417161136865616,
|
||||||
|
"loss_akhir": 0.0018767464207485318
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"model": "model_rawit_pasar_wirolegi.h5",
|
||||||
|
"scaler": "scaler_rawit_pasar_wirolegi.pkl",
|
||||||
|
"config": "config.json"
|
||||||
|
},
|
||||||
|
"created_at": "2026-09-19T12:25:27.010573"
|
||||||
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,11 @@
|
||||||
|
Flask
|
||||||
|
flask-cors
|
||||||
|
pandas
|
||||||
|
numpy
|
||||||
|
scikit-learn
|
||||||
|
joblib
|
||||||
|
tensorflow==2.10.0
|
||||||
|
pytest
|
||||||
|
PyMySQL
|
||||||
|
requests
|
||||||
|
lxml
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
|
||||||
|
$apiDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||||
|
$python = "C:\Users\alfia\anaconda3\envs\tf_gpu39\python.exe"
|
||||||
|
$port = 8000
|
||||||
|
|
||||||
|
if (-not (Test-Path $python)) {
|
||||||
|
throw "Python environment tf_gpu39 tidak ditemukan: $python"
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Host "Menghentikan proses backend PrediksiHargaCabe..."
|
||||||
|
$projectProcesses = Get-CimInstance Win32_Process | Where-Object {
|
||||||
|
$_.ProcessId -ne $PID -and
|
||||||
|
$_.CommandLine -and
|
||||||
|
$_.CommandLine -match [regex]::Escape($apiDir) -and
|
||||||
|
$_.CommandLine -match "main\.py|flask|waitress|gunicorn"
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($process in $projectProcesses) {
|
||||||
|
Write-Host " Stop PID $($process.ProcessId)"
|
||||||
|
Stop-Process -Id $process.ProcessId -Force -ErrorAction SilentlyContinue
|
||||||
|
}
|
||||||
|
|
||||||
|
$portConnections = Get-NetTCPConnection -LocalPort $port -State Listen -ErrorAction SilentlyContinue
|
||||||
|
foreach ($connection in $portConnections) {
|
||||||
|
if ($connection.OwningProcess -and $connection.OwningProcess -ne $PID) {
|
||||||
|
Write-Host " Membebaskan port $port dari PID $($connection.OwningProcess)"
|
||||||
|
Stop-Process -Id $connection.OwningProcess -Force -ErrorAction SilentlyContinue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Start-Sleep -Seconds 1
|
||||||
|
Set-Location $apiDir
|
||||||
|
$env:FLASK_DEBUG = "1"
|
||||||
|
$env:PYTHONUNBUFFERED = "1"
|
||||||
|
|
||||||
|
Write-Host "Menjalankan backend dengan auto-reloader dari $apiDir"
|
||||||
|
Write-Host "Perubahan file Python akan dimuat otomatis. Tekan Ctrl+C untuk berhenti."
|
||||||
|
& $python "main.py"
|
||||||
|
|
@ -0,0 +1,89 @@
|
||||||
|
"""Import the existing research CSV files into the configured MySQL database.
|
||||||
|
|
||||||
|
This is a one-time/manual import utility. It does not schedule scraping.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pymysql
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||||
|
|
||||||
|
from app.config import Config
|
||||||
|
|
||||||
|
|
||||||
|
BASE_DIR = Path(__file__).resolve().parents[1]
|
||||||
|
CSV_FILES = {
|
||||||
|
"rawit": next(BASE_DIR.glob("harga_cabai_rawit_*.csv")),
|
||||||
|
"merah": next(BASE_DIR.glob("harga_cabe_merah_besar_*.csv")),
|
||||||
|
"keriting": next(BASE_DIR.glob("harga_cabai_merah_keriting_*.csv")),
|
||||||
|
}
|
||||||
|
MARKETS = (
|
||||||
|
"pasar_tanjung",
|
||||||
|
"pasar_kreongan",
|
||||||
|
"pasar_tegal_besar",
|
||||||
|
"pasar_mangli",
|
||||||
|
"pasar_wirolegi",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def normalized_market_column(market: str) -> str:
|
||||||
|
return "Pasar_" + "_".join(part.title() for part in market.removeprefix("pasar_").split("_"))
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
connection = pymysql.connect(
|
||||||
|
host=Config.MYSQL_HOST,
|
||||||
|
port=Config.MYSQL_PORT,
|
||||||
|
database=Config.MYSQL_DATABASE,
|
||||||
|
user=Config.MYSQL_USER,
|
||||||
|
password=Config.MYSQL_PASSWORD,
|
||||||
|
connect_timeout=Config.MYSQL_CONNECT_TIMEOUT,
|
||||||
|
)
|
||||||
|
inserted = 0
|
||||||
|
try:
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute("SELECT id, slug FROM commodities")
|
||||||
|
commodities = {slug: row_id for row_id, slug in cursor.fetchall()}
|
||||||
|
cursor.execute("SELECT id, slug FROM markets")
|
||||||
|
markets = {slug: row_id for row_id, slug in cursor.fetchall()}
|
||||||
|
|
||||||
|
for commodity, csv_path in CSV_FILES.items():
|
||||||
|
frame = pd.read_csv(csv_path)
|
||||||
|
for _, row in frame.iterrows():
|
||||||
|
price_date = pd.to_datetime(row["Tanggal"], errors="coerce")
|
||||||
|
if pd.isna(price_date):
|
||||||
|
continue
|
||||||
|
for market in MARKETS:
|
||||||
|
value = pd.to_numeric(row.get(normalized_market_column(market)), errors="coerce")
|
||||||
|
if pd.isna(value) or float(value) <= 0:
|
||||||
|
continue
|
||||||
|
cursor.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO price_history
|
||||||
|
(commodity_id, market_id, price_date, price, source)
|
||||||
|
VALUES (%s, %s, %s, %s, 'csv_import')
|
||||||
|
ON DUPLICATE KEY UPDATE
|
||||||
|
price = VALUES(price), source = VALUES(source)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
commodities[commodity],
|
||||||
|
markets[market],
|
||||||
|
price_date.date(),
|
||||||
|
float(value) * 1000,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
inserted += cursor.rowcount
|
||||||
|
connection.commit()
|
||||||
|
except Exception:
|
||||||
|
connection.rollback()
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
print(f"Import selesai. Baris diproses: {inserted}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,28 @@
|
||||||
|
import numpy as np
|
||||||
|
from sklearn.preprocessing import MinMaxScaler
|
||||||
|
|
||||||
|
from app.services.forecaster import forecast_recursive
|
||||||
|
|
||||||
|
|
||||||
|
class IncrementModel:
|
||||||
|
def predict(self, model_input, verbose=0):
|
||||||
|
return np.asarray([[model_input[0, -1, 0] + 0.1]])
|
||||||
|
|
||||||
|
|
||||||
|
def test_recursive_forecast_uses_scaled_feedback_and_keeps_prefix():
|
||||||
|
scaler = MinMaxScaler().fit(np.asarray([[0.0], [100.0]]))
|
||||||
|
window = np.asarray([10.0, 20.0, 30.0])
|
||||||
|
prices, out_of_range = forecast_recursive(IncrementModel(), scaler, window, 5)
|
||||||
|
|
||||||
|
assert prices == [40, 50, 60, 70, 80]
|
||||||
|
assert out_of_range is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_forecast_never_returns_negative_prices():
|
||||||
|
class NegativeModel:
|
||||||
|
def predict(self, model_input, verbose=0):
|
||||||
|
return np.asarray([[-2.0]])
|
||||||
|
|
||||||
|
scaler = MinMaxScaler().fit(np.asarray([[0.0], [100.0]]))
|
||||||
|
prices, _ = forecast_recursive(NegativeModel(), scaler, np.asarray([10.0, 20.0, 30.0]), 1)
|
||||||
|
assert prices == [0]
|
||||||
|
|
@ -0,0 +1,30 @@
|
||||||
|
import json
|
||||||
|
|
||||||
|
import app.services.model_registry as registry_module
|
||||||
|
from app.services.model_registry import ModelRegistry
|
||||||
|
|
||||||
|
|
||||||
|
class FakeModel:
|
||||||
|
input_shape = (None, 7, 1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_skips_incomplete_artifacts(tmp_path, monkeypatch):
|
||||||
|
market_dir = tmp_path / "rawit" / "pasar_tanjung"
|
||||||
|
market_dir.mkdir(parents=True)
|
||||||
|
(market_dir / "config.json").write_text(
|
||||||
|
json.dumps({"model": {"time_step": 7}, "files": {"model": "model.h5", "scaler": "scaler.pkl"}}),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(market_dir / "model.h5").write_text("model", encoding="utf-8")
|
||||||
|
(market_dir / "scaler.pkl").write_bytes(b"scaler")
|
||||||
|
|
||||||
|
class FakeScaler:
|
||||||
|
data_min_ = [0]
|
||||||
|
data_max_ = [1]
|
||||||
|
|
||||||
|
monkeypatch.setattr(registry_module, "load_model", lambda *args, **kwargs: FakeModel())
|
||||||
|
monkeypatch.setattr(registry_module.joblib, "load", lambda *args, **kwargs: FakeScaler())
|
||||||
|
model_registry = ModelRegistry(tmp_path)
|
||||||
|
|
||||||
|
assert model_registry.get("rawit", "pasar_tanjung") is not None
|
||||||
|
assert any(item["market"] == "pasar_kreongan" for item in model_registry.skipped)
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
from app import create_app
|
||||||
|
from app.config import Config
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
class TestConfig(Config):
|
||||||
|
TESTING = True
|
||||||
|
MODELS_DIR = Path("tests/fixtures/empty-models")
|
||||||
|
DATA_DIR = Path("tests/fixtures/empty-data")
|
||||||
|
|
||||||
|
|
||||||
|
def test_route_contract_and_error_format(monkeypatch):
|
||||||
|
monkeypatch.setattr("app.ModelRegistry.scan", lambda self: None)
|
||||||
|
app = create_app(TestConfig)
|
||||||
|
client = app.test_client()
|
||||||
|
|
||||||
|
assert client.get("/api/v1/health").status_code == 200
|
||||||
|
response = client.get("/api/v1/predict?commodity=unknown&market=pasar_tanjung")
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert response.json["error"]["code"] == "UNKNOWN_COMMODITY"
|
||||||
|
|
||||||
|
response = client.get("/api/v1/predict?commodity=rawit&market=pasar_tanjung&horizon=2")
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.json["error"]["code"] == "INVALID_HORIZON"
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,164 @@
|
||||||
|
CREATE DATABASE IF NOT EXISTS prediksicabe
|
||||||
|
CHARACTER SET utf8mb4
|
||||||
|
COLLATE utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
USE prediksicabe;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS commodities (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
name VARCHAR(100) NOT NULL,
|
||||||
|
slug VARCHAR(50) NOT NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uq_commodities_slug (slug)
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS markets (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
name VARCHAR(100) NOT NULL,
|
||||||
|
slug VARCHAR(50) NOT NULL,
|
||||||
|
latitude DECIMAL(10,7) NOT NULL,
|
||||||
|
longitude DECIMAL(10,7) NOT NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uq_markets_slug (slug)
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS price_history (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
commodity_id BIGINT UNSIGNED NOT NULL,
|
||||||
|
market_id BIGINT UNSIGNED NOT NULL,
|
||||||
|
price_date DATE NOT NULL,
|
||||||
|
price DECIMAL(15,2) NOT NULL,
|
||||||
|
source VARCHAR(100) NOT NULL DEFAULT 'csv_import',
|
||||||
|
scraping_log_id BIGINT UNSIGNED NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uq_price_history_observation (commodity_id, market_id, price_date),
|
||||||
|
KEY idx_price_history_date (price_date),
|
||||||
|
KEY idx_price_history_commodity_market (commodity_id, market_id),
|
||||||
|
CONSTRAINT fk_price_history_commodity FOREIGN KEY (commodity_id) REFERENCES commodities(id) ON UPDATE CASCADE ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT fk_price_history_market FOREIGN KEY (market_id) REFERENCES markets(id) ON UPDATE CASCADE ON DELETE RESTRICT
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS scraping_logs (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
started_at DATETIME NOT NULL,
|
||||||
|
finished_at DATETIME NULL,
|
||||||
|
status ENUM('running', 'success', 'failed') NOT NULL,
|
||||||
|
records_inserted INT UNSIGNED NOT NULL DEFAULT 0,
|
||||||
|
records_updated INT UNSIGNED NOT NULL DEFAULT 0,
|
||||||
|
error_message TEXT NULL,
|
||||||
|
source VARCHAR(100) NOT NULL DEFAULT 'manual',
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
KEY idx_scraping_logs_status (status),
|
||||||
|
KEY idx_scraping_logs_started_at (started_at)
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
ALTER TABLE price_history
|
||||||
|
ADD KEY idx_price_history_scraping_log (scraping_log_id),
|
||||||
|
ADD CONSTRAINT fk_price_history_scraping_log
|
||||||
|
FOREIGN KEY (scraping_log_id) REFERENCES scraping_logs(id)
|
||||||
|
ON UPDATE CASCADE ON DELETE CASCADE;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS users (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
name VARCHAR(120) NOT NULL,
|
||||||
|
email VARCHAR(190) NOT NULL,
|
||||||
|
password_hash VARCHAR(255) NOT NULL,
|
||||||
|
role ENUM('user', 'admin') NOT NULL DEFAULT 'user',
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uq_users_email (email)
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS preprocessing_runs (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
dataset_source VARCHAR(100) NOT NULL,
|
||||||
|
started_at DATETIME NOT NULL,
|
||||||
|
finished_at DATETIME NULL,
|
||||||
|
status ENUM('pending', 'running', 'success', 'failed') NOT NULL,
|
||||||
|
rows_before INT UNSIGNED NOT NULL DEFAULT 0,
|
||||||
|
rows_after INT UNSIGNED NOT NULL DEFAULT 0,
|
||||||
|
parameters_json JSON NULL,
|
||||||
|
output_path VARCHAR(500) NULL,
|
||||||
|
error_message TEXT NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
KEY idx_preprocessing_status (status)
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS training_runs (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
commodity_id BIGINT UNSIGNED NOT NULL,
|
||||||
|
started_at DATETIME NOT NULL,
|
||||||
|
finished_at DATETIME NULL,
|
||||||
|
status ENUM('pending', 'running', 'success', 'failed') NOT NULL,
|
||||||
|
seed INT NULL,
|
||||||
|
train_rows INT UNSIGNED NOT NULL DEFAULT 0,
|
||||||
|
test_rows INT UNSIGNED NOT NULL DEFAULT 0,
|
||||||
|
metrics_json JSON NULL,
|
||||||
|
history_path VARCHAR(500) NULL,
|
||||||
|
prediction_result_path VARCHAR(500) NULL,
|
||||||
|
error_message TEXT NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
KEY idx_training_status (status),
|
||||||
|
CONSTRAINT fk_training_commodity FOREIGN KEY (commodity_id) REFERENCES commodities(id) ON UPDATE CASCADE ON DELETE RESTRICT
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS model_versions (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
commodity_id BIGINT UNSIGNED NOT NULL,
|
||||||
|
version VARCHAR(80) NOT NULL,
|
||||||
|
model_path VARCHAR(500) NOT NULL,
|
||||||
|
scaler_path VARCHAR(500) NOT NULL,
|
||||||
|
config_path VARCHAR(500) NULL,
|
||||||
|
metrics_json JSON NULL,
|
||||||
|
training_run_id BIGINT UNSIGNED NULL,
|
||||||
|
is_active BOOLEAN NOT NULL DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uq_model_version (commodity_id, version),
|
||||||
|
KEY idx_model_active (commodity_id, is_active),
|
||||||
|
CONSTRAINT fk_model_commodity FOREIGN KEY (commodity_id) REFERENCES commodities(id) ON UPDATE CASCADE ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT fk_model_training FOREIGN KEY (training_run_id) REFERENCES training_runs(id) ON UPDATE CASCADE ON DELETE SET NULL
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS predictions (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||||
|
model_version_id BIGINT UNSIGNED NULL,
|
||||||
|
commodity_id BIGINT UNSIGNED NOT NULL,
|
||||||
|
market_id BIGINT UNSIGNED NOT NULL,
|
||||||
|
prediction_date DATE NOT NULL,
|
||||||
|
forecast_date DATE NOT NULL,
|
||||||
|
predicted_price DECIMAL(15,2) NOT NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uq_prediction (commodity_id, market_id, prediction_date, forecast_date),
|
||||||
|
CONSTRAINT fk_prediction_model FOREIGN KEY (model_version_id) REFERENCES model_versions(id) ON UPDATE CASCADE ON DELETE SET NULL,
|
||||||
|
CONSTRAINT fk_prediction_commodity FOREIGN KEY (commodity_id) REFERENCES commodities(id) ON UPDATE CASCADE ON DELETE RESTRICT,
|
||||||
|
CONSTRAINT fk_prediction_market FOREIGN KEY (market_id) REFERENCES markets(id) ON UPDATE CASCADE ON DELETE RESTRICT
|
||||||
|
) ENGINE=InnoDB;
|
||||||
|
|
||||||
|
INSERT INTO commodities (name, slug) VALUES
|
||||||
|
('Cabai Rawit', 'rawit'),
|
||||||
|
('Cabai Merah Besar', 'merah'),
|
||||||
|
('Cabai Merah Keriting', 'keriting')
|
||||||
|
ON DUPLICATE KEY UPDATE name = VALUES(name);
|
||||||
|
|
||||||
|
INSERT INTO markets (name, slug, latitude, longitude) VALUES
|
||||||
|
('Pasar Tanjung', 'pasar_tanjung', -8.1738152, 113.6954052),
|
||||||
|
('Pasar Kreongan', 'pasar_kreongan', -8.1556968, 113.7008754),
|
||||||
|
('Pasar Tegal Besar', 'pasar_tegal_besar', -8.1926152, 113.6987226),
|
||||||
|
('Pasar Mangli', 'pasar_mangli', -8.1902978, 113.6493539),
|
||||||
|
('Pasar Wirolegi', 'pasar_wirolegi', -8.1872926, 113.7431422)
|
||||||
|
ON DUPLICATE KEY UPDATE
|
||||||
|
name = VALUES(name),
|
||||||
|
latitude = VALUES(latitude),
|
||||||
|
longitude = VALUES(longitude);
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
{
|
||||||
|
"name": "PrediksiHargaCabe",
|
||||||
|
"lockfileVersion": 3,
|
||||||
|
"requires": true,
|
||||||
|
"packages": {
|
||||||
|
"node_modules/@react-leaflet/core": {
|
||||||
|
"version": "3.0.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/@react-leaflet/core/-/core-3.0.0.tgz",
|
||||||
|
"integrity": "sha512-3EWmekh4Nz+pGcr+xjf0KNyYfC3U2JjnkWsh0zcqaexYqmmB5ZhH37kz41JXGmKzpaMZCnPofBBm64i+YrEvGQ==",
|
||||||
|
"license": "Hippocratic-2.1",
|
||||||
|
"peerDependencies": {
|
||||||
|
"leaflet": "^1.9.0",
|
||||||
|
"react": "^19.0.0",
|
||||||
|
"react-dom": "^19.0.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/leaflet": {
|
||||||
|
"version": "1.9.4",
|
||||||
|
"resolved": "https://registry.npmjs.org/leaflet/-/leaflet-1.9.4.tgz",
|
||||||
|
"integrity": "sha512-nxS1ynzJOmOlHp+iL3FyWqK89GtNL8U8rvlMOsQdTTssxZwCXh8N2NB3GDQOL+YR3XnWyZAxwQixURb+FA74PA==",
|
||||||
|
"license": "BSD-2-Clause"
|
||||||
|
},
|
||||||
|
"node_modules/react": {
|
||||||
|
"version": "19.3.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/react/-/react-19.3.0.tgz",
|
||||||
|
"integrity": "sha512-E8LUcbtBWt20bbl2YoHfx4ZDBdxVTfOKtCZn9cDSJ4l6/nuoApcpIBcj47t2wZoVX8g2ZHuMHbiShgCR1T5Sog==",
|
||||||
|
"license": "MIT",
|
||||||
|
"peer": true,
|
||||||
|
"engines": {
|
||||||
|
"node": ">=0.10.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/react-dom": {
|
||||||
|
"version": "19.3.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.3.0.tgz",
|
||||||
|
"integrity": "sha512-JDk8dgif51OjFoDE70+OT9ICyYr+69HlmihNwp1+Nsfbna3t5sIiCa9ZJktDmQ4/1b/rn26hIAR2uYXDMr5r0Q==",
|
||||||
|
"license": "MIT",
|
||||||
|
"peer": true,
|
||||||
|
"dependencies": {
|
||||||
|
"scheduler": "^0.28.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"react": "^19.3.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/react-leaflet": {
|
||||||
|
"version": "5.0.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/react-leaflet/-/react-leaflet-5.0.0.tgz",
|
||||||
|
"integrity": "sha512-CWbTpr5vcHw5bt9i4zSlPEVQdTVcML390TjeDG0cK59z1ylexpqC6M1PJFjV8jD7CF+ACBFsLIDs6DRMoLEofw==",
|
||||||
|
"license": "Hippocratic-2.1",
|
||||||
|
"dependencies": {
|
||||||
|
"@react-leaflet/core": "^3.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"leaflet": "^1.9.0",
|
||||||
|
"react": "^19.0.0",
|
||||||
|
"react-dom": "^19.0.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/scheduler": {
|
||||||
|
"version": "0.28.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.28.0.tgz",
|
||||||
|
"integrity": "sha512-juorfCmIkIw8tT+p5BXSm6PJjQF/ycEYmKyzURCIt/RaZIhL+PulbQ9Yu2z1HdOJDdqDTlxA1+xKBmHXJsczAw==",
|
||||||
|
"license": "MIT",
|
||||||
|
"peer": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
@react-leaflet/core Copyright 2020 Paul Le Cam and contributors (“Licensor”)
|
||||||
|
|
||||||
|
Hippocratic License Version Number: 2.1.
|
||||||
|
|
||||||
|
Purpose. The purpose of this License is for the Licensor named above to permit the Licensee (as defined below) broad permission, if consistent with Human Rights Laws and Human Rights Principles (as each is defined below), to use and work with the Software (as defined below) within the full scope of Licensor’s copyright and patent rights, if any, in the Software, while ensuring attribution and protecting the Licensor from liability.
|
||||||
|
|
||||||
|
Permission and Conditions. The Licensor grants permission by this license (“License”), free of charge, to the extent of Licensor’s rights under applicable copyright and patent law, to any person or entity (the “Licensee”) obtaining a copy of this software and associated documentation files (the “Software”), to do everything with the Software that would otherwise infringe (i) the Licensor’s copyright in the Software or (ii) any patent claims to the Software that the Licensor can license or becomes able to license, subject to all of the following terms and conditions:
|
||||||
|
|
||||||
|
- Acceptance. This License is automatically offered to every person and entity subject to its terms and conditions. Licensee accepts this License and agrees to its terms and conditions by taking any action with the Software that, absent this License, would infringe any intellectual property right held by Licensor.
|
||||||
|
|
||||||
|
- Notice. Licensee must ensure that everyone who gets a copy of any part of this Software from Licensee, with or without changes, also receives the License and the above copyright notice (and if included by the Licensor, patent, trademark and attribution notice). Licensee must cause any modified versions of the Software to carry prominent notices stating that Licensee changed the Software. For clarity, although Licensee is free to create modifications of the Software and distribute only the modified portion created by Licensee with additional or different terms, the portion of the Software not modified must be distributed pursuant to this License. If anyone notifies Licensee in writing that Licensee has not complied with this Notice section, Licensee can keep this License by taking all practical steps to comply within 30 days after the notice. If Licensee does not do so, Licensee’s License (and all rights licensed hereunder) shall end immediately.
|
||||||
|
|
||||||
|
- Compliance with Human Rights Principles and Human Rights Laws.
|
||||||
|
|
||||||
|
1. Human Rights Principles.
|
||||||
|
|
||||||
|
(a) Licensee is advised to consult the articles of the United Nations Universal Declaration of Human Rights and the United Nations Global Compact that define recognized principles of international human rights (the “Human Rights Principles”). Licensee shall use the Software in a manner consistent with Human Rights Principles.
|
||||||
|
|
||||||
|
(b) Unless the Licensor and Licensee agree otherwise, any dispute, controversy, or claim arising out of or relating to (i) Section 1(a) regarding Human Rights Principles, including the breach of Section 1(a), termination of this License for breach of the Human Rights Principles, or invalidity of Section 1(a) or (ii) a determination of whether any Law is consistent or in conflict with Human Rights Principles pursuant to Section 2, below, shall be settled by arbitration in accordance with the Hague Rules on Business and Human Rights Arbitration (the “Rules”); provided, however, that Licensee may elect not to participate in such arbitration, in which event this License (and all rights licensed hereunder) shall end immediately. The number of arbitrators shall be one unless the Rules require otherwise.
|
||||||
|
|
||||||
|
Unless both the Licensor and Licensee agree to the contrary: (1) All documents and information concerning the arbitration shall be public and may be disclosed by any party; (2) The repository referred to under Article 43 of the Rules shall make available to the public in a timely manner all documents concerning the arbitration which are communicated to it, including all submissions of the parties, all evidence admitted into the record of the proceedings, all transcripts or other recordings of hearings and all orders, decisions and awards of the arbitral tribunal, subject only to the arbitral tribunal's powers to take such measures as may be necessary to safeguard the integrity of the arbitral process pursuant to Articles 18, 33, 41 and 42 of the Rules; and (3) Article 26(6) of the Rules shall not apply.
|
||||||
|
|
||||||
|
2. Human Rights Laws. The Software shall not be used by any person or entity for any systems, activities, or other uses that violate any Human Rights Laws. “Human Rights Laws” means any applicable laws, regulations, or rules (collectively, “Laws”) that protect human, civil, labor, privacy, political, environmental, security, economic, due process, or similar rights; provided, however, that such Laws are consistent and not in conflict with Human Rights Principles (a dispute over the consistency or a conflict between Laws and Human Rights Principles shall be determined by arbitration as stated above). Where the Human Rights Laws of more than one jurisdiction are applicable or in conflict with respect to the use of the Software, the Human Rights Laws that are most protective of the individuals or groups harmed shall apply.
|
||||||
|
|
||||||
|
3. Indemnity. Licensee shall hold harmless and indemnify Licensor (and any other contributor) against all losses, damages, liabilities, deficiencies, claims, actions, judgments, settlements, interest, awards, penalties, fines, costs, or expenses of whatever kind, including Licensor’s reasonable attorneys’ fees, arising out of or relating to Licensee’s use of the Software in violation of Human Rights Laws or Human Rights Principles.
|
||||||
|
|
||||||
|
- Failure to Comply. Any failure of Licensee to act according to the terms and conditions of this License is both a breach of the License and an infringement of the intellectual property rights of the Licensor (subject to exceptions under Laws, e.g., fair use). In the event of a breach or infringement, the terms and conditions of this License may be enforced by Licensor under the Laws of any jurisdiction to which Licensee is subject. Licensee also agrees that the Licensor may enforce the terms and conditions of this License against Licensee through specific performance (or similar remedy under Laws) to the extent permitted by Laws. For clarity, except in the event of a breach of this License, infringement, or as otherwise stated in this License, Licensor may not terminate this License with Licensee.
|
||||||
|
|
||||||
|
- Enforceability and Interpretation. If any term or provision of this License is determined to be invalid, illegal, or unenforceable by a court of competent jurisdiction, then such invalidity, illegality, or unenforceability shall not affect any other term or provision of this License or invalidate or render unenforceable such term or provision in any other jurisdiction; provided, however, subject to a court modification pursuant to the immediately following sentence, if any term or provision of this License pertaining to Human Rights Laws or Human Rights Principles is deemed invalid, illegal, or unenforceable against Licensee by a court of competent jurisdiction, all rights in the Software granted to Licensee shall be deemed null and void as between Licensor and Licensee. Upon a determination that any term or provision is invalid, illegal, or unenforceable, to the extent permitted by Laws, the court may modify this License to affect the original purpose that the Software be used in compliance with Human Rights Principles and Human Rights Laws as closely as possible. The language in this License shall be interpreted as to its fair meaning and not strictly for or against any party.
|
||||||
|
|
||||||
|
- Disclaimer. TO THE FULL EXTENT ALLOWED BY LAW, THIS SOFTWARE COMES “AS IS,” WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED, AND LICENSOR AND ANY OTHER CONTRIBUTOR SHALL NOT BE LIABLE TO ANYONE FOR ANY DAMAGES OR OTHER LIABILITY ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THIS LICENSE, UNDER ANY KIND OF LEGAL CLAIM.
|
||||||
|
|
||||||
|
This Hippocratic License is an Ethical Source license (https://ethicalsource.dev) and is offered for use by licensors and licensees at their own risk, on an “AS IS” basis, and with no warranties express or implied, to the maximum extent permitted by Laws.
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
# React Leaflet core
|
||||||
|
|
||||||
|
[Documentation](https://react-leaflet.js.org/docs/core-introduction)
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
import type { Map as LeafletMap } from 'leaflet';
|
||||||
|
export declare function useAttribution(map: LeafletMap, attribution: string | null | undefined): void;
|
||||||
|
|
@ -0,0 +1,18 @@
|
||||||
|
import { useEffect, useRef } from 'react';
|
||||||
|
export function useAttribution(map, attribution) {
|
||||||
|
const attributionRef = useRef(attribution);
|
||||||
|
useEffect(function updateAttribution() {
|
||||||
|
if (attribution !== attributionRef.current && map.attributionControl != null) {
|
||||||
|
if (attributionRef.current != null) {
|
||||||
|
map.attributionControl.removeAttribution(attributionRef.current);
|
||||||
|
}
|
||||||
|
if (attribution != null) {
|
||||||
|
map.attributionControl.addAttribution(attribution);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
attributionRef.current = attribution;
|
||||||
|
}, [
|
||||||
|
map,
|
||||||
|
attribution
|
||||||
|
]);
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue