Initial commit
This commit is contained in:
el
commit
f3b1f5fc84
|
|
@ -0,0 +1,27 @@
|
||||||
|
.DS_Store
|
||||||
|
|
||||||
|
# Python caches
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
.venv/
|
||||||
|
env/
|
||||||
|
|
||||||
|
# Environment and local secrets
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
|
||||||
|
# Streamlit/local runtime
|
||||||
|
.streamlit/secrets.toml
|
||||||
|
|
||||||
|
# App runtime state/logs
|
||||||
|
auto_crawl_log.json
|
||||||
|
crawler_state.json
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# Notebook/IPython
|
||||||
|
.ipynb_checkpoints/
|
||||||
|
|
@ -0,0 +1,235 @@
|
||||||
|
# 📊 Perbaikan Dashboard Sentimen - Data Realtime
|
||||||
|
|
||||||
|
## 🎯 Ringkasan Perubahan
|
||||||
|
|
||||||
|
Semua fungsi di dashboard telah diperbaiki untuk **menggunakan data realtime yang dinamis** bukan data dummy. Sistem sekarang secara otomatis merefresh data ketika ada perubahan di database dan menampilkan timestamp setiap kali data diupdate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔄 Perbaikan Utama
|
||||||
|
|
||||||
|
### 1. **Database Functions** (`database.py`)
|
||||||
|
✅ Ditambahkan helper functions untuk deteksi data baru:
|
||||||
|
- `get_tweet_count()` - Menghitung total tweet di database
|
||||||
|
- `get_latest_crawl_time()` - Mendapatkan waktu crawl terakhir
|
||||||
|
|
||||||
|
**Fungsi:** Digunakan untuk cache invalidation yang akurat
|
||||||
|
|
||||||
|
### 2. **Sentiment Analysis Page** (`page_modules/sentiment_page.py`)
|
||||||
|
|
||||||
|
#### Perbaikan Cache:
|
||||||
|
- ✅ Cache sekarang menggunakan `get_tweet_count()` bukan `len(df)`
|
||||||
|
- ✅ Cache otomatis dihapus saat ada tweet baru
|
||||||
|
- ✅ Prediksi sentimen direfresh otomatis saat database berubah
|
||||||
|
- ✅ Session state `_last_total_tweets` melacak versi database
|
||||||
|
|
||||||
|
#### Timestamp Dinamis:
|
||||||
|
- ✅ **Pie Chart** menampilkan waktu update: "Diupdate: DD/MM/YYYY HH:MM:SS"
|
||||||
|
- ✅ **Bar Chart** menampilkan waktu update + persentase sentimen
|
||||||
|
- ✅ **Trend Chart** menampilkan waktu update realtime
|
||||||
|
- ✅ Semua metric selalu menampilkan timestamp terkini
|
||||||
|
|
||||||
|
#### Import Baru:
|
||||||
|
```python
|
||||||
|
from database import engine, get_tweet_count, get_latest_crawl_time
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. **Crawling Page** (`page_modules/crawling_page.py`)
|
||||||
|
|
||||||
|
#### Metrics Update:
|
||||||
|
- ✅ `_render_metrics()` sekarang menampilkan timestamp: "🔄 Data diperbarui: DD/MM/YYYY HH:MM:SS"
|
||||||
|
- ✅ Total tweet, hari, tanggal selalu diambil fresh dari database
|
||||||
|
- ✅ Setiap kali halaman dimuat, data diambil ulang
|
||||||
|
|
||||||
|
### 4. **Preprocessing Page** (`page_modules/preprocessing_page.py`)
|
||||||
|
|
||||||
|
#### Cache Improvement:
|
||||||
|
- ✅ Cache menggunakan `get_tweet_count()` untuk invalidation
|
||||||
|
- ✅ Session state `_pp_last_total_tweets` melacak versi database
|
||||||
|
- ✅ Old cache entries otomatis dihapus
|
||||||
|
- ✅ Preprocessing direfresh saat ada data baru
|
||||||
|
|
||||||
|
### 5. **Main App** (`app.py`)
|
||||||
|
|
||||||
|
#### Smart Refresh Interval:
|
||||||
|
- ✅ Auto-refresh interval dinamis:
|
||||||
|
- 30 detik jika ada data baru dalam 5 menit terakhir
|
||||||
|
- 60 detik jika tidak ada aktivitas crawler baru
|
||||||
|
- ✅ Sidebar menampilkan status crawler
|
||||||
|
- ✅ **Sidebar baru:** Widget "🔄 Update Terbaru" menampilkan:
|
||||||
|
- Tanggal dan waktu update terakhir
|
||||||
|
- Format: DD/MM/YYYY HH:MM:SS WIB
|
||||||
|
|
||||||
|
#### Auto-refresh Configuration:
|
||||||
|
```python
|
||||||
|
st_autorefresh(
|
||||||
|
interval=refresh_interval, # Dynamic berdasarkan aktivitas
|
||||||
|
key="auto_refresh_dashboard"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Data Flow Realtime
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────┐
|
||||||
|
│ Crawler Service │ ← Data baru setiap N menit
|
||||||
|
│ (python3 crawler) │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────┐
|
||||||
|
│ SQLite Database │ ← 429+ tweets dengan timestamp
|
||||||
|
│ (data_sentimen...) │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
│
|
||||||
|
┌─────┴──────┬────────────┬──────────┐
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
┌──────────┐ ┌─────────┐ ┌──────────┐ ┌────────┐
|
||||||
|
│Crawling │ │Preprocess│ │Sentiment │ │Sidebar │
|
||||||
|
│Page │ │ Page │ │ Page │ │ Info │
|
||||||
|
└──────────┘ └─────────┘ └──────────┘ └────────┘
|
||||||
|
│ │ │ │
|
||||||
|
└────────────┴───────────┴───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────────────┐
|
||||||
|
│ Smart Cache Busting │
|
||||||
|
│ (get_tweet_count) │
|
||||||
|
└──────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────────────┐
|
||||||
|
│ Fresh Data Show │
|
||||||
|
│ + Realtime Timestamp │
|
||||||
|
└──────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Cara Kerja Cache Invalidation
|
||||||
|
|
||||||
|
### Sebelumnya (Dummy/Stale):
|
||||||
|
```
|
||||||
|
Cache Key = "sent_realtime_2026-05-09_2026-05-09_450"
|
||||||
|
├─ Bergantung pada `len(df)` - tidak akurat
|
||||||
|
├─ Jika ada tweet baru tidak terdeteksi
|
||||||
|
└─ Data lama tetap ditampilkan
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sesudahnya (Realtime & Dynamic):
|
||||||
|
```
|
||||||
|
Cache Key = "sent_realtime_2026-05-09_2026-05-09_429"
|
||||||
|
├─ Bergantung pada total tweet di database
|
||||||
|
├─ Saat tweet baru masuk (429 → 430):
|
||||||
|
│ └─ Cache key berubah otomatis
|
||||||
|
│ └─ Prediksi sentimen refresh
|
||||||
|
│ └─ Timestamp diupdate ke waktu terbaru
|
||||||
|
└─ Data selalu fresh ✓
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔍 Timestamp Ditampilkan Di
|
||||||
|
|
||||||
|
1. ✅ **Ambil Data Twitter**
|
||||||
|
- Caption: "🔄 Data diperbarui: DD/MM/YYYY HH:MM:SS"
|
||||||
|
|
||||||
|
2. ✅ **Bersihkan Data**
|
||||||
|
- Tidak ada perubahan (inherited dari session state)
|
||||||
|
|
||||||
|
3. ✅ **Analisis Sentimen**
|
||||||
|
- Pie Chart: "Diupdate: DD/MM/YYYY HH:MM:SS"
|
||||||
|
- Bar Chart: "Diupdate: DD/MM/YYYY HH:MM:SS" + persentase
|
||||||
|
- Trend Chart: "Diupdate: DD/MM/YYYY HH:MM:SS"
|
||||||
|
|
||||||
|
4. ✅ **Sidebar**
|
||||||
|
- "🔄 Update Terbaru" widget menampilkan waktu crawl terakhir
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📈 Data yang Sekarang Realtime
|
||||||
|
|
||||||
|
| Data | Sebelumnya | Sesudahnya |
|
||||||
|
|------|-----------|-----------|
|
||||||
|
| **Total Tweet** | Dummy | ✅ Fresh dari DB |
|
||||||
|
| **Tanggal Awal** | Hardcoded | ✅ Dinamis dari data |
|
||||||
|
| **Tanggal Akhir** | Hardcoded | ✅ Dinamis dari data |
|
||||||
|
| **Sentimen Positif** | Cached | ✅ Recomputed setiap data baru |
|
||||||
|
| **Sentimen Netral** | Cached | ✅ Recomputed setiap data baru |
|
||||||
|
| **Sentimen Negatif** | Cached | ✅ Recomputed setiap data baru |
|
||||||
|
| **Pie Chart** | Stale | ✅ Refresh + Timestamp |
|
||||||
|
| **Bar Chart** | Stale | ✅ Refresh + Timestamp + % |
|
||||||
|
| **Trend Chart** | Stale | ✅ Refresh + Timestamp |
|
||||||
|
| **Word Cloud** | Cached | ✅ Recomputed setiap data baru |
|
||||||
|
| **Waktu Update** | ❌ Tidak ada | ✅ Ditampilkan di sidebar |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Cara Menggunakan
|
||||||
|
|
||||||
|
### 1. Jalankan Crawler (di terminal terpisah)
|
||||||
|
```bash
|
||||||
|
python3 crawler.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Jalankan Dashboard
|
||||||
|
```bash
|
||||||
|
streamlit run app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Monitor Data
|
||||||
|
- Sidebar menampilkan status crawler dan waktu update terbaru
|
||||||
|
- Dashboard auto-refresh setiap 30-60 detik
|
||||||
|
- Saat ada data baru, prediksi sentimen otomatis direfresh
|
||||||
|
- Timestamp ditampilkan di setiap chart/metric
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Verifikasi
|
||||||
|
|
||||||
|
Untuk memastikan perbaikan bekerja:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Check database functions
|
||||||
|
python3 -c "from database import get_tweet_count, get_latest_crawl_time; print('Count:', get_tweet_count()); print('Latest:', get_latest_crawl_time())"
|
||||||
|
|
||||||
|
# 2. Run syntax check
|
||||||
|
python3 -m py_compile app.py page_modules/*.py
|
||||||
|
|
||||||
|
# 3. Start dashboard
|
||||||
|
streamlit run app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📝 File yang Dimodifikasi
|
||||||
|
|
||||||
|
1. ✅ `database.py` - Tambah helper functions
|
||||||
|
2. ✅ `app.py` - Smart refresh + sidebar info
|
||||||
|
3. ✅ `page_modules/sentiment_page.py` - Cache busting + timestamps
|
||||||
|
4. ✅ `page_modules/crawling_page.py` - Metrics timestamp
|
||||||
|
5. ✅ `page_modules/preprocessing_page.py` - Cache busting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Hasil Akhir
|
||||||
|
|
||||||
|
**SEBELUM:**
|
||||||
|
- Data dummy / stale
|
||||||
|
- Cache tidak pernah di-refresh
|
||||||
|
- Timestamp tidak ditampilkan
|
||||||
|
- Perlu manual refresh untuk melihat data baru
|
||||||
|
|
||||||
|
**SESUDAHNYA:**
|
||||||
|
- ✅ Data selalu realtime fresh
|
||||||
|
- ✅ Cache auto-refresh saat ada data baru
|
||||||
|
- ✅ Timestamp ditampilkan di semua metric dan chart
|
||||||
|
- ✅ Auto-refresh setiap 30-60 detik
|
||||||
|
- ✅ Dashboard responsif terhadap perubahan data
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Status: ✅ COMPLETED**
|
||||||
|
Semua fungsi telah diperbaiki untuk menggunakan data realtime yang dinamis sesuai fungsinya.
|
||||||
|
|
@ -0,0 +1,982 @@
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from html import escape
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import streamlit as st
|
||||||
|
from streamlit_autorefresh import st_autorefresh
|
||||||
|
|
||||||
|
from database import init_db, get_latest_crawl_time
|
||||||
|
from timezone_utils import (
|
||||||
|
INDONESIA_TIMEZONES,
|
||||||
|
browser_offset_to_choice,
|
||||||
|
browser_timezone_to_choice,
|
||||||
|
get_default_timezone,
|
||||||
|
get_timezone_label,
|
||||||
|
get_timezone_name,
|
||||||
|
parse_dt_with_source_tz,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
load_dotenv()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
st.set_page_config(
|
||||||
|
page_title="Dashboard Sentimen Twitter",
|
||||||
|
layout="wide",
|
||||||
|
page_icon="📊",
|
||||||
|
initial_sidebar_state="expanded"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Initialize timezone selection in session state
|
||||||
|
if "user_timezone" not in st.session_state:
|
||||||
|
st.session_state.user_timezone = get_default_timezone()
|
||||||
|
|
||||||
|
if "follow_device_timezone" not in st.session_state:
|
||||||
|
st.session_state.follow_device_timezone = True
|
||||||
|
|
||||||
|
|
||||||
|
def _get_query_param(name):
|
||||||
|
try:
|
||||||
|
value = st.query_params.get(name)
|
||||||
|
except Exception:
|
||||||
|
value = None
|
||||||
|
|
||||||
|
if isinstance(value, list):
|
||||||
|
return value[0] if value else None
|
||||||
|
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _get_context_timezone():
|
||||||
|
try:
|
||||||
|
return st.context.timezone
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _get_context_offset():
|
||||||
|
try:
|
||||||
|
offset = st.context.timezone_offset
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if offset is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
# st.context.timezone_offset follows JavaScript getTimezoneOffset:
|
||||||
|
# UTC+8 is -480. The app mapping uses UTC offset minutes: UTC+8 is 480.
|
||||||
|
return str(-int(offset))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_timezone_from_browser():
|
||||||
|
browser_timezone = _get_context_timezone() or _get_query_param("browser_tz")
|
||||||
|
browser_offset = _get_context_offset() or _get_query_param("browser_offset")
|
||||||
|
mapped_timezone = (
|
||||||
|
browser_timezone_to_choice(browser_timezone)
|
||||||
|
or browser_offset_to_choice(browser_offset)
|
||||||
|
)
|
||||||
|
|
||||||
|
st.session_state.browser_timezone = browser_timezone
|
||||||
|
st.session_state.browser_offset = browser_offset
|
||||||
|
|
||||||
|
if st.session_state.get("follow_device_timezone", True):
|
||||||
|
timezone_choice = mapped_timezone or get_default_timezone()
|
||||||
|
|
||||||
|
if st.session_state.user_timezone != timezone_choice:
|
||||||
|
st.session_state.user_timezone = timezone_choice
|
||||||
|
|
||||||
|
|
||||||
|
_sync_timezone_from_browser()
|
||||||
|
|
||||||
|
# Determine refresh interval berdasarkan aktivitas crawler
|
||||||
|
def get_refresh_interval():
|
||||||
|
"""Refresh lebih cepat jika ada data baru dari crawler"""
|
||||||
|
try:
|
||||||
|
latest_crawl = get_latest_crawl_time()
|
||||||
|
if latest_crawl:
|
||||||
|
timezone_choice = st.session_state.get("user_timezone", get_default_timezone())
|
||||||
|
latest_time = parse_dt_with_source_tz(
|
||||||
|
[latest_crawl],
|
||||||
|
timezone_choice,
|
||||||
|
os.getenv("APP_TIMEZONE", "Asia/Makassar")
|
||||||
|
).iloc[0]
|
||||||
|
now = pd.Timestamp.now(tz=get_timezone_name(timezone_choice)).tz_localize(None)
|
||||||
|
time_diff = (now - latest_time).total_seconds()
|
||||||
|
# Jika data baru < 5 menit, refresh setiap 30 detik, jika tidak setiap 60 detik
|
||||||
|
return 30000 if time_diff < 300 else 60000
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return 60000
|
||||||
|
|
||||||
|
refresh_interval = get_refresh_interval()
|
||||||
|
|
||||||
|
st_autorefresh(
|
||||||
|
interval=refresh_interval,
|
||||||
|
key="auto_refresh_dashboard"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_crawler_alive():
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from crawler import NRT_INTERVAL_MINUTES, get_crawler_state
|
||||||
|
|
||||||
|
threshold = timedelta(minutes=max((NRT_INTERVAL_MINUTES * 2) + 1, 3))
|
||||||
|
|
||||||
|
def parse_dt(value):
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(value)
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||||
|
return parsed.astimezone(timezone.utc)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
state = get_crawler_state()
|
||||||
|
if not state:
|
||||||
|
return False
|
||||||
|
|
||||||
|
updated_at = parse_dt(state.get("updated_at"))
|
||||||
|
heartbeat_at = parse_dt(state.get("heartbeat_at"))
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
if (
|
||||||
|
state.get("is_running", False)
|
||||||
|
and updated_at is not None
|
||||||
|
and now - updated_at <= threshold
|
||||||
|
):
|
||||||
|
return True
|
||||||
|
|
||||||
|
return (
|
||||||
|
state.get("service_active", False)
|
||||||
|
and heartbeat_at is not None
|
||||||
|
and now - heartbeat_at <= threshold
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
st.session_state._scheduler_started = _check_crawler_alive()
|
||||||
|
|
||||||
|
try:
|
||||||
|
from crawler import NRT_INTERVAL_MINUTES
|
||||||
|
st.session_state._scheduler_interval = NRT_INTERVAL_MINUTES
|
||||||
|
except Exception:
|
||||||
|
st.session_state._scheduler_interval = 0
|
||||||
|
|
||||||
|
|
||||||
|
st.markdown("""
|
||||||
|
<style>
|
||||||
|
@import url('https://fonts.googleapis.com/css2?family=Plus+Jakarta+Sans:wght@300;400;500;600;700;800&display=swap');
|
||||||
|
|
||||||
|
:root {
|
||||||
|
--blue-primary: #3b6cf7;
|
||||||
|
--blue-light: #eef2ff;
|
||||||
|
--green: #16a34a;
|
||||||
|
--red: #ef4444;
|
||||||
|
--text: #0f172a;
|
||||||
|
--text-secondary: #475569;
|
||||||
|
--text-tertiary: #64748b;
|
||||||
|
--border: #e2e8f0;
|
||||||
|
--bg: #f8fafc;
|
||||||
|
--card: #ffffff;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
GLOBAL STYLING
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
* {
|
||||||
|
font-family: 'Plus Jakarta Sans', sans-serif !important;
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stIconMaterial"],
|
||||||
|
[data-testid="stIconMaterial"] *,
|
||||||
|
.material-icons,
|
||||||
|
.material-symbols-rounded,
|
||||||
|
.material-symbols-outlined,
|
||||||
|
.material-symbols-sharp {
|
||||||
|
font-family: "Material Symbols Rounded", "Material Icons" !important;
|
||||||
|
font-weight: normal !important;
|
||||||
|
font-style: normal !important;
|
||||||
|
line-height: 1 !important;
|
||||||
|
letter-spacing: normal !important;
|
||||||
|
text-transform: none !important;
|
||||||
|
white-space: nowrap !important;
|
||||||
|
word-wrap: normal !important;
|
||||||
|
direction: ltr !important;
|
||||||
|
-webkit-font-feature-settings: "liga" !important;
|
||||||
|
-webkit-font-smoothing: antialiased !important;
|
||||||
|
font-feature-settings: "liga" !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
html, body, .stApp, [data-testid="stAppViewContainer"] {
|
||||||
|
background: var(--bg) !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stMain"] {
|
||||||
|
background: var(--bg) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.main > div {
|
||||||
|
padding: 1.5rem 2rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
SIDEBAR
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
[data-testid="stSidebar"] {
|
||||||
|
background: var(--card) !important;
|
||||||
|
box-shadow: 2px 0 8px rgba(15,23,42,0.08) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stSidebar"] > div:first-child {
|
||||||
|
padding: 1.5rem 1.25rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stSidebar"] * {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
BUTTONS & NAVIGATION
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.stButton > button {
|
||||||
|
background: var(--card) !important;
|
||||||
|
color: var(--text-tertiary) !important;
|
||||||
|
border: 1.5px solid var(--border) !important;
|
||||||
|
border-radius: 12px !important;
|
||||||
|
padding: 0.75rem 1.25rem !important;
|
||||||
|
font-weight: 600 !important;
|
||||||
|
font-size: 0.9375rem !important;
|
||||||
|
transition: all 0.2s ease !important;
|
||||||
|
width: 100% !important;
|
||||||
|
text-align: center !important;
|
||||||
|
display: flex !important;
|
||||||
|
align-items: center !important;
|
||||||
|
justify-content: center !important;
|
||||||
|
margin-bottom: 0.5rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stSidebar"] .stButton > button {
|
||||||
|
text-align: left !important;
|
||||||
|
justify-content: flex-start !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stButton > button *,
|
||||||
|
[data-testid="stDownloadButton"] button * {
|
||||||
|
color: inherit !important;
|
||||||
|
line-height: 1.25 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stSidebar"] .stButton > button[kind="primary"],
|
||||||
|
[data-testid="stSidebar"] .stButton > button[kind="primary"] * {
|
||||||
|
color: #ffffff !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stButton > button:hover {
|
||||||
|
background: var(--blue-light) !important;
|
||||||
|
color: var(--blue-primary) !important;
|
||||||
|
border-color: var(--blue-primary) !important;
|
||||||
|
box-shadow: 0 4px 12px rgba(59, 108, 247, 0.1) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stButton > button[kind="primary"] {
|
||||||
|
background: var(--blue-primary) !important;
|
||||||
|
color: #ffffff !important;
|
||||||
|
border-color: var(--blue-primary) !important;
|
||||||
|
font-weight: 700 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stButton > button[kind="primary"]:hover {
|
||||||
|
background: #2d59d1 !important;
|
||||||
|
box-shadow: 0 4px 12px rgba(59, 108, 247, 0.2) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stButton > button:disabled,
|
||||||
|
.stButton > button:disabled * {
|
||||||
|
background: #e2e8f0 !important;
|
||||||
|
color: var(--text-secondary) !important;
|
||||||
|
border-color: #cbd5e1 !important;
|
||||||
|
opacity: 1 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
HEADERS & TITLES
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.top-header {
|
||||||
|
background: var(--card);
|
||||||
|
padding: 1.5rem;
|
||||||
|
border-radius: 16px;
|
||||||
|
margin-bottom: 1.5rem;
|
||||||
|
box-shadow: 0 2px 8px rgba(15,23,42,0.06);
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.top-header * {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.page-title {
|
||||||
|
font-size: 1.875rem !important;
|
||||||
|
font-weight: 800 !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
margin: 0 !important;
|
||||||
|
line-height: 1.2;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1, h2, h3, h4, h5, h6 {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
CARDS & CONTAINERS
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.card {
|
||||||
|
background: var(--card);
|
||||||
|
border: 1.5px solid var(--border);
|
||||||
|
border-radius: 16px;
|
||||||
|
padding: 1.5rem;
|
||||||
|
box-shadow: 0 2px 8px rgba(15,23,42,0.06);
|
||||||
|
margin-bottom: 1.25rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.section-header {
|
||||||
|
background: var(--card);
|
||||||
|
border: 1.5px solid var(--border);
|
||||||
|
border-radius: 14px;
|
||||||
|
padding: 1rem 1.25rem;
|
||||||
|
margin-bottom: 0.75rem;
|
||||||
|
box-shadow: 0 2px 6px rgba(15,23,42,0.07);
|
||||||
|
}
|
||||||
|
|
||||||
|
.section-header-title {
|
||||||
|
font-size: 0.95rem;
|
||||||
|
font-weight: 700;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.section-header-subtitle {
|
||||||
|
font-size: 0.78rem;
|
||||||
|
color: var(--text-tertiary) !important;
|
||||||
|
margin-top: 3px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
METRICS & PILLS
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
[data-testid="stMetric"] {
|
||||||
|
background: var(--card) !important;
|
||||||
|
border: 1.5px solid var(--border) !important;
|
||||||
|
border-radius: 14px !important;
|
||||||
|
padding: 1.25rem !important;
|
||||||
|
box-shadow: 0 2px 6px rgba(15,23,42,0.05) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stMetric"] label,
|
||||||
|
[data-testid="stMetricLabel"],
|
||||||
|
[data-testid="stMetricLabel"] * {
|
||||||
|
color: var(--text-secondary) !important;
|
||||||
|
font-weight: 700 !important;
|
||||||
|
font-size: 0.7875rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stMetricValue"],
|
||||||
|
[data-testid="stMetricValue"] * {
|
||||||
|
color: var(--text) !important;
|
||||||
|
font-weight: 900 !important;
|
||||||
|
font-size: 1.75rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
TEXT & CONTENT
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.stMarkdown {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
p:not([style]), li:not([style]) {
|
||||||
|
color: var(--text-secondary) !important;
|
||||||
|
line-height: 1.6 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
span:not([style]) {
|
||||||
|
color: inherit !important;
|
||||||
|
line-height: 1.6 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stMarkdown p:not([style]),
|
||||||
|
.stMarkdown li:not([style]) {
|
||||||
|
color: var(--text-secondary) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stMarkdown span:not([style]) {
|
||||||
|
color: inherit !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stButton > button p,
|
||||||
|
.stButton > button span,
|
||||||
|
[data-testid="stDownloadButton"] button p,
|
||||||
|
[data-testid="stDownloadButton"] button span {
|
||||||
|
color: inherit !important;
|
||||||
|
line-height: 1.25 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
EXPANDERS
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
[data-testid="stExpander"] {
|
||||||
|
background: var(--card) !important;
|
||||||
|
border: 1.5px solid var(--border) !important;
|
||||||
|
border-radius: 12px !important;
|
||||||
|
box-shadow: 0 2px 6px rgba(15,23,42,0.05) !important;
|
||||||
|
overflow: hidden !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stExpander"] details {
|
||||||
|
border: 0 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stExpander"] summary {
|
||||||
|
background: var(--card) !important;
|
||||||
|
min-height: 48px !important;
|
||||||
|
padding: 0.75rem 1rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stExpander"] summary:hover {
|
||||||
|
background: #f8fafc !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stExpander"] summary *,
|
||||||
|
[data-testid="stExpander"] [data-testid="stIconMaterial"] {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stCaption, .stCaptionContainer, .stCaptionContainer * {
|
||||||
|
color: var(--text-tertiary) !important;
|
||||||
|
font-size: 0.8125rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
DATA & TABLES
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
[data-testid="stDataFrame"] {
|
||||||
|
border-radius: 12px !important;
|
||||||
|
border: 1px solid var(--border) !important;
|
||||||
|
overflow: hidden !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stDataFrame"] * {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
ALERTS & WARNINGS
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.stAlert {
|
||||||
|
border-radius: 12px !important;
|
||||||
|
margin-bottom: 1rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stWarning {
|
||||||
|
background-color: #fffbeb !important;
|
||||||
|
border-color: #fcd34d !important;
|
||||||
|
color: #78350f !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stSuccess {
|
||||||
|
background-color: #f0fdf4 !important;
|
||||||
|
border-color: #86efac !important;
|
||||||
|
color: #14532d !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stError {
|
||||||
|
background-color: #fef2f2 !important;
|
||||||
|
border-color: #fecaca !important;
|
||||||
|
color: #7f1d1d !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stInfo {
|
||||||
|
background-color: #f0f9ff !important;
|
||||||
|
border-color: #bae6fd !important;
|
||||||
|
color: #0c4a6e !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
FORM ELEMENTS
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.stTextInput input,
|
||||||
|
.stDateInput input,
|
||||||
|
.stSelectbox select,
|
||||||
|
[data-testid="stTextInput"] input,
|
||||||
|
[data-testid="stDateInput"] input,
|
||||||
|
[data-baseweb="input"] input {
|
||||||
|
background: #ffffff !important;
|
||||||
|
border-radius: 10px !important;
|
||||||
|
border: 1.5px solid var(--border) !important;
|
||||||
|
padding: 0.75rem !important;
|
||||||
|
font-size: 0.9375rem !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
-webkit-text-fill-color: var(--text) !important;
|
||||||
|
caret-color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stTextInput input::placeholder,
|
||||||
|
.stDateInput input::placeholder,
|
||||||
|
[data-testid="stTextInput"] input::placeholder,
|
||||||
|
[data-testid="stDateInput"] input::placeholder {
|
||||||
|
color: #94a3b8 !important;
|
||||||
|
-webkit-text-fill-color: #94a3b8 !important;
|
||||||
|
opacity: 1 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="input"],
|
||||||
|
[data-baseweb="select"] > div {
|
||||||
|
background: #ffffff !important;
|
||||||
|
border-color: var(--border) !important;
|
||||||
|
border-radius: 10px !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="select"] span,
|
||||||
|
[data-baseweb="select"] div,
|
||||||
|
[data-baseweb="input"] div {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.stTextInput input:focus,
|
||||||
|
.stSelectbox select:focus,
|
||||||
|
.stDateInput input:focus,
|
||||||
|
[data-testid="stTextInput"] input:focus,
|
||||||
|
[data-testid="stDateInput"] input:focus {
|
||||||
|
border-color: var(--blue-primary) !important;
|
||||||
|
box-shadow: 0 0 0 3px rgba(59, 108, 247, 0.1) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="popover"],
|
||||||
|
[data-baseweb="popover"] > div,
|
||||||
|
[data-baseweb="popover"] > div > div,
|
||||||
|
[data-baseweb="popover"] [role="dialog"],
|
||||||
|
[data-baseweb="menu"],
|
||||||
|
[data-baseweb="calendar"] {
|
||||||
|
background: #ffffff !important;
|
||||||
|
background-color: #ffffff !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="menu"] *,
|
||||||
|
[data-baseweb="calendar"] * {
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="popover"] * {
|
||||||
|
background-color: #ffffff !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="popover"] [role="dialog"],
|
||||||
|
[data-baseweb="popover"] [role="dialog"] > div,
|
||||||
|
[data-baseweb="popover"] [role="dialog"] > div > div,
|
||||||
|
[data-baseweb="popover"] [role="grid"],
|
||||||
|
[data-baseweb="popover"] [role="row"],
|
||||||
|
[data-baseweb="popover"] [role="gridcell"],
|
||||||
|
[data-baseweb="popover"] [role="columnheader"] {
|
||||||
|
background: #ffffff !important;
|
||||||
|
background-color: #ffffff !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="calendar"],
|
||||||
|
[data-baseweb="calendar"] > div,
|
||||||
|
[data-baseweb="calendar"] div,
|
||||||
|
[data-baseweb="calendar"] [role="grid"],
|
||||||
|
[data-baseweb="calendar"] [role="row"],
|
||||||
|
[data-baseweb="calendar"] [role="columnheader"],
|
||||||
|
[data-baseweb="calendar"] [role="gridcell"],
|
||||||
|
[data-baseweb="popover"] [data-baseweb="calendar"],
|
||||||
|
[data-baseweb="popover"] [data-baseweb="calendar"] div {
|
||||||
|
background: #ffffff !important;
|
||||||
|
background-color: #ffffff !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="calendar"] [role="columnheader"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Sunday"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Monday"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Tuesday"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Wednesday"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Thursday"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Friday"],
|
||||||
|
[data-baseweb="calendar"] [aria-label="Saturday"] {
|
||||||
|
color: var(--text-secondary) !important;
|
||||||
|
font-weight: 700 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="menu"] li:hover,
|
||||||
|
[data-baseweb="menu"] [role="option"]:hover {
|
||||||
|
background: var(--blue-light) !important;
|
||||||
|
color: var(--blue-primary) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="calendar"] button,
|
||||||
|
[data-baseweb="calendar"] [role="button"],
|
||||||
|
[data-baseweb="popover"] button,
|
||||||
|
[data-baseweb="popover"] select {
|
||||||
|
background: #ffffff !important;
|
||||||
|
background-color: #ffffff !important;
|
||||||
|
color: var(--text) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="calendar"] [aria-disabled="true"],
|
||||||
|
[data-baseweb="calendar"] [aria-disabled="true"] * {
|
||||||
|
background: #ffffff !important;
|
||||||
|
background-color: #ffffff !important;
|
||||||
|
color: #cbd5e1 !important;
|
||||||
|
-webkit-text-fill-color: #cbd5e1 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="calendar"] [role="gridcell"]:hover,
|
||||||
|
[data-baseweb="calendar"] button:hover {
|
||||||
|
background: var(--blue-light) !important;
|
||||||
|
background-color: var(--blue-light) !important;
|
||||||
|
color: var(--blue-primary) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-baseweb="calendar"] [aria-selected="true"],
|
||||||
|
[data-baseweb="calendar"] [aria-selected="true"] *,
|
||||||
|
[data-baseweb="calendar"] button[aria-selected="true"],
|
||||||
|
[data-baseweb="calendar"] button[aria-selected="true"] * {
|
||||||
|
background: var(--blue-primary) !important;
|
||||||
|
background-color: var(--blue-primary) !important;
|
||||||
|
color: #ffffff !important;
|
||||||
|
-webkit-text-fill-color: #ffffff !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
PILLS & CUSTOM ELEMENTS
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
.pill-container {
|
||||||
|
display: flex;
|
||||||
|
gap: 1rem;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
margin-bottom: 1.25rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pill {
|
||||||
|
background: var(--card);
|
||||||
|
border: 1.5px solid var(--border);
|
||||||
|
border-radius: 14px;
|
||||||
|
padding: 1.25rem;
|
||||||
|
text-align: center;
|
||||||
|
flex: 1;
|
||||||
|
min-width: 200px;
|
||||||
|
box-shadow: 0 2px 6px rgba(15,23,42,0.05);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* =========================
|
||||||
|
LAYOUT FIX
|
||||||
|
========================= */
|
||||||
|
|
||||||
|
#MainMenu, footer, header {
|
||||||
|
visibility: hidden !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stHeader"] {
|
||||||
|
background: transparent !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stDownloadButton"] button {
|
||||||
|
background: var(--card) !important;
|
||||||
|
color: var(--blue-primary) !important;
|
||||||
|
border: 1.5px solid var(--border) !important;
|
||||||
|
border-radius: 10px !important;
|
||||||
|
font-weight: 600 !important;
|
||||||
|
transition: all 0.2s ease !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
[data-testid="stDownloadButton"] button:hover {
|
||||||
|
background: var(--blue-light) !important;
|
||||||
|
border-color: var(--blue-primary) !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
</style>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
|
||||||
|
if "page" not in st.session_state:
|
||||||
|
st.session_state.page = "crawling"
|
||||||
|
|
||||||
|
if st.session_state.get("current_page"):
|
||||||
|
st.session_state.page = st.session_state.current_page
|
||||||
|
st.session_state.scroll_to_top = True
|
||||||
|
del st.session_state["current_page"]
|
||||||
|
|
||||||
|
if (
|
||||||
|
st.session_state.get("scroll_to_top")
|
||||||
|
or st.session_state.get("_last_rendered_page") != st.session_state.page
|
||||||
|
):
|
||||||
|
st.session_state.scroll_to_top = False
|
||||||
|
st.session_state._last_rendered_page = st.session_state.page
|
||||||
|
|
||||||
|
|
||||||
|
with st.sidebar:
|
||||||
|
|
||||||
|
st.markdown("""
|
||||||
|
<div style="display:flex;align-items:center;gap:0.75rem;padding:0.5rem 0.5rem 1.5rem;">
|
||||||
|
<div style="width:38px;height:38px;background:linear-gradient(135deg,#3b6cf7,#6389f8);
|
||||||
|
border-radius:10px;display:flex;align-items:center;justify-content:center;
|
||||||
|
font-size:1.1rem;flex-shrink:0;color:white;">📊</div>
|
||||||
|
<div>
|
||||||
|
<div style="font-size:1rem;font-weight:800;color:#0f172a;line-height:1.2;">SentimenX</div>
|
||||||
|
<div style="font-size:0.7rem;color:#94a3b8;font-weight:500;">Twitter Analytics</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
sched_ok = st.session_state.get("_scheduler_started", False)
|
||||||
|
sched_interval = st.session_state.get("_scheduler_interval", 0)
|
||||||
|
|
||||||
|
sched_color = "#16a34a" if sched_ok else "#f59e0b"
|
||||||
|
|
||||||
|
sched_label = (
|
||||||
|
f"Aktif — diperbarui tiap {sched_interval} menit"
|
||||||
|
if sched_ok
|
||||||
|
else "Nonaktif — jalankan: python3 crawler.py"
|
||||||
|
)
|
||||||
|
|
||||||
|
sched_dot = "●" if sched_ok else "○"
|
||||||
|
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:#f8fafc;border:1.5px solid #e2e8f0;border-radius:10px;
|
||||||
|
padding:0.625rem 0.75rem;margin-bottom:1.25rem;">
|
||||||
|
<div style="display:flex;align-items:center;gap:0.5rem;margin-bottom:2px;">
|
||||||
|
<span style="color:{sched_color};font-size:0.9rem;">{sched_dot}</span>
|
||||||
|
<span style="font-size:0.75rem;font-weight:700;color:#334155;">Crawler Service</span>
|
||||||
|
</div>
|
||||||
|
<div style="font-size:0.7rem;color:#64748b;line-height:1.5;">{sched_label}</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
# Tampilkan waktu update terakhir data
|
||||||
|
try:
|
||||||
|
latest_crawl = get_latest_crawl_time()
|
||||||
|
if latest_crawl:
|
||||||
|
crawl_dt = parse_dt_with_source_tz(
|
||||||
|
[latest_crawl],
|
||||||
|
st.session_state.user_timezone,
|
||||||
|
os.getenv("APP_TIMEZONE", "Asia/Makassar")
|
||||||
|
).iloc[0]
|
||||||
|
tz_label = get_timezone_label(st.session_state.user_timezone)
|
||||||
|
update_date = crawl_dt.strftime("%d/%m/%Y")
|
||||||
|
update_time = f"{crawl_dt.strftime('%H:%M:%S')} {tz_label}"
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:#eef2ff;border:1.5px solid #c7d2fe;border-radius:10px;
|
||||||
|
padding:0.625rem 0.75rem;margin-bottom:1.25rem;">
|
||||||
|
<div style="font-size:0.7rem;font-weight:700;color:#1e3a8a;margin-bottom:2px;">
|
||||||
|
🔄 Update Terbaru</div>
|
||||||
|
<div style="font-size:0.7rem;color:#1e40af;line-height:1.5;">
|
||||||
|
{update_date}<br/>{update_time}</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
st.markdown("""
|
||||||
|
<div style="font-size:0.65rem;font-weight:700;color:#94a3b8;
|
||||||
|
letter-spacing:0.08em;text-transform:uppercase;
|
||||||
|
padding:0 0.5rem;margin-bottom:0.5rem;">MENU UTAMA</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
current_page = st.session_state.get("page", "crawling")
|
||||||
|
|
||||||
|
nav_items = [
|
||||||
|
("crawling", "🔄 Ambil Data Twitter"),
|
||||||
|
("preprocessing", "🧹 Bersihkan Data"),
|
||||||
|
("sentiment", "📈 Analisis Sentimen"),
|
||||||
|
]
|
||||||
|
|
||||||
|
for page_key, label in nav_items:
|
||||||
|
is_active = current_page == page_key
|
||||||
|
|
||||||
|
if st.button(
|
||||||
|
label,
|
||||||
|
width="stretch",
|
||||||
|
type="primary" if is_active else "secondary",
|
||||||
|
key=f"nav_{page_key}"
|
||||||
|
):
|
||||||
|
st.session_state.page = page_key
|
||||||
|
st.session_state.scroll_to_top = True
|
||||||
|
st.rerun()
|
||||||
|
|
||||||
|
st.markdown("<br>", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
# Timezone selector
|
||||||
|
st.markdown("""
|
||||||
|
<div style="font-size:0.65rem;font-weight:700;color:#94a3b8;
|
||||||
|
letter-spacing:0.08em;text-transform:uppercase;
|
||||||
|
padding:0 0.5rem;margin-bottom:0.5rem;">⏰ ZONA WAKTU</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
follow_device = st.toggle(
|
||||||
|
"Ikuti timezone device",
|
||||||
|
value=st.session_state.get("follow_device_timezone", True),
|
||||||
|
key="follow_device_timezone_toggle",
|
||||||
|
help="Jika aktif, dashboard membaca zona waktu dari browser/device."
|
||||||
|
)
|
||||||
|
|
||||||
|
if follow_device != st.session_state.follow_device_timezone:
|
||||||
|
st.session_state.follow_device_timezone = follow_device
|
||||||
|
|
||||||
|
if follow_device:
|
||||||
|
mapped_timezone = (
|
||||||
|
browser_timezone_to_choice(
|
||||||
|
st.session_state.get("browser_timezone")
|
||||||
|
)
|
||||||
|
or browser_offset_to_choice(st.session_state.get("browser_offset"))
|
||||||
|
)
|
||||||
|
st.session_state.user_timezone = mapped_timezone or get_default_timezone()
|
||||||
|
|
||||||
|
st.rerun()
|
||||||
|
|
||||||
|
timezone_options = list(INDONESIA_TIMEZONES.keys())
|
||||||
|
|
||||||
|
if (
|
||||||
|
not st.session_state.follow_device_timezone
|
||||||
|
and st.session_state.user_timezone not in timezone_options
|
||||||
|
):
|
||||||
|
st.session_state.user_timezone = get_default_timezone()
|
||||||
|
|
||||||
|
if st.session_state.follow_device_timezone:
|
||||||
|
raw_detected_tz = st.session_state.get("browser_timezone")
|
||||||
|
detected_offset = st.session_state.get("browser_offset")
|
||||||
|
detected_tz = raw_detected_tz or (
|
||||||
|
"Timezone dari offset browser" if detected_offset else "Belum terdeteksi"
|
||||||
|
)
|
||||||
|
mapped_timezone = (
|
||||||
|
browser_timezone_to_choice(raw_detected_tz)
|
||||||
|
or browser_offset_to_choice(detected_offset)
|
||||||
|
)
|
||||||
|
active_label = mapped_timezone or get_default_timezone()
|
||||||
|
st.session_state.user_timezone = active_label
|
||||||
|
try:
|
||||||
|
offset_minutes = int(detected_offset)
|
||||||
|
sign = "+" if offset_minutes >= 0 else "-"
|
||||||
|
abs_minutes = abs(offset_minutes)
|
||||||
|
offset_label = f"UTC{sign}{abs_minutes // 60:02d}:{abs_minutes % 60:02d}"
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
offset_label = "offset belum terdeteksi"
|
||||||
|
st.caption(f"Device: {detected_tz} · {offset_label} · Aktif: {active_label}")
|
||||||
|
else:
|
||||||
|
selected_tz = st.selectbox(
|
||||||
|
"Pilih zona waktu Indonesia",
|
||||||
|
timezone_options,
|
||||||
|
index=timezone_options.index(st.session_state.user_timezone),
|
||||||
|
label_visibility="collapsed",
|
||||||
|
key="tz_selector"
|
||||||
|
)
|
||||||
|
|
||||||
|
if selected_tz != st.session_state.user_timezone:
|
||||||
|
st.session_state.user_timezone = selected_tz
|
||||||
|
st.rerun()
|
||||||
|
|
||||||
|
active_tz_name = get_timezone_name(st.session_state.user_timezone)
|
||||||
|
active_now = pd.Timestamp.now(tz=active_tz_name)
|
||||||
|
active_clock = active_now.strftime("%d/%m/%Y %H:%M:%S")
|
||||||
|
st.markdown(
|
||||||
|
f"""
|
||||||
|
<div style="background:#fff;border:1.5px solid #e2e8f0;border-radius:10px;
|
||||||
|
padding:0.625rem 0.75rem;margin:0.6rem 0 1.25rem;
|
||||||
|
font-family:'Plus Jakarta Sans',sans-serif;">
|
||||||
|
<div style="font-size:0.7rem;font-weight:800;color:#334155;margin-bottom:2px;">
|
||||||
|
Jam Aktif
|
||||||
|
</div>
|
||||||
|
<div style="font-size:0.82rem;font-weight:800;color:#0f172a;line-height:1.5;">
|
||||||
|
{active_clock}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""",
|
||||||
|
unsafe_allow_html=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
if (
|
||||||
|
st.session_state.follow_device_timezone
|
||||||
|
and (
|
||||||
|
browser_timezone_to_choice(st.session_state.get("browser_timezone"))
|
||||||
|
or browser_offset_to_choice(st.session_state.get("browser_offset"))
|
||||||
|
) is None
|
||||||
|
and st.session_state.get("browser_timezone")
|
||||||
|
):
|
||||||
|
st.caption("Timezone device belum valid dari browser, memakai default aplikasi.")
|
||||||
|
|
||||||
|
if not st.session_state.follow_device_timezone and selected_tz != st.session_state.user_timezone:
|
||||||
|
st.session_state.user_timezone = selected_tz
|
||||||
|
st.rerun()
|
||||||
|
|
||||||
|
st.markdown("<br>", unsafe_allow_html=True)
|
||||||
|
crawler_query = os.getenv(
|
||||||
|
"QUERY",
|
||||||
|
'komdigi (ongkir OR "gratis ongkir" OR "free ongkir") OR "pembatasan gratis ongkir" OR "gratis ongkir dibatasi"'
|
||||||
|
)
|
||||||
|
|
||||||
|
query_terms = [
|
||||||
|
term.strip().strip('"')
|
||||||
|
for term in crawler_query.split(" OR ")
|
||||||
|
if term.strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
query_html = "<br/>".join(
|
||||||
|
f"• {escape(term)}" for term in query_terms
|
||||||
|
) or f"• {escape(crawler_query)}"
|
||||||
|
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:#f0f4fb;border-radius:12px;padding:1rem;margin-top:auto;">
|
||||||
|
<div style="font-size:0.7rem;font-weight:700;color:#0f172a;text-transform:uppercase;
|
||||||
|
letter-spacing:0.06em;margin-bottom:0.625rem;">🔍 Query Crawler Aktif</div>
|
||||||
|
<div style="font-size:0.78rem;color:#475569;line-height:1.9;">
|
||||||
|
{query_html}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
st.markdown("""
|
||||||
|
<div style="font-size:0.68rem;color:#94a3b8;text-align:center;margin-top:1.5rem;
|
||||||
|
padding-top:1rem;border-top:1px solid #e2e8f0;line-height:1.6;">
|
||||||
|
Data bersumber dari X (Twitter)<br/>via tweet-harvest
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
|
||||||
|
from page_modules import crawling_page, preprocessing_page, sentiment_page
|
||||||
|
|
||||||
|
if st.session_state.page == "crawling":
|
||||||
|
crawling_page.show()
|
||||||
|
|
||||||
|
elif st.session_state.page == "preprocessing":
|
||||||
|
preprocessing_page.show()
|
||||||
|
|
||||||
|
elif st.session_state.page == "sentiment":
|
||||||
|
sentiment_page.show()
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,137 @@
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from scraper_worker import scrape_once
|
||||||
|
|
||||||
|
AUTO_CRAWL_INTERVAL_HOURS = 0.083
|
||||||
|
NRT_INTERVAL_MINUTES = 5
|
||||||
|
|
||||||
|
LOG_FILE = "auto_crawl_log.json"
|
||||||
|
STATE_FILE = "crawler_state.json"
|
||||||
|
|
||||||
|
|
||||||
|
def get_crawler_state():
|
||||||
|
if not os.path.exists(STATE_FILE):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(STATE_FILE, "r") as f:
|
||||||
|
return json.load(f)
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def get_auto_crawl_logs(limit=10):
|
||||||
|
if not os.path.exists(LOG_FILE):
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
with open(LOG_FILE, "r") as f:
|
||||||
|
return json.load(f)[:limit]
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def save_auto_crawl_log(status="success", total_saved=0, error=None):
|
||||||
|
logs = get_auto_crawl_logs(50)
|
||||||
|
|
||||||
|
logs.insert(0, {
|
||||||
|
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"status": status,
|
||||||
|
"total_saved": total_saved,
|
||||||
|
"error": error,
|
||||||
|
})
|
||||||
|
|
||||||
|
with open(LOG_FILE, "w") as f:
|
||||||
|
json.dump(logs[:50], f, indent=2)
|
||||||
|
|
||||||
|
|
||||||
|
def set_crawler_state(is_running=False, service_active=None):
|
||||||
|
previous_state = get_crawler_state()
|
||||||
|
now = datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
if service_active is None:
|
||||||
|
service_active = previous_state.get("service_active", False)
|
||||||
|
|
||||||
|
state = {
|
||||||
|
"is_running": bool(is_running),
|
||||||
|
"service_active": bool(service_active),
|
||||||
|
"updated_at": now,
|
||||||
|
"heartbeat_at": now if service_active else None,
|
||||||
|
"last_job_started_at": previous_state.get("last_job_started_at"),
|
||||||
|
"last_job_finished_at": previous_state.get("last_job_finished_at"),
|
||||||
|
}
|
||||||
|
|
||||||
|
if is_running:
|
||||||
|
state["last_job_started_at"] = now
|
||||||
|
elif previous_state.get("is_running"):
|
||||||
|
state["last_job_finished_at"] = now
|
||||||
|
|
||||||
|
with open(STATE_FILE, "w") as f:
|
||||||
|
json.dump(state, f, indent=2)
|
||||||
|
|
||||||
|
|
||||||
|
def auto_crawl_job():
|
||||||
|
print("=" * 50)
|
||||||
|
print("CRAWLING MULAI:", datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
|
||||||
|
print("=" * 50)
|
||||||
|
|
||||||
|
service_active = get_crawler_state().get("service_active", False)
|
||||||
|
set_crawler_state(True, service_active=service_active)
|
||||||
|
|
||||||
|
try:
|
||||||
|
total = scrape_once(
|
||||||
|
limit=int(os.getenv("SCRAPE_LIMIT", "50"))
|
||||||
|
)
|
||||||
|
|
||||||
|
if total is None:
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
save_auto_crawl_log(
|
||||||
|
status="success",
|
||||||
|
total_saved=total,
|
||||||
|
error=None
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Selesai. {total} tweet baru tersimpan.")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
save_auto_crawl_log(
|
||||||
|
status="error",
|
||||||
|
total_saved=0,
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
|
print("Error crawler:", e)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
set_crawler_state(False, service_active=service_active)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("Crawler realtime aktif.")
|
||||||
|
print(f"Interval: {NRT_INTERVAL_MINUTES} menit")
|
||||||
|
|
||||||
|
set_crawler_state(False, service_active=True)
|
||||||
|
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
auto_crawl_job()
|
||||||
|
print(f"Menunggu {NRT_INTERVAL_MINUTES} menit...")
|
||||||
|
|
||||||
|
wait_until = time.time() + (NRT_INTERVAL_MINUTES * 60)
|
||||||
|
|
||||||
|
while time.time() < wait_until:
|
||||||
|
set_crawler_state(False, service_active=True)
|
||||||
|
time.sleep(min(30, max(1, wait_until - time.time())))
|
||||||
|
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("Crawler dihentikan.")
|
||||||
|
|
||||||
|
finally:
|
||||||
|
set_crawler_state(False, service_active=False)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,157 @@
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
import pandas as pd
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
DB_PATH = os.getenv("DB_PATH", "data_sentimen_komdigi.db")
|
||||||
|
|
||||||
|
engine = create_engine(f"sqlite:///{DB_PATH}", echo=False)
|
||||||
|
|
||||||
|
|
||||||
|
def init_db():
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
cursor = conn.cursor()
|
||||||
|
|
||||||
|
cursor.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS tweets (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
tweet_id TEXT UNIQUE,
|
||||||
|
text TEXT,
|
||||||
|
created_at TEXT,
|
||||||
|
crawled_at TEXT,
|
||||||
|
crawl_type TEXT DEFAULT 'realtime'
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
|
||||||
|
conn.commit()
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def save_tweets(data, crawl_type="realtime"):
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
cursor = conn.cursor()
|
||||||
|
|
||||||
|
saved = 0
|
||||||
|
|
||||||
|
for tweet in data:
|
||||||
|
try:
|
||||||
|
cursor.execute("""
|
||||||
|
INSERT OR IGNORE INTO tweets
|
||||||
|
(tweet_id, text, created_at, crawled_at, crawl_type)
|
||||||
|
VALUES (?, ?, ?, ?, ?)
|
||||||
|
""", (
|
||||||
|
str(tweet.get("tweet_id", "")),
|
||||||
|
str(tweet.get("text", "")),
|
||||||
|
str(tweet.get("created_at", "")),
|
||||||
|
str(tweet.get("crawled_at", "")),
|
||||||
|
crawl_type,
|
||||||
|
))
|
||||||
|
|
||||||
|
if cursor.rowcount > 0:
|
||||||
|
saved += 1
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print("Gagal simpan tweet:", e)
|
||||||
|
|
||||||
|
conn.commit()
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
return saved
|
||||||
|
|
||||||
|
|
||||||
|
def load_tweets():
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
|
||||||
|
df = pd.read_sql_query("""
|
||||||
|
SELECT *
|
||||||
|
FROM tweets
|
||||||
|
ORDER BY created_at DESC
|
||||||
|
""", conn)
|
||||||
|
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
def insert_tweets(df):
|
||||||
|
data = []
|
||||||
|
|
||||||
|
for _, row in df.iterrows():
|
||||||
|
full_text = row.get("full_text", row.get("text", ""))
|
||||||
|
|
||||||
|
data.append({
|
||||||
|
"tweet_id": row.get(
|
||||||
|
"tweet_id",
|
||||||
|
f"{row.get('username', '')}_{hash(full_text)}"
|
||||||
|
),
|
||||||
|
"text": full_text,
|
||||||
|
"created_at": row.get("created_at", ""),
|
||||||
|
"crawled_at": row.get("crawled_at", ""),
|
||||||
|
})
|
||||||
|
|
||||||
|
return save_tweets(data, "realtime")
|
||||||
|
|
||||||
|
|
||||||
|
def get_tweet_count():
|
||||||
|
"""Dapatkan jumlah tweet terbaru dari database untuk cache invalidation"""
|
||||||
|
init_db()
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cursor.execute("SELECT COUNT(*) FROM tweets")
|
||||||
|
count = cursor.fetchone()[0]
|
||||||
|
conn.close()
|
||||||
|
return count
|
||||||
|
|
||||||
|
|
||||||
|
def get_existing_tweet_ids(tweet_ids):
|
||||||
|
"""Ambil tweet_id yang sudah ada agar crawler bisa melewati duplikat."""
|
||||||
|
ids = [str(tweet_id) for tweet_id in tweet_ids if str(tweet_id).strip()]
|
||||||
|
|
||||||
|
if not ids:
|
||||||
|
return set()
|
||||||
|
|
||||||
|
init_db()
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
cursor = conn.cursor()
|
||||||
|
existing = set()
|
||||||
|
|
||||||
|
for start in range(0, len(ids), 500):
|
||||||
|
chunk = ids[start:start + 500]
|
||||||
|
placeholders = ",".join(["?"] * len(chunk))
|
||||||
|
|
||||||
|
cursor.execute(
|
||||||
|
f"SELECT tweet_id FROM tweets WHERE tweet_id IN ({placeholders})",
|
||||||
|
chunk
|
||||||
|
)
|
||||||
|
|
||||||
|
existing.update(str(row[0]) for row in cursor.fetchall())
|
||||||
|
|
||||||
|
conn.close()
|
||||||
|
return existing
|
||||||
|
|
||||||
|
|
||||||
|
def get_latest_crawl_time():
|
||||||
|
"""Dapatkan waktu crawl terakhir untuk mendeteksi data baru"""
|
||||||
|
init_db()
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
df = pd.read_sql_query("""
|
||||||
|
SELECT MAX(crawled_at) as latest_crawl
|
||||||
|
FROM tweets
|
||||||
|
""", conn)
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
if df.empty or df['latest_crawl'].isna().all():
|
||||||
|
return None
|
||||||
|
return df['latest_crawl'].iloc[0]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
init_db()
|
||||||
|
print("Database berhasil dibuat.")
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,113 @@
|
||||||
|
import sqlite3
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
DB_FILE = "data_sentimen_komdigi.db"
|
||||||
|
EXCEL_FILE = "hasil_prediksi_sentimen.xlsx"
|
||||||
|
TABLE_NAME = "tweets"
|
||||||
|
|
||||||
|
|
||||||
|
def get_value(row, possible_names, default=""):
|
||||||
|
for name in possible_names:
|
||||||
|
if name in row and pd.notna(row[name]):
|
||||||
|
return row[name]
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def parse_date(value):
|
||||||
|
if pd.isna(value) or value == "":
|
||||||
|
return datetime.now().isoformat()
|
||||||
|
|
||||||
|
try:
|
||||||
|
return pd.to_datetime(value).isoformat()
|
||||||
|
except Exception:
|
||||||
|
return str(value)
|
||||||
|
|
||||||
|
|
||||||
|
df = pd.read_excel(EXCEL_FILE)
|
||||||
|
|
||||||
|
conn = sqlite3.connect(DB_FILE)
|
||||||
|
cur = conn.cursor()
|
||||||
|
|
||||||
|
cur.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS tweets (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
tweet_id TEXT UNIQUE,
|
||||||
|
text TEXT,
|
||||||
|
created_at TEXT,
|
||||||
|
crawled_at TEXT,
|
||||||
|
crawl_type TEXT
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
|
||||||
|
inserted = 0
|
||||||
|
skipped = 0
|
||||||
|
|
||||||
|
for index, row in df.iterrows():
|
||||||
|
text = str(get_value(row, [
|
||||||
|
"text",
|
||||||
|
"full_text",
|
||||||
|
"Isi Tweet",
|
||||||
|
"tweet",
|
||||||
|
"Tweet",
|
||||||
|
"clean_text",
|
||||||
|
"teks",
|
||||||
|
"komentar"
|
||||||
|
])).strip()
|
||||||
|
|
||||||
|
if not text or text.lower() == "nan":
|
||||||
|
skipped += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
tweet_id = str(get_value(row, [
|
||||||
|
"tweet_id",
|
||||||
|
"id_str",
|
||||||
|
"ID Tweet",
|
||||||
|
"id",
|
||||||
|
"Id"
|
||||||
|
], f"historis_{index}_{abs(hash(text))}"))
|
||||||
|
|
||||||
|
created_at = parse_date(get_value(row, [
|
||||||
|
"created_at",
|
||||||
|
"Tanggal Tweet",
|
||||||
|
"tanggal",
|
||||||
|
"date",
|
||||||
|
"Date",
|
||||||
|
"waktu"
|
||||||
|
], datetime.now().isoformat()))
|
||||||
|
|
||||||
|
crawled_at = parse_date(get_value(row, [
|
||||||
|
"crawled_at",
|
||||||
|
"Masuk Database"
|
||||||
|
], datetime.now().isoformat()))
|
||||||
|
|
||||||
|
try:
|
||||||
|
cur.execute("""
|
||||||
|
INSERT OR IGNORE INTO tweets
|
||||||
|
(tweet_id, text, created_at, crawled_at, crawl_type)
|
||||||
|
VALUES (?, ?, ?, ?, ?)
|
||||||
|
""", (
|
||||||
|
tweet_id,
|
||||||
|
text,
|
||||||
|
created_at,
|
||||||
|
crawled_at,
|
||||||
|
"historis"
|
||||||
|
))
|
||||||
|
|
||||||
|
if cur.rowcount > 0:
|
||||||
|
inserted += 1
|
||||||
|
else:
|
||||||
|
skipped += 1
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print("Gagal insert:", e)
|
||||||
|
skipped += 1
|
||||||
|
|
||||||
|
conn.commit()
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
print("Import selesai.")
|
||||||
|
print("Berhasil masuk:", inserted)
|
||||||
|
print("Dilewati:", skipped)
|
||||||
|
|
@ -0,0 +1,758 @@
|
||||||
|
ada
|
||||||
|
adalah
|
||||||
|
adanya
|
||||||
|
adapun
|
||||||
|
agak
|
||||||
|
agaknya
|
||||||
|
agar
|
||||||
|
akan
|
||||||
|
akankah
|
||||||
|
akhir
|
||||||
|
akhiri
|
||||||
|
akhirnya
|
||||||
|
aku
|
||||||
|
akulah
|
||||||
|
amat
|
||||||
|
amatlah
|
||||||
|
anda
|
||||||
|
andalah
|
||||||
|
antar
|
||||||
|
antara
|
||||||
|
antaranya
|
||||||
|
apa
|
||||||
|
apaan
|
||||||
|
apabila
|
||||||
|
apakah
|
||||||
|
apalagi
|
||||||
|
apatah
|
||||||
|
artinya
|
||||||
|
asal
|
||||||
|
asalkan
|
||||||
|
atas
|
||||||
|
atau
|
||||||
|
ataukah
|
||||||
|
ataupun
|
||||||
|
awal
|
||||||
|
awalnya
|
||||||
|
bagai
|
||||||
|
bagaikan
|
||||||
|
bagaimana
|
||||||
|
bagaimanakah
|
||||||
|
bagaimanapun
|
||||||
|
bagi
|
||||||
|
bagian
|
||||||
|
bahkan
|
||||||
|
bahwa
|
||||||
|
bahwasanya
|
||||||
|
baik
|
||||||
|
bakal
|
||||||
|
bakalan
|
||||||
|
balik
|
||||||
|
banyak
|
||||||
|
bapak
|
||||||
|
baru
|
||||||
|
bawah
|
||||||
|
beberapa
|
||||||
|
begini
|
||||||
|
beginian
|
||||||
|
beginikah
|
||||||
|
beginilah
|
||||||
|
begitu
|
||||||
|
begitukah
|
||||||
|
begitulah
|
||||||
|
begitupun
|
||||||
|
bekerja
|
||||||
|
belakang
|
||||||
|
belakangan
|
||||||
|
belum
|
||||||
|
belumlah
|
||||||
|
benar
|
||||||
|
benarkah
|
||||||
|
benarlah
|
||||||
|
berada
|
||||||
|
berakhir
|
||||||
|
berakhirlah
|
||||||
|
berakhirnya
|
||||||
|
berapa
|
||||||
|
berapakah
|
||||||
|
berapalah
|
||||||
|
berapapun
|
||||||
|
berarti
|
||||||
|
berawal
|
||||||
|
berbagai
|
||||||
|
berdatangan
|
||||||
|
beri
|
||||||
|
berikan
|
||||||
|
berikut
|
||||||
|
berikutnya
|
||||||
|
berjumlah
|
||||||
|
berkali-kali
|
||||||
|
berkata
|
||||||
|
berkehendak
|
||||||
|
berkeinginan
|
||||||
|
berkenaan
|
||||||
|
berlainan
|
||||||
|
berlalu
|
||||||
|
berlangsung
|
||||||
|
berlebihan
|
||||||
|
bermacam
|
||||||
|
bermacam-macam
|
||||||
|
bermaksud
|
||||||
|
bermula
|
||||||
|
bersama
|
||||||
|
bersama-sama
|
||||||
|
bersiap
|
||||||
|
bersiap-siap
|
||||||
|
bertanya
|
||||||
|
bertanya-tanya
|
||||||
|
berturut
|
||||||
|
berturut-turut
|
||||||
|
bertutur
|
||||||
|
berujar
|
||||||
|
berupa
|
||||||
|
besar
|
||||||
|
betul
|
||||||
|
betulkah
|
||||||
|
biasa
|
||||||
|
biasanya
|
||||||
|
bila
|
||||||
|
bilakah
|
||||||
|
bisa
|
||||||
|
bisakah
|
||||||
|
boleh
|
||||||
|
bolehkah
|
||||||
|
bolehlah
|
||||||
|
buat
|
||||||
|
bukan
|
||||||
|
bukankah
|
||||||
|
bukanlah
|
||||||
|
bukannya
|
||||||
|
bulan
|
||||||
|
bung
|
||||||
|
cara
|
||||||
|
caranya
|
||||||
|
cukup
|
||||||
|
cukupkah
|
||||||
|
cukuplah
|
||||||
|
cuma
|
||||||
|
dahulu
|
||||||
|
dalam
|
||||||
|
dan
|
||||||
|
dapat
|
||||||
|
dari
|
||||||
|
daripada
|
||||||
|
datang
|
||||||
|
dekat
|
||||||
|
demi
|
||||||
|
demikian
|
||||||
|
demikianlah
|
||||||
|
dengan
|
||||||
|
depan
|
||||||
|
di
|
||||||
|
dia
|
||||||
|
diakhiri
|
||||||
|
diakhirinya
|
||||||
|
dialah
|
||||||
|
diantara
|
||||||
|
diantaranya
|
||||||
|
diberi
|
||||||
|
diberikan
|
||||||
|
diberikannya
|
||||||
|
dibuat
|
||||||
|
dibuatnya
|
||||||
|
didapat
|
||||||
|
didatangkan
|
||||||
|
digunakan
|
||||||
|
diibaratkan
|
||||||
|
diibaratkannya
|
||||||
|
diingat
|
||||||
|
diingatkan
|
||||||
|
diinginkan
|
||||||
|
dijawab
|
||||||
|
dijelaskan
|
||||||
|
dijelaskannya
|
||||||
|
dikarenakan
|
||||||
|
dikatakan
|
||||||
|
dikatakannya
|
||||||
|
dikerjakan
|
||||||
|
diketahui
|
||||||
|
diketahuinya
|
||||||
|
dikira
|
||||||
|
dilakukan
|
||||||
|
dilalui
|
||||||
|
dilihat
|
||||||
|
dimaksud
|
||||||
|
dimaksudkan
|
||||||
|
dimaksudkannya
|
||||||
|
dimaksudnya
|
||||||
|
diminta
|
||||||
|
dimintai
|
||||||
|
dimisalkan
|
||||||
|
dimulai
|
||||||
|
dimulailah
|
||||||
|
dimulainya
|
||||||
|
dimungkinkan
|
||||||
|
dini
|
||||||
|
dipastikan
|
||||||
|
diperbuat
|
||||||
|
diperbuatnya
|
||||||
|
dipergunakan
|
||||||
|
diperkirakan
|
||||||
|
diperlihatkan
|
||||||
|
diperlukan
|
||||||
|
diperlukannya
|
||||||
|
dipersoalkan
|
||||||
|
dipertanyakan
|
||||||
|
dipunyai
|
||||||
|
diri
|
||||||
|
dirinya
|
||||||
|
disampaikan
|
||||||
|
disebut
|
||||||
|
disebutkan
|
||||||
|
disebutkannya
|
||||||
|
disini
|
||||||
|
disinilah
|
||||||
|
ditambahkan
|
||||||
|
ditandaskan
|
||||||
|
ditanya
|
||||||
|
ditanyai
|
||||||
|
ditanyakan
|
||||||
|
ditegaskan
|
||||||
|
ditujukan
|
||||||
|
ditunjuk
|
||||||
|
ditunjuki
|
||||||
|
ditunjukkan
|
||||||
|
ditunjukkannya
|
||||||
|
ditunjuknya
|
||||||
|
dituturkan
|
||||||
|
dituturkannya
|
||||||
|
diucapkan
|
||||||
|
diucapkannya
|
||||||
|
diungkapkan
|
||||||
|
dong
|
||||||
|
dua
|
||||||
|
dulu
|
||||||
|
empat
|
||||||
|
enggak
|
||||||
|
enggaknya
|
||||||
|
entah
|
||||||
|
entahlah
|
||||||
|
guna
|
||||||
|
gunakan
|
||||||
|
hal
|
||||||
|
hampir
|
||||||
|
hanya
|
||||||
|
hanyalah
|
||||||
|
hari
|
||||||
|
harus
|
||||||
|
haruslah
|
||||||
|
harusnya
|
||||||
|
hendak
|
||||||
|
hendaklah
|
||||||
|
hendaknya
|
||||||
|
hingga
|
||||||
|
ia
|
||||||
|
ialah
|
||||||
|
ibarat
|
||||||
|
ibaratkan
|
||||||
|
ibaratnya
|
||||||
|
ibu
|
||||||
|
ikut
|
||||||
|
ingat
|
||||||
|
ingat-ingat
|
||||||
|
ingin
|
||||||
|
inginkah
|
||||||
|
inginkan
|
||||||
|
ini
|
||||||
|
inikah
|
||||||
|
inilah
|
||||||
|
itu
|
||||||
|
itukah
|
||||||
|
itulah
|
||||||
|
jadi
|
||||||
|
jadilah
|
||||||
|
jadinya
|
||||||
|
jangan
|
||||||
|
jangankan
|
||||||
|
janganlah
|
||||||
|
jauh
|
||||||
|
jawab
|
||||||
|
jawaban
|
||||||
|
jawabnya
|
||||||
|
jelas
|
||||||
|
jelaskan
|
||||||
|
jelaslah
|
||||||
|
jelasnya
|
||||||
|
jika
|
||||||
|
jikalau
|
||||||
|
juga
|
||||||
|
jumlah
|
||||||
|
jumlahnya
|
||||||
|
justru
|
||||||
|
kala
|
||||||
|
kalau
|
||||||
|
kalaulah
|
||||||
|
kalaupun
|
||||||
|
kalian
|
||||||
|
kami
|
||||||
|
kamilah
|
||||||
|
kamu
|
||||||
|
kamulah
|
||||||
|
kan
|
||||||
|
kapan
|
||||||
|
kapankah
|
||||||
|
kapanpun
|
||||||
|
karena
|
||||||
|
karenanya
|
||||||
|
kasus
|
||||||
|
kata
|
||||||
|
katakan
|
||||||
|
katakanlah
|
||||||
|
katanya
|
||||||
|
ke
|
||||||
|
keadaan
|
||||||
|
kebetulan
|
||||||
|
kecil
|
||||||
|
kedua
|
||||||
|
keduanya
|
||||||
|
keinginan
|
||||||
|
kelamaan
|
||||||
|
kelihatan
|
||||||
|
kelihatannya
|
||||||
|
kelima
|
||||||
|
keluar
|
||||||
|
kembali
|
||||||
|
kemudian
|
||||||
|
kemungkinan
|
||||||
|
kemungkinannya
|
||||||
|
kenapa
|
||||||
|
kepada
|
||||||
|
kepadanya
|
||||||
|
kesampaian
|
||||||
|
keseluruhan
|
||||||
|
keseluruhannya
|
||||||
|
keterlaluan
|
||||||
|
ketika
|
||||||
|
khususnya
|
||||||
|
kini
|
||||||
|
kinilah
|
||||||
|
kira
|
||||||
|
kira-kira
|
||||||
|
kiranya
|
||||||
|
kita
|
||||||
|
kitalah
|
||||||
|
kok
|
||||||
|
kurang
|
||||||
|
lagi
|
||||||
|
lagian
|
||||||
|
lah
|
||||||
|
lain
|
||||||
|
lainnya
|
||||||
|
lalu
|
||||||
|
lama
|
||||||
|
lamanya
|
||||||
|
lanjut
|
||||||
|
lanjutnya
|
||||||
|
lebih
|
||||||
|
lewat
|
||||||
|
lima
|
||||||
|
luar
|
||||||
|
macam
|
||||||
|
maka
|
||||||
|
makanya
|
||||||
|
makin
|
||||||
|
malah
|
||||||
|
malahan
|
||||||
|
mampu
|
||||||
|
mampukah
|
||||||
|
mana
|
||||||
|
manakala
|
||||||
|
manalagi
|
||||||
|
masa
|
||||||
|
masalah
|
||||||
|
masalahnya
|
||||||
|
masih
|
||||||
|
masihkah
|
||||||
|
masing
|
||||||
|
masing-masing
|
||||||
|
mau
|
||||||
|
maupun
|
||||||
|
melainkan
|
||||||
|
melakukan
|
||||||
|
melalui
|
||||||
|
melihat
|
||||||
|
melihatnya
|
||||||
|
memang
|
||||||
|
memastikan
|
||||||
|
memberi
|
||||||
|
memberikan
|
||||||
|
membuat
|
||||||
|
memerlukan
|
||||||
|
memihak
|
||||||
|
meminta
|
||||||
|
memintakan
|
||||||
|
memisalkan
|
||||||
|
memperbuat
|
||||||
|
mempergunakan
|
||||||
|
memperkirakan
|
||||||
|
memperlihatkan
|
||||||
|
mempersiapkan
|
||||||
|
mempersoalkan
|
||||||
|
mempertanyakan
|
||||||
|
mempunyai
|
||||||
|
memulai
|
||||||
|
memungkinkan
|
||||||
|
menaiki
|
||||||
|
menambahkan
|
||||||
|
menandaskan
|
||||||
|
menanti
|
||||||
|
menanti-nanti
|
||||||
|
menantikan
|
||||||
|
menanya
|
||||||
|
menanyai
|
||||||
|
menanyakan
|
||||||
|
mendapat
|
||||||
|
mendapatkan
|
||||||
|
mendatang
|
||||||
|
mendatangi
|
||||||
|
mendatangkan
|
||||||
|
menegaskan
|
||||||
|
mengakhiri
|
||||||
|
mengapa
|
||||||
|
mengatakan
|
||||||
|
mengatakannya
|
||||||
|
mengenai
|
||||||
|
mengerjakan
|
||||||
|
mengetahui
|
||||||
|
menggunakan
|
||||||
|
menghendaki
|
||||||
|
mengibaratkan
|
||||||
|
mengibaratkannya
|
||||||
|
mengingat
|
||||||
|
mengingatkan
|
||||||
|
menginginkan
|
||||||
|
mengira
|
||||||
|
mengucapkan
|
||||||
|
mengucapkannya
|
||||||
|
mengungkapkan
|
||||||
|
menjadi
|
||||||
|
menjawab
|
||||||
|
menjelaskan
|
||||||
|
menuju
|
||||||
|
menunjuk
|
||||||
|
menunjuki
|
||||||
|
menunjukkan
|
||||||
|
menunjuknya
|
||||||
|
menurut
|
||||||
|
menuturkan
|
||||||
|
menyampaikan
|
||||||
|
menyangkut
|
||||||
|
menyatakan
|
||||||
|
menyebutkan
|
||||||
|
menyeluruh
|
||||||
|
menyiapkan
|
||||||
|
merasa
|
||||||
|
mereka
|
||||||
|
merekalah
|
||||||
|
merupakan
|
||||||
|
meski
|
||||||
|
meskipun
|
||||||
|
meyakini
|
||||||
|
meyakinkan
|
||||||
|
minta
|
||||||
|
mirip
|
||||||
|
misal
|
||||||
|
misalkan
|
||||||
|
misalnya
|
||||||
|
mula
|
||||||
|
mulai
|
||||||
|
mulailah
|
||||||
|
mulanya
|
||||||
|
mungkin
|
||||||
|
mungkinkah
|
||||||
|
nah
|
||||||
|
naik
|
||||||
|
namun
|
||||||
|
nanti
|
||||||
|
nantinya
|
||||||
|
nyaris
|
||||||
|
nyatanya
|
||||||
|
oleh
|
||||||
|
olehnya
|
||||||
|
pada
|
||||||
|
padahal
|
||||||
|
padanya
|
||||||
|
pak
|
||||||
|
paling
|
||||||
|
panjang
|
||||||
|
pantas
|
||||||
|
para
|
||||||
|
pasti
|
||||||
|
pastilah
|
||||||
|
penting
|
||||||
|
pentingnya
|
||||||
|
per
|
||||||
|
percuma
|
||||||
|
perlu
|
||||||
|
perlukah
|
||||||
|
perlunya
|
||||||
|
pernah
|
||||||
|
persoalan
|
||||||
|
pertama
|
||||||
|
pertama-tama
|
||||||
|
pertanyaan
|
||||||
|
pertanyakan
|
||||||
|
pihak
|
||||||
|
pihaknya
|
||||||
|
pukul
|
||||||
|
pula
|
||||||
|
pun
|
||||||
|
punya
|
||||||
|
rasa
|
||||||
|
rasanya
|
||||||
|
rata
|
||||||
|
rupanya
|
||||||
|
saat
|
||||||
|
saatnya
|
||||||
|
saja
|
||||||
|
sajalah
|
||||||
|
saling
|
||||||
|
sama
|
||||||
|
sama-sama
|
||||||
|
sambil
|
||||||
|
sampai
|
||||||
|
sampai-sampai
|
||||||
|
sampaikan
|
||||||
|
sana
|
||||||
|
sangat
|
||||||
|
sangatlah
|
||||||
|
satu
|
||||||
|
saya
|
||||||
|
sayalah
|
||||||
|
se
|
||||||
|
sebab
|
||||||
|
sebabnya
|
||||||
|
sebagai
|
||||||
|
sebagaimana
|
||||||
|
sebagainya
|
||||||
|
sebagian
|
||||||
|
sebaik
|
||||||
|
sebaik-baiknya
|
||||||
|
sebaiknya
|
||||||
|
sebaliknya
|
||||||
|
sebanyak
|
||||||
|
sebegini
|
||||||
|
sebegitu
|
||||||
|
sebelum
|
||||||
|
sebelumnya
|
||||||
|
sebenarnya
|
||||||
|
seberapa
|
||||||
|
sebesar
|
||||||
|
sebetulnya
|
||||||
|
sebisanya
|
||||||
|
sebuah
|
||||||
|
sebut
|
||||||
|
sebutlah
|
||||||
|
sebutnya
|
||||||
|
secara
|
||||||
|
secukupnya
|
||||||
|
sedang
|
||||||
|
sedangkan
|
||||||
|
sedemikian
|
||||||
|
sedikit
|
||||||
|
sedikitnya
|
||||||
|
seenaknya
|
||||||
|
segala
|
||||||
|
segalanya
|
||||||
|
segera
|
||||||
|
seharusnya
|
||||||
|
sehingga
|
||||||
|
seingat
|
||||||
|
sejak
|
||||||
|
sejauh
|
||||||
|
sejenak
|
||||||
|
sejumlah
|
||||||
|
sekadar
|
||||||
|
sekadarnya
|
||||||
|
sekali
|
||||||
|
sekali-kali
|
||||||
|
sekalian
|
||||||
|
sekaligus
|
||||||
|
sekalipun
|
||||||
|
sekarang
|
||||||
|
sekarang
|
||||||
|
sekecil
|
||||||
|
seketika
|
||||||
|
sekiranya
|
||||||
|
sekitar
|
||||||
|
sekitarnya
|
||||||
|
sekurang-kurangnya
|
||||||
|
sekurangnya
|
||||||
|
sela
|
||||||
|
selain
|
||||||
|
selaku
|
||||||
|
selalu
|
||||||
|
selama
|
||||||
|
selama-lamanya
|
||||||
|
selamanya
|
||||||
|
selanjutnya
|
||||||
|
seluruh
|
||||||
|
seluruhnya
|
||||||
|
semacam
|
||||||
|
semakin
|
||||||
|
semampu
|
||||||
|
semampunya
|
||||||
|
semasa
|
||||||
|
semasih
|
||||||
|
semata
|
||||||
|
semata-mata
|
||||||
|
semaunya
|
||||||
|
sementara
|
||||||
|
semisal
|
||||||
|
semisalnya
|
||||||
|
sempat
|
||||||
|
semua
|
||||||
|
semuanya
|
||||||
|
semula
|
||||||
|
sendiri
|
||||||
|
sendirian
|
||||||
|
sendirinya
|
||||||
|
seolah
|
||||||
|
seolah-olah
|
||||||
|
seorang
|
||||||
|
sepanjang
|
||||||
|
sepantasnya
|
||||||
|
sepantasnyalah
|
||||||
|
seperlunya
|
||||||
|
seperti
|
||||||
|
sepertinya
|
||||||
|
sepihak
|
||||||
|
sering
|
||||||
|
seringnya
|
||||||
|
serta
|
||||||
|
serupa
|
||||||
|
sesaat
|
||||||
|
sesama
|
||||||
|
sesampai
|
||||||
|
sesegera
|
||||||
|
sesekali
|
||||||
|
seseorang
|
||||||
|
sesuatu
|
||||||
|
sesuatunya
|
||||||
|
sesudah
|
||||||
|
sesudahnya
|
||||||
|
setelah
|
||||||
|
setempat
|
||||||
|
setengah
|
||||||
|
seterusnya
|
||||||
|
setiap
|
||||||
|
setiba
|
||||||
|
setibanya
|
||||||
|
setidak-tidaknya
|
||||||
|
setidaknya
|
||||||
|
setinggi
|
||||||
|
seusai
|
||||||
|
sewaktu
|
||||||
|
siap
|
||||||
|
siapa
|
||||||
|
siapakah
|
||||||
|
siapapun
|
||||||
|
sini
|
||||||
|
sinilah
|
||||||
|
soal
|
||||||
|
soalnya
|
||||||
|
suatu
|
||||||
|
sudah
|
||||||
|
sudahkah
|
||||||
|
sudahlah
|
||||||
|
supaya
|
||||||
|
tadi
|
||||||
|
tadinya
|
||||||
|
tahu
|
||||||
|
tahun
|
||||||
|
tak
|
||||||
|
tambah
|
||||||
|
tambahnya
|
||||||
|
tampak
|
||||||
|
tampaknya
|
||||||
|
tandas
|
||||||
|
tandasnya
|
||||||
|
tanpa
|
||||||
|
tanya
|
||||||
|
tanyakan
|
||||||
|
tanyanya
|
||||||
|
tapi
|
||||||
|
tegas
|
||||||
|
tegasnya
|
||||||
|
telah
|
||||||
|
tempat
|
||||||
|
tengah
|
||||||
|
tentang
|
||||||
|
tentu
|
||||||
|
tentulah
|
||||||
|
tentunya
|
||||||
|
tepat
|
||||||
|
terakhir
|
||||||
|
terasa
|
||||||
|
terbanyak
|
||||||
|
terdahulu
|
||||||
|
terdapat
|
||||||
|
terdiri
|
||||||
|
terhadap
|
||||||
|
terhadapnya
|
||||||
|
teringat
|
||||||
|
teringat-ingat
|
||||||
|
terjadi
|
||||||
|
terjadilah
|
||||||
|
terjadinya
|
||||||
|
terkira
|
||||||
|
terlalu
|
||||||
|
terlebih
|
||||||
|
terlihat
|
||||||
|
termasuk
|
||||||
|
ternyata
|
||||||
|
tersampaikan
|
||||||
|
tersebut
|
||||||
|
tersebutlah
|
||||||
|
tertentu
|
||||||
|
tertuju
|
||||||
|
terus
|
||||||
|
terutama
|
||||||
|
tetap
|
||||||
|
tetapi
|
||||||
|
tiap
|
||||||
|
tiba
|
||||||
|
tiba-tiba
|
||||||
|
tidak
|
||||||
|
tidakkah
|
||||||
|
tidaklah
|
||||||
|
tiga
|
||||||
|
tinggi
|
||||||
|
toh
|
||||||
|
tunjuk
|
||||||
|
turut
|
||||||
|
tutur
|
||||||
|
tuturnya
|
||||||
|
ucap
|
||||||
|
ucapnya
|
||||||
|
ujar
|
||||||
|
ujarnya
|
||||||
|
umum
|
||||||
|
umumnya
|
||||||
|
ungkap
|
||||||
|
ungkapnya
|
||||||
|
untuk
|
||||||
|
usah
|
||||||
|
usai
|
||||||
|
waduh
|
||||||
|
wah
|
||||||
|
wahai
|
||||||
|
waktu
|
||||||
|
waktunya
|
||||||
|
walau
|
||||||
|
walaupun
|
||||||
|
wong
|
||||||
|
yaitu
|
||||||
|
yakin
|
||||||
|
yakni
|
||||||
|
yang
|
||||||
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,700 @@
|
||||||
|
import streamlit as st
|
||||||
|
import pandas as pd
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import string
|
||||||
|
from collections import Counter
|
||||||
|
import plotly.graph_objects as go
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from database import engine, get_tweet_count, get_latest_crawl_time
|
||||||
|
from page_modules.table_utils import render_standard_table
|
||||||
|
from timezone_utils import (
|
||||||
|
parse_dt_with_tz,
|
||||||
|
parse_dt_with_source_tz,
|
||||||
|
get_timezone_label,
|
||||||
|
get_timezone_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_dt(series):
|
||||||
|
return parse_dt_with_tz(
|
||||||
|
series,
|
||||||
|
st.session_state.get("user_timezone", "WIB (UTC+7)")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_crawled_dt(series):
|
||||||
|
return parse_dt_with_source_tz(
|
||||||
|
series,
|
||||||
|
st.session_state.get("user_timezone", "WIB (UTC+7)"),
|
||||||
|
os.getenv("APP_TIMEZONE", "Asia/Makassar")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def format_dt(value):
|
||||||
|
if value is None or pd.isna(value):
|
||||||
|
return "Belum ada"
|
||||||
|
|
||||||
|
try:
|
||||||
|
tz_label = get_timezone_label(
|
||||||
|
st.session_state.get("user_timezone", "WIB (UTC+7)")
|
||||||
|
)
|
||||||
|
return f"{value.strftime('%d/%m/%Y %H:%M')} {tz_label}"
|
||||||
|
except Exception:
|
||||||
|
return "Belum ada"
|
||||||
|
|
||||||
|
|
||||||
|
def user_today():
|
||||||
|
timezone_choice = st.session_state.get("user_timezone", "WIB (UTC+7)")
|
||||||
|
return pd.Timestamp.now(tz=get_timezone_name(timezone_choice)).date()
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_dynamic_period():
|
||||||
|
mode = st.session_state.get("analysis_mode")
|
||||||
|
today = user_today()
|
||||||
|
|
||||||
|
configs = {
|
||||||
|
"realtime": (
|
||||||
|
today - timedelta(days=6),
|
||||||
|
today,
|
||||||
|
"Tweet Terkini — 7 Hari Terakhir",
|
||||||
|
),
|
||||||
|
"30days": (
|
||||||
|
today - timedelta(days=29),
|
||||||
|
today,
|
||||||
|
"30 Hari Terakhir",
|
||||||
|
),
|
||||||
|
"captured": (
|
||||||
|
today,
|
||||||
|
today,
|
||||||
|
"Tweet Hari Ini",
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
if mode not in configs:
|
||||||
|
return
|
||||||
|
|
||||||
|
start_day, end_day, mode_display = configs[mode]
|
||||||
|
dt_start = datetime.combine(start_day, datetime.min.time())
|
||||||
|
dt_end = datetime.combine(
|
||||||
|
end_day,
|
||||||
|
datetime.max.time().replace(microsecond=0)
|
||||||
|
)
|
||||||
|
|
||||||
|
st.session_state.filter_start_date = dt_start
|
||||||
|
st.session_state.filter_end_date = dt_end
|
||||||
|
st.session_state.filter_label = (
|
||||||
|
f"{dt_start.strftime('%d/%m/%Y')} s/d {dt_end.strftime('%d/%m/%Y')}"
|
||||||
|
)
|
||||||
|
st.session_state.mode_display = mode_display
|
||||||
|
st.session_state.filter_date_column = "created_at"
|
||||||
|
|
||||||
|
|
||||||
|
def _load_stopwords():
|
||||||
|
stopword_file = "indonesian-stopwords-complete.txt"
|
||||||
|
base = set()
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(stopword_file, "r", encoding="utf-8") as f:
|
||||||
|
base = set(f.read().splitlines())
|
||||||
|
|
||||||
|
for kata in ["tidak", "bukan", "jangan", "kurang", "lebih"]:
|
||||||
|
base.discard(kata)
|
||||||
|
|
||||||
|
except FileNotFoundError:
|
||||||
|
base = {
|
||||||
|
"yang", "dan", "di", "ke", "dari", "ini", "itu",
|
||||||
|
"dengan", "untuk", "pada", "adalah", "oleh", "ada",
|
||||||
|
"ya", "akan", "atau", "juga", "sama", "karena",
|
||||||
|
"jika", "sudah", "telah"
|
||||||
|
}
|
||||||
|
|
||||||
|
base.update({
|
||||||
|
"rt", "amp", "https", "http", "co", "t",
|
||||||
|
"wkwk", "wkwkwk", "haha", "hehe",
|
||||||
|
"yg", "dgn", "utk", "dr", "krn", "tp", "jd", "sdh",
|
||||||
|
"aja", "doang", "banget", "bgt", "nih", "sih", "dong", "deh",
|
||||||
|
})
|
||||||
|
|
||||||
|
return base
|
||||||
|
|
||||||
|
|
||||||
|
def _load_stemmer():
|
||||||
|
try:
|
||||||
|
from Sastrawi.Stemmer.StemmerFactory import StemmerFactory
|
||||||
|
return StemmerFactory().create_stemmer()
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
NORMALISASI = {
|
||||||
|
"gk": "tidak",
|
||||||
|
"ga": "tidak",
|
||||||
|
"gak": "tidak",
|
||||||
|
"nggak": "tidak",
|
||||||
|
"ngga": "tidak",
|
||||||
|
"tdk": "tidak",
|
||||||
|
"tak": "tidak",
|
||||||
|
"yg": "yang",
|
||||||
|
"dgn": "dengan",
|
||||||
|
"utk": "untuk",
|
||||||
|
"org": "orang",
|
||||||
|
"krn": "karena",
|
||||||
|
"dr": "dari",
|
||||||
|
"tp": "tapi",
|
||||||
|
"tpi": "tapi",
|
||||||
|
"sm": "sama",
|
||||||
|
"jd": "jadi",
|
||||||
|
"sdh": "sudah",
|
||||||
|
"blm": "belum",
|
||||||
|
"emg": "memang",
|
||||||
|
"emang": "memang",
|
||||||
|
"gimana": "bagaimana",
|
||||||
|
"gitu": "begitu",
|
||||||
|
"gini": "begini",
|
||||||
|
"bgt": "banget",
|
||||||
|
"ongkir": "ongkos kirim",
|
||||||
|
"freeongkir": "gratis ongkir",
|
||||||
|
"free": "gratis",
|
||||||
|
"ecommerce": "e commerce",
|
||||||
|
"komdigi": "komdigi",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def step1_cleaning(text):
|
||||||
|
text = str(text).lower()
|
||||||
|
text = re.sub(r"http\S+|www\S+|https\S+", "", text)
|
||||||
|
text = re.sub(r"@\w+", "", text)
|
||||||
|
text = re.sub(r"#", "", text)
|
||||||
|
text = re.sub(r"\d+", "", text)
|
||||||
|
text = text.translate(
|
||||||
|
str.maketrans("", "", string.punctuation)
|
||||||
|
)
|
||||||
|
text = re.sub(r"[^a-zA-Z\s]", "", text)
|
||||||
|
text = re.sub(r"\s+", " ", text).strip()
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def step2_normalization(text):
|
||||||
|
return " ".join(
|
||||||
|
NORMALISASI.get(word, word)
|
||||||
|
for word in text.split()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def step3_stopword_removal(text, stopwords):
|
||||||
|
return " ".join(
|
||||||
|
word
|
||||||
|
for word in text.split()
|
||||||
|
if word not in stopwords and len(word) > 2
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def step4_stemming(text, stemmer):
|
||||||
|
if stemmer is None:
|
||||||
|
return text
|
||||||
|
|
||||||
|
return stemmer.stem(text)
|
||||||
|
|
||||||
|
|
||||||
|
def full_preprocessing(text, stopwords, stemmer):
|
||||||
|
s1 = step1_cleaning(text)
|
||||||
|
s2 = step2_normalization(s1)
|
||||||
|
s3 = step3_stopword_removal(s2, stopwords)
|
||||||
|
s4 = step4_stemming(s3, stemmer)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"setelah_cleaning": s1,
|
||||||
|
"setelah_normalisasi": s2,
|
||||||
|
"setelah_stopword": s3,
|
||||||
|
"clean_text": s4,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _section_header(title, subtitle=""):
|
||||||
|
"""Render consistent section header card"""
|
||||||
|
sub_html = (
|
||||||
|
f'<div style="font-size:0.78rem;color:#64748b;margin-top:4px;line-height:1.5;">{subtitle}</div>'
|
||||||
|
if subtitle else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:#ffffff;border:1.5px solid #e2e8f0;border-radius:14px;
|
||||||
|
padding:1rem 1.25rem;margin-bottom:1rem;
|
||||||
|
box-shadow:0 2px 6px rgba(15,23,42,0.06);">
|
||||||
|
<div style="font-size:0.95rem;font-weight:700;color:#0f172a;letter-spacing:0.02em;">{title}</div>
|
||||||
|
{sub_html}
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_preprocessing_styles():
|
||||||
|
st.markdown("""
|
||||||
|
<style>
|
||||||
|
.st-key-preprocessing_chart_panel,
|
||||||
|
.st-key-preprocessing_download_panel {
|
||||||
|
background: #ffffff !important;
|
||||||
|
border: 1.5px solid #e2e8f0 !important;
|
||||||
|
border-radius: 12px !important;
|
||||||
|
box-shadow: 0 2px 6px rgba(15,23,42,0.06) !important;
|
||||||
|
padding: 1.1rem 1.2rem 1rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.st-key-preprocessing_download_panel [data-testid="stDownloadButton"] button {
|
||||||
|
min-height: 46px !important;
|
||||||
|
margin-bottom: 0 !important;
|
||||||
|
border-radius: 10px !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.preprocessing-section-gap-sm { height: 1rem; }
|
||||||
|
.preprocessing-section-gap-md { height: 1.45rem; }
|
||||||
|
.preprocessing-section-gap-lg { height: 1.9rem; }
|
||||||
|
</style>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _section_gap(size="md"):
|
||||||
|
st.markdown(
|
||||||
|
f'<div class="preprocessing-section-gap-{size}"></div>',
|
||||||
|
unsafe_allow_html=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _pill(col, icon, bg, color, dark, label, val, sub):
|
||||||
|
"""Render metric pill card with consistent styling"""
|
||||||
|
with col:
|
||||||
|
fs = "1.25rem" if len(str(val)) <= 8 else "1rem"
|
||||||
|
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:{bg};border:1.5px solid {color};border-radius:14px;
|
||||||
|
padding:1.5rem 1.25rem;text-align:center;
|
||||||
|
box-shadow:0 2px 8px {color}15;transition:all 0.2s ease;
|
||||||
|
min-height:178px;margin-bottom:0.65rem;">
|
||||||
|
<div style="width:44px;height:44px;background:{color};border-radius:12px;
|
||||||
|
display:flex;align-items:center;justify-content:center;
|
||||||
|
font-size:1.25rem;margin:0 auto 0.875rem;flex-shrink:0;color:white;">{icon}</div>
|
||||||
|
<div style="font-size:0.7rem;font-weight:700;color:{dark};
|
||||||
|
text-transform:uppercase;letter-spacing:0.06em;
|
||||||
|
margin-bottom:0.4rem;line-height:1.3;">{label}</div>
|
||||||
|
<div style="font-size:{fs};font-weight:800;color:{dark};line-height:1.2;margin-bottom:0.5rem;">{val}</div>
|
||||||
|
<div style="font-size:0.7rem;color:{color};font-weight:600;line-height:1.4;">{sub}</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
|
||||||
|
def show():
|
||||||
|
st.markdown("""
|
||||||
|
<div class="top-header">
|
||||||
|
<div style="display:flex;align-items:center;gap:0.75rem;">
|
||||||
|
<div style="width:36px;height:36px;background:#f0fdf4;border-radius:10px;
|
||||||
|
display:flex;align-items:center;justify-content:center;font-size:1.1rem;">
|
||||||
|
🧹
|
||||||
|
</div>
|
||||||
|
<h1 class="page-title">Bersihkan Data</h1>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
_render_preprocessing_styles()
|
||||||
|
|
||||||
|
st.markdown("""
|
||||||
|
<div style="background:#fff;border:1.5px solid #e2e8f0;border-radius:16px;
|
||||||
|
padding:1.25rem 1.5rem;box-shadow:0 2px 6px rgba(15,23,42,0.07);
|
||||||
|
margin-bottom:1.25rem;">
|
||||||
|
<div style="font-size:1rem;font-weight:700;color:#0f172a;margin-bottom:0.5rem;">
|
||||||
|
📖 Apa yang dilakukan halaman ini?
|
||||||
|
</div>
|
||||||
|
<div style="font-size:0.8375rem;color:#475569;line-height:1.75;">
|
||||||
|
Tweet dibersihkan melalui <strong>4 tahap preprocessing</strong>:
|
||||||
|
<br>
|
||||||
|
<strong>① Cleaning</strong> — hapus URL, mention, hashtag, angka, tanda baca
|
||||||
|
→
|
||||||
|
<strong>② Normalisasi</strong> — ubah kata tidak baku
|
||||||
|
→
|
||||||
|
<strong>③ Stopword Removal</strong> — hapus kata tidak bermakna
|
||||||
|
→
|
||||||
|
<strong>④ Stemming</strong> — ubah kata ke bentuk dasar.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
if "analysis_mode" not in st.session_state:
|
||||||
|
st.warning("⚠️ Silakan pilih mode tampilan di halaman Ambil Data Twitter terlebih dahulu.")
|
||||||
|
return
|
||||||
|
|
||||||
|
_sync_dynamic_period()
|
||||||
|
|
||||||
|
start_date = st.session_state.get("filter_start_date")
|
||||||
|
end_date = st.session_state.get("filter_end_date")
|
||||||
|
filter_label = st.session_state.get("filter_label", "-")
|
||||||
|
mode_display = st.session_state.get("mode_display", "-")
|
||||||
|
|
||||||
|
if start_date is None or end_date is None:
|
||||||
|
st.warning("⚠️ Silakan buka halaman Ambil Data Twitter terlebih dahulu untuk memilih periode.")
|
||||||
|
return
|
||||||
|
|
||||||
|
mode_meta = {
|
||||||
|
"realtime": ("#16a34a", "📡"),
|
||||||
|
"30days": ("#3b6cf7", "📅"),
|
||||||
|
"captured": ("#0284c7", "📆"),
|
||||||
|
"custom": ("#d97706", "🔍"),
|
||||||
|
}
|
||||||
|
|
||||||
|
mode_color, mode_icon = mode_meta.get(
|
||||||
|
st.session_state.analysis_mode,
|
||||||
|
("#3b6cf7", "📊")
|
||||||
|
)
|
||||||
|
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:#fff;border-left:4px solid {mode_color};
|
||||||
|
border-top:1.5px solid #e2e8f0;border-right:1.5px solid #e2e8f0;
|
||||||
|
border-bottom:1.5px solid #e2e8f0;border-radius:0 12px 12px 0;
|
||||||
|
padding:0.875rem 1.25rem;margin-bottom:1.25rem;
|
||||||
|
box-shadow:0 2px 6px rgba(15,23,42,0.07);
|
||||||
|
display:flex;align-items:center;gap:0.75rem;">
|
||||||
|
<span style="font-size:1.375rem;">{mode_icon}</span>
|
||||||
|
<div>
|
||||||
|
<div style="font-size:0.875rem;font-weight:700;color:#0f172a;">{mode_display}</div>
|
||||||
|
<div style="font-size:0.78rem;color:#475569;margin-top:2px;">
|
||||||
|
Periode: <strong style="color:#0f172a;">{filter_label}</strong>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
try:
|
||||||
|
df_all = pd.read_sql(
|
||||||
|
"SELECT * FROM tweets ORDER BY created_at DESC",
|
||||||
|
engine
|
||||||
|
)
|
||||||
|
|
||||||
|
if df_all.empty:
|
||||||
|
st.warning("⚠️ Belum ada data. Kembali ke halaman Ambil Data Twitter.")
|
||||||
|
return
|
||||||
|
|
||||||
|
df_all["created_at"] = parse_dt(df_all["created_at"])
|
||||||
|
if "crawled_at" in df_all.columns:
|
||||||
|
df_all["crawled_at"] = parse_crawled_dt(df_all["crawled_at"])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
st.error(f"❌ {e}")
|
||||||
|
return
|
||||||
|
|
||||||
|
s_dt = pd.Timestamp(start_date)
|
||||||
|
e_dt = pd.Timestamp(end_date)
|
||||||
|
|
||||||
|
df = df_all[
|
||||||
|
(df_all["created_at"] >= s_dt)
|
||||||
|
&
|
||||||
|
(df_all["created_at"] <= e_dt)
|
||||||
|
].copy()
|
||||||
|
|
||||||
|
if df.empty:
|
||||||
|
st.warning(
|
||||||
|
f"⚠️ Tidak ada tweet dengan tanggal asli dalam periode {filter_label}."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
total_tweets_in_db = get_tweet_count()
|
||||||
|
latest_crawl_marker = get_latest_crawl_time() or "no-crawl"
|
||||||
|
data_marker = (total_tweets_in_db, latest_crawl_marker)
|
||||||
|
|
||||||
|
cache_key = (
|
||||||
|
f"pp_{st.session_state.analysis_mode}_"
|
||||||
|
f"{start_date}_{end_date}_{total_tweets_in_db}_{latest_crawl_marker}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Clear old cache entries
|
||||||
|
for old_key in list(st.session_state.keys()):
|
||||||
|
if old_key.startswith("pp_") and old_key != cache_key:
|
||||||
|
del st.session_state[old_key]
|
||||||
|
|
||||||
|
force_refresh = data_marker != st.session_state.get("_pp_last_data_marker")
|
||||||
|
|
||||||
|
if cache_key not in st.session_state or force_refresh:
|
||||||
|
stemmer = _load_stemmer()
|
||||||
|
stopwords = _load_stopwords()
|
||||||
|
|
||||||
|
with st.spinner("🧹 Sedang memproses 4 tahap preprocessing..."):
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for _, row in df.iterrows():
|
||||||
|
r = full_preprocessing(
|
||||||
|
row["text"],
|
||||||
|
stopwords,
|
||||||
|
stemmer
|
||||||
|
)
|
||||||
|
|
||||||
|
r["tweet_id"] = row.get("tweet_id", "")
|
||||||
|
r["text_asli"] = row["text"]
|
||||||
|
r["created_at"] = row["created_at"]
|
||||||
|
r["crawled_at"] = row.get("crawled_at")
|
||||||
|
|
||||||
|
results.append(r)
|
||||||
|
|
||||||
|
df_c = pd.DataFrame(results)
|
||||||
|
|
||||||
|
df_c = df_c[
|
||||||
|
df_c["clean_text"].str.strip().str.len() > 0
|
||||||
|
].copy()
|
||||||
|
|
||||||
|
st.session_state[cache_key] = df_c
|
||||||
|
st.session_state[cache_key + "_stemmer_ok"] = stemmer is not None
|
||||||
|
st.session_state["_pp_last_data_marker"] = data_marker
|
||||||
|
|
||||||
|
df_c = st.session_state[cache_key]
|
||||||
|
stemmer_ok = st.session_state.get(
|
||||||
|
cache_key + "_stemmer_ok",
|
||||||
|
False
|
||||||
|
)
|
||||||
|
|
||||||
|
avg_b = df["text"].astype(str).str.len().mean()
|
||||||
|
avg_a = df_c["clean_text"].astype(str).str.len().mean()
|
||||||
|
red = ((avg_b - avg_a) / avg_b * 100) if avg_b > 0 else 0
|
||||||
|
rmv = len(df) - len(df_c)
|
||||||
|
|
||||||
|
_section_header(
|
||||||
|
"📌 Ringkasan Preprocessing",
|
||||||
|
f"Berdasarkan tanggal asli tweet · {filter_label}"
|
||||||
|
)
|
||||||
|
|
||||||
|
_section_gap("sm")
|
||||||
|
|
||||||
|
c1, c2, c3, c4 = st.columns(4, gap="medium")
|
||||||
|
|
||||||
|
_pill(
|
||||||
|
c1, "📊", "#eef2ff", "#3b6cf7", "#1e3a8a",
|
||||||
|
"Tweet Siap Dianalisis", f"{len(df_c):,}",
|
||||||
|
"Setelah 4 tahap preprocessing"
|
||||||
|
)
|
||||||
|
|
||||||
|
_pill(
|
||||||
|
c2, "📏", "#f0fdf4", "#16a34a", "#14532d",
|
||||||
|
"Panjang Sebelum", f"{avg_b:.0f} karakter",
|
||||||
|
"Rata-rata per tweet"
|
||||||
|
)
|
||||||
|
|
||||||
|
_pill(
|
||||||
|
c3, "✨", "#fff7ed", "#ea580c", "#7c2d12",
|
||||||
|
"Panjang Sesudah", f"{avg_a:.0f} karakter",
|
||||||
|
"Rata-rata per tweet"
|
||||||
|
)
|
||||||
|
|
||||||
|
_pill(
|
||||||
|
c4, "🗑️", "#fef2f2", "#ef4444", "#7f1d1d",
|
||||||
|
"Reduksi Teks", f"{red:.1f}%",
|
||||||
|
f"{rmv} tweet terlalu pendek dibuang"
|
||||||
|
)
|
||||||
|
|
||||||
|
_section_gap("lg")
|
||||||
|
|
||||||
|
steps = [
|
||||||
|
(
|
||||||
|
"#eef2ff", "#3b6cf7", "#1e3a8a",
|
||||||
|
"① Cleaning",
|
||||||
|
"Ubah ke huruf kecil · Hapus URL, mention, hashtag, angka, tanda baca, karakter non-alfabet"
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"#f0fdf4", "#16a34a", "#14532d",
|
||||||
|
"② Normalisasi Kata",
|
||||||
|
"Ubah singkatan/kata gaul seperti gk → tidak, ongkir → ongkos kirim."
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"#fff7ed", "#ea580c", "#7c2d12",
|
||||||
|
"③ Stopword Removal",
|
||||||
|
"Hapus kata tidak bermakna, tetapi pertahankan kata negasi seperti tidak dan bukan."
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"#fefce8", "#ca8a04", "#713f12",
|
||||||
|
"④ Stemming",
|
||||||
|
f"Ubah kata ke bentuk dasar. {'✅ Aktif' if stemmer_ok else '⚠️ Nonaktif'}"
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
cols = st.columns(4, gap="medium")
|
||||||
|
|
||||||
|
for col, (bg, color, dark, title, desc) in zip(cols, steps):
|
||||||
|
with col:
|
||||||
|
st.markdown(f"""
|
||||||
|
<div style="background:{bg};border:1.5px solid {color}44;border-radius:14px;
|
||||||
|
padding:1rem;height:100%;">
|
||||||
|
<div style="font-size:0.8rem;font-weight:800;color:{dark};
|
||||||
|
margin-bottom:0.5rem;">{title}</div>
|
||||||
|
<div style="font-size:0.75rem;color:{dark};opacity:0.85;line-height:1.6;">
|
||||||
|
{desc}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
_section_gap("lg")
|
||||||
|
|
||||||
|
_section_header(
|
||||||
|
"📋 Perbandingan Teks per Tahap Preprocessing",
|
||||||
|
f"{len(df_c):,} tweet · {filter_label}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if "crawled_at" not in df_c.columns:
|
||||||
|
df_c = df_c.copy()
|
||||||
|
df_c["crawled_at"] = pd.NaT
|
||||||
|
|
||||||
|
disp = df_c[
|
||||||
|
[
|
||||||
|
"tweet_id",
|
||||||
|
"created_at",
|
||||||
|
"crawled_at",
|
||||||
|
"text_asli",
|
||||||
|
"setelah_cleaning",
|
||||||
|
"setelah_normalisasi",
|
||||||
|
"setelah_stopword",
|
||||||
|
"clean_text",
|
||||||
|
]
|
||||||
|
].copy()
|
||||||
|
|
||||||
|
disp.columns = [
|
||||||
|
"ID Tweet",
|
||||||
|
"Tanggal Tweet",
|
||||||
|
"Masuk Database",
|
||||||
|
"Teks Asli",
|
||||||
|
"① Setelah Cleaning",
|
||||||
|
"② Setelah Normalisasi",
|
||||||
|
"③ Setelah Stopword",
|
||||||
|
"④ Hasil Akhir",
|
||||||
|
]
|
||||||
|
|
||||||
|
disp["Tanggal Tweet"] = disp["Tanggal Tweet"].apply(format_dt)
|
||||||
|
disp["Masuk Database"] = disp["Masuk Database"].apply(format_dt)
|
||||||
|
|
||||||
|
render_standard_table(
|
||||||
|
disp,
|
||||||
|
height=350,
|
||||||
|
min_width=1580,
|
||||||
|
nowrap=["ID Tweet", "Tanggal Tweet", "Masuk Database"],
|
||||||
|
wide_columns=[
|
||||||
|
"Teks Asli",
|
||||||
|
"① Setelah Cleaning",
|
||||||
|
"② Setelah Normalisasi",
|
||||||
|
"③ Setelah Stopword",
|
||||||
|
"④ Hasil Akhir",
|
||||||
|
],
|
||||||
|
column_widths={
|
||||||
|
"ID Tweet": "160px",
|
||||||
|
"Tanggal Tweet": "170px",
|
||||||
|
"Masuk Database": "170px",
|
||||||
|
"Teks Asli": "300px",
|
||||||
|
"① Setelah Cleaning": "260px",
|
||||||
|
"② Setelah Normalisasi": "260px",
|
||||||
|
"③ Setelah Stopword": "260px",
|
||||||
|
"④ Hasil Akhir": "260px",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
_section_gap("lg")
|
||||||
|
|
||||||
|
with st.container(border=True, key="preprocessing_download_panel"):
|
||||||
|
c1, c2 = st.columns(2, gap="medium", vertical_alignment="bottom")
|
||||||
|
|
||||||
|
with c1:
|
||||||
|
st.download_button(
|
||||||
|
"📥 Unduh Hasil Preprocessing Lengkap",
|
||||||
|
df_c.to_csv(index=False).encode("utf-8"),
|
||||||
|
f"clean_tweet_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv",
|
||||||
|
"text/csv",
|
||||||
|
width="stretch"
|
||||||
|
)
|
||||||
|
|
||||||
|
with c2:
|
||||||
|
out2 = df_c[
|
||||||
|
[
|
||||||
|
"tweet_id",
|
||||||
|
"text_asli",
|
||||||
|
"clean_text"
|
||||||
|
]
|
||||||
|
].copy()
|
||||||
|
|
||||||
|
out2.columns = [
|
||||||
|
"tweet_id",
|
||||||
|
"tweet",
|
||||||
|
"clean_text"
|
||||||
|
]
|
||||||
|
|
||||||
|
st.download_button(
|
||||||
|
"📥 Unduh Teks Bersih Saja",
|
||||||
|
out2.to_csv(index=False).encode("utf-8"),
|
||||||
|
f"teks_bersih_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv",
|
||||||
|
"text/csv",
|
||||||
|
width="stretch"
|
||||||
|
)
|
||||||
|
|
||||||
|
_section_gap("lg")
|
||||||
|
|
||||||
|
_section_header(
|
||||||
|
"📊 Kata-Kata yang Paling Sering Muncul",
|
||||||
|
f"Dari {len(df_c):,} tweet yang sudah bersih"
|
||||||
|
)
|
||||||
|
|
||||||
|
all_words = " ".join(
|
||||||
|
df_c["clean_text"].fillna("")
|
||||||
|
).split()
|
||||||
|
|
||||||
|
stop_extra = {
|
||||||
|
"ongkos",
|
||||||
|
"kirim",
|
||||||
|
"gratis",
|
||||||
|
"komdigi"
|
||||||
|
}
|
||||||
|
|
||||||
|
filtered_words = [
|
||||||
|
w for w in all_words
|
||||||
|
if len(w) > 2 and w not in stop_extra
|
||||||
|
]
|
||||||
|
|
||||||
|
word_freq = Counter(
|
||||||
|
filtered_words
|
||||||
|
).most_common(20)
|
||||||
|
|
||||||
|
if not word_freq:
|
||||||
|
st.info("⚠️ Kata tidak cukup untuk ditampilkan.")
|
||||||
|
|
||||||
|
else:
|
||||||
|
words = [w[0] for w in word_freq]
|
||||||
|
counts = [w[1] for w in word_freq]
|
||||||
|
|
||||||
|
fig = go.Figure(data=[
|
||||||
|
go.Bar(
|
||||||
|
y=words[::-1],
|
||||||
|
x=counts[::-1],
|
||||||
|
orientation="h",
|
||||||
|
marker=dict(color="#3b6cf7"),
|
||||||
|
text=[str(c) for c in counts[::-1]],
|
||||||
|
textposition="outside",
|
||||||
|
hovertemplate="<b>%{y}</b><br>Muncul %{x} kali<extra></extra>",
|
||||||
|
)
|
||||||
|
])
|
||||||
|
|
||||||
|
fig.update_layout(
|
||||||
|
height=500,
|
||||||
|
margin=dict(l=0, r=60, t=8, b=8),
|
||||||
|
paper_bgcolor="rgba(0,0,0,0)",
|
||||||
|
plot_bgcolor="rgba(0,0,0,0)",
|
||||||
|
xaxis=dict(showgrid=True, gridcolor="rgba(203,213,225,0.8)"),
|
||||||
|
yaxis=dict(showgrid=False),
|
||||||
|
)
|
||||||
|
|
||||||
|
with st.container(border=True, key="preprocessing_chart_panel"):
|
||||||
|
st.plotly_chart(
|
||||||
|
fig,
|
||||||
|
width="stretch",
|
||||||
|
config={"displayModeBar": False}
|
||||||
|
)
|
||||||
|
|
||||||
|
_section_gap("lg")
|
||||||
|
|
||||||
|
st.session_state["preprocessed_df"] = df_c
|
||||||
|
|
||||||
|
if st.button(
|
||||||
|
"📈 Lanjut ke Analisis Sentimen →",
|
||||||
|
type="primary",
|
||||||
|
width="stretch"
|
||||||
|
):
|
||||||
|
st.session_state.current_page = "sentiment"
|
||||||
|
st.rerun()
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,271 @@
|
||||||
|
from html import escape
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import streamlit as st
|
||||||
|
|
||||||
|
|
||||||
|
TABLE_CSS = """
|
||||||
|
<style>
|
||||||
|
.sd-table-card {
|
||||||
|
background: #ffffff;
|
||||||
|
border: 1.5px solid #e2e8f0;
|
||||||
|
border-radius: 12px;
|
||||||
|
box-shadow: 0 2px 6px rgba(15,23,42,0.06);
|
||||||
|
overflow: hidden;
|
||||||
|
margin-top: 0.25rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table-scroll {
|
||||||
|
overflow: auto;
|
||||||
|
background: #ffffff;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table {
|
||||||
|
width: 100%;
|
||||||
|
border-collapse: separate;
|
||||||
|
border-spacing: 0;
|
||||||
|
color: #0f172a;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table thead th {
|
||||||
|
position: sticky;
|
||||||
|
top: 0;
|
||||||
|
z-index: 2;
|
||||||
|
background: #f8fafc;
|
||||||
|
border-bottom: 1px solid #e2e8f0;
|
||||||
|
color: #475569;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
font-weight: 800;
|
||||||
|
padding: 0.72rem 0.78rem;
|
||||||
|
text-align: left;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table tbody td {
|
||||||
|
border-top: 1px solid #eef2f7;
|
||||||
|
color: #334155;
|
||||||
|
font-size: 0.78rem;
|
||||||
|
line-height: 1.55;
|
||||||
|
padding: 0.78rem;
|
||||||
|
vertical-align: top;
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table tbody tr:first-child td {
|
||||||
|
border-top: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table tbody tr:hover td {
|
||||||
|
background: #f8fafc;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table .sd-table-right {
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table .sd-table-nowrap {
|
||||||
|
white-space: nowrap;
|
||||||
|
overflow-wrap: normal;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table .sd-table-wide {
|
||||||
|
min-width: 240px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-table-badge {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
min-width: 88px;
|
||||||
|
border-radius: 999px;
|
||||||
|
padding: 0.25rem 0.62rem;
|
||||||
|
font-size: 0.74rem;
|
||||||
|
font-weight: 800;
|
||||||
|
line-height: 1.2 !important;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-badge-green {
|
||||||
|
background: #f0fdf4;
|
||||||
|
border: 1px solid #bbf7d0;
|
||||||
|
color: #16a34a !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-badge-red {
|
||||||
|
background: #fef2f2;
|
||||||
|
border: 1px solid #fecaca;
|
||||||
|
color: #dc2626 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-badge-slate {
|
||||||
|
background: #f8fafc;
|
||||||
|
border: 1px solid #cbd5e1;
|
||||||
|
color: #475569 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sd-badge-blue {
|
||||||
|
background: #eef2ff;
|
||||||
|
border: 1px solid #c7d2fe;
|
||||||
|
color: #3b6cf7 !important;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
TEXT_COLUMN_HINTS = (
|
||||||
|
"tweet",
|
||||||
|
"teks",
|
||||||
|
"isi",
|
||||||
|
"keterangan",
|
||||||
|
"bersih",
|
||||||
|
"clean",
|
||||||
|
"hasil akhir",
|
||||||
|
)
|
||||||
|
|
||||||
|
NOWRAP_COLUMN_HINTS = (
|
||||||
|
"id",
|
||||||
|
"no",
|
||||||
|
"tanggal",
|
||||||
|
"waktu",
|
||||||
|
"status",
|
||||||
|
"sentimen",
|
||||||
|
"keyakinan",
|
||||||
|
"database",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _plain_value(value):
|
||||||
|
if value is None:
|
||||||
|
return "-"
|
||||||
|
|
||||||
|
try:
|
||||||
|
if pd.isna(value):
|
||||||
|
return "-"
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
return str(value)
|
||||||
|
|
||||||
|
|
||||||
|
def _badge_class(value):
|
||||||
|
text = value.lower()
|
||||||
|
|
||||||
|
if "negatif" in text or "gagal" in text:
|
||||||
|
return "sd-badge-red"
|
||||||
|
|
||||||
|
if "positif" in text or "berhasil" in text:
|
||||||
|
return "sd-badge-green"
|
||||||
|
|
||||||
|
if "netral" in text:
|
||||||
|
return "sd-badge-slate"
|
||||||
|
|
||||||
|
return "sd-badge-blue"
|
||||||
|
|
||||||
|
|
||||||
|
def _is_text_column(column):
|
||||||
|
name = column.lower()
|
||||||
|
return any(hint in name for hint in TEXT_COLUMN_HINTS)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_nowrap_column(column):
|
||||||
|
name = column.lower()
|
||||||
|
return any(hint in name for hint in NOWRAP_COLUMN_HINTS)
|
||||||
|
|
||||||
|
|
||||||
|
def render_standard_table(
|
||||||
|
df,
|
||||||
|
*,
|
||||||
|
height=380,
|
||||||
|
min_width=760,
|
||||||
|
right_align=None,
|
||||||
|
nowrap=None,
|
||||||
|
badge_columns=None,
|
||||||
|
wide_columns=None,
|
||||||
|
column_widths=None,
|
||||||
|
):
|
||||||
|
if df.empty:
|
||||||
|
st.info("Tidak ada data untuk ditampilkan.")
|
||||||
|
return
|
||||||
|
|
||||||
|
right_align = set(right_align or [])
|
||||||
|
nowrap = set(nowrap or [])
|
||||||
|
badge_columns = set(badge_columns or [])
|
||||||
|
wide_columns = set(wide_columns or [])
|
||||||
|
column_widths = column_widths or {}
|
||||||
|
|
||||||
|
columns = list(df.columns)
|
||||||
|
|
||||||
|
colgroup = ""
|
||||||
|
if column_widths:
|
||||||
|
col_tags = []
|
||||||
|
|
||||||
|
for column in columns:
|
||||||
|
width = column_widths.get(column)
|
||||||
|
style = f' style="width:{escape(str(width))};"' if width else ""
|
||||||
|
col_tags.append(f"<col{style}>")
|
||||||
|
|
||||||
|
colgroup = f"<colgroup>{''.join(col_tags)}</colgroup>"
|
||||||
|
|
||||||
|
header_cells = []
|
||||||
|
|
||||||
|
for column in columns:
|
||||||
|
classes = []
|
||||||
|
|
||||||
|
if column in right_align:
|
||||||
|
classes.append("sd-table-right")
|
||||||
|
|
||||||
|
header_cells.append(
|
||||||
|
f'<th class="{" ".join(classes)}">{escape(str(column))}</th>'
|
||||||
|
)
|
||||||
|
|
||||||
|
body_rows = []
|
||||||
|
|
||||||
|
for _, row in df.iterrows():
|
||||||
|
cells = []
|
||||||
|
|
||||||
|
for column in columns:
|
||||||
|
value = _plain_value(row[column])
|
||||||
|
classes = []
|
||||||
|
|
||||||
|
if column in right_align:
|
||||||
|
classes.append("sd-table-right")
|
||||||
|
|
||||||
|
if column in nowrap or _is_nowrap_column(str(column)):
|
||||||
|
classes.append("sd-table-nowrap")
|
||||||
|
|
||||||
|
if column in wide_columns or _is_text_column(str(column)):
|
||||||
|
classes.append("sd-table-wide")
|
||||||
|
|
||||||
|
class_attr = f' class="{" ".join(classes)}"' if classes else ""
|
||||||
|
|
||||||
|
if column in badge_columns:
|
||||||
|
badge_class = _badge_class(value)
|
||||||
|
cell_html = (
|
||||||
|
f'<span class="sd-table-badge {badge_class}">'
|
||||||
|
f"{escape(value)}</span>"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cell_html = escape(value)
|
||||||
|
|
||||||
|
cells.append(f"<td{class_attr}>{cell_html}</td>")
|
||||||
|
|
||||||
|
body_rows.append(f"<tr>{''.join(cells)}</tr>")
|
||||||
|
|
||||||
|
table_html = f"""
|
||||||
|
{TABLE_CSS}
|
||||||
|
<div class="sd-table-card">
|
||||||
|
<div class="sd-table-scroll" style="max-height:{int(height)}px;">
|
||||||
|
<table class="sd-table" style="min-width:{int(min_width)}px;">
|
||||||
|
{colgroup}
|
||||||
|
<thead>
|
||||||
|
<tr>{''.join(header_cells)}</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{''.join(body_rows)}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
|
||||||
|
st.markdown(table_html, unsafe_allow_html=True)
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
streamlit
|
||||||
|
streamlit-autorefresh
|
||||||
|
pandas
|
||||||
|
plotly
|
||||||
|
joblib
|
||||||
|
scikit-learn
|
||||||
|
Sastrawi
|
||||||
|
python-dotenv
|
||||||
|
sqlalchemy
|
||||||
|
wordcloud
|
||||||
|
matplotlib
|
||||||
|
openpyxl
|
||||||
|
playwright
|
||||||
|
pymysql
|
||||||
|
numpy
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
from crawler import main
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,364 @@
|
||||||
|
import os
|
||||||
|
import glob
|
||||||
|
import hashlib
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from pandas.errors import EmptyDataError
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
from database import save_tweets, init_db, get_existing_tweet_ids
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
AUTH_TOKEN = os.getenv("TWITTER_AUTH_TOKEN")
|
||||||
|
|
||||||
|
QUERY = os.getenv(
|
||||||
|
"QUERY",
|
||||||
|
'komdigi (ongkir OR "gratis ongkir" OR "free ongkir") OR "pembatasan gratis ongkir" OR "gratis ongkir dibatasi"'
|
||||||
|
)
|
||||||
|
|
||||||
|
SCRAPE_LIMIT = int(
|
||||||
|
os.getenv("SCRAPE_LIMIT", "50")
|
||||||
|
)
|
||||||
|
|
||||||
|
RECENT_DAYS = int(
|
||||||
|
os.getenv("RECENT_DAYS", "2")
|
||||||
|
)
|
||||||
|
|
||||||
|
APP_TIMEZONE = os.getenv("APP_TIMEZONE", "Asia/Makassar")
|
||||||
|
|
||||||
|
SCRAPE_TABS = [
|
||||||
|
tab.strip().upper()
|
||||||
|
for tab in os.getenv("SCRAPE_TABS", "LATEST,TOP").split(",")
|
||||||
|
if tab.strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def get_recent_window():
|
||||||
|
today = datetime.now(ZoneInfo(APP_TIMEZONE)).date()
|
||||||
|
since_day = today - timedelta(days=max(RECENT_DAYS - 1, 0))
|
||||||
|
until_day = today + timedelta(days=1)
|
||||||
|
|
||||||
|
return since_day, until_day
|
||||||
|
|
||||||
|
|
||||||
|
def build_recent_query():
|
||||||
|
query_lower = f" {QUERY.lower()}"
|
||||||
|
|
||||||
|
if " since:" in query_lower or " until:" in query_lower:
|
||||||
|
return QUERY
|
||||||
|
|
||||||
|
since_day, until_day = get_recent_window()
|
||||||
|
|
||||||
|
return f"({QUERY}) since:{since_day.isoformat()} until:{until_day.isoformat()}"
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_tweet_date(value):
|
||||||
|
parsed = pd.to_datetime(value, errors="coerce", utc=True)
|
||||||
|
|
||||||
|
if pd.isna(parsed):
|
||||||
|
# Selalu gunakan UTC timezone untuk konsistensi
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
return parsed.isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def is_in_recent_window(value):
|
||||||
|
parsed = pd.to_datetime(value, errors="coerce", utc=True)
|
||||||
|
|
||||||
|
if pd.isna(parsed):
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
local_dt = parsed.tz_convert(APP_TIMEZONE)
|
||||||
|
except Exception:
|
||||||
|
local_dt = parsed
|
||||||
|
|
||||||
|
since_day, until_day = get_recent_window()
|
||||||
|
tweet_day = local_dt.date()
|
||||||
|
|
||||||
|
return since_day <= tweet_day < until_day
|
||||||
|
|
||||||
|
|
||||||
|
def stable_fallback_id(row, full_text):
|
||||||
|
username = str(row.get("username", "") or "unknown").strip() or "unknown"
|
||||||
|
digest = hashlib.sha1(
|
||||||
|
full_text.encode("utf-8", errors="ignore")
|
||||||
|
).hexdigest()[:16]
|
||||||
|
|
||||||
|
return f"{username}_{digest}"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_tweet_id(row, full_text):
|
||||||
|
for column in ("id_str", "tweet_id", "id"):
|
||||||
|
value = row.get(column)
|
||||||
|
|
||||||
|
if pd.notna(value) and str(value).strip():
|
||||||
|
return str(value).strip()
|
||||||
|
|
||||||
|
return stable_fallback_id(row, full_text)
|
||||||
|
|
||||||
|
|
||||||
|
def is_header_like_row(row, tweet_id, full_text):
|
||||||
|
text = str(full_text or "").strip().lower()
|
||||||
|
tweet_id_text = str(tweet_id or "").strip().lower()
|
||||||
|
created_at = str(row.get("created_at", "") or "").strip().lower()
|
||||||
|
|
||||||
|
header_values = {
|
||||||
|
"full_text",
|
||||||
|
"text",
|
||||||
|
"id_str",
|
||||||
|
"tweet_id",
|
||||||
|
"created_at",
|
||||||
|
"conversation_id_str",
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
text in header_values
|
||||||
|
or tweet_id_text in header_values
|
||||||
|
or created_at == "created_at"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def ambil_file_csv_terbaru(min_mtime=None, output_name=None):
|
||||||
|
if output_name:
|
||||||
|
output_path = os.path.join("tweets-data", output_name)
|
||||||
|
root, ext = os.path.splitext(output_path)
|
||||||
|
files = [
|
||||||
|
path for path in (
|
||||||
|
output_path,
|
||||||
|
f"{root}.old{ext}",
|
||||||
|
)
|
||||||
|
if os.path.exists(path)
|
||||||
|
]
|
||||||
|
else:
|
||||||
|
files = glob.glob("tweets-data/*.csv")
|
||||||
|
|
||||||
|
if not files:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if min_mtime is not None:
|
||||||
|
files = [
|
||||||
|
path for path in files
|
||||||
|
if os.path.getmtime(path) >= min_mtime
|
||||||
|
]
|
||||||
|
|
||||||
|
if not files:
|
||||||
|
return None
|
||||||
|
|
||||||
|
non_empty_files = [
|
||||||
|
path for path in files
|
||||||
|
if os.path.getsize(path) > 2
|
||||||
|
]
|
||||||
|
|
||||||
|
if non_empty_files:
|
||||||
|
files = non_empty_files
|
||||||
|
|
||||||
|
return max(
|
||||||
|
files,
|
||||||
|
key=os.path.getmtime
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def scrape_tab(tab, effective_query, limit):
|
||||||
|
output_name = f"hasil_{tab.lower()}.csv"
|
||||||
|
started_at = time.time() - 1
|
||||||
|
|
||||||
|
command = [
|
||||||
|
"npx",
|
||||||
|
"--yes",
|
||||||
|
"tweet-harvest@2.6.1",
|
||||||
|
"-o",
|
||||||
|
output_name,
|
||||||
|
"-s",
|
||||||
|
effective_query,
|
||||||
|
"--tab",
|
||||||
|
tab,
|
||||||
|
"-l",
|
||||||
|
str(limit),
|
||||||
|
"--token",
|
||||||
|
AUTH_TOKEN,
|
||||||
|
]
|
||||||
|
|
||||||
|
print(f"Scraping tab {tab}...")
|
||||||
|
try:
|
||||||
|
result = subprocess.run(command, capture_output=True, text=True, timeout=180)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
print(f"tweet-harvest tab {tab} timed out after 180 seconds")
|
||||||
|
raise RuntimeError(
|
||||||
|
f"tweet-harvest tab {tab} timed out (likely no results atau network issue)"
|
||||||
|
)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
error_msg = result.stderr if result.stderr else result.stdout
|
||||||
|
print(f"tweet-harvest stderr: {error_msg}")
|
||||||
|
raise RuntimeError(
|
||||||
|
f"tweet-harvest tab {tab} gagal dengan exit code {result.returncode}: {error_msg}"
|
||||||
|
)
|
||||||
|
|
||||||
|
latest_file = ambil_file_csv_terbaru(started_at, output_name)
|
||||||
|
|
||||||
|
if latest_file is None:
|
||||||
|
print(f"CSV baru untuk tab {tab} tidak ditemukan")
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
# Cek ukuran file sebelum membaca
|
||||||
|
try:
|
||||||
|
file_size = os.path.getsize(latest_file)
|
||||||
|
if file_size <= 2:
|
||||||
|
print(f"CSV tab {tab} kosong atau tidak valid (ukuran: {file_size} bytes)")
|
||||||
|
return pd.DataFrame()
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Gagal memeriksa ukuran file {latest_file}: {e}")
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
try:
|
||||||
|
df = pd.read_csv(latest_file)
|
||||||
|
except EmptyDataError as e:
|
||||||
|
print(f"CSV tab {tab} kosong atau tidak valid: {e}")
|
||||||
|
return pd.DataFrame()
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Gagal membaca CSV tab {tab}: {e}")
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
if df.empty:
|
||||||
|
print(f"Data tab {tab} kosong")
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
if "full_text" not in df.columns:
|
||||||
|
print(f"Kolom full_text tidak ditemukan pada tab {tab}")
|
||||||
|
print(df.columns.tolist())
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
df["_source_tab"] = tab
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
def scrape_once(limit=SCRAPE_LIMIT):
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
if not AUTH_TOKEN:
|
||||||
|
raise ValueError(
|
||||||
|
"TWITTER_AUTH_TOKEN belum diisi di file .env"
|
||||||
|
)
|
||||||
|
|
||||||
|
effective_query = build_recent_query()
|
||||||
|
tabs = SCRAPE_TABS or ["LATEST"]
|
||||||
|
|
||||||
|
print("Scraping tweet realtime...")
|
||||||
|
print("Query:", effective_query)
|
||||||
|
print("Tab:", ", ".join(tabs))
|
||||||
|
|
||||||
|
frames = []
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
for tab in tabs:
|
||||||
|
try:
|
||||||
|
tab_df = scrape_tab(tab, effective_query, limit)
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(f"{tab}: {e}")
|
||||||
|
print(f"Tab {tab} gagal:", e)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not tab_df.empty:
|
||||||
|
frames.append(tab_df)
|
||||||
|
|
||||||
|
if not frames:
|
||||||
|
# Jika tidak ada data baru dari scraper, kembalikan 0 daripada error
|
||||||
|
# Ini normal jika tidak ada tweet baru yang sesuai kriteria
|
||||||
|
if errors:
|
||||||
|
print("Warning - Scrape failed with errors:", " | ".join(errors))
|
||||||
|
else:
|
||||||
|
print("No new data found in twitter search")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
df = pd.concat(frames, ignore_index=True)
|
||||||
|
print(f"Total hasil mentah dari crawler: {len(df)} tweet")
|
||||||
|
|
||||||
|
df["_full_text_clean"] = df["full_text"].fillna("").astype(str)
|
||||||
|
df = df[df["_full_text_clean"].str.strip().ne("")].copy()
|
||||||
|
|
||||||
|
data = []
|
||||||
|
outside_window = 0
|
||||||
|
|
||||||
|
for _, row in df.iterrows():
|
||||||
|
full_text = str(row.get("_full_text_clean", ""))
|
||||||
|
tweet_id = extract_tweet_id(row, full_text)
|
||||||
|
|
||||||
|
if is_header_like_row(row, tweet_id, full_text):
|
||||||
|
continue
|
||||||
|
|
||||||
|
created_at = normalize_tweet_date(
|
||||||
|
row.get("created_at", datetime.now().isoformat())
|
||||||
|
)
|
||||||
|
|
||||||
|
if not is_in_recent_window(created_at):
|
||||||
|
outside_window += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
data.append({
|
||||||
|
"tweet_id": tweet_id,
|
||||||
|
"text": full_text,
|
||||||
|
"created_at": created_at,
|
||||||
|
"crawled_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
})
|
||||||
|
|
||||||
|
if outside_window:
|
||||||
|
print(
|
||||||
|
f"{outside_window} tweet dilewati karena di luar rentang realtime"
|
||||||
|
)
|
||||||
|
|
||||||
|
before_dedupe = len(data)
|
||||||
|
deduped = {}
|
||||||
|
|
||||||
|
for tweet in data:
|
||||||
|
deduped.setdefault(tweet["tweet_id"], tweet)
|
||||||
|
|
||||||
|
data = list(deduped.values())
|
||||||
|
duplicate_between_tabs = before_dedupe - len(data)
|
||||||
|
|
||||||
|
if duplicate_between_tabs:
|
||||||
|
print(
|
||||||
|
f"{duplicate_between_tabs} tweet duplikat dari LATEST/TOP dilewati"
|
||||||
|
)
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
print("Tidak ada tweet hari ini yang bisa disimpan")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
existing_ids = get_existing_tweet_ids(
|
||||||
|
[tweet["tweet_id"] for tweet in data]
|
||||||
|
)
|
||||||
|
|
||||||
|
new_data = [
|
||||||
|
tweet for tweet in data
|
||||||
|
if tweet["tweet_id"] not in existing_ids
|
||||||
|
]
|
||||||
|
|
||||||
|
skipped = len(data) - len(new_data)
|
||||||
|
|
||||||
|
if skipped:
|
||||||
|
print(f"{skipped} tweet dilewati karena sudah ada di database")
|
||||||
|
|
||||||
|
if not new_data:
|
||||||
|
print("Tidak ada tweet baru untuk disimpan")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
saved = save_tweets(
|
||||||
|
new_data,
|
||||||
|
"realtime"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"{saved} tweet baru berhasil disimpan"
|
||||||
|
)
|
||||||
|
|
||||||
|
return saved
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
scrape_once()
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
import re
|
||||||
|
import string
|
||||||
|
import joblib
|
||||||
|
|
||||||
|
model = joblib.load("model_naive_bayes.pkl")
|
||||||
|
tfidf = joblib.load("tfidf_vectorizer.pkl")
|
||||||
|
|
||||||
|
|
||||||
|
def bersihkan_teks(text):
|
||||||
|
|
||||||
|
text = str(text).lower()
|
||||||
|
|
||||||
|
text = re.sub(
|
||||||
|
r"http\S+|www\S+|@\w+|#|\d+",
|
||||||
|
"",
|
||||||
|
text
|
||||||
|
)
|
||||||
|
|
||||||
|
text = text.translate(
|
||||||
|
str.maketrans("", "", string.punctuation)
|
||||||
|
)
|
||||||
|
|
||||||
|
text = re.sub(r"\s+", " ", text).strip()
|
||||||
|
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def prediksi_sentimen(list_text):
|
||||||
|
|
||||||
|
clean_texts = [
|
||||||
|
bersihkan_teks(text)
|
||||||
|
for text in list_text
|
||||||
|
]
|
||||||
|
|
||||||
|
vectors = tfidf.transform(clean_texts)
|
||||||
|
|
||||||
|
predictions = model.predict(vectors)
|
||||||
|
|
||||||
|
return clean_texts, predictions
|
||||||
Binary file not shown.
|
|
@ -0,0 +1,224 @@
|
||||||
|
"""
|
||||||
|
Timezone utilities untuk dashboard sentimen.
|
||||||
|
Mendukung 3 zona waktu Indonesia: WIB, WITA, WIT
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
from datetime import datetime
|
||||||
|
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
# Mapping timezone Indonesia
|
||||||
|
INDONESIA_TIMEZONES = {
|
||||||
|
"WIB (UTC+7)": "Asia/Jakarta", # Western Indonesia (Barat)
|
||||||
|
"WITA (UTC+8)": "Asia/Makassar", # Central Indonesia (Tengah)
|
||||||
|
"WIT (UTC+9)": "Asia/Jayapura", # Eastern Indonesia (Timur)
|
||||||
|
}
|
||||||
|
|
||||||
|
BROWSER_TIMEZONE_TO_CHOICE = {
|
||||||
|
"Asia/Jakarta": "WIB (UTC+7)",
|
||||||
|
"Asia/Pontianak": "WIB (UTC+7)",
|
||||||
|
"Asia/Makassar": "WITA (UTC+8)",
|
||||||
|
"Asia/Ujung_Pandang": "WITA (UTC+8)",
|
||||||
|
"Asia/Jayapura": "WIT (UTC+9)",
|
||||||
|
"Asia/Singapore": "WITA (UTC+8)",
|
||||||
|
"Asia/Kuala_Lumpur": "WITA (UTC+8)",
|
||||||
|
"Asia/Brunei": "WITA (UTC+8)",
|
||||||
|
}
|
||||||
|
|
||||||
|
TIMEZONE_NAME_TO_CHOICE = {
|
||||||
|
value: key for key, value in INDONESIA_TIMEZONES.items()
|
||||||
|
}
|
||||||
|
TIMEZONE_NAME_TO_CHOICE.update(BROWSER_TIMEZONE_TO_CHOICE)
|
||||||
|
|
||||||
|
BROWSER_OFFSET_TO_CHOICE = {
|
||||||
|
420: "WIB (UTC+7)",
|
||||||
|
480: "WITA (UTC+8)",
|
||||||
|
540: "WIT (UTC+9)",
|
||||||
|
}
|
||||||
|
|
||||||
|
TIMEZONE_DISPLAY = {
|
||||||
|
"Asia/Jakarta": "WIB",
|
||||||
|
"Asia/Makassar": "WITA",
|
||||||
|
"Asia/Jayapura": "WIT",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_default_timezone():
|
||||||
|
"""Get default timezone from APP_TIMEZONE, falling back to WITA."""
|
||||||
|
return TIMEZONE_NAME_TO_CHOICE.get(
|
||||||
|
os.getenv("APP_TIMEZONE", "Asia/Makassar"),
|
||||||
|
"WITA (UTC+8)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_timezone_label(timezone_choice="WIB (UTC+7)"):
|
||||||
|
"""Return short timezone label for a user timezone choice."""
|
||||||
|
target_tz = get_timezone_name(timezone_choice)
|
||||||
|
|
||||||
|
if target_tz in TIMEZONE_DISPLAY:
|
||||||
|
return TIMEZONE_DISPLAY[target_tz]
|
||||||
|
|
||||||
|
try:
|
||||||
|
return datetime.now(ZoneInfo(target_tz)).strftime("%Z") or target_tz
|
||||||
|
except Exception:
|
||||||
|
return target_tz
|
||||||
|
|
||||||
|
|
||||||
|
def get_timezone_name(timezone_choice="WIB (UTC+7)"):
|
||||||
|
"""Return IANA timezone name for a user timezone choice."""
|
||||||
|
if timezone_choice in INDONESIA_TIMEZONES:
|
||||||
|
return INDONESIA_TIMEZONES[timezone_choice]
|
||||||
|
|
||||||
|
if is_valid_timezone_name(timezone_choice):
|
||||||
|
return timezone_choice
|
||||||
|
|
||||||
|
return INDONESIA_TIMEZONES.get(get_default_timezone(), "Asia/Makassar")
|
||||||
|
|
||||||
|
|
||||||
|
def is_valid_timezone_name(timezone_name):
|
||||||
|
"""Return True if value is a valid IANA timezone name."""
|
||||||
|
if not timezone_name or not isinstance(timezone_name, str):
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
ZoneInfo(timezone_name)
|
||||||
|
return True
|
||||||
|
except ZoneInfoNotFoundError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def browser_timezone_to_choice(browser_timezone):
|
||||||
|
"""Map browser IANA timezone to an Indonesian choice or the original IANA name."""
|
||||||
|
if browser_timezone in BROWSER_TIMEZONE_TO_CHOICE:
|
||||||
|
return BROWSER_TIMEZONE_TO_CHOICE[browser_timezone]
|
||||||
|
|
||||||
|
if is_valid_timezone_name(browser_timezone):
|
||||||
|
return browser_timezone
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def browser_offset_to_choice(offset_minutes):
|
||||||
|
"""Map browser UTC offset in minutes to one of the Indonesian choices."""
|
||||||
|
try:
|
||||||
|
offset = int(offset_minutes)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
return BROWSER_OFFSET_TO_CHOICE.get(offset)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_dt_with_tz(series, timezone_choice="WIB (UTC+7)"):
|
||||||
|
"""
|
||||||
|
Parse datetime series dan konversi ke timezone pilihan user.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
series: pandas Series dengan datetime values
|
||||||
|
timezone_choice: User's timezone choice (format: "WIB (UTC+7)")
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
pandas Series dengan datetime yang sudah dikonversi ke timezone lokal
|
||||||
|
"""
|
||||||
|
return parse_dt_with_source_tz(series, timezone_choice, "UTC")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_dt_with_source_tz(series, timezone_choice="WIB (UTC+7)", naive_source_tz="UTC"):
|
||||||
|
"""
|
||||||
|
Parse datetime series and convert to user's timezone.
|
||||||
|
|
||||||
|
Datetime values with explicit timezone keep their own timezone. Naive values
|
||||||
|
are interpreted as `naive_source_tz` before conversion.
|
||||||
|
"""
|
||||||
|
target_tz = get_timezone_name(timezone_choice)
|
||||||
|
|
||||||
|
def convert_one(value):
|
||||||
|
if value is None or pd.isna(value):
|
||||||
|
return pd.NaT
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = pd.to_datetime(value, errors="coerce", format="mixed")
|
||||||
|
except Exception:
|
||||||
|
return pd.NaT
|
||||||
|
|
||||||
|
if pd.isna(parsed):
|
||||||
|
return pd.NaT
|
||||||
|
|
||||||
|
try:
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
parsed = parsed.tz_localize(naive_source_tz)
|
||||||
|
|
||||||
|
return parsed.tz_convert(target_tz).tz_localize(None)
|
||||||
|
except Exception:
|
||||||
|
return pd.NaT
|
||||||
|
|
||||||
|
return pd.Series(series).apply(convert_one)
|
||||||
|
|
||||||
|
|
||||||
|
def format_datetime(dt_value, timezone_choice="WIB (UTC+7)", format_str="%d/%m/%Y %H:%M:%S"):
|
||||||
|
"""
|
||||||
|
Format datetime value dengan timezone dan menampilkan label zona.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
dt_value: datetime value
|
||||||
|
timezone_choice: User's timezone choice
|
||||||
|
format_str: Format string untuk strftime
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted string dengan timezone label
|
||||||
|
"""
|
||||||
|
if pd.isna(dt_value):
|
||||||
|
return "N/A"
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Parse to UTC first if needed
|
||||||
|
if isinstance(dt_value, str):
|
||||||
|
dt_value = pd.to_datetime(dt_value, errors="coerce", utc=True, format="mixed")
|
||||||
|
|
||||||
|
# Convert to target timezone
|
||||||
|
target_tz = get_timezone_name(timezone_choice)
|
||||||
|
|
||||||
|
if pd.isna(dt_value):
|
||||||
|
return "N/A"
|
||||||
|
|
||||||
|
# If no timezone info, assume UTC
|
||||||
|
if dt_value.tz is None:
|
||||||
|
dt_value = pd.Timestamp(dt_value, tz="UTC")
|
||||||
|
|
||||||
|
# Convert to target timezone
|
||||||
|
converted = dt_value.tz_convert(target_tz)
|
||||||
|
tz_label = get_timezone_label(timezone_choice)
|
||||||
|
|
||||||
|
return f"{converted.strftime(format_str)} {tz_label}"
|
||||||
|
except Exception as e:
|
||||||
|
return str(dt_value)
|
||||||
|
|
||||||
|
|
||||||
|
def format_datetime_short(dt_value, timezone_choice="WIB (UTC+7)"):
|
||||||
|
"""Format datetime in short format: HH:MM:SS TIMEZONE"""
|
||||||
|
return format_datetime(dt_value, timezone_choice, "%H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
|
def format_date_and_time(dt_value, timezone_choice="WIB (UTC+7)"):
|
||||||
|
"""Format datetime in date and time separately"""
|
||||||
|
if pd.isna(dt_value):
|
||||||
|
return "N/A", "N/A"
|
||||||
|
|
||||||
|
try:
|
||||||
|
if isinstance(dt_value, str):
|
||||||
|
dt_value = pd.to_datetime(dt_value, errors="coerce", utc=True, format="mixed")
|
||||||
|
|
||||||
|
target_tz = get_timezone_name(timezone_choice)
|
||||||
|
tz_label = get_timezone_label(timezone_choice)
|
||||||
|
|
||||||
|
if dt_value.tz is None:
|
||||||
|
dt_value = pd.Timestamp(dt_value, tz="UTC")
|
||||||
|
|
||||||
|
converted = dt_value.tz_convert(target_tz)
|
||||||
|
|
||||||
|
date_str = converted.strftime("%d/%m/%Y")
|
||||||
|
time_str = f"{converted.strftime('%H:%M:%S')} {tz_label}"
|
||||||
|
|
||||||
|
return date_str, time_str
|
||||||
|
except Exception:
|
||||||
|
return "N/A", "N/A"
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
"conversation_id_str","created_at","favorite_count","full_text","id_str","image_url","in_reply_to_screen_name","lang","location","quote_count","reply_count","retweet_count","tweet_url","user_id_str","username"
|
||||||
|
"2052443461857812806","Thu May 07 17:40:28 +0000 2026","0","komdigi gratis ongkir","2052443461857812806","","","in","","0","0","0","https://x.com/undefined/status/2052443461857812806","1303393431654260737",
|
||||||
|
"2052443021707469159","Thu May 07 17:38:43 +0000 2026","0","komdigi gratis ongkir gk jelas jir","2052443021707469159","","","in","","0","0","0","https://x.com/undefined/status/2052443021707469159","1303393431654260737",
|
||||||
|
"2027305700427522148","Mon Mar 02 01:26:27 +0000 2026","1","@_katakita08 @doctor__fabre Komdigi sibuk ngurusin fre ongkir","2028280745362653688","","_katakita08","in","","0","0","0","https://x.com/undefined/status/2028280745362653688","1374686160748830724",
|
||||||
|
"2016738074940805382","Thu Jan 29 05:08:55 +0000 2026","66","@tempodotco Sudah berapa kali data pribadi WNI bocor dan apa yg komdigi lakukan mereka lebih tertarik mengatur 1 orang 1 akun sosmed pembatasan gratis ongkir blokir grok SIM card maksimal 3 per NIK","2016740315689029739","","tempodotco","in","","0","0","3","https://x.com/undefined/status/2016740315689029739","1829801785609109504",
|
||||||
|
"1995631281837953172","Mon Dec 01 23:09:09 +0000 2025","0","Gitu komdigi sok2 an ngatur diskon / subsidi ongkir di marketplace dengan alasan supaya persaingan bisnis lebih fair. Tp jelas2 ada kayak gini diem.","1995631281837953172","","","in","","0","0","0","https://x.com/undefined/status/1995631281837953172","1696021858645344256",
|
||||||
|
"1993480321187373349","Wed Nov 26 00:48:48 +0000 2025","0","@txtdrimedia Komdigi lagi sibuk satu orang satu akun yang punya hp wajib daftar pakai pemindai wajah mebatasi free ongkir di e-commerce dan sibuk nyalahin Cloudflare maraknya judol","1993482034753491122","","txtdrimedia","in","","0","0","0","https://x.com/undefined/status/1993482034753491122","1829801785609109504",
|
||||||
|
"1984282936938467824","Sat Nov 01 06:23:56 +0000 2025","4","@__AnakKolong @meutya_hafid @kemkomdigi @KemensetnegRI @setkabgoid Menteri Free Ongkir ini sering buat kegaduhan dengan statement nya yang super TOLOL. Menteri Komdigi tapi ga ngerti IT samgat bodoh sekali","1984506677421543929","","__AnakKolong","in","","0","0","0","https://x.com/undefined/status/1984506677421543929","1410466956134608899",
|
||||||
|
"1953352953361756412","Thu Aug 07 08:16:51 +0000 2025","0","@tanyakanrl Soal batasin ongkar ongkir sih paling gercep tuh si komdigi mah. Soal ginian melempem","1953369736168735098","","tanyakanrl","in","","0","0","0","https://x.com/undefined/status/1953369736168735098","1691576611261542401",
|
||||||
|
"1951265147789648203","Mon Aug 04 10:51:53 +0000 2025","0","@PPATK Yg anda bekuin ntu tabungan warga malih bukan tabunga judol. Bedakan itu dlu deh. Kalau mau sasar judol kenapa engga kerjasama sama komdigi? Oh lupa komdigi ngurusin perihal ongkir ya? Ckckckc kerjaannya apa eh malah kerja ini. Bilang aja engha bisa kerja.","1952321588067455142","","PPATK","in","","0","0","0","https://x.com/undefined/status/1952321588067455142","1450056695728664576",
|
||||||
|
"1948343267055333546","Thu Jul 24 16:30:52 +0000 2025","5","@__AnakKolong @KemensetnegRI @meutya_hafid @kemkomdigi @prabowo nahh keguoblokan Komdigi mulai ditunjukkan sudah lah kalau memang tak mau diganti menterinya. lebih baik suruh diam saja urus tuh free ongkir","1948420627892756653","","__AnakKolong","in","","0","0","0","https://x.com/undefined/status/1948420627892756653","1410466956134608899",
|
||||||
|
"1943472326781415902","Fri Jul 11 01:02:11 +0000 2025","4","@V3g3L Komdigi sibuk ngurusin gratis ongkir Polisi sibuk menanam jagung ","1943475874608906577","","P3gEl","in","","0","0","0","https://x.com/undefined/status/1943475874608906577","1374686160748830724",
|
||||||
|
"1943264388397568315","Thu Jul 10 23:59:17 +0000 2025","1","@TaliUdeng @puanmaharani_ri Komdigi suruh ngurusin judol aja daripada ngurusin gratis ongkir","1943460046597427506","","TaliUdeng","in","","0","0","0","https://x.com/undefined/status/1943460046597427506","1374686160748830724",
|
||||||
|
"1943107751561040062","Thu Jul 10 00:55:24 +0000 2025","4","@DS_yantie @puanmaharani_ri peran dari komdigi? hanya menghabiskan apbn saja dan jangan lupa program andalan komdigi 'PEMBATASN FREE ONGKIR' Itu lah keguoblokan dan ketololan menterinya https://t.co/Z9qWycSiwV","1943111782664671388","https://pbs.twimg.com/media/GvdRqA0bwAAjex4.png","DS_yantie","in","","0","1","0","https://x.com/undefined/status/1943111782664671388","1410466956134608899",
|
||||||
|
"1942884216951419006","Thu Jul 10 00:41:25 +0000 2025","2","Komdigi masih sibuk ngurusin gratis ongkir kayaknya ya https://t.co/59D5qMexS1","1943108262205010384","","DS_yantie","in","","0","0","0","https://x.com/undefined/status/1943108262205010384","1374686160748830724",
|
||||||
|
"1942961310884610558","Wed Jul 09 15:05:22 +0000 2025","2","@MARQUEZ__93 Buat AI hanya untuk membatasi FREE ONGKIR saja? itu lah menteri give away setiap hari menunjukkan keguoblokan dan kedunguan nya. yang mengherankan 1 kantor komdigi mengapa diam saja? apakah mereka semua sama level keguoblokannya?","1942963292944973994","","MARQUEZ__93","in","","0","1","1","https://x.com/undefined/status/1942963292944973994","1410466956134608899",
|
||||||
|
"1937347075077021966","Tue Jun 24 03:08:31 +0000 2025","0","Apa ini propaganda dari wacana awal komdigi yang mau batasi gratis ongkir itu? Hmm terus jadinya disuruh patungan deh user oren gila emang tapi udah masuk lingkaran mau gimana lagi","1937347075077021966","","","in","","0","1","0","https://x.com/undefined/status/1937347075077021966","835321350",
|
||||||
|
"1935660141225152631","Thu Jun 19 11:42:11 +0000 2025","1","@seruanhl @Anak__Ogi @AntoniusCDN @DS_yantie @ch_chotimah2 @Urrangawak @FirzaHusain @HumorJonTampan @kenhans03 @denni_sauya @Cintada16 itu tugas @kemkomdigi gimana nih komdigi? hanya mengurusi ttg Free Ongkir saja ? dan take down twit di X ? https://t.co/SqaObOCeiR","1935664405536198896","","seruanhl","in","","0","0","0","https://x.com/undefined/status/1935664405536198896","1410466956134608899",
|
||||||
|
"1935493341237699002","Thu Jun 19 01:58:20 +0000 2025","2","@DS_yantie karna disetiap kota/kabupaten tidak mempunyai data center hingga pendaftaran via online (yg jauh lebih cepat dan sederhana) tidak mampu kapasitas loading nya jika hanya mengandalkan data ke pusat semua. Komdigi yg seharusnya berpikir jalan keluarnya malah hanya mikir free ongkir","1935517475124429113","","DS_yantie","in","","0","1","0","https://x.com/undefined/status/1935517475124429113","1410466956134608899",
|
||||||
|
"1934063492786762088","Sun Jun 15 14:53:08 +0000 2025","0","@DS_yantie @puanmaharani_ri nah itu menteri give away yg ga penting malah diurusi yang penting malah dilupakan. Nama kementeriannya keren Komdigi = Kementerian Komunikasi dan Digital tapi hanya ngurusi free ongkir saja? HOIII Menteri Tolol diurus tuh Data Center jangan sampai kebobolan lagi","1934262908369993873","","DS_yantie","in","","0","0","0","https://x.com/undefined/status/1934262908369993873","1410466956134608899",
|
||||||
|
"1934063492786762088","Sun Jun 15 01:40:44 +0000 2025","63","Karena Komdigi nya sibuk ngurusin e-SIM dan gratis ongkir @puanmaharani_ri apresiasi dukungan meta buat RI perangi Judol Indonesia diketahui tercatat sebagai salah satu negara dengan jumlah konten judol terbanyak yang diblokir di platform Meta. https://t.co/ItDYKXOOXN","1934063492786762088","https://pbs.twimg.com/media/GtcsfxXbUAA0FQe.jpg","","in","","2","9","30","https://x.com/undefined/status/1934063492786762088","1374686160748830724",
|
||||||
|
|
|
@ -0,0 +1,21 @@
|
||||||
|
"conversation_id_str","created_at","favorite_count","full_text","id_str","image_url","in_reply_to_screen_name","lang","location","quote_count","reply_count","retweet_count","tweet_url","user_id_str","username"
|
||||||
|
"2052443461857812806","Thu May 07 17:40:28 +0000 2026","0","komdigi gratis ongkir","2052443461857812806","","","in","","0","0","0","https://x.com/undefined/status/2052443461857812806","1303393431654260737",
|
||||||
|
"2052443021707469159","Thu May 07 17:38:43 +0000 2026","0","komdigi gratis ongkir gk jelas jir","2052443021707469159","","","in","","0","0","0","https://x.com/undefined/status/2052443021707469159","1303393431654260737",
|
||||||
|
"2027305700427522148","Mon Mar 02 01:26:27 +0000 2026","1","@_katakita08 @doctor__fabre Komdigi sibuk ngurusin fre ongkir","2028280745362653688","","_katakita08","in","","0","0","0","https://x.com/undefined/status/2028280745362653688","1374686160748830724",
|
||||||
|
"2016738074940805382","Thu Jan 29 05:08:55 +0000 2026","66","@tempodotco Sudah berapa kali data pribadi WNI bocor dan apa yg komdigi lakukan mereka lebih tertarik mengatur 1 orang 1 akun sosmed pembatasan gratis ongkir blokir grok SIM card maksimal 3 per NIK","2016740315689029739","","tempodotco","in","","0","0","3","https://x.com/undefined/status/2016740315689029739","1829801785609109504",
|
||||||
|
"1995631281837953172","Mon Dec 01 23:09:09 +0000 2025","0","Gitu komdigi sok2 an ngatur diskon / subsidi ongkir di marketplace dengan alasan supaya persaingan bisnis lebih fair. Tp jelas2 ada kayak gini diem.","1995631281837953172","","","in","","0","0","0","https://x.com/undefined/status/1995631281837953172","1696021858645344256",
|
||||||
|
"1993480321187373349","Wed Nov 26 00:48:48 +0000 2025","0","@txtdrimedia Komdigi lagi sibuk satu orang satu akun yang punya hp wajib daftar pakai pemindai wajah mebatasi free ongkir di e-commerce dan sibuk nyalahin Cloudflare maraknya judol","1993482034753491122","","txtdrimedia","in","","0","0","0","https://x.com/undefined/status/1993482034753491122","1829801785609109504",
|
||||||
|
"1984282936938467824","Sat Nov 01 06:23:56 +0000 2025","4","@__AnakKolong @meutya_hafid @kemkomdigi @KemensetnegRI @setkabgoid Menteri Free Ongkir ini sering buat kegaduhan dengan statement nya yang super TOLOL. Menteri Komdigi tapi ga ngerti IT samgat bodoh sekali","1984506677421543929","","__AnakKolong","in","","0","0","0","https://x.com/undefined/status/1984506677421543929","1410466956134608899",
|
||||||
|
"1953352953361756412","Thu Aug 07 08:16:51 +0000 2025","0","@tanyakanrl Soal batasin ongkar ongkir sih paling gercep tuh si komdigi mah. Soal ginian melempem","1953369736168735098","","tanyakanrl","in","","0","0","0","https://x.com/undefined/status/1953369736168735098","1691576611261542401",
|
||||||
|
"1951265147789648203","Mon Aug 04 10:51:53 +0000 2025","0","@PPATK Yg anda bekuin ntu tabungan warga malih bukan tabunga judol. Bedakan itu dlu deh. Kalau mau sasar judol kenapa engga kerjasama sama komdigi? Oh lupa komdigi ngurusin perihal ongkir ya? Ckckckc kerjaannya apa eh malah kerja ini. Bilang aja engha bisa kerja.","1952321588067455142","","PPATK","in","","0","0","0","https://x.com/undefined/status/1952321588067455142","1450056695728664576",
|
||||||
|
"1948343267055333546","Thu Jul 24 16:30:52 +0000 2025","5","@__AnakKolong @KemensetnegRI @meutya_hafid @kemkomdigi @prabowo nahh keguoblokan Komdigi mulai ditunjukkan sudah lah kalau memang tak mau diganti menterinya. lebih baik suruh diam saja urus tuh free ongkir","1948420627892756653","","__AnakKolong","in","","0","0","0","https://x.com/undefined/status/1948420627892756653","1410466956134608899",
|
||||||
|
"1943472326781415902","Fri Jul 11 01:02:11 +0000 2025","4","@V3g3L Komdigi sibuk ngurusin gratis ongkir Polisi sibuk menanam jagung ","1943475874608906577","","P3gEl","in","","0","0","0","https://x.com/undefined/status/1943475874608906577","1374686160748830724",
|
||||||
|
"1943264388397568315","Thu Jul 10 23:59:17 +0000 2025","1","@TaliUdeng @puanmaharani_ri Komdigi suruh ngurusin judol aja daripada ngurusin gratis ongkir","1943460046597427506","","TaliUdeng","in","","0","0","0","https://x.com/undefined/status/1943460046597427506","1374686160748830724",
|
||||||
|
"1943107751561040062","Thu Jul 10 00:55:24 +0000 2025","4","@DS_yantie @puanmaharani_ri peran dari komdigi? hanya menghabiskan apbn saja dan jangan lupa program andalan komdigi 'PEMBATASN FREE ONGKIR' Itu lah keguoblokan dan ketololan menterinya https://t.co/Z9qWycSiwV","1943111782664671388","https://pbs.twimg.com/media/GvdRqA0bwAAjex4.png","DS_yantie","in","","0","1","0","https://x.com/undefined/status/1943111782664671388","1410466956134608899",
|
||||||
|
"1942884216951419006","Thu Jul 10 00:41:25 +0000 2025","2","Komdigi masih sibuk ngurusin gratis ongkir kayaknya ya https://t.co/59D5qMexS1","1943108262205010384","","DS_yantie","in","","0","0","0","https://x.com/undefined/status/1943108262205010384","1374686160748830724",
|
||||||
|
"1942961310884610558","Wed Jul 09 15:05:22 +0000 2025","2","@MARQUEZ__93 Buat AI hanya untuk membatasi FREE ONGKIR saja? itu lah menteri give away setiap hari menunjukkan keguoblokan dan kedunguan nya. yang mengherankan 1 kantor komdigi mengapa diam saja? apakah mereka semua sama level keguoblokannya?","1942963292944973994","","MARQUEZ__93","in","","0","1","1","https://x.com/undefined/status/1942963292944973994","1410466956134608899",
|
||||||
|
"1937347075077021966","Tue Jun 24 03:08:31 +0000 2025","0","Apa ini propaganda dari wacana awal komdigi yang mau batasi gratis ongkir itu? Hmm terus jadinya disuruh patungan deh user oren gila emang tapi udah masuk lingkaran mau gimana lagi","1937347075077021966","","","in","","0","1","0","https://x.com/undefined/status/1937347075077021966","835321350",
|
||||||
|
"1935660141225152631","Thu Jun 19 11:42:11 +0000 2025","1","@seruanhl @Anak__Ogi @AntoniusCDN @DS_yantie @ch_chotimah2 @Urrangawak @FirzaHusain @HumorJonTampan @kenhans03 @denni_sauya @Cintada16 itu tugas @kemkomdigi gimana nih komdigi? hanya mengurusi ttg Free Ongkir saja ? dan take down twit di X ? https://t.co/SqaObOCeiR","1935664405536198896","","seruanhl","in","","0","0","0","https://x.com/undefined/status/1935664405536198896","1410466956134608899",
|
||||||
|
"1935493341237699002","Thu Jun 19 01:58:20 +0000 2025","2","@DS_yantie karna disetiap kota/kabupaten tidak mempunyai data center hingga pendaftaran via online (yg jauh lebih cepat dan sederhana) tidak mampu kapasitas loading nya jika hanya mengandalkan data ke pusat semua. Komdigi yg seharusnya berpikir jalan keluarnya malah hanya mikir free ongkir","1935517475124429113","","DS_yantie","in","","0","1","0","https://x.com/undefined/status/1935517475124429113","1410466956134608899",
|
||||||
|
"1934063492786762088","Sun Jun 15 14:53:08 +0000 2025","0","@DS_yantie @puanmaharani_ri nah itu menteri give away yg ga penting malah diurusi yang penting malah dilupakan. Nama kementeriannya keren Komdigi = Kementerian Komunikasi dan Digital tapi hanya ngurusi free ongkir saja? HOIII Menteri Tolol diurus tuh Data Center jangan sampai kebobolan lagi","1934262908369993873","","DS_yantie","in","","0","0","0","https://x.com/undefined/status/1934262908369993873","1410466956134608899",
|
||||||
|
"1934063492786762088","Sun Jun 15 01:40:44 +0000 2025","63","Karena Komdigi nya sibuk ngurusin e-SIM dan gratis ongkir @puanmaharani_ri apresiasi dukungan meta buat RI perangi Judol Indonesia diketahui tercatat sebagai salah satu negara dengan jumlah konten judol terbanyak yang diblokir di platform Meta. https://t.co/ItDYKXOOXN","1934063492786762088","https://pbs.twimg.com/media/GtcsfxXbUAA0FQe.jpg","","in","","2","9","30","https://x.com/undefined/status/1934063492786762088","1374686160748830724",
|
||||||
|
|
|
@ -0,0 +1 @@
|
||||||
|
|
||||||
|
|
|
|
@ -0,0 +1 @@
|
||||||
|
|
||||||
|
|
|
|
@ -0,0 +1,8 @@
|
||||||
|
"conversation_id_str","created_at","favorite_count","full_text","id_str","image_url","in_reply_to_screen_name","lang","location","quote_count","reply_count","retweet_count","tweet_url","user_id_str","username"
|
||||||
|
"2053330583707914695","Sun May 10 04:25:34 +0000 2026","0","komdigi gaada kerjaan ngurus ongkir","2053330583707914695","","","in","","0","0","0","https://x.com/undefined/status/2053330583707914695","1916470659917758464",
|
||||||
|
"2053330483665375476","Sun May 10 04:25:10 +0000 2026","0","komdigi ngurusin gratis ongkir doang","2053330483665375476","","","in","","0","0","0","https://x.com/undefined/status/2053330483665375476","1916470659917758464",
|
||||||
|
"2053067823627518260","Sat May 09 11:01:27 +0000 2026","0","komdigi gratis ongkir ya","2053067823627518260","","","in","","0","0","0","https://x.com/undefined/status/2053067823627518260","1916470659917758464",
|
||||||
|
"2052921293289496669","Sat May 09 01:19:12 +0000 2026","0","komdigi gratis ongkir","2052921293289496669","","","in","","0","0","0","https://x.com/undefined/status/2052921293289496669","1916470659917758464",
|
||||||
|
"2052921254785786307","Sat May 09 01:19:03 +0000 2026","0","komdigi ongkir","2052921254785786307","","","in","","0","0","0","https://x.com/undefined/status/2052921254785786307","1916470659917758464",
|
||||||
|
"2053046421184827846","Sat May 09 09:36:25 +0000 2026","0","komdigi gratis ongkir ga jelas","2053046421184827846","","","in","","0","0","0","https://x.com/undefined/status/2053046421184827846","1916470659917758464",
|
||||||
|
"2053046378579050583","Sat May 09 09:36:15 +0000 2026","0","komdigi gratis ongkir tai","2053046378579050583","","","in","","0","0","0","https://x.com/undefined/status/2053046378579050583","1916470659917758464",
|
||||||
|
Loading…
Reference in New Issue