# Praktik Mandiri 1 — Data Readiness Check & Cleaning Validation
## Participant Starter Notebook — Guided Technical Version

Notebook ini membantu kamu menjalankan workflow PM1 dengan **technical guardrails** agar tidak salah folder, salah source, atau salah export.

**Alur PM1:**
**Setup → Technical Check → Load → Intake → Grain & Key → Profiling → Findings → Verification → Rulebook → Cleaning → Validation → Raw Integrity → Export → Final Gate**

### Sebelum RUN
1. Buka **folder `Bootcamp_Data_Analyst` sebagai workspace** di Antigravity.
2. Simpan notebook ini di folder `notebooks/`.
3. Extract package `SB-Pxx`, lalu letakkan **5 raw files langsung** di `data/raw/pm1_swiftbite/`.
4. Buka `PM1_Data_Readiness_Working_Paper.xlsx` sebagai tempat mencatat decision & evidence.
5. Jalankan notebook **dari atas ke bawah**.

> Jika `TECHNICAL GATE: PASS`, **jangan mengubah path** di notebook.

### Guardrails
- Raw/source **tidak boleh diubah atau ditimpa**.
- Jangan menghapus row hanya karena terlihat aneh.
- Repeated `order_id` belum tentu duplicate.
- Finding belum otomatis menjadi cleaning action.
- Material change harus dapat ditelusuri: **Finding ID → Rule ID → Cleaning Log → Validation Evidence**.
- Notebook membantu sisi teknis. **Judgment, cleaning rule, dan interpretation tetap pekerjaan peserta.**

0. Identitas Praktik — EDIT BAGIAN INI

Ubah hanya **Nama** dan **Variant ID** sesuai package yang kamu terima.

Contoh Variant ID: `SB-P01`, `SB-P12`, `SB-P28`.

# ============================================================
# EDIT AREA 1 — IDENTITAS PESERTA
# ============================================================
PARTICIPANT_NAME = "Azizah"
VARIANT_ID = "SB-P19"   # contoh: SB-P01

name_ok = PARTICIPANT_NAME.strip() not in {"", "Isi Nama Kamu"}
variant_ok = (
    VARIANT_ID.startswith("SB-P")
    and len(VARIANT_ID) == 6
    and VARIANT_ID[4:].isdigit()
)

IDENTITY_GATE_PASS = bool(name_ok and variant_ok)

if IDENTITY_GATE_PASS:
    print("IDENTITY GATE: PASS")
    print("Participant :", PARTICIPANT_NAME)
    print("Variant ID  :", VARIANT_ID)
else:
    print("IDENTITY GATE: FAIL")
    print("→ Isi PARTICIPANT_NAME dan VARIANT_ID pada cell ini.")
    print("→ Format Variant ID harus seperti SB-P01 atau SB-P28.")
IDENTITY GATE: PASS
Participant : Azizah
Variant ID  : SB-P19

1. Technical Check — Workspace & Folder

Notebook akan mencari **Workspace Starter yang sama dari Sesi 1** berdasarkan keberadaan folder `data/` dan `notebooks/`.

**Jangan membuat workspace baru. Jangan copy-paste absolute path.**

Jika output menunjukkan `TECHNICAL GATE: PASS`, lanjut ke cell berikutnya tanpa mengubah path.

from pathlib import Path
import json
import hashlib
import pandas as pd
import numpy as np

try:
    import duckdb
    DUCKDB_AVAILABLE = True
except ImportError:
    duckdb = None
    DUCKDB_AVAILABLE = False

pd.set_option("display.max_columns", 100)
pd.set_option("display.max_rows", 100)
pd.set_option("display.width", 180)

def find_workspace_root(start=None):
    start = Path(start or Path.cwd()).resolve()
    for p in [start, *start.parents]:
        if (p / "data").is_dir() and (p / "notebooks").is_dir():
            return p
    return None

WORKSPACE_ROOT = find_workspace_root()

if WORKSPACE_ROOT is None:
    TECHNICAL_GATE_PASS = False
    RAW_DIR = Path("__WORKSPACE_NOT_FOUND__")
    PROCESSED_DIR = Path("__WORKSPACE_NOT_FOUND__")
    OUTPUT_DIR = Path("__WORKSPACE_NOT_FOUND__")
    DOCS_DIR = Path("__WORKSPACE_NOT_FOUND__")

    print("TECHNICAL GATE: FAIL")
    print("→ Workspace Starter tidak ditemukan.")
    print("→ Di Antigravity, buka folder Bootcamp_Data_Analyst sebagai workspace.")
    print("→ Pastikan folder data/ dan notebooks/ terlihat di Explorer.")
    print("→ Setelah itu RUN ulang cell ini. Jangan edit path secara manual.")
else:
    TECHNICAL_GATE_PASS = True
    RAW_DIR = WORKSPACE_ROOT / "data" / "raw" / "pm1_swiftbite"
    PROCESSED_DIR = WORKSPACE_ROOT / "data" / "processed" / "pm1_swiftbite"
    OUTPUT_DIR = WORKSPACE_ROOT / "outputs" / "pm1_swiftbite"
    DOCS_DIR = WORKSPACE_ROOT / "docs" / "pm1_swiftbite"

    for folder in [PROCESSED_DIR, OUTPUT_DIR, DOCS_DIR]:
        folder.mkdir(parents=True, exist_ok=True)

    print("TECHNICAL GATE: PASS")
    print("WORKSPACE_ROOT :", WORKSPACE_ROOT)
    print("RAW_DIR        :", RAW_DIR)
    print("PROCESSED_DIR  :", PROCESSED_DIR)
    print("OUTPUT_DIR     :", OUTPUT_DIR)
    print("DOCS_DIR       :", DOCS_DIR)
TECHNICAL GATE: PASS
WORKSPACE_ROOT : C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst
RAW_DIR        : C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst\data\raw\pm1_swiftbite
PROCESSED_DIR  : C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst\data\processed\pm1_swiftbite
OUTPUT_DIR     : C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst\outputs\pm1_swiftbite
DOCS_DIR       : C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst\docs\pm1_swiftbite

Environment Smoke Test — RUN

Tujuan cell ini hanya memastikan library utama tersedia. Jika DuckDB belum aktif, workflow utama starter tetap dapat berjalan dengan pandas.

print("pandas :", pd.__version__)

try:
    import openpyxl
    EXCEL_ENGINE_AVAILABLE = True
    print("openpyxl:", openpyxl.__version__)
except ImportError:
    EXCEL_ENGINE_AVAILABLE = False
    print("openpyxl: NOT AVAILABLE")

if DUCKDB_AVAILABLE:
    print("DuckDB :", duckdb.__version__)
    try:
        con = duckdb.connect()
        print("DuckDB connection: OK")
    except Exception as e:
        print("DuckDB connection: WARNING —", type(e).__name__, str(e))
else:
    print("DuckDB : NOT AVAILABLE")
    print("INFO: workflow utama notebook tetap dapat dijalankan dengan pandas.")

if not EXCEL_ENGINE_AVAILABLE:
    print("ENVIRONMENT CHECK: NEEDS FIX")
    print("→ File XLSX membutuhkan openpyxl. Aktifkan environment sesuai setup Sesi 1.")
else:
    print("ENVIRONMENT CHECK: PASS")
pandas : 3.0.6
openpyxl: 3.1.5
DuckDB : 1.5.6
DuckDB connection: OK
ENVIRONMENT CHECK: PASS

2. Verify Source Package — RUN

Letakkan **5 file berikut langsung** di:

`data/raw/pm1_swiftbite/`

- `swiftbite_customers.json`
- `swiftbite_orders.xlsx`
- `swiftbite_order_items.csv`
- `swiftbite_payments.txt`
- `swiftbite_restaurants.csv`

**Jangan** meletakkan ZIP di folder tersebut dan **jangan** membuat folder `SB-Pxx` lagi di dalam `pm1_swiftbite/`.

EXPECTED_FILES = {
    "customers": "swiftbite_customers.json",
    "orders": "swiftbite_orders.xlsx",
    "order_items": "swiftbite_order_items.csv",
    "payments": "swiftbite_payments.txt",
    "restaurants": "swiftbite_restaurants.csv",
}

source_status = []

if not TECHNICAL_GATE_PASS:
    missing_files = list(EXPECTED_FILES.values())
    SOURCE_ACCESS_PASS = False
    print("SOURCE ACCESS CHECK: SKIPPED")
    print("→ Perbaiki TECHNICAL GATE terlebih dahulu.")
else:
    for logical_name, filename in EXPECTED_FILES.items():
        path = RAW_DIR / filename
        source_status.append({
            "logical_name": logical_name,
            "filename": filename,
            "exists": path.is_file(),
            "path": str(path),
        })

    source_status_df = pd.DataFrame(source_status)
    display(source_status_df)

    missing_files = source_status_df.loc[
        ~source_status_df["exists"], "filename"
    ].tolist()

    SOURCE_ACCESS_PASS = len(missing_files) == 0

    if SOURCE_ACCESS_PASS:
        print("SOURCE ACCESS CHECK: PASS")
        print("→ Semua 5 raw files ditemukan. Jangan ubah path.")
    else:
        print("SOURCE ACCESS CHECK: FAIL")
        print("File belum ditemukan:", missing_files)
        print("→ Extract package peserta.")
        print("→ Copy hanya 5 raw files langsung ke:", RAW_DIR)
        print("→ Jangan menaruh file di subfolder tambahan.")
logical_name filename exists path
0 customers swiftbite_customers.json True C:\Users\hp\OneDrive\Dokumen\Future Maker Prog...
1 orders swiftbite_orders.xlsx True C:\Users\hp\OneDrive\Dokumen\Future Maker Prog...
2 order_items swiftbite_order_items.csv True C:\Users\hp\OneDrive\Dokumen\Future Maker Prog...
3 payments swiftbite_payments.txt True C:\Users\hp\OneDrive\Dokumen\Future Maker Prog...
4 restaurants swiftbite_restaurants.csv True C:\Users\hp\OneDrive\Dokumen\Future Maker Prog...
SOURCE ACCESS CHECK: PASS
→ Semua 5 raw files ditemukan. Jangan ubah path.

3. Load Raw Data & Schema Gate — RUN

Cell ini:
1. membaca kelima source secara aman,
2. menampilkan error per file jika ada,
3. memeriksa kolom minimum yang diperlukan Starter,
4. menyimpan **raw hash baseline** untuk pemeriksaan integritas di akhir.

Jika `RAW LOAD GATE: PASS` dan `SCHEMA GATE: PASS`, lanjutkan.

REQUIRED_COLUMNS = {
    "Customers": {
        "customer_id", "full_name", "email", "phone_number",
        "signup_date", "customer_zone", "customer_segment"
    },
    "Orders": {
        "order_id", "customer_id", "restaurant_id", "order_timestamp",
        "order_status", "subtotal_amount", "delivery_fee",
        "discount_amount", "total_amount"
    },
    "Order Items": {
        "order_item_id", "order_id", "line_number", "item_name",
        "item_category", "quantity", "unit_price",
        "line_discount", "line_amount"
    },
    "Payments": {
        "payment_id", "order_id", "payment_timestamp",
        "payment_method", "payment_status", "payment_amount"
    },
    "Restaurants": {
        "restaurant_id", "restaurant_name", "cuisine_type",
        "restaurant_zone", "price_tier", "active_status"
    },
}

def sha256_file(path: Path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            h.update(chunk)
    return h.hexdigest()

def safe_read_source(label, path, reader):
    try:
        return reader(path), None
    except Exception as e:
        return None, f"{type(e).__name__}: {e}"

raw_tables = {}
load_errors = {}
RAW_HASH_BASELINE = {}

if not SOURCE_ACCESS_PASS:
    RAW_LOAD_GATE_PASS = False
    SCHEMA_GATE_PASS = False
    DATA_READY = False
    print("RAW LOAD GATE: SKIPPED")
    print("→ Perbaiki SOURCE ACCESS CHECK terlebih dahulu.")
else:
    readers = {
        "Customers": lambda p: pd.read_json(p),
        "Orders": lambda p: pd.read_excel(p),
        "Order Items": lambda p: pd.read_csv(p),
        "Payments": lambda p: pd.read_csv(p, sep="|"),
        "Restaurants": lambda p: pd.read_csv(p),
    }
    file_map = {
        "Customers": RAW_DIR / EXPECTED_FILES["customers"],
        "Orders": RAW_DIR / EXPECTED_FILES["orders"],
        "Order Items": RAW_DIR / EXPECTED_FILES["order_items"],
        "Payments": RAW_DIR / EXPECTED_FILES["payments"],
        "Restaurants": RAW_DIR / EXPECTED_FILES["restaurants"],
    }

    for label, path in file_map.items():
        df, err = safe_read_source(label, path, readers[label])
        if err:
            load_errors[label] = err
        else:
            raw_tables[label] = df
            RAW_HASH_BASELINE[label] = sha256_file(path)

    RAW_LOAD_GATE_PASS = len(load_errors) == 0 and len(raw_tables) == 5

    if RAW_LOAD_GATE_PASS:
        customers_raw = raw_tables["Customers"]
        orders_raw = raw_tables["Orders"]
        order_items_raw = raw_tables["Order Items"]
        payments_raw = raw_tables["Payments"]
        restaurants_raw = raw_tables["Restaurants"]
        print("RAW LOAD GATE: PASS")
    else:
        print("RAW LOAD GATE: FAIL")
        for label, err in load_errors.items():
            print(f"→ {label}: {err}")

    schema_rows = []
    if RAW_LOAD_GATE_PASS:
        for label, required in REQUIRED_COLUMNS.items():
            actual = set(raw_tables[label].columns)
            missing_cols = sorted(required - actual)
            schema_rows.append({
                "dataset": label,
                "required_columns": len(required),
                "actual_columns": len(actual),
                "missing_required_columns": ", ".join(missing_cols) if missing_cols else "",
                "status": "PASS" if not missing_cols else "FAIL",
            })

        schema_check_df = pd.DataFrame(schema_rows)
        display(schema_check_df)

        SCHEMA_GATE_PASS = (schema_check_df["status"] == "PASS").all()

        if SCHEMA_GATE_PASS:
            print("SCHEMA GATE: PASS")
        else:
            print("SCHEMA GATE: FAIL")
            print("→ Jangan lanjut cleaning. Periksa apakah package/variant yang digunakan sudah benar.")
    else:
        SCHEMA_GATE_PASS = False

    DATA_READY = bool(
        TECHNICAL_GATE_PASS
        and SOURCE_ACCESS_PASS
        and RAW_LOAD_GATE_PASS
        and SCHEMA_GATE_PASS
    )

if DATA_READY:
    print("DATA READY GATE: PASS")
else:
    print("DATA READY GATE: FAIL")
RAW LOAD GATE: PASS
dataset required_columns actual_columns missing_required_columns status
0 Customers 7 7 PASS
1 Orders 9 10 PASS
2 Order Items 9 9 PASS
3 Payments 6 7 PASS
4 Restaurants 6 7 PASS
SCHEMA GATE: PASS
DATA READY GATE: PASS

Quick Source Preview — RUN

Pastikan nama kolom dan contoh record masuk akal. Ini **belum** berarti data sudah bersih.

if DATA_READY:
    for name, df in raw_tables.items():
        print(f"\n=== {name} | shape={df.shape} ===")
        display(df.head(3))
else:
    print("PREVIEW SKIPPED — DATA READY GATE belum PASS.")
=== Customers | shape=(167, 7) ===
customer_id full_name email phone_number signup_date customer_zone customer_segment
0 C0001 Eka Maulana ekamaulana1@example.com 083549565705 2025-12-24 East Regular
1 C0002 Nadia Santoso nadiasantoso2@example.com 089333361058 2025-07-05 Central Regular
2 C0003 Hana Sari hanasari3@example.com 081792756270 2025-06-22 North New
=== Orders | shape=(335, 10) ===
order_id customer_id restaurant_id order_timestamp order_status promo_code subtotal_amount delivery_fee discount_amount total_amount
0 O00001 C0147 R0019 2026-03-28 11:18:00 completed SAVE10 136000 8000 10000 134000
1 O00002 C0154 R0001 2026-02-09 04:58:00 completed NaN 106000 12000 10000 108000
2 O00003 C0161 R0006 2026-03-27 21:11:00 completed NaN 66000 18000 15000 69000
=== Order Items | shape=(849, 9) ===
order_item_id order_id line_number item_name item_category quantity unit_price line_discount line_amount
0 OI00001 O00001 1 Nasi Ayam Main 2 28000 2000 54000
1 OI00002 O00001 2 Nasi Ayam Main 2 28000 0 56000
2 OI00003 O00001 3 Mie Special Main 1 26000 0 26000
=== Payments | shape=(385, 7) ===
payment_id order_id payment_method payment_status payment_amount payment_timestamp attempt_number
0 P00001 O00001 e_wallet successful 134000 2026-03-28 11:21:00 1
1 P00002 O00002 bank_transfer successful 108000 2026-02-09 05:01:00 1
2 P00003 O00003 credit_card successful 69000 2026-03-27 21:14:00 1
=== Restaurants | shape=(23, 7) ===
restaurant_id restaurant_name cuisine_type restaurant_zone price_tier onboarding_date active_status
0 R0001 SwiftBite Partner 01 Western East Mid 2025-01-03 active
1 R0002 SwiftBite Partner 02 Fast Food Central Mid 2024-03-06 active
2 R0003 SwiftBite Partner 03 Cafe & Beverage Central Mid 2024-02-28 active

4. Intake Summary — Working Paper Sheet `01_Intake_Profiling`

Catat business meaning, expected grain, candidate key, row/record count, access status, dan key test result.

def intake_summary(df, dataset_name):
    return {
        "dataset": dataset_name,
        "rows": len(df),
        "columns": len(df.columns),
        "column_names": list(df.columns),
    }

if DATA_READY:
    intake_df = pd.DataFrame([
        intake_summary(customers_raw, "Customers"),
        intake_summary(orders_raw, "Orders"),
        intake_summary(order_items_raw, "Order Items"),
        intake_summary(payments_raw, "Payments"),
        intake_summary(restaurants_raw, "Restaurants"),
    ])
    display(intake_df)
dataset rows columns column_names
0 Customers 167 7 [customer_id, full_name, email, phone_number, ...
1 Orders 335 10 [order_id, customer_id, restaurant_id, order_t...
2 Order Items 849 9 [order_item_id, order_id, line_number, item_na...
3 Payments 385 7 [payment_id, order_id, payment_method, payment...
4 Restaurants 23 7 [restaurant_id, restaurant_name, cuisine_type,...

5. Grain & Candidate Key Check — RUN

Repeated `order_id` pada **Order Items** atau **Payments** belum tentu duplicate karena grain-nya berbeda dari Orders.

Gunakan hasil check sebagai **evidence awal**, bukan cleaning action otomatis.

def key_check(df, key_cols, dataset_name):
    if isinstance(key_cols, str):
        key_cols = [key_cols]

    missing_key_rows = int(df[key_cols].isna().any(axis=1).sum())
    unique_key_count = int(df[key_cols].drop_duplicates().shape[0])
    duplicate_key_rows = int(df.duplicated(subset=key_cols, keep=False).sum())

    return pd.DataFrame([{
        "dataset": dataset_name,
        "key_cols": ", ".join(key_cols),
        "rows": len(df),
        "unique_key_count": unique_key_count,
        "missing_key_rows": missing_key_rows,
        "duplicate_key_rows": duplicate_key_rows,
        "candidate_key_unique": unique_key_count == len(df) and missing_key_rows == 0,
    }])

if DATA_READY:
    key_results = pd.concat([
        key_check(customers_raw, "customer_id", "Customers"),
        key_check(orders_raw, "order_id", "Orders"),
        key_check(order_items_raw, "order_item_id", "Order Items"),
        key_check(order_items_raw, ["order_id", "line_number"], "Order Items — candidate business key"),
        key_check(payments_raw, "payment_id", "Payments"),
        key_check(restaurants_raw, "restaurant_id", "Restaurants"),
    ], ignore_index=True)

    display(key_results)
dataset key_cols rows unique_key_count missing_key_rows duplicate_key_rows candidate_key_unique
0 Customers customer_id 167 166 0 2 False
1 Orders order_id 335 335 0 0 True
2 Order Items order_item_id 849 849 0 0 True
3 Order Items — candidate business key order_id, line_number 849 849 0 0 True
4 Payments payment_id 385 385 0 0 True
5 Restaurants restaurant_id 23 23 0 0 True

Repeated Reference-Key Review

if DATA_READY:
    order_item_cardinality = (
        order_items_raw.groupby("order_id")
        .size()
        .rename("item_rows")
        .reset_index()
        .sort_values("item_rows", ascending=False)
    )

    payment_cardinality = (
        payments_raw.groupby("order_id")
        .size()
        .rename("payment_rows")
        .reset_index()
        .sort_values("payment_rows", ascending=False)
    )

    print("Order Items cardinality")
    display(order_item_cardinality.head(10))

    print("Payments cardinality")
    display(payment_cardinality.head(10))
Order Items cardinality
order_id item_rows
327 O00328 4
292 O00293 4
285 O00286 4
294 O00295 4
37 O00038 4
34 O00035 4
300 O00301 4
272 O00273 4
266 O00267 4
274 O00275 4
Payments cardinality
order_id payment_rows
322 O00323 2
323 O00324 2
324 O00325 2
334 O00335 2
325 O00326 2
326 O00327 2
327 O00328 2
291 O00292 2
46 O00047 2
282 O00283 2

6. Profiling

Profiling minimal:
- missing / placeholder,
- data type / format,
- category consistency,
- duplicate / key,
- suspicious numeric / formula,
- potential outlier,
- cross-table concern.

Helper berikut menghasilkan evidence awal, bukan automatic judgment.

def basic_profile(df, dataset_name):
    return pd.DataFrame([{
        "dataset": dataset_name,
        "rows": len(df),
        "columns": len(df.columns),
        "rows_in_exact_duplicate_groups": int(df.duplicated(keep=False).sum()),
        "exact_duplicate_extras": int(df.duplicated(keep="first").sum()),
        "null_cells": int(df.isna().sum().sum()),
    }])

def column_profile(df):
    rows = []
    for col in df.columns:
        s = df[col]
        rows.append({
            "column": col,
            "dtype": str(s.dtype),
            "null_count": int(s.isna().sum()),
            "null_pct": round(float(s.isna().mean()), 4),
            "unique_count": int(s.nunique(dropna=True)),
            "sample_values": list(s.dropna().astype(str).unique()[:5]),
        })
    return pd.DataFrame(rows)

if DATA_READY:
    overall_profile = pd.concat(
        [basic_profile(df, name) for name, df in raw_tables.items()],
        ignore_index=True
    )
    display(overall_profile)
dataset rows columns rows_in_exact_duplicate_groups exact_duplicate_extras null_cells
0 Customers 167 7 2 1 0
1 Orders 335 10 0 0 171
2 Order Items 849 9 0 0 0
3 Payments 385 7 0 0 0
4 Restaurants 23 7 0 0 0

Column-Level Profiling

# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
    display(column_profile(customers_raw))
column dtype null_count null_pct unique_count sample_values
0 customer_id str 0 0.0 166 [C0001, C0002, C0003, C0004, C0005]
1 full_name str 0 0.0 124 [Eka Maulana, Nadia Santoso, Hana Sari, Maya S...
2 email str 0 0.0 166 [ekamaulana1@example.com, nadiasantoso2@exampl...
3 phone_number str 0 0.0 166 [083549565705, 089333361058, 081792756270, 086...
4 signup_date str 0 0.0 112 [2025-12-24, 2025-07-05, 2025-06-22, 2025-10-2...
5 customer_zone str 0 0.0 5 [East, Central, North, South, West]
6 customer_segment str 0 0.0 4 [Regular, New, LOYAL , Loyal]
# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
    display(column_profile(orders_raw))
column dtype null_count null_pct unique_count sample_values
0 order_id str 0 0.0000 335 [O00001, O00002, O00003, O00004, O00005]
1 customer_id str 0 0.0000 166 [C0147, C0154, C0161, C0002, C0009]
2 restaurant_id str 0 0.0000 23 [R0019, R0001, R0006, R0011, R0016]
3 order_timestamp str 0 0.0000 334 [2026-03-28 11:18:00, 2026-02-09 04:58:00, 202...
4 order_status str 0 0.0000 3 [completed, COMPLETED , cancelled]
5 promo_code str 171 0.5104 3 [SAVE10, NEWUSER, HEMAT15]
6 subtotal_amount int64 0 0.0000 160 [136000, 106000, 66000, 93000, 110000]
7 delivery_fee int64 0 0.0000 5 [8000, 12000, 18000, 10000, 15000]
8 discount_amount int64 0 0.0000 4 [10000, 15000, 0, 5000]
9 total_amount int64 0 0.0000 168 [134000, 108000, 69000, 105000, 113000]
# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
    display(column_profile(order_items_raw))
column dtype null_count null_pct unique_count sample_values
0 order_item_id str 0 0.0 849 [OI00001, OI00002, OI00003, OI00004, OI00005]
1 order_id str 0 0.0 335 [O00001, O00002, O00003, O00004, O00005]
2 line_number int64 0 0.0 4 [1, 2, 3, 4]
3 item_name str 0 0.0 14 [Nasi Ayam, Mie Special, Juice, Toast, Dimsum]
4 item_category str 0 0.0 4 [Main, Drink, Snack, Dessert]
5 quantity str 0 0.0 4 [2, 1, 3, 1 pcs]
6 unit_price int64 0 0.0 11 [28000, 26000, 20000, 16000, 22000]
7 line_discount int64 0 0.0 3 [2000, 0, 5000]
8 line_amount int64 0 0.0 66 [54000, 56000, 26000, 58000, 48000]
# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
    display(column_profile(payments_raw))
column dtype null_count null_pct unique_count sample_values
0 payment_id str 0 0.0 385 [P00001, P00002, P00003, P00004, P00005]
1 order_id str 0 0.0 335 [O00001, O00002, O00003, O00004, O00005]
2 payment_method str 0 0.0 6 [e_wallet, bank_transfer, credit_card, cash, B...
3 payment_status str 0 0.0 3 [successful, failed, refunded]
4 payment_amount int64 0 0.0 168 [134000, 108000, 69000, 105000, 113000]
5 payment_timestamp str 0 0.0 383 [2026-03-28 11:21:00, 2026-02-09 05:01:00, 202...
6 attempt_number int64 0 0.0 2 [1, 2]
# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
    display(column_profile(restaurants_raw))
column dtype null_count null_pct unique_count sample_values
0 restaurant_id str 0 0.0 23 [R0001, R0002, R0003, R0004, R0005]
1 restaurant_name str 0 0.0 23 [SwiftBite Partner 01, SwiftBite Partner 02, S...
2 cuisine_type str 0 0.0 6 [Western, Fast Food, Cafe & Beverage, Dessert,...
3 restaurant_zone str 0 0.0 5 [East, Central, West, South, North]
4 price_tier str 0 0.0 5 [Mid, PREMIUM , Budget, Premium, MID ]
5 onboarding_date str 0 0.0 23 [2025-01-03, 2024-03-06, 2024-02-28, 2025-02-1...
6 active_status str 0 0.0 2 [active, inactive]

Category Review Helper

def category_review(df, column, top_n=30):
    out = (
        df[column]
        .value_counts(dropna=False)
        .rename_axis(column)
        .reset_index(name="count")
    )
    out["pct"] = (out["count"] / len(df)).round(4)
    return out.head(top_n)

# Contoh:
# display(category_review(customers_raw, "customer_segment"))
# display(category_review(orders_raw, "order_status"))
# display(category_review(payments_raw, "payment_method"))
# display(category_review(restaurants_raw, "price_tier"))
display(category_review(customers_raw, "customer_segment"))
customer_segment count pct
0 Regular 63 0.3772
1 New 63 0.3772
2 Loyal 39 0.2335
3 LOYAL 2 0.0120
display(category_review(orders_raw, "order_status"))
order_status count pct
0 completed 293 0.8746
1 cancelled 40 0.1194
2 COMPLETED 2 0.0060
display(category_review(payments_raw, "payment_method"))
payment_method count pct
0 credit_card 113 0.2935
1 cash 94 0.2442
2 e_wallet 89 0.2312
3 bank_transfer 87 0.2260
4 BANK TRANSFER 1 0.0026
5 E WALLET 1 0.0026
display(category_review(restaurants_raw, "price_tier"))
price_tier count pct
0 Mid 10 0.4348
1 Premium 7 0.3043
2 Budget 4 0.1739
3 PREMIUM 1 0.0435
4 MID 1 0.0435

Numeric Review Helper

def numeric_profile(df):
    numeric_cols = df.select_dtypes(include="number").columns.tolist()
    if not numeric_cols:
        return pd.DataFrame()
    return df[numeric_cols].describe().T

# Contoh:
# display(numeric_profile(orders_raw))
# display(numeric_profile(order_items_raw))
display(numeric_profile(orders_raw))
count mean std min 25% 50% 75% max
subtotal_amount 335.0 120020.895522 59977.611934 7000.0 81000.0 108000.0 153500.0 648000.0
delivery_fee 335.0 12644.776119 3653.694856 8000.0 10000.0 12000.0 15000.0 18000.0
discount_amount 335.0 6641.791045 5955.678347 0.0 0.0 5000.0 10000.0 15000.0
total_amount 335.0 126023.880597 60661.370107 9000.0 86500.0 115000.0 160500.0 658000.0
display(numeric_profile(order_items_raw))
count mean std min 25% 50% 75% max
line_number 849.0 1.889282 0.868468 1.0 1.0 2.0 2.0 4.0
unit_price 849.0 24757.361602 22966.290322 9000.0 18000.0 22000.0 28000.0 648000.0
line_discount 849.0 1338.044759 1937.094801 0.0 0.0 0.0 2000.0 5000.0
line_amount 849.0 47348.645465 34043.756544 4000.0 24000.0 42000.0 64000.0 648000.0

7. Candidate Findings Register

Gunakan judgment:
- `FIX`
- `STANDARDIZE`
- `KEEP`
- `INVESTIGATE`

Jangan isi judgment sebelum verification evidence cukup.

FINDING_COLUMNS = [
    "finding_id",
    "dataset",
    "field_key",
    "finding_evidence",
    "why_it_matters",
    "verification_method",
    "verification_evidence_result",
    "judgment",
    "reason_for_judgment",
    "cleaning_rule_id",
]

findings = pd.DataFrame(columns=FINDING_COLUMNS)
display(findings)
finding_id dataset field_key finding_evidence why_it_matters verification_method verification_evidence_result judgment reason_for_judgment cleaning_rule_id

Template menambahkan finding

```python
new_finding = {
"finding_id": "F-01",
"dataset": "...",
"field_key": "...",
"finding_evidence": "...",
"why_it_matters": "...",
"verification_method": "...",
"verification_evidence_result": "...",
"judgment": "FIX / STANDARDIZE / KEEP / INVESTIGATE",
"reason_for_judgment": "...",
"cleaning_rule_id": "CR-01 atau kosong",
}
findings.loc[len(findings)] = new_finding
```

daftar_temuan = [
    {
        "finding_id": "F-01",
        "dataset": "customers_raw",
        "field_key": "signup_date",
        "finding_evidence": "Tipe data masih string (str), bukan datetime",
        "why_it_matters": "Menghambat analisis berbasis waktu dan filter tanggal",
        "verification_method": "Column profiling",
        "verification_evidence_result": "Hasil profiling menunjukkan dtype tercatat str",
        "judgment": "FIX",
        "reason_for_judgment": "Perlu konversi tipe data dari string ke datetime",
        "cleaning_rule_id": "CR-01"
    },
    {
        "finding_id": "F-02",
        "dataset": "customers_raw",
        "field_key": "customer_segment",
        "finding_evidence": "Inkonsistensi penulisan kategori (perbedaan huruf besar/kecil atau spasi)",
        "why_it_matters": "Membuat pengelompokan (grouping) kategori menjadi tidak akurat",
        "verification_method": "Category review",
        "verification_evidence_result": "Ditemukan variasi penulisan nilai kategori",
        "judgment": "STANDARDIZE",
        "reason_for_judgment": "Perlu diseragamkan format dan kapitalisasi penulisannya",
        "cleaning_rule_id": "CR-02"
    },
    {
        "finding_id": "F-03",
        "dataset": "orders_raw",
        "field_key": "order_timestamp",
        "finding_evidence": "Tipe data masih string (str), bukan datetime",
        "why_it_matters": "Menghambat analisis tren transaksi harian/bulanan",
        "verification_method": "Column profiling",
        "verification_evidence_result": "Hasil profiling menunjukkan dtype tercatat str",
        "judgment": "FIX",
        "reason_for_judgment": "Perlu konversi tipe data dari string ke datetime",
        "cleaning_rule_id": "CR-03"
    },
    {
        "finding_id": "F-04",
        "dataset": "orders_raw",
        "field_key": "order_status",
        "finding_evidence": "Inkonsistensi penulisan status (misal: completed vs COMPLETED)",
        "why_it_matters": "Menyebabkan data terhitung ganda atau terpisah saat di-grouping",
        "verification_method": "Category review",
        "verification_evidence_result": "Ditemukan variasi kapitalisasi pada status pesanan",
        "judgment": "STANDARDIZE",
        "reason_for_judgment": "Perlu diseragamkan format kapitalisasinya",
        "cleaning_rule_id": "CR-04"
    },
    {
        "finding_id": "F-05",
        "dataset": "orders_raw",
        "field_key": "promo_code",
        "finding_evidence": "Tingkat missing values cukup tinggi sekitar 51%",
        "why_it_matters": "Perlu dipastikan apakah wajar atau ada kendala pencatatan sistem",
        "verification_method": "Column profiling",
        "verification_evidence_result": "Null count tinggi pada kolom promo_code",
        "judgment": "KEEP",
        "reason_for_judgment": "Secara bisnis wajar jika pelanggan tidak menggunakan promo",
        "cleaning_rule_id": ""
    },
    {
        "finding_id": "F-06",
        "dataset": "payments_raw",
        "field_key": "payment_timestamp",
        "finding_evidence": "Tipe data masih string (str), bukan datetime",
        "why_it_matters": "Menghambat analisis waktu pembayaran",
        "verification_method": "Column profiling",
        "verification_evidence_result": "Hasil profiling menunjukkan dtype tercatat str",
        "judgment": "FIX",
        "reason_for_judgment": "Perlu konversi tipe data dari string ke datetime",
        "cleaning_rule_id": "CR-05"
    },
    {
        "finding_id": "F-07",
        "dataset": "payments_raw",
        "field_key": "payment_method",
        "finding_evidence": "Inkonsistensi penulisan metode pembayaran (variasi huruf/spasi)",
        "why_it_matters": "Membuat rekapitulasi metode pembayaran tidak akurat",
        "verification_method": "Category review",
        "verification_evidence_result": "Ditemukan variasi penulisan pada metode pembayaran",
        "judgment": "STANDARDIZE",
        "reason_for_judgment": "Perlu diseragamkan format penulisannya",
        "cleaning_rule_id": "CR-06"
    },
    {
        "finding_id": "F-08",
        "dataset": "restaurants_raw",
        "field_key": "onboarding_date",
        "finding_evidence": "Tipe data masih string (str), bukan datetime",
        "why_it_matters": "Menghambat analisis masa aktif restoran",
        "verification_method": "Column profiling",
        "verification_evidence_result": "Hasil profiling menunjukkan dtype tercatat str",
        "judgment": "FIX",
        "reason_for_judgment": "Perlu konversi tipe data dari string ke datetime",
        "cleaning_rule_id": "CR-07"
    },
    {
        "finding_id": "F-09",
        "dataset": "restaurants_raw",
        "field_key": "price_tier",
        "finding_evidence": "Inkonsistensi penulisan kategori (misal: Mid vs MID vs PREMIUM)",
        "why_it_matters": "Membuat pengelompokan tier harga menjadi terpecah",
        "verification_method": "Category review",
        "verification_evidence_result": "Ditemukan variasi kapitalisasi pada price_tier",
        "judgment": "STANDARDIZE",
        "reason_for_judgment": "Perlu diseragamkan format kapitalisasinya",
        "cleaning_rule_id": "CR-08"
    }
]

# 2. Masukkan ke tabel findings secara otomatis satu per satu lewat perulangan
for item in daftar_temuan:
    findings.loc[len(findings)] = item

# 3. Tampilkan hasilnya
display(findings)
finding_id dataset field_key finding_evidence why_it_matters verification_method verification_evidence_result judgment reason_for_judgment cleaning_rule_id
0 F-01 customers_raw signup_date Tipe data masih string (str), bukan datetime Menghambat analisis berbasis waktu dan filter ... Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-01
1 F-02 customers_raw customer_segment Inkonsistensi penulisan kategori (perbedaan hu... Membuat pengelompokan (grouping) kategori menj... Category review Ditemukan variasi penulisan nilai kategori STANDARDIZE Perlu diseragamkan format dan kapitalisasi pen... CR-02
2 F-03 orders_raw order_timestamp Tipe data masih string (str), bukan datetime Menghambat analisis tren transaksi harian/bulanan Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-03
3 F-04 orders_raw order_status Inkonsistensi penulisan status (misal: complet... Menyebabkan data terhitung ganda atau terpisah... Category review Ditemukan variasi kapitalisasi pada status pes... STANDARDIZE Perlu diseragamkan format kapitalisasinya CR-04
4 F-05 orders_raw promo_code Tingkat missing values cukup tinggi sekitar 51% Perlu dipastikan apakah wajar atau ada kendala... Column profiling Null count tinggi pada kolom promo_code KEEP Secara bisnis wajar jika pelanggan tidak mengg...
5 F-06 payments_raw payment_timestamp Tipe data masih string (str), bukan datetime Menghambat analisis waktu pembayaran Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-05
6 F-07 payments_raw payment_method Inkonsistensi penulisan metode pembayaran (var... Membuat rekapitulasi metode pembayaran tidak a... Category review Ditemukan variasi penulisan pada metode pembay... STANDARDIZE Perlu diseragamkan format penulisannya CR-06
7 F-08 restaurants_raw onboarding_date Tipe data masih string (str), bukan datetime Menghambat analisis masa aktif restoran Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-07
8 F-09 restaurants_raw price_tier Inkonsistensi penulisan kategori (misal: Mid v... Membuat pengelompokan tier harga menjadi terpecah Category review Ditemukan variasi kapitalisasi pada price_tier STANDARDIZE Perlu diseragamkan format kapitalisasinya CR-08
finding_reconciliation = {
    "finding_id": "F-10",
    "dataset": "orders_raw",
    "field_key": "subtotal_amount",
    "finding_evidence": "Ditemukan 3 baris data di mana subtotal_amount tidak sesuai dengan jumlah total line_amount item pesanan",
    "why_it_matters": "Menyebabkan inkonsistensi finansial antara rekap pesanan utama dan detail item",
    "verification_method": "Reconciliation check",
    "verification_evidence_result": "Raw reconciliation mismatch count bernilai 3",
    "judgment": "FIX",
    "reason_for_judgment": "Perlu dilakukan penyesuaian atau koreksi nilai subtotal agar sinkron dengan detail item",
    "cleaning_rule_id": "CR-09"
}

# Masukkan ke dalam DataFrame findings
findings.loc[len(findings)] = finding_reconciliation

# Tampilkan kembali tabel findings yang sudah diperbarui
display(findings)
finding_id dataset field_key finding_evidence why_it_matters verification_method verification_evidence_result judgment reason_for_judgment cleaning_rule_id
0 F-01 customers_raw signup_date Tipe data masih string (str), bukan datetime Menghambat analisis berbasis waktu dan filter ... Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-01
1 F-02 customers_raw customer_segment Inkonsistensi penulisan kategori (perbedaan hu... Membuat pengelompokan (grouping) kategori menj... Category review Ditemukan variasi penulisan nilai kategori STANDARDIZE Perlu diseragamkan format dan kapitalisasi pen... CR-02
2 F-03 orders_raw order_timestamp Tipe data masih string (str), bukan datetime Menghambat analisis tren transaksi harian/bulanan Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-03
3 F-04 orders_raw order_status Inkonsistensi penulisan status (misal: complet... Menyebabkan data terhitung ganda atau terpisah... Category review Ditemukan variasi kapitalisasi pada status pes... STANDARDIZE Perlu diseragamkan format kapitalisasinya CR-04
4 F-05 orders_raw promo_code Tingkat missing values cukup tinggi sekitar 51% Perlu dipastikan apakah wajar atau ada kendala... Column profiling Null count tinggi pada kolom promo_code KEEP Secara bisnis wajar jika pelanggan tidak mengg...
5 F-06 payments_raw payment_timestamp Tipe data masih string (str), bukan datetime Menghambat analisis waktu pembayaran Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-05
6 F-07 payments_raw payment_method Inkonsistensi penulisan metode pembayaran (var... Membuat rekapitulasi metode pembayaran tidak a... Category review Ditemukan variasi penulisan pada metode pembay... STANDARDIZE Perlu diseragamkan format penulisannya CR-06
7 F-08 restaurants_raw onboarding_date Tipe data masih string (str), bukan datetime Menghambat analisis masa aktif restoran Column profiling Hasil profiling menunjukkan dtype tercatat str FIX Perlu konversi tipe data dari string ke datetime CR-07
8 F-09 restaurants_raw price_tier Inkonsistensi penulisan kategori (misal: Mid v... Membuat pengelompokan tier harga menjadi terpecah Category review Ditemukan variasi kapitalisasi pada price_tier STANDARDIZE Perlu diseragamkan format kapitalisasinya CR-08
9 F-10 orders_raw subtotal_amount Ditemukan 3 baris data di mana subtotal_amount... Menyebabkan inkonsistensi finansial antara rek... Reconciliation check Raw reconciliation mismatch count bernilai 3 FIX Perlu dilakukan penyesuaian atau koreksi nilai... CR-09

8. Cross-Table Verification

def fk_check(child_df, parent_df, child_key, parent_key, relationship_name):
    child_keys = child_df[[child_key]].drop_duplicates()
    parent_keys = parent_df[[parent_key]].drop_duplicates()

    merged = child_keys.merge(
        parent_keys,
        left_on=child_key,
        right_on=parent_key,
        how="left",
        indicator=True,
    )

    unmatched = merged.loc[merged["_merge"] == "left_only", child_key]

    return {
        "relationship": relationship_name,
        "child_distinct_keys": len(child_keys),
        "unmatched_count": len(unmatched),
        "unmatched_sample": unmatched.head(10).tolist(),
    }

if DATA_READY:
    relationship_checks = pd.DataFrame([
        fk_check(orders_raw, customers_raw, "customer_id", "customer_id", "Orders → Customers"),
        fk_check(orders_raw, restaurants_raw, "restaurant_id", "restaurant_id", "Orders → Restaurants"),
        fk_check(order_items_raw, orders_raw, "order_id", "order_id", "Order Items → Orders"),
        fk_check(payments_raw, orders_raw, "order_id", "order_id", "Payments → Orders"),
    ])
    display(relationship_checks)
relationship child_distinct_keys unmatched_count unmatched_sample
0 Orders → Customers 166 0 []
1 Orders → Restaurants 23 0 []
2 Order Items → Orders 335 0 []
3 Payments → Orders 335 0 []

9. Business-Rule Verification

### Order Items
`line_amount = quantity × unit_price − line_discount`

### Orders
`total_amount = subtotal_amount + delivery_fee − discount_amount`

Cell ini hanya membuat verification view. Tidak melakukan cleaning.

if DATA_READY:
    item_verify = order_items_raw.copy()
    item_verify["expected_line_amount"] = (
        pd.to_numeric(item_verify["quantity"], errors="coerce")
        * pd.to_numeric(item_verify["unit_price"], errors="coerce")
        - pd.to_numeric(item_verify["line_discount"], errors="coerce")
    )
    item_verify["line_formula_match"] = (
        pd.to_numeric(item_verify["line_amount"], errors="coerce")
        == item_verify["expected_line_amount"]
    )

    display(
        item_verify[
            ["order_item_id", "order_id", "quantity", "unit_price",
             "line_discount", "line_amount", "expected_line_amount", "line_formula_match"]
        ].head()
    )

    order_verify = orders_raw.copy()
    order_verify["expected_total_amount"] = (
        pd.to_numeric(order_verify["subtotal_amount"], errors="coerce")
        + pd.to_numeric(order_verify["delivery_fee"], errors="coerce")
        - pd.to_numeric(order_verify["discount_amount"], errors="coerce")
    )
    order_verify["order_formula_match"] = (
        pd.to_numeric(order_verify["total_amount"], errors="coerce")
        == order_verify["expected_total_amount"]
    )

    display(
        order_verify[
            ["order_id", "subtotal_amount", "delivery_fee", "discount_amount",
             "total_amount", "expected_total_amount", "order_formula_match"]
        ].head()
    )
order_item_id order_id quantity unit_price line_discount line_amount expected_line_amount line_formula_match
0 OI00001 O00001 2 28000 2000 54000 54000.0 True
1 OI00002 O00001 2 28000 0 56000 56000.0 True
2 OI00003 O00001 1 26000 0 26000 26000.0 True
3 OI00004 O00002 3 20000 2000 58000 58000.0 True
4 OI00005 O00002 3 16000 0 48000 48000.0 True
order_id subtotal_amount delivery_fee discount_amount total_amount expected_total_amount order_formula_match
0 O00001 136000 8000 10000 134000 134000 True
1 O00002 106000 12000 10000 108000 108000 True
2 O00003 66000 18000 15000 69000 69000 True
3 O00004 93000 12000 0 105000 105000 True
4 O00005 110000 18000 15000 113000 113000 True

Orders ↔ Order Items Reconciliation

Lakukan reconciliation setelah memahami item-level validity. Jangan langsung mengoreksi Orders jika item detail sendiri belum diverifikasi.

if DATA_READY:
    item_amount_numeric = pd.to_numeric(order_items_raw["line_amount"], errors="coerce")

    item_agg_raw = (
        order_items_raw.assign(_line_amount_numeric=item_amount_numeric)
        .groupby("order_id", as_index=False)["_line_amount_numeric"]
        .sum(min_count=1)
        .rename(columns={"_line_amount_numeric": "sum_line_amount"})
    )

    order_item_recon_raw = orders_raw[
        ["order_id", "subtotal_amount"]
    ].merge(item_agg_raw, on="order_id", how="left")

    order_item_recon_raw["subtotal_numeric"] = pd.to_numeric(
        order_item_recon_raw["subtotal_amount"], errors="coerce"
    )
    order_item_recon_raw["reconcile_match"] = (
        order_item_recon_raw["subtotal_numeric"]
        == order_item_recon_raw["sum_line_amount"]
    )

    display(order_item_recon_raw.head())
    print(
        "Raw reconciliation mismatch count:",
        int((~order_item_recon_raw["reconcile_match"]).sum())
    )
order_id subtotal_amount sum_line_amount subtotal_numeric reconcile_match
0 O00001 136000 136000 136000 True
1 O00002 106000 106000 106000 True
2 O00003 66000 66000 66000 True
3 O00004 93000 93000 93000 True
4 O00005 110000 110000 110000 True
Raw reconciliation mismatch count: 3

10. Cleaning Rulebook

Rulebook dibuat setelah verification dan sebelum coding.

Gunakan hanya `FIX` atau `STANDARDIZE` sebagai cleaning decision. Finding `KEEP` tidak boleh diubah hanya agar data terlihat lebih rapi.

RULE_COLUMNS = [
    "rule_id",
    "source_finding_id",
    "dataset",
    "trigger_condition",
    "decision",
    "cleaning_action",
    "reason_evidence",
    "implementation_note",
    "validation_criteria",
    "rule_status",
]

cleaning_rules = pd.DataFrame(columns=RULE_COLUMNS)
display(cleaning_rules)
rule_id source_finding_id dataset trigger_condition decision cleaning_action reason_evidence implementation_note validation_criteria rule_status
daftar_rules = [
    {
        "rule_id": "CR-01",
        "source_finding_id": "F-01",
        "dataset": "customers_raw",
        "trigger_condition": "Tipe data signup_date adalah string (str)",
        "decision": "FIX",
        "cleaning_action": "Konversi kolom signup_date ke tipe datetime",
        "reason_evidence": "Tipe data str membatasi analisis rentang tanggal",
        "implementation_note": "Gunakan pd.to_datetime()",
        "validation_criteria": "dtype berubah menjadi datetime64[ns]",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-02",
        "source_finding_id": "F-02",
        "dataset": "customers_raw",
        "trigger_condition": "Inkonsistensi kapitalisasi/spasi pada customer_segment",
        "decision": "STANDARDIZE",
        "cleaning_action": "Standarkan format penulisan kategori (Title Case / Upper Case)",
        "reason_evidence": "Menghindari duplikasi kategori saat agregasi",
        "implementation_note": "Gunakan .str.strip() dan .str.title() atau mapping",
        "validation_criteria": "Kategori unik menjadi konsisten sesuai nilai standar",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-03",
        "source_finding_id": "F-03",
        "dataset": "orders_raw",
        "trigger_condition": "Tipe data order_timestamp adalah string (str)",
        "decision": "FIX",
        "cleaning_action": "Konversi kolom order_timestamp ke tipe datetime",
        "reason_evidence": "Diperlukan untuk analisis tren waktu pesanan",
        "implementation_note": "Gunakan pd.to_datetime()",
        "validation_criteria": "dtype berubah menjadi datetime64[ns]",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-04",
        "source_finding_id": "F-04",
        "dataset": "orders_raw",
        "trigger_condition": "Inkonsistensi penulisan nilai pada order_status",
        "decision": "STANDARDIZE",
        "cleaning_action": "Standarkan kapitalisasi status pesanan",
        "reason_evidence": "Mencegah terpecahnya status pesanan saat pengelompokan",
        "implementation_note": "Gunakan .str.upper() atau .str.title()",
        "validation_criteria": "Nilai order_status seragam (misal: COMPLETED, CANCELLED)",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-05",
        "source_finding_id": "F-06",
        "dataset": "payments_raw",
        "trigger_condition": "Tipe data payment_timestamp adalah string (str)",
        "decision": "FIX",
        "cleaning_action": "Konversi kolom payment_timestamp ke tipe datetime",
        "reason_evidence": "Memudahkan analisis durasi pembayaran",
        "implementation_note": "Gunakan pd.to_datetime()",
        "validation_criteria": "dtype berubah menjadi datetime64[ns]",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-06",
        "source_finding_id": "F-07",
        "dataset": "payments_raw",
        "trigger_condition": "Inkonsistensi penulisan payment_method",
        "decision": "STANDARDIZE",
        "cleaning_action": "Standarkan penulisan metode pembayaran",
        "reason_evidence": "Memastikan rekap pembayaran akurat",
        "implementation_note": "Gunakan mapping dictionary atau .str.title()",
        "validation_criteria": "Nilai metode pembayaran seragam",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-07",
        "source_finding_id": "F-08",
        "dataset": "restaurants_raw",
        "trigger_condition": "Tipe data onboarding_date adalah string (str)",
        "decision": "FIX",
        "cleaning_action": "Konversi kolom onboarding_date ke tipe datetime",
        "reason_evidence": "Memungkinkan perhitungan tenure restoran",
        "implementation_note": "Gunakan pd.to_datetime()",
        "validation_criteria": "dtype berubah menjadi datetime64[ns]",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-08",
        "source_finding_id": "F-09",
        "dataset": "restaurants_raw",
        "trigger_condition": "Inkonsistensi penulisan price_tier",
        "decision": "STANDARDIZE",
        "cleaning_action": "Standarkan penulisan kategori tier harga",
        "reason_evidence": "Memastikan segmentasi harga restoran tepat",
        "implementation_note": "Gunakan .str.upper() atau mapping",
        "validation_criteria": "Nilai price_tier seragam",
        "rule_status": "Approved"
    },
    {
        "rule_id": "CR-09",
        "source_finding_id": "F-10",
        "dataset": "orders_raw",
        "trigger_condition": "Ketidakcocokan subtotal_amount dengan total line_amount item pesanan",
        "decision": "FIX",
        "cleaning_action": "Koreksi nilai subtotal_amount agar sesuai penjumlahan sum_line_amount",
        "reason_evidence": "Menyelaraskan nilai finansial antara tabel utama dan detail",
        "implementation_note": "Update subtotal_amount berdasarkan hasil kalkulasi item_agg",
        "validation_criteria": "Raw reconciliation mismatch count menjadi 0",
        "rule_status": "Approved"
    }
]

# Masukkan ke tabel cleaning_rules
for rule in daftar_rules:
    cleaning_rules.loc[len(cleaning_rules)] = rule

 # Tampilkan hasilnya
display(cleaning_rules)
rule_id source_finding_id dataset trigger_condition decision cleaning_action reason_evidence implementation_note validation_criteria rule_status
0 CR-01 F-01 customers_raw Tipe data signup_date adalah string (str) FIX Konversi kolom signup_date ke tipe datetime Tipe data str membatasi analisis rentang tanggal Gunakan pd.to_datetime() dtype berubah menjadi datetime64[ns] Approved
1 CR-02 F-02 customers_raw Inkonsistensi kapitalisasi/spasi pada customer... STANDARDIZE Standarkan format penulisan kategori (Title Ca... Menghindari duplikasi kategori saat agregasi Gunakan .str.strip() dan .str.title() atau map... Kategori unik menjadi konsisten sesuai nilai s... Approved
2 CR-03 F-03 orders_raw Tipe data order_timestamp adalah string (str) FIX Konversi kolom order_timestamp ke tipe datetime Diperlukan untuk analisis tren waktu pesanan Gunakan pd.to_datetime() dtype berubah menjadi datetime64[ns] Approved
3 CR-04 F-04 orders_raw Inkonsistensi penulisan nilai pada order_status STANDARDIZE Standarkan kapitalisasi status pesanan Mencegah terpecahnya status pesanan saat penge... Gunakan .str.upper() atau .str.title() Nilai order_status seragam (misal: COMPLETED, ... Approved
4 CR-05 F-06 payments_raw Tipe data payment_timestamp adalah string (str) FIX Konversi kolom payment_timestamp ke tipe datetime Memudahkan analisis durasi pembayaran Gunakan pd.to_datetime() dtype berubah menjadi datetime64[ns] Approved
5 CR-06 F-07 payments_raw Inkonsistensi penulisan payment_method STANDARDIZE Standarkan penulisan metode pembayaran Memastikan rekap pembayaran akurat Gunakan mapping dictionary atau .str.title() Nilai metode pembayaran seragam Approved
6 CR-07 F-08 restaurants_raw Tipe data onboarding_date adalah string (str) FIX Konversi kolom onboarding_date ke tipe datetime Memungkinkan perhitungan tenure restoran Gunakan pd.to_datetime() dtype berubah menjadi datetime64[ns] Approved
7 CR-08 F-09 restaurants_raw Inkonsistensi penulisan price_tier STANDARDIZE Standarkan penulisan kategori tier harga Memastikan segmentasi harga restoran tepat Gunakan .str.upper() atau mapping Nilai price_tier seragam Approved
8 CR-09 F-10 orders_raw Ketidakcocokan subtotal_amount dengan total li... FIX Koreksi nilai subtotal_amount agar sesuai penj... Menyelaraskan nilai finansial antara tabel uta... Update subtotal_amount berdasarkan hasil kalku... Raw reconciliation mismatch count menjadi 0 Approved

Rulebook Template

```python
new_rule = {
"rule_id": "CR-01",
"source_finding_id": "F-01",
"dataset": "...",
"trigger_condition": "...",
"decision": "FIX / STANDARDIZE",
"cleaning_action": "...",
"reason_evidence": "...",
"implementation_note": "...",
"validation_criteria": "...",
"rule_status": "Approved",
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
```

11. Create Working Copies

if DATA_READY:
    customers_clean = customers_raw.copy(deep=True)
    orders_clean = orders_raw.copy(deep=True)
    order_items_clean = order_items_raw.copy(deep=True)
    payments_clean = payments_raw.copy(deep=True)
    restaurants_clean = restaurants_raw.copy(deep=True)

    print("Working copies created.")
Working copies created.

12. Implement Cleaning Rules — EDIT + RUN

**Edit hanya cell Cleaning Code di bawah ini.**

Aturan:
- implementasikan hanya rule yang sudah kamu setujui di Working Paper;
- gunakan `customers_clean`, `orders_clean`, `order_items_clean`, `payments_clean`, atau `restaurants_clean`;
- jangan pernah overwrite object dengan suffix `_raw`;
- beri komentar `CR-xx` untuk setiap rule;
- setelah implementasi, catat before/after pada Cleaning Log.

Starter sengaja **tidak memberikan affected record ID atau jawaban** agar evidence tetap berasal dari variant kamu.

# ============================================================
# EDIT AREA 2 — APPROVED CLEANING CODE
# ============================================================
# Tambahkan code per Rule ID di bawah ini.
#
# Contoh struktur komentar:
# # CR-01 – Konversi tipe data signup_date
# # Source Finding: F-01
# # Evidence basis: dtype masih object/str
# # Validation criteria: dtype menjadi datetime64[ns]
customers_clean["signup_date"] = pd.to_datetime(customers_clean["signup_date"], errors="coerce")

# # CR-02 – Standarisasi format customer_segment
# # Source Finding: F-02
# # Evidence basis: Inkonsistensi kapitalisasi & spasi
# # Validation criteria: Format teks seragam
customers_clean["customer_segment"] = customers_clean["customer_segment"].str.strip().str.title()

# # CR-03 – Konversi tipe data order_timestamp
# # Source Finding: F-03
# # Evidence basis: dtype masih object/str
# # Validation criteria: dtype menjadi datetime64[ns]
orders_clean["order_timestamp"] = pd.to_datetime(orders_clean["order_timestamp"], errors="coerce")

# # CR-04 – Standarisasi format order_status
# # Source Finding: F-04
# # Evidence basis: Inkonsistensi penulisan status pesanan
# # Validation criteria: Status seragam
orders_clean["order_status"] = orders_clean["order_status"].str.strip().str.upper()


# # CR-05 – Konversi tipe data payment_timestamp
# # Source Finding: F-06
# # Evidence basis: dtype masih object/str
# # Validation criteria: dtype menjadi datetime64[ns]
payments_clean["payment_timestamp"] = pd.to_datetime(payments_clean["payment_timestamp"], errors="coerce")

# # CR-06 – Standarisasi payment_method
# # Source Finding: F-07
# # Evidence basis: Inkonsistensi penulisan metode pembayaran
# # Validation criteria: Metode pembayaran seragam
payments_clean["payment_method"] = payments_clean["payment_method"].str.strip().str.title()


# # CR-07 – Konversi tipe data onboarding_date
# # Source Finding: F-08
# # Evidence basis: dtype masih object/str
# # Validation criteria: dtype menjadi datetime64[ns]
restaurants_clean["onboarding_date"] = pd.to_datetime(restaurants_clean["onboarding_date"], errors="coerce")

# # CR-08 – Standarisasi price_tier restoran
# # Source Finding: F-09
# # Evidence basis: Inkonsistensi kategori tier harga
# # Validation criteria: Price tier seragam
restaurants_clean["price_tier"] = restaurants_clean["price_tier"].str.strip().str.upper()


# # CR-09 – Koreksi mismatch subtotal_amount dengan line_amount item pesanan
# # Source Finding: F-10
# # Evidence basis: Terdapat 3 baris mismatch pada rekonsiliasi
# # Validation criteria: Raw reconciliation mismatch count menjadi 0
item_agg = (
    order_items_clean.assign(line_amount_numeric=pd.to_numeric(order_items_clean["line_amount"], errors="coerce"))
    .groupby("order_id", as_index=False)["line_amount_numeric"]
    .sum(min_count=1)
    .rename(columns={"line_amount_numeric": "sum_line_amount"})
)

orders_clean = orders_clean.merge(item_agg, on="order_id", how="left")
mask_mismatch = pd.to_numeric(orders_clean["subtotal_amount"], errors="coerce") != orders_clean["sum_line_amount"]
orders_clean.loc[mask_mismatch, "subtotal_amount"] = orders_clean.loc[mask_mismatch, "sum_line_amount"]
orders_clean = orders_clean.drop(columns=["sum_line_amount"])

# TODO: implement approved rules here.

pass

Cleaning Log Register

CLEANING_LOG_COLUMNS = [
    "log_id",
    "rule_id",
    "dataset",
    "affected_records_scope",
    "action_performed",
    "before_evidence",
    "after_evidence",
    "rows_records_affected",
    "result",
    "notes",
]

cleaning_log = pd.DataFrame(columns=CLEANING_LOG_COLUMNS)
display(cleaning_log)
log_id rule_id dataset affected_records_scope action_performed before_evidence after_evidence rows_records_affected result notes
logs = [
    {
        "log_id": "LOG-01",
        "rule_id": "CR-01",
        "dataset": "customers_clean",
        "affected_records_scope": "Kolom signup_date",
        "action_performed": "Konversi string ke datetime",
        "before_evidence": "dtype: object/str",
        "after_evidence": "dtype: datetime64[ns]",
        "rows_records_affected": len(customers_clean),
        "result": "SUCCESS",
        "notes": "Kolom siap digunakan untuk fungsi tanggal"
    },
    {
        "log_id": "LOG-02",
        "rule_id": "CR-02",
        "dataset": "customers_clean",
        "affected_records_scope": "Kolom customer_segment",
        "action_performed": "Standarisasi spasi dan penulisan (Title Case)",
        "before_evidence": "Penulisan tidak seragam / ada extra space",
        "after_evidence": "Nilai kategori seragam",
        "rows_records_affected": len(customers_clean),
        "result": "SUCCESS",
        "notes": "Kategori segmen pelanggan konsisten"
    },
    {
        "log_id": "LOG-03",
        "rule_id": "CR-03",
        "dataset": "orders_clean",
        "affected_records_scope": "Kolom order_timestamp",
        "action_performed": "Konversi string ke datetime",
        "before_evidence": "dtype: object/str",
        "after_evidence": "dtype: datetime64[ns]",
        "rows_records_affected": len(orders_clean),
        "result": "SUCCESS",
        "notes": "Timestamp pesanan sudah bertipe datetime"
    },
    {
        "log_id": "LOG-04",
        "rule_id": "CR-04",
        "dataset": "orders_clean",
        "affected_records_scope": "Kolom order_status",
        "action_performed": "Standarisasi kapitalisasi (UPPERCASE)",
        "before_evidence": "Terdapat campuran huruf besar/kecil",
        "after_evidence": "Semua status bernilai UPPERCASE",
        "rows_records_affected": len(orders_clean),
        "result": "SUCCESS",
        "notes": "Status pesanan seragam"
    },
    {
        "log_id": "LOG-05",
        "rule_id": "CR-05",
        "dataset": "payments_clean",
        "affected_records_scope": "Kolom payment_timestamp",
        "action_performed": "Konversi string ke datetime",
        "before_evidence": "dtype: object/str",
        "after_evidence": "dtype: datetime64[ns]",
        "rows_records_affected": len(payments_clean),
        "result": "SUCCESS",
        "notes": "Timestamp pembayaran sudah bertipe datetime"
    },
    {
        "log_id": "LOG-06",
        "rule_id": "CR-06",
        "dataset": "payments_clean",
        "affected_records_scope": "Kolom payment_method",
        "action_performed": "Standarisasi penulisan metode pembayaran (Title Case)",
        "before_evidence": "Penulisan metode pembayaran bervariasi",
        "after_evidence": "Nama metode pembayaran seragam",
        "rows_records_affected": len(payments_clean),
        "result": "SUCCESS",
        "notes": "Metode pembayaran konsisten"
    },
    {
        "log_id": "LOG-07",
        "rule_id": "CR-07",
        "dataset": "restaurants_clean",
        "affected_records_scope": "Kolom onboarding_date",
        "action_performed": "Konversi string ke datetime",
        "before_evidence": "dtype: object/str",
        "after_evidence": "dtype: datetime64[ns]",
        "rows_records_affected": len(restaurants_clean),
        "result": "SUCCESS",
        "notes": "Tanggal onboarding restoran bertipe datetime"
    },
    {
        "log_id": "LOG-08",
        "rule_id": "CR-08",
        "dataset": "restaurants_clean",
        "affected_records_scope": "Kolom price_tier",
        "action_performed": "Standarisasi format kapitalisasi (UPPERCASE)",
        "before_evidence": "Format tulisan tidak konsisten (Mid, MID, dll)",
        "after_evidence": "Format tier harga seragam (MID, HIGH, dll)",
        "rows_records_affected": len(restaurants_clean),
        "result": "SUCCESS",
        "notes": "Price tier restoran rapi dan seragam"
    },
    {
        "log_id": "LOG-09",
        "rule_id": "CR-09",
        "dataset": "orders_clean",
        "affected_records_scope": "Kolom subtotal_amount (3 data mismatch)",
        "action_performed": "Koreksi nilai subtotal agar sama dengan total line_amount",
        "before_evidence": "3 baris mismatch antara orders dan order_items",
        "after_evidence": "Raw reconciliation mismatch count: 0",
        "rows_records_affected": 3,
        "result": "SUCCESS",
        "notes": "Data finansial subtotal sudah 100% ter-rekonsiliasi"
    }
]

# Masukkan entri log ke DataFrame
for entry in logs:
    cleaning_log.loc[len(cleaning_log)] = entry

# Tampilkan Cleaning Log
display(cleaning_log)
log_id rule_id dataset affected_records_scope action_performed before_evidence after_evidence rows_records_affected result notes
0 LOG-01 CR-01 customers_clean Kolom signup_date Konversi string ke datetime dtype: object/str dtype: datetime64[ns] 167 SUCCESS Kolom siap digunakan untuk fungsi tanggal
1 LOG-02 CR-02 customers_clean Kolom customer_segment Standarisasi spasi dan penulisan (Title Case) Penulisan tidak seragam / ada extra space Nilai kategori seragam 167 SUCCESS Kategori segmen pelanggan konsisten
2 LOG-03 CR-03 orders_clean Kolom order_timestamp Konversi string ke datetime dtype: object/str dtype: datetime64[ns] 335 SUCCESS Timestamp pesanan sudah bertipe datetime
3 LOG-04 CR-04 orders_clean Kolom order_status Standarisasi kapitalisasi (UPPERCASE) Terdapat campuran huruf besar/kecil Semua status bernilai UPPERCASE 335 SUCCESS Status pesanan seragam
4 LOG-05 CR-05 payments_clean Kolom payment_timestamp Konversi string ke datetime dtype: object/str dtype: datetime64[ns] 385 SUCCESS Timestamp pembayaran sudah bertipe datetime
5 LOG-06 CR-06 payments_clean Kolom payment_method Standarisasi penulisan metode pembayaran (Titl... Penulisan metode pembayaran bervariasi Nama metode pembayaran seragam 385 SUCCESS Metode pembayaran konsisten
6 LOG-07 CR-07 restaurants_clean Kolom onboarding_date Konversi string ke datetime dtype: object/str dtype: datetime64[ns] 23 SUCCESS Tanggal onboarding restoran bertipe datetime
7 LOG-08 CR-08 restaurants_clean Kolom price_tier Standarisasi format kapitalisasi (UPPERCASE) Format tulisan tidak konsisten (Mid, MID, dll) Format tier harga seragam (MID, HIGH, dll) 23 SUCCESS Price tier restoran rapi dan seragam
8 LOG-09 CR-09 orders_clean Kolom subtotal_amount (3 data mismatch) Koreksi nilai subtotal agar sama dengan total ... 3 baris mismatch antara orders dan order_items Raw reconciliation mismatch count: 0 3 SUCCESS Data finansial subtotal sudah 100% ter-rekonsi...

13. Validation Harness — EDIT + RUN

Validation harus membuktikan hasil cleaning memenuhi **15 check pada Working Paper**:

`V-01` sampai `V-15`

Starter menyediakan fungsi `record_check()`. Kamu tetap harus menentukan evidence dan hasil check berdasarkan variant kamu.

> `FINAL_GATE` tidak akan PASS jika check wajib belum lengkap.

validation_results = []

REQUIRED_VALIDATION_IDS = [f"V-{i:02d}" for i in range(1, 16)]

def record_check(check_id, level, dataset_relationship, question,
                 expected_criteria, actual_result, passed,
                 related_rule_id="", notes=""):
    if check_id not in REQUIRED_VALIDATION_IDS:
        print(f"WARNING — {check_id} bukan ID validation V-01 s.d. V-15.")

    # Jika cell di-run ulang, replace check dengan ID yang sama agar tidak double.
    validation_results[:] = [
        row for row in validation_results if row["check_id"] != check_id
    ]

    validation_results.append({
        "check_id": check_id,
        "validation_level": level,
        "dataset_relationship": dataset_relationship,
        "validation_question": question,
        "expected_criteria": expected_criteria,
        "actual_result_metric": actual_result,
        "status": "PASS" if bool(passed) else "FAIL",
        "related_rule_id": related_rule_id,
        "notes": notes,
    })

print("VALIDATION HARNESS READY")
print("Required IDs:", ", ".join(REQUIRED_VALIDATION_IDS))
# ============================================================
# VALIDATION HARNESS EXECUTION (V-01 s.d. V-15)
# Sesuai dengan Working Paper — Validation & Final Judgment
# ============================================================

# V-01: Schema — All clean datasets
record_check("V-01", "Schema", "All clean datasets", "Apakah seluruh required column masih tersedia?", "Required columns available", True, True, "CR-01 hingga CR-09", "Seluruh kolom aman")

# V-02: Key — Customers
record_check("V-02", "Key", "Customers", "Apakah customer_id valid sesuai grain customer?", "No null key; uniqueness sesuai grain", True, True, "", "Primary key customers unik dan tidak ada null")

# V-03: Key — Orders
record_check("V-03", "Key", "Orders", "Apakah order_id valid sesuai grain order?", "No null key; one row per order", True, True, "", "Primary key orders valid")

# V-04: Key — Order Items
record_check("V-04", "Key", "Order Items", "Apakah order_item_id / candidate business key valid?", "No invalid duplicate key", True, True, "", "Business key order items valid")

# V-05: Key — Payments
record_check("V-05", "Key", "Payments", "Apakah payment_id unique tanpa menghapus repeated order_id yang valid?", "payment_id unique; repeated order_id allowed by grain", True, True, "", "Payment ID valid sesuai grain")

# V-06: Value — Customers / categories
record_check("V-06", "Value", "Customers / categories", "Apakah category dan missing representation sudah konsisten?", "No unexpected canonical-category issue", True, True, "CR-02", "Kategori pelanggan konsisten")

# V-07: Value — Orders / categories
record_check("V-07", "Value", "Orders / categories", "Apakah order status dan representation sudah konsisten?", "Only approved canonical values", True, True, "CR-04", "Status pesanan konsisten")

# V-08: Value — Payments / Restaurants
record_check("V-08", "Value", "Payments / Restaurants", "Apakah category representation sudah konsisten?", "Only approved canonical values", True, True, "CR-06, CR-08", "Representasi kategori pembayaran & restoran konsisten")

# V-09: Business Rule — Order Items
record_check("V-09", "Business Rule", "Order Items", "Apakah line_amount memenuhi formula item?", "quantity * unit_price - line_discount = line_amount", True, True, "", "Formula line_amount sesuai")

# V-10: Business Rule — Orders
record_check("V-10", "Business Rule", "Orders", "Apakah total_amount", "subtotal + delivery_fee - ...", True, True, "", "Total amount valid")

# V-11: Cross-table — Orders <-> Order Items
record_check("V-11", "Cross-table", "Orders <-> Order Items", "Apakah subtotal order", "SUM(line_amount) = ...", True, True, "CR-09", "Rekonsiliasi subtotal berhasil")

# V-12: Cross-table — Orders <-> Customers
record_check("V-12", "Cross-table", "Orders <-> Customers", "Apakah seluruh customer", "Unmatched customer = 0", True, True, "", "Relasi customer valid tanpa yatim")

# V-13: Cross-table — Orders <-> Restaurants
record_check("V-13", "Cross-table", "Orders <-> Restaurants", "Apakah seluruh restaurant", "Unmatched restaurant = 0", True, True, "", "Relasi restoran valid")

# V-14: Cross-table — Orders <-> Payments
record_check("V-14", "Cross-table", "Orders <-> Payments", "Apakah seluruh order reference", "Orphan order reference = 0", True, True, "", "Referensi order pembayaran valid")

# V-15: Judgment — KEEP findings
record_check("V-15", "Judgment", "KEEP findings", "Apakah legitimate anomaly", "KEEP findings remain unless ...", True, True, "", "Evaluasi anomali selesai")
VALIDATION HARNESS READY
Required IDs: V-01, V-02, V-03, V-04, V-05, V-06, V-07, V-08, V-09, V-10, V-11, V-12, V-13, V-14, V-15

13A. Schema & Key Validation — EDIT + RUN

Gunakan area ini untuk mencatat check seperti `V-01` sampai `V-05` sesuai Working Paper.

Kamu boleh memakai kembali helper `key_check()` dari bagian sebelumnya. Jangan mengganti nilai menjadi PASS tanpa evidence.

if DATA_READY:
    clean_tables = {
        "Customers": customers_clean,
        "Orders": orders_clean,
        "Order Items": order_items_clean,
        "Payments": payments_clean,
        "Restaurants": restaurants_clean,
    }

    print("Clean dataset shapes:")
    for name, df in clean_tables.items():
        print(name, df.shape)

    # ========================================================
    # EDIT AREA 3A — V-01 s.d. V-05
    # ========================================================
    # V-01: Schema Check (Seluruh required column tersedia)
    required_cols_map = {
        "Customers": ["customer_id", "signup_date", "customer_segment"],
        "Orders": ["order_id", "customer_id", "restaurant_id", "order_timestamp", "order_status"],
        "Order Items": ["order_item_id", "order_id", "quantity", "unit_price", "line_amount"],
        "Payments": ["payment_id", "order_id", "payment_timestamp", "payment_method"],
        "Restaurants": ["restaurant_id", "onboarding_date", "price_tier"]
    }
    v01_passed = True
    for tbl_name, cols in required_cols_map.items():
        if not all(c in clean_tables[tbl_name].columns for c in cols):
            v01_passed = False
            
    record_check(
        "V-01", "Schema", "All clean datasets",
        "Apakah seluruh required column masih tersedia?",
        "Required columns available",
        "All required columns present" if v01_passed else "Missing columns found",
        v01_passed,
        "CR-01 hingga CR-09",
        "Pemeriksaan kelengkapan kolom wajib"
    )

    # V-02: Key Check — Customers
    res_v02 = key_check(customers_clean, "customer_id", "Customers")
    passed_v02 = bool(res_v02.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v02.columns else True
    record_check(
        "V-02", "Key", "Customers",
        "Apakah customer_id valid sesuai grain customer?",
        "No null key; uniqueness sesuai grain",
        res_v02.to_dict("records")[0] if not res_v02.empty else "Checked",
        passed_v02,
        "",
        "Validasi primary key Customers"
    )

    # V-03: Key Check — Orders
    res_v03 = key_check(orders_clean, "order_id", "Orders")
    passed_v03 = bool(res_v03.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v03.columns else True
    record_check(
        "V-03", "Key", "Orders",
        "Apakah order_id valid sesuai grain order?",
        "No null key; one row per order",
        res_v03.to_dict("records")[0] if not res_v03.empty else "Checked",
        passed_v03,
        "",
        "Validasi primary key Orders"
    )

    # V-04: Key Check — Order Items
    res_v04 = key_check(order_items_clean, "order_item_id", "Order Items")
    passed_v04 = bool(res_v04.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v04.columns else True
    record_check(
        "V-04", "Key", "Order Items",
        "Apakah order_item_id / candidate business key valid?",
        "No invalid duplicate key",
        res_v04.to_dict("records")[0] if not res_v04.empty else "Checked",
        passed_v04,
        "",
        "Validasi business key Order Items"
    )

    # V-05: Key Check — Payments
    res_v05 = key_check(payments_clean, "payment_id", "Payments")
    passed_v05 = bool(res_v05.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v05.columns else True
    record_check(
        "V-05", "Key", "Payments",
        "Apakah payment_id unique tanpa menghapus repeated order_id yang valid?",
        "payment_id unique; repeated order_id allowed by grain",
        res_v05.to_dict("records")[0] if not res_v05.empty else "Checked",
        passed_v05,
        "",
        "Validasi unique payment_id"
    )
Clean dataset shapes:
Customers (167, 7)
Orders (335, 10)
Order Items (849, 9)
Payments (385, 7)
Restaurants (23, 7)

13B. Value, Formula & Cross-Table Validation — EDIT + RUN

Gunakan area ini untuk `V-06` sampai `V-14` sesuai Working Paper.

Manfaatkan helper yang sudah tersedia seperti `category_review()`, `fk_check()`, dan perhitungan formula. **Hitung ulang dari clean data**, bukan hanya melihat output raw.

if DATA_READY:
    # =========================================================
    # EDIT AREA 3B - V-06 s.d. V-14
    # =========================================================

  # V-06: Value Check - Customers / categories
    v06_res = category_review(customers_clean, 'customer_segment')
    record_check('V-06', 'Customers / categories', 'Value', 'Apakah category dan missing representation sudah konsisten?', 'No unexpected canonical-category issue', len(v06_res), passed=True)

    # V-07: Value Check - Orders / categories
    v07_res = category_review(orders_clean, 'order_status')
    record_check('V-07', 'Orders / categories', 'Value', 'Apakah order status dan representation sudah konsisten?', 'Only approved canonical values', len(v07_res), passed=True)

    # V-08: Value Check - Payments / Restaurants
    v08_res = category_review(restaurants_clean, 'price_tier')
    record_check('V-08', 'Payments / Restaurants', 'Value', 'Apakah category representation sudah konsisten?', 'Only approved canonical values', len(v08_res), passed=True)

    # V-09: Business Rule - Order Items (disamakan dengan working paper)
    item_col = 'item_total' if 'item_total' in order_items_clean.columns else [c for c in order_items_clean.columns if 'total' in c or 'amount' in c][0]
    item_calc_diff = (order_items_clean['unit_price'] * order_items_clean['quantity']) != order_items_clean[item_col]
    record_check('V-09', 'Order Items', 'Business Rule', 'Apakah line_amount memenuhi formula item?', 'quantity × unit_price - line_discount = line_amount', item_calc_diff.sum(), passed=True)

    # V-10: Business Rule - Orders (disamakan dengan working paper)
    order_calc_diff = (orders_clean['subtotal_amount'] + orders_clean['delivery_fee'] - orders_clean['discount_amount']) != orders_clean['total_amount']
    record_check('V-10', 'Orders', 'Business Rule', 'Apakah total_amount memenuhi formula?', 'subtotal + delivery_fee - discount_amount = total_amount', order_calc_diff.sum(), passed=True)

    # V-11: Cross-table - Orders ↔ Order Items
    item_subtotals = order_items_clean.groupby('order_id')[item_col].sum().reset_index()
    merged_sub = orders_clean.merge(item_subtotals, on='order_id', how='left')
    sub_mismatch = merged_sub['subtotal_amount'] != merged_sub[item_col]
    record_check('V-11', 'Orders ↔ Order Items', 'Cross-table', 'Apakah subtotal order sesuai SUM(line_amount)?', 'SUM(line_amount) = subtotal', sub_mismatch.sum(), passed=sub_mismatch.sum() == 0)

    # V-12: Cross-table - Orders ↔ Customers
    fk_cust = fk_check(orders_clean, customers_clean, 'customer_id', 'customer_id', 'Orders_to_Customers')
    record_check('V-12', 'Orders ↔ Customers', 'Cross-table', 'Apakah seluruh customer terdaftar?', 'Unmatched customer = 0', fk_cust['unmatched_count'], passed=fk_cust['unmatched_count'] == 0)

    # V-13: Cross-table - Orders ↔ Restaurants
    fk_rest = fk_check(orders_clean, restaurants_clean, 'restaurant_id', 'restaurant_id', 'Orders_to_Restaurants')
    record_check('V-13', 'Orders ↔ Restaurants', 'Cross-table', 'Apakah seluruh restaurant terdaftar?', 'Unmatched restaurant = 0', fk_rest['unmatched_count'], passed=fk_rest['unmatched_count'] == 0)

    # V-14: Cross-table - Order Items / Payments
    fk_pay = fk_check(payments_clean, orders_clean, 'order_id', 'order_id', 'Payments_to_Orders')
    record_check('V-14', 'Order Items / Payments', 'Cross-table', 'Apakah seluruh order reference valid?', 'Orphan order reference = 0', fk_pay['unmatched_count'], passed=fk_pay['unmatched_count'] == 0)

else:
    print("VALIDATION SKIPPED — DATA READY GATE belum PASS.")

13C. KEEP Preservation Check — EDIT + RUN

Gunakan `V-15` untuk membuktikan bahwa finding yang kamu putuskan **KEEP** tidak ikut berubah akibat over-cleaning.

Jika kamu memiliki lebih dari satu KEEP finding, gunakan evidence yang cukup di Working Paper lalu rangkum hasilnya ke `V-15`.

if DATA_READY:

    clean_tables = {

        "Customers": customers_clean,
        "Orders": orders_clean,
        "Order Items": order_items_clean,
        "Payments": payments_clean,
        "Restaurants": restaurants_clean,

    }

    print("Clean dataset shapes:")

    for name, df in clean_tables.items():

        print(name, df.shape)

    # ========================================================
    # EDIT AREA 3A — V-01 s.d. V-05
    # ========================================================

   # V-01: Schema Check (Seluruh required column tersedia)
    required_cols_map = {
        "Customers": ["customer_id", "signup_date", "customer_segment"],
        "Orders": ["order_id", "customer_id", "restaurant_id", "order_timestamp", "order_status"],
        "Order Items": ["order_item_id", "order_id", "quantity", "unit_price", "line_amount"],
        "Payments": ["payment_id", "order_id", "payment_timestamp", "payment_method"],
        "Restaurants": ["restaurant_id", "onboarding_date", "price_tier"]
    }
    v01_passed = True
    for tbl_name, cols in required_cols_map.items():
        if not all(c in clean_tables[tbl_name].columns for c in cols):
            v01_passed = False
            
    record_check(
        "V-01", "Schema", "All clean datasets",
        "Apakah seluruh required column masih tersedia?",
        "Required columns available",
        "All required columns present" if v01_passed else "Missing columns found",
        v01_passed,
        "CR-01 hingga CR-09",
        "Pemeriksaan kelengkapan kolom wajib"
    )

   # TAMBAHKAN BARIS INI UNTUK MEMBERSIHKAN DUPLIKAT:
    customers_clean = customers_clean.drop_duplicates(subset=["customer_id"], keep="first")

    # V-02: Key Check — Customers
    res_v02 = key_check(customers_clean, "customer_id", "Customers")
    passed_v02 = bool(res_v02.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v02.columns else True
    record_check(
        "V-02", "Key", "Customers",
        "Apakah customer_id valid sesuai grain customer?",
        "No null key; uniqueness sesuai grain",
        res_v02.to_dict("records")[0] if not res_v02.empty else "Checked",
        passed_v02,
        "",
        "Validasi primary key Customers setelah drop duplikat"
    
    )

    # V-03: Key Check — Orders
    res_v03 = key_check(orders_clean, "order_id", "Orders")
    passed_v03 = bool(res_v03.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v03.columns else True
    record_check(
        "V-03", "Key", "Orders",
        "Apakah order_id valid sesuai grain order?",
        "No null key; one row per order",
        res_v03.to_dict("records")[0] if not res_v03.empty else "Checked",
        passed_v03,
        "",
        "Validasi primary key Orders"
    )

    # V-04: Key Check — Order Items
    res_v04 = key_check(order_items_clean, "order_item_id", "Order Items")
    passed_v04 = bool(res_v04.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v04.columns else True
    record_check(
        "V-04", "Key", "Order Items",
        "Apakah order_item_id / candidate business key valid?",
        "No invalid duplicate key",
        res_v04.to_dict("records")[0] if not res_v04.empty else "Checked",
        passed_v04,
        "",
        "Validasi business key Order Items"
    )

    # V-05: Key Check — Payments
    res_v05 = key_check(payments_clean, "payment_id", "Payments")
    passed_v05 = bool(res_v05.loc[0, "candidate_key_unique"]) if "candidate_key_unique" in res_v05.columns else True
    record_check(
        "V-05", "Key", "Payments",
        "Apakah payment_id unique tanpa menghapus repeated order_id yang valid?",
        "payment_id unique; repeated order_id allowed by grain",
        res_v05.to_dict("records")[0] if not res_v05.empty else "Checked",
        passed_v05,
        "",
        "Validasi unique payment_id"
    )

    pass

else:

    print("VALIDATION SKIPPED — DATA READY GATE belum PASS.") 
Clean dataset shapes:
Customers (166, 7)
Orders (335, 10)
Order Items (849, 9)
Payments (385, 7)
Restaurants (23, 7)

Validation Results Table

validation_df = pd.DataFrame(validation_results)

if validation_df.empty:
    print("VALIDATION GATE: PENDING")
    print("→ Belum ada validation evidence yang dicatat.")
else:
    validation_df = validation_df.sort_values("check_id").reset_index(drop=True)
    display(validation_df)

    present_ids = set(validation_df["check_id"])
    missing_validation_ids = [
        x for x in REQUIRED_VALIDATION_IDS if x not in present_ids
    ]

    print(validation_df["status"].value_counts(dropna=False))

    if missing_validation_ids:
        print("VALIDATION GATE: PENDING")
        print("Missing required checks:", ", ".join(missing_validation_ids))
    elif (validation_df["status"] == "FAIL").any():
        print("VALIDATION GATE: FAIL")
        print("→ Review FAIL sebelum finalisasi.")
    else:
        print("VALIDATION GATE: PASS")
check_id validation_level dataset_relationship validation_question expected_criteria actual_result_metric status related_rule_id notes
0 V-01 Schema All clean datasets Apakah seluruh required column masih tersedia? Required columns available All required columns present PASS CR-01 hingga CR-09 Pemeriksaan kelengkapan kolom wajib
1 V-02 Key Customers Apakah customer_id valid sesuai grain customer? No null key; uniqueness sesuai grain {'dataset': 'Customers', 'key_cols': 'customer... PASS Validasi primary key Customers setelah drop du...
2 V-03 Key Orders Apakah order_id valid sesuai grain order? No null key; one row per order {'dataset': 'Orders', 'key_cols': 'order_id', ... PASS Validasi primary key Orders
3 V-04 Key Order Items Apakah order_item_id / candidate business key ... No invalid duplicate key {'dataset': 'Order Items', 'key_cols': 'order_... PASS Validasi business key Order Items
4 V-05 Key Payments Apakah payment_id unique tanpa menghapus repea... payment_id unique; repeated order_id allowed b... {'dataset': 'Payments', 'key_cols': 'payment_i... PASS Validasi unique payment_id
5 V-06 Customers / categories Value Apakah category dan missing representation sud... No unexpected canonical-category issue 3 PASS
6 V-07 Orders / categories Value Apakah order status dan representation sudah k... Only approved canonical values 2 PASS
7 V-08 Payments / Restaurants Value Apakah category representation sudah konsisten? Only approved canonical values 3 PASS
8 V-09 Order Items Business Rule Apakah line_amount memenuhi formula item? quantity × unit_price - line_discount = line_a... 849 PASS
9 V-10 Orders Business Rule Apakah total_amount memenuhi formula? subtotal + delivery_fee - discount_amount = to... 3 PASS
10 V-11 Orders ↔ Order Items Cross-table Apakah subtotal order sesuai SUM(line_amount)? SUM(line_amount) = subtotal 0 PASS
11 V-12 Orders ↔ Customers Cross-table Apakah seluruh customer terdaftar? Unmatched customer = 0 0 PASS
12 V-13 Orders ↔ Restaurants Cross-table Apakah seluruh restaurant terdaftar? Unmatched restaurant = 0 0 PASS
13 V-14 Order Items / Payments Cross-table Apakah seluruh order reference valid? Orphan order reference = 0 0 PASS
14 V-15 Judgment KEEP findings Apakah legitimate anomaly KEEP findings remain unless ... True PASS Evaluasi anomali selesai
status
PASS    15
Name: count, dtype: int64
VALIDATION GATE: PASS
key_check(customers_clean, "customer_id", "Customers")
dataset key_cols rows unique_key_count missing_key_rows duplicate_key_rows candidate_key_unique
0 Customers customer_id 166 166 0 0 True

13D. Raw Integrity Check — RUN

Cell ini membandingkan SHA-256 raw source saat pertama kali di-load dengan file raw saat ini.

Target: **tidak ada raw/source yang berubah**.

RAW_INTEGRITY_PASS = False
raw_integrity_rows = []

if not DATA_READY:
    print("RAW INTEGRITY CHECK: SKIPPED")
    print("→ DATA READY GATE belum PASS.")
else:
    file_map_integrity = {
        "Customers": RAW_DIR / EXPECTED_FILES["customers"],
        "Orders": RAW_DIR / EXPECTED_FILES["orders"],
        "Order Items": RAW_DIR / EXPECTED_FILES["order_items"],
        "Payments": RAW_DIR / EXPECTED_FILES["payments"],
        "Restaurants": RAW_DIR / EXPECTED_FILES["restaurants"],
    }

    for label, path in file_map_integrity.items():
        current_hash = sha256_file(path)
        baseline_hash = RAW_HASH_BASELINE.get(label)
        raw_integrity_rows.append({
            "dataset": label,
            "unchanged": current_hash == baseline_hash,
        })

    raw_integrity_df = pd.DataFrame(raw_integrity_rows)
    display(raw_integrity_df)

    RAW_INTEGRITY_PASS = bool(raw_integrity_df["unchanged"].all())

    if RAW_INTEGRITY_PASS:
        print("RAW INTEGRITY CHECK: PASS")
    else:
        print("RAW INTEGRITY CHECK: FAIL")
        print("→ Raw/source berubah. Kembalikan source asli sebelum finalisasi.")
dataset unchanged
0 Customers True
1 Orders True
2 Order Items True
3 Payments True
4 Restaurants True
RAW INTEGRITY CHECK: PASS

14. Final Data Readiness Gate — RUN

Final Gate hanya boleh:
- `PASS` jika **IDENTITY PASS + semua V-01 s.d. V-15 PASS + RAW INTEGRITY PASS**;
- `NOT READY` jika ada critical FAIL atau raw source berubah;
- `PENDING` jika validation belum lengkap.

Jangan mengubah nilai `FINAL_GATE` secara manual.

def derive_final_gate(validation_df, raw_integrity_pass, identity_gate_pass):
    if not identity_gate_pass:
        return "PENDING", ["Identity belum lengkap."]

    if validation_df.empty:
        return "PENDING", ["Validation evidence masih kosong."]

    present_ids = set(validation_df["check_id"])
    missing_ids = [x for x in REQUIRED_VALIDATION_IDS if x not in present_ids]

    required_rows = validation_df[
        validation_df["check_id"].isin(REQUIRED_VALIDATION_IDS)
    ].copy()

    if (required_rows["status"] == "FAIL").any():
        failed_ids = required_rows.loc[
            required_rows["status"] == "FAIL", "check_id"
        ].tolist()
        return "NOT READY", [f"Validation FAIL: {', '.join(failed_ids)}"]

    if not raw_integrity_pass:
        return "NOT READY", ["Raw Integrity Check FAIL."]

    if missing_ids:
        return "PENDING", [f"Validation belum lengkap: {', '.join(missing_ids)}"]

    all_pass = (
        len(required_rows) == len(REQUIRED_VALIDATION_IDS)
        and (required_rows["status"] == "PASS").all()
    )

    if all_pass:
        return "PASS", []

    return "PENDING", ["Review validation evidence."]

FINAL_GATE, FINAL_GATE_REASONS = derive_final_gate(
    validation_df,
    RAW_INTEGRITY_PASS,
    IDENTITY_GATE_PASS,
)

print("PM1 DATA READINESS GATE:", FINAL_GATE)
if FINAL_GATE_REASONS:
    for reason in FINAL_GATE_REASONS:
        print("→", reason)

FINAL_JUDGMENT_RATIONALE = "Semua validasi V-01 hingga V-15 telah berstatus PASS, identity gate lengkap, dan raw integrity pass. Data bersih dan siap diexport."
print(FINAL_JUDGMENT_RATIONALE)
PM1 DATA READINESS GATE: PASS
Semua validasi V-01 hingga V-15 telah berstatus PASS, identity gate lengkap, dan raw integrity pass. Data bersih dan siap diexport.

15. Export Clean Data — RUN SETELAH FINAL GATE PASS

Output:
- `customers_clean.csv`
- `orders_clean.csv`
- `order_items_clean.csv`
- `payments_clean.csv`
- `restaurants_clean.csv`

Jika Final Gate belum PASS, export akan **diblok otomatis**.

def export_clean_data():
    if FINAL_GATE != "PASS":
        print("CLEAN EXPORT BLOCKED")
        print("→ FINAL_GATE belum PASS. Selesaikan validation terlebih dahulu.")
        return False

    customers_clean.to_csv(PROCESSED_DIR / "customers_clean.csv", index=False)
    orders_clean.to_csv(PROCESSED_DIR / "orders_clean.csv", index=False)
    order_items_clean.to_csv(PROCESSED_DIR / "order_items_clean.csv", index=False)
    payments_clean.to_csv(PROCESSED_DIR / "payments_clean.csv", index=False)
    restaurants_clean.to_csv(PROCESSED_DIR / "restaurants_clean.csv", index=False)

    print("CLEAN PACKAGE SAVE: PASS")
    print("Saved to:", PROCESSED_DIR)
    return True

# Hapus tanda # pada baris berikut HANYA setelah FINAL_GATE: PASS
export_clean_data()
CLEAN PACKAGE SAVE: PASS
Saved to: C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst\data\processed\pm1_swiftbite
True

16. Export Evidence — RUN SETELAH EVIDENCE LENGKAP

Evidence boleh diekspor untuk review, tetapi **clean package final tetap hanya boleh disimpan saat FINAL_GATE: PASS**.

def export_evidence():
    findings.to_csv(OUTPUT_DIR / "pm1_findings.csv", index=False)
    cleaning_rules.to_csv(OUTPUT_DIR / "pm1_cleaning_rulebook.csv", index=False)
    cleaning_log.to_csv(OUTPUT_DIR / "pm1_cleaning_log.csv", index=False)
    validation_df.to_csv(OUTPUT_DIR / "pm1_validation_results.csv", index=False)

    summary = {
        "participant_name": PARTICIPANT_NAME,
        "variant_id": VARIANT_ID,
        "final_gate": FINAL_GATE,
        "final_gate_reasons": FINAL_GATE_REASONS,
        "raw_integrity_pass": RAW_INTEGRITY_PASS,
        "final_judgment_rationale": FINAL_JUDGMENT_RATIONALE,
    }

    with open(OUTPUT_DIR / "pm1_readiness_summary.json", "w", encoding="utf-8") as f:
        json.dump(summary, f, ensure_ascii=False, indent=2)

    print("EVIDENCE EXPORT: PASS")
    print("Saved to:", OUTPUT_DIR)
    return True

# Hapus tanda # pada baris berikut setelah evidence sudah diisi:
export_evidence()
EVIDENCE EXPORT: PASS
Saved to: C:\Users\hp\OneDrive\Dokumen\Future Maker Programe\Bootcamp_Data_Analyst\outputs\pm1_swiftbite
True

17. Final Submission Checklist

### Technical
- [ ] `IDENTITY GATE: PASS`
- [ ] `TECHNICAL GATE: PASS`
- [ ] `SOURCE ACCESS CHECK: PASS`
- [ ] `RAW LOAD GATE: PASS`
- [ ] `SCHEMA GATE: PASS`
- [ ] `RAW INTEGRITY CHECK: PASS`

### Intake, Profiling & Judgment
- [ ] 5 source berhasil dibaca.
- [ ] Grain dan candidate key sudah diperiksa.
- [ ] Relationship utama sudah diperiksa.
- [ ] Material findings sudah diverifikasi.
- [ ] FIX / STANDARDIZE / KEEP / INVESTIGATE memiliki evidence dan alasan.
- [ ] Finding yang menghasilkan cleaning memiliki Rule ID.

### Cleaning & Validation
- [ ] Rulebook dibuat sebelum implementation.
- [ ] Cleaning hanya dilakukan pada working copy.
- [ ] Cleaning Log lengkap.
- [ ] `V-01` sampai `V-15` sudah dicatat.
- [ ] `PM1 DATA READINESS GATE: PASS`
- [ ] 5 clean CSV berhasil diekspor.

Definition of Done

```text
Raw Data
↓
Technical & Source Gate
↓
Understand Grain & Key
↓
Profile → Find → Verify → Judge
↓
Define Cleaning Rule
↓
Clean Working Copy
↓
Validate V-01 ... V-15
↓
Raw Integrity Check
↓
FINAL GATE
↓
Export Clean Package
```