# Praktik Mandiri 1 — Data Readiness Check & Cleaning Validation
## Participant Starter Notebook — Guided Technical Version
Notebook ini membantu kamu menjalankan workflow PM1 dengan **technical guardrails** agar tidak salah folder, salah source, atau salah export.
**Alur PM1:**
**Setup → Technical Check → Load → Intake → Grain & Key → Profiling → Findings → Verification → Rulebook → Cleaning → Validation → Raw Integrity → Export → Final Gate**
### Sebelum RUN
1. Buka **folder `Bootcamp_Data_Analyst` sebagai workspace** di Antigravity.
2. Simpan notebook ini di folder `notebooks/`.
3. Extract package `SB-Pxx`, lalu letakkan **5 raw files langsung** di `data/raw/pm1_swiftbite/`.
4. Buka `PM1_Data_Readiness_Working_Paper.xlsx` sebagai tempat mencatat decision & evidence.
5. Jalankan notebook **dari atas ke bawah**.
> Jika `TECHNICAL GATE: PASS`, **jangan mengubah path** di notebook.
### Guardrails
- Raw/source **tidak boleh diubah atau ditimpa**.
- Jangan menghapus row hanya karena terlihat aneh.
- Repeated `order_id` belum tentu duplicate.
- Finding belum otomatis menjadi cleaning action.
- Material change harus dapat ditelusuri: **Finding ID → Rule ID → Cleaning Log → Validation Evidence**.
- Notebook membantu sisi teknis. **Judgment, cleaning rule, dan interpretation tetap pekerjaan peserta.**
Ubah hanya **Nama** dan **Variant ID** sesuai package yang kamu terima.
Contoh Variant ID: `SB-P01`, `SB-P12`, `SB-P28`.
# ============================================================
# EDIT AREA 1 — IDENTITAS PESERTA
# ============================================================
PARTICIPANT_NAME = "Bening"
VARIANT_ID = "SB-P11" # contoh: SB-P01
name_ok = PARTICIPANT_NAME.strip() not in {"", "Isi Nama Kamu"}
variant_ok = (
VARIANT_ID.startswith("SB-P")
and len(VARIANT_ID) == 6
and VARIANT_ID[4:].isdigit()
)
IDENTITY_GATE_PASS = bool(name_ok and variant_ok)
if IDENTITY_GATE_PASS:
print("IDENTITY GATE: PASS")
print("Participant :", PARTICIPANT_NAME)
print("Variant ID :", VARIANT_ID)
else:
print("IDENTITY GATE: FAIL")
print("→ Isi PARTICIPANT_NAME dan VARIANT_ID pada cell ini.")
print("→ Format Variant ID harus seperti SB-P01 atau SB-P28.")
IDENTITY GATE: PASS Participant : Bening Variant ID : SB-P11
Notebook akan mencari **Workspace Starter yang sama dari Sesi 1** berdasarkan keberadaan folder `data/` dan `notebooks/`.
**Jangan membuat workspace baru. Jangan copy-paste absolute path.**
Jika output menunjukkan `TECHNICAL GATE: PASS`, lanjut ke cell berikutnya tanpa mengubah path.
from pathlib import Path
import json
import hashlib
import pandas as pd
import numpy as np
try:
import duckdb
DUCKDB_AVAILABLE = True
except ImportError:
duckdb = None
DUCKDB_AVAILABLE = False
pd.set_option("display.max_columns", 100)
pd.set_option("display.max_rows", 100)
pd.set_option("display.width", 180)
def find_workspace_root(start=None):
start = Path(start or Path.cwd()).resolve()
for p in [start, *start.parents]:
if (p / "data").is_dir() and (p / "notebooks").is_dir():
return p
return None
WORKSPACE_ROOT = find_workspace_root()
if WORKSPACE_ROOT is None:
TECHNICAL_GATE_PASS = False
RAW_DIR = Path("__WORKSPACE_NOT_FOUND__")
PROCESSED_DIR = Path("__WORKSPACE_NOT_FOUND__")
OUTPUT_DIR = Path("__WORKSPACE_NOT_FOUND__")
DOCS_DIR = Path("__WORKSPACE_NOT_FOUND__")
print("TECHNICAL GATE: FAIL")
print("→ Workspace Starter tidak ditemukan.")
print("→ Di Antigravity, buka folder Bootcamp_Data_Analyst sebagai workspace.")
print("→ Pastikan folder data/ dan notebooks/ terlihat di Explorer.")
print("→ Setelah itu RUN ulang cell ini. Jangan edit path secara manual.")
else:
TECHNICAL_GATE_PASS = True
RAW_DIR = WORKSPACE_ROOT / "data" / "raw" / "pm1_swiftbite"
PROCESSED_DIR = WORKSPACE_ROOT / "data" / "processed" / "pm1_swiftbite"
OUTPUT_DIR = WORKSPACE_ROOT / "outputs" / "pm1_swiftbite"
DOCS_DIR = WORKSPACE_ROOT / "docs" / "pm1_swiftbite"
for folder in [PROCESSED_DIR, OUTPUT_DIR, DOCS_DIR]:
folder.mkdir(parents=True, exist_ok=True)
print("TECHNICAL GATE: PASS")
print("WORKSPACE_ROOT :", WORKSPACE_ROOT)
print("RAW_DIR :", RAW_DIR)
print("PROCESSED_DIR :", PROCESSED_DIR)
print("OUTPUT_DIR :", OUTPUT_DIR)
print("DOCS_DIR :", DOCS_DIR)
TECHNICAL GATE: PASS WORKSPACE_ROOT : C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst RAW_DIR : C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst\data\raw\pm1_swiftbite PROCESSED_DIR : C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst\data\processed\pm1_swiftbite OUTPUT_DIR : C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst\outputs\pm1_swiftbite DOCS_DIR : C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst\docs\pm1_swiftbite
Tujuan cell ini hanya memastikan library utama tersedia. Jika DuckDB belum aktif, workflow utama starter tetap dapat berjalan dengan pandas.
print("pandas :", pd.__version__)
try:
import openpyxl
EXCEL_ENGINE_AVAILABLE = True
print("openpyxl:", openpyxl.__version__)
except ImportError:
EXCEL_ENGINE_AVAILABLE = False
print("openpyxl: NOT AVAILABLE")
if DUCKDB_AVAILABLE:
print("DuckDB :", duckdb.__version__)
try:
con = duckdb.connect()
print("DuckDB connection: OK")
except Exception as e:
print("DuckDB connection: WARNING —", type(e).__name__, str(e))
else:
print("DuckDB : NOT AVAILABLE")
print("INFO: workflow utama notebook tetap dapat dijalankan dengan pandas.")
if not EXCEL_ENGINE_AVAILABLE:
print("ENVIRONMENT CHECK: NEEDS FIX")
print("→ File XLSX membutuhkan openpyxl. Aktifkan environment sesuai setup Sesi 1.")
else:
print("ENVIRONMENT CHECK: PASS")
pandas : 2.3.0 openpyxl: 3.1.5 DuckDB : 1.5.6 DuckDB connection: OK ENVIRONMENT CHECK: PASS
Letakkan **5 file berikut langsung** di:
`data/raw/pm1_swiftbite/`
- `swiftbite_customers.json`
- `swiftbite_orders.xlsx`
- `swiftbite_order_items.csv`
- `swiftbite_payments.txt`
- `swiftbite_restaurants.csv`
**Jangan** meletakkan ZIP di folder tersebut dan **jangan** membuat folder `SB-Pxx` lagi di dalam `pm1_swiftbite/`.
EXPECTED_FILES = {
"customers": "swiftbite_customers.json",
"orders": "swiftbite_orders.xlsx",
"order_items": "swiftbite_order_items.csv",
"payments": "swiftbite_payments.txt",
"restaurants": "swiftbite_restaurants.csv",
}
source_status = []
if not TECHNICAL_GATE_PASS:
missing_files = list(EXPECTED_FILES.values())
SOURCE_ACCESS_PASS = False
print("SOURCE ACCESS CHECK: SKIPPED")
print("→ Perbaiki TECHNICAL GATE terlebih dahulu.")
else:
for logical_name, filename in EXPECTED_FILES.items():
path = RAW_DIR / filename
source_status.append({
"logical_name": logical_name,
"filename": filename,
"exists": path.is_file(),
"path": str(path),
})
source_status_df = pd.DataFrame(source_status)
display(source_status_df)
missing_files = source_status_df.loc[
~source_status_df["exists"], "filename"
].tolist()
SOURCE_ACCESS_PASS = len(missing_files) == 0
if SOURCE_ACCESS_PASS:
print("SOURCE ACCESS CHECK: PASS")
print("→ Semua 5 raw files ditemukan. Jangan ubah path.")
else:
print("SOURCE ACCESS CHECK: FAIL")
print("File belum ditemukan:", missing_files)
print("→ Extract package peserta.")
print("→ Copy hanya 5 raw files langsung ke:", RAW_DIR)
print("→ Jangan menaruh file di subfolder tambahan.")
| logical_name | filename | exists | path | |
|---|---|---|---|---|
| 0 | customers | swiftbite_customers.json | True | C:\Users\Bening JPA\OneDrive\Documents\Bootcam... |
| 1 | orders | swiftbite_orders.xlsx | True | C:\Users\Bening JPA\OneDrive\Documents\Bootcam... |
| 2 | order_items | swiftbite_order_items.csv | True | C:\Users\Bening JPA\OneDrive\Documents\Bootcam... |
| 3 | payments | swiftbite_payments.txt | True | C:\Users\Bening JPA\OneDrive\Documents\Bootcam... |
| 4 | restaurants | swiftbite_restaurants.csv | True | C:\Users\Bening JPA\OneDrive\Documents\Bootcam... |
SOURCE ACCESS CHECK: PASS → Semua 5 raw files ditemukan. Jangan ubah path.
Cell ini:
1. membaca kelima source secara aman,
2. menampilkan error per file jika ada,
3. memeriksa kolom minimum yang diperlukan Starter,
4. menyimpan **raw hash baseline** untuk pemeriksaan integritas di akhir.
Jika `RAW LOAD GATE: PASS` dan `SCHEMA GATE: PASS`, lanjutkan.
REQUIRED_COLUMNS = {
"Customers": {
"customer_id", "full_name", "email", "phone_number",
"signup_date", "customer_zone", "customer_segment"
},
"Orders": {
"order_id", "customer_id", "restaurant_id", "order_timestamp",
"order_status", "subtotal_amount", "delivery_fee",
"discount_amount", "total_amount"
},
"Order Items": {
"order_item_id", "order_id", "line_number", "item_name",
"item_category", "quantity", "unit_price",
"line_discount", "line_amount"
},
"Payments": {
"payment_id", "order_id", "payment_timestamp",
"payment_method", "payment_status", "payment_amount"
},
"Restaurants": {
"restaurant_id", "restaurant_name", "cuisine_type",
"restaurant_zone", "price_tier", "active_status"
},
}
def sha256_file(path: Path):
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
h.update(chunk)
return h.hexdigest()
def safe_read_source(label, path, reader):
try:
return reader(path), None
except Exception as e:
return None, f"{type(e).__name__}: {e}"
raw_tables = {}
load_errors = {}
RAW_HASH_BASELINE = {}
if not SOURCE_ACCESS_PASS:
RAW_LOAD_GATE_PASS = False
SCHEMA_GATE_PASS = False
DATA_READY = False
print("RAW LOAD GATE: SKIPPED")
print("→ Perbaiki SOURCE ACCESS CHECK terlebih dahulu.")
else:
readers = {
"Customers": lambda p: pd.read_json(p),
"Orders": lambda p: pd.read_excel(p),
"Order Items": lambda p: pd.read_csv(p),
"Payments": lambda p: pd.read_csv(p, sep="|"),
"Restaurants": lambda p: pd.read_csv(p),
}
file_map = {
"Customers": RAW_DIR / EXPECTED_FILES["customers"],
"Orders": RAW_DIR / EXPECTED_FILES["orders"],
"Order Items": RAW_DIR / EXPECTED_FILES["order_items"],
"Payments": RAW_DIR / EXPECTED_FILES["payments"],
"Restaurants": RAW_DIR / EXPECTED_FILES["restaurants"],
}
for label, path in file_map.items():
df, err = safe_read_source(label, path, readers[label])
if err:
load_errors[label] = err
else:
raw_tables[label] = df
RAW_HASH_BASELINE[label] = sha256_file(path)
RAW_LOAD_GATE_PASS = len(load_errors) == 0 and len(raw_tables) == 5
if RAW_LOAD_GATE_PASS:
customers_raw = raw_tables["Customers"]
orders_raw = raw_tables["Orders"]
order_items_raw = raw_tables["Order Items"]
payments_raw = raw_tables["Payments"]
restaurants_raw = raw_tables["Restaurants"]
print("RAW LOAD GATE: PASS")
else:
print("RAW LOAD GATE: FAIL")
for label, err in load_errors.items():
print(f"→ {label}: {err}")
schema_rows = []
if RAW_LOAD_GATE_PASS:
for label, required in REQUIRED_COLUMNS.items():
actual = set(raw_tables[label].columns)
missing_cols = sorted(required - actual)
schema_rows.append({
"dataset": label,
"required_columns": len(required),
"actual_columns": len(actual),
"missing_required_columns": ", ".join(missing_cols) if missing_cols else "",
"status": "PASS" if not missing_cols else "FAIL",
})
schema_check_df = pd.DataFrame(schema_rows)
display(schema_check_df)
SCHEMA_GATE_PASS = (schema_check_df["status"] == "PASS").all()
if SCHEMA_GATE_PASS:
print("SCHEMA GATE: PASS")
else:
print("SCHEMA GATE: FAIL")
print("→ Jangan lanjut cleaning. Periksa apakah package/variant yang digunakan sudah benar.")
else:
SCHEMA_GATE_PASS = False
DATA_READY = bool(
TECHNICAL_GATE_PASS
and SOURCE_ACCESS_PASS
and RAW_LOAD_GATE_PASS
and SCHEMA_GATE_PASS
)
if DATA_READY:
print("DATA READY GATE: PASS")
else:
print("DATA READY GATE: FAIL")
RAW LOAD GATE: PASS
| dataset | required_columns | actual_columns | missing_required_columns | status | |
|---|---|---|---|---|---|
| 0 | Customers | 7 | 7 | PASS | |
| 1 | Orders | 9 | 10 | PASS | |
| 2 | Order Items | 9 | 9 | PASS | |
| 3 | Payments | 6 | 7 | PASS | |
| 4 | Restaurants | 6 | 7 | PASS |
SCHEMA GATE: PASS DATA READY GATE: PASS
Pastikan nama kolom dan contoh record masuk akal. Ini **belum** berarti data sudah bersih.
if DATA_READY:
for name, df in raw_tables.items():
print(f"\n=== {name} | shape={df.shape} ===")
display(df.head(3))
else:
print("PREVIEW SKIPPED — DATA READY GATE belum PASS.")
=== Customers | shape=(174, 7) ===
| customer_id | full_name | phone_number | signup_date | customer_zone | customer_segment | ||
|---|---|---|---|---|---|---|---|
| 0 | C0001 | Eka Sari | ekasari1@example.com | 085526299937 | 2025-06-12 | South | Loyal |
| 1 | C0002 | Farah Kusuma | farahkusuma2@example.com | 081784596265 | 2025-08-20 | East | New |
| 2 | C0003 | Dimas Sari | dimassari3@example.com | 084948484369 | 2025-09-15 | Central | Loyal |
=== Orders | shape=(339, 10) ===
| order_id | customer_id | restaurant_id | order_timestamp | order_status | promo_code | subtotal_amount | delivery_fee | discount_amount | total_amount | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | O00001 | C0102 | R0005 | 2026-03-28 03:08:00 | completed | NaN | 149000 | 18000 | 10000 | 157000 |
| 1 | O00002 | C0109 | R0010 | 2026-01-11 03:32:00 | completed | SAVE10 | 133000 | 12000 | 10000 | 135000 |
| 2 | O00003 | C0116 | R0015 | 2026-01-11 22:52:00 | completed | NEWUSER | 45000 | 18000 | 15000 | 48000 |
=== Order Items | shape=(893, 9) ===
| order_item_id | order_id | line_number | item_name | item_category | quantity | unit_price | line_discount | line_amount | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | OI00001 | O00001 | 1 | Toast | Snack | 3 | 16000 | 2000 | 46000 |
| 1 | OI00002 | O00001 | 2 | Milk Tea | Drink | 3 | 22000 | 5000 | 61000 |
| 2 | OI00003 | O00001 | 3 | Chicken Set | Main | 1 | 42000 | 0 | 42000 |
=== Payments | shape=(387, 7) ===
| payment_id | order_id | payment_method | payment_status | payment_amount | payment_timestamp | attempt_number | |
|---|---|---|---|---|---|---|---|
| 0 | P00001 | O00001 | e_wallet | successful | 157000 | 2026-03-28 03:11:00 | 1 |
| 1 | P00002 | O00002 | bank_transfer | successful | 135000 | 2026-01-11 03:35:00 | 1 |
| 2 | P00003 | O00003 | credit_card | successful | 48000 | 2026-01-11 22:55:00 | 1 |
=== Restaurants | shape=(24, 7) ===
| restaurant_id | restaurant_name | cuisine_type | restaurant_zone | price_tier | onboarding_date | active_status | |
|---|---|---|---|---|---|---|---|
| 0 | R0001 | SwiftBite Partner 01 | Indonesian | North | Premium | 2025-02-18 | active |
| 1 | R0002 | SwiftBite Partner 02 | Asian | East | Mid | 2024-10-13 | active |
| 2 | R0003 | SwiftBite Partner 03 | Western | East | Budget | 2024-04-05 | active |
Catat business meaning, expected grain, candidate key, row/record count, access status, dan key test result.
def intake_summary(df, dataset_name):
return {
"dataset": dataset_name,
"rows": len(df),
"columns": len(df.columns),
"column_names": list(df.columns),
}
if DATA_READY:
intake_df = pd.DataFrame([
intake_summary(customers_raw, "Customers"),
intake_summary(orders_raw, "Orders"),
intake_summary(order_items_raw, "Order Items"),
intake_summary(payments_raw, "Payments"),
intake_summary(restaurants_raw, "Restaurants"),
])
display(intake_df)
| dataset | rows | columns | column_names | |
|---|---|---|---|---|
| 0 | Customers | 174 | 7 | [customer_id, full_name, email, phone_number, ... |
| 1 | Orders | 339 | 10 | [order_id, customer_id, restaurant_id, order_t... |
| 2 | Order Items | 893 | 9 | [order_item_id, order_id, line_number, item_na... |
| 3 | Payments | 387 | 7 | [payment_id, order_id, payment_method, payment... |
| 4 | Restaurants | 24 | 7 | [restaurant_id, restaurant_name, cuisine_type,... |
Repeated `order_id` pada **Order Items** atau **Payments** belum tentu duplicate karena grain-nya berbeda dari Orders.
Gunakan hasil check sebagai **evidence awal**, bukan cleaning action otomatis.
def key_check(df, key_cols, dataset_name):
if isinstance(key_cols, str):
key_cols = [key_cols]
missing_key_rows = int(df[key_cols].isna().any(axis=1).sum())
unique_key_count = int(df[key_cols].drop_duplicates().shape[0])
duplicate_key_rows = int(df.duplicated(subset=key_cols, keep=False).sum())
return pd.DataFrame([{
"dataset": dataset_name,
"key_cols": ", ".join(key_cols),
"rows": len(df),
"unique_key_count": unique_key_count,
"missing_key_rows": missing_key_rows,
"duplicate_key_rows": duplicate_key_rows,
"candidate_key_unique": unique_key_count == len(df) and missing_key_rows == 0,
}])
if DATA_READY:
key_results = pd.concat([
key_check(customers_raw, "customer_id", "Customers"),
key_check(orders_raw, "order_id", "Orders"),
key_check(order_items_raw, "order_item_id", "Order Items"),
key_check(order_items_raw, ["order_id", "line_number"], "Order Items — candidate business key"),
key_check(payments_raw, "payment_id", "Payments"),
key_check(restaurants_raw, "restaurant_id", "Restaurants"),
], ignore_index=True)
display(key_results)
| dataset | key_cols | rows | unique_key_count | missing_key_rows | duplicate_key_rows | candidate_key_unique | |
|---|---|---|---|---|---|---|---|
| 0 | Customers | customer_id | 174 | 173 | 0 | 2 | False |
| 1 | Orders | order_id | 339 | 339 | 0 | 0 | True |
| 2 | Order Items | order_item_id | 893 | 893 | 0 | 0 | True |
| 3 | Order Items — candidate business key | order_id, line_number | 893 | 893 | 0 | 0 | True |
| 4 | Payments | payment_id | 387 | 387 | 0 | 0 | True |
| 5 | Restaurants | restaurant_id | 24 | 24 | 0 | 0 | True |
if DATA_READY:
order_item_cardinality = (
order_items_raw.groupby("order_id")
.size()
.rename("item_rows")
.reset_index()
.sort_values("item_rows", ascending=False)
)
payment_cardinality = (
payments_raw.groupby("order_id")
.size()
.rename("payment_rows")
.reset_index()
.sort_values("payment_rows", ascending=False)
)
print("Order Items cardinality")
display(order_item_cardinality.head(10))
print("Payments cardinality")
display(payment_cardinality.head(10))
Order Items cardinality
| order_id | item_rows | |
|---|---|---|
| 11 | O00012 | 5 |
| 68 | O00069 | 5 |
| 246 | O00247 | 5 |
| 83 | O00084 | 5 |
| 139 | O00140 | 5 |
| 136 | O00137 | 5 |
| 198 | O00199 | 5 |
| 320 | O00321 | 5 |
| 14 | O00015 | 4 |
| 296 | O00297 | 4 |
Payments cardinality
| order_id | payment_rows | |
|---|---|---|
| 111 | O00112 | 2 |
| 300 | O00301 | 2 |
| 104 | O00105 | 2 |
| 83 | O00084 | 2 |
| 105 | O00106 | 2 |
| 147 | O00148 | 2 |
| 166 | O00167 | 2 |
| 69 | O00070 | 2 |
| 70 | O00071 | 2 |
| 211 | O00212 | 2 |
Profiling minimal:
- missing / placeholder,
- data type / format,
- category consistency,
- duplicate / key,
- suspicious numeric / formula,
- potential outlier,
- cross-table concern.
Helper berikut menghasilkan evidence awal, bukan automatic judgment.
def basic_profile(df, dataset_name):
return pd.DataFrame([{
"dataset": dataset_name,
"rows": len(df),
"columns": len(df.columns),
"rows_in_exact_duplicate_groups": int(df.duplicated(keep=False).sum()),
"exact_duplicate_extras": int(df.duplicated(keep="first").sum()),
"null_cells": int(df.isna().sum().sum()),
}])
def column_profile(df):
rows = []
for col in df.columns:
s = df[col]
rows.append({
"column": col,
"dtype": str(s.dtype),
"null_count": int(s.isna().sum()),
"null_pct": round(float(s.isna().mean()), 4),
"unique_count": int(s.nunique(dropna=True)),
"sample_values": list(s.dropna().astype(str).unique()[:5]),
})
return pd.DataFrame(rows)
if DATA_READY:
overall_profile = pd.concat(
[basic_profile(df, name) for name, df in raw_tables.items()],
ignore_index=True
)
display(overall_profile)
| dataset | rows | columns | rows_in_exact_duplicate_groups | exact_duplicate_extras | null_cells | |
|---|---|---|---|---|---|---|
| 0 | Customers | 174 | 7 | 2 | 1 | 0 |
| 1 | Orders | 339 | 10 | 0 | 0 | 189 |
| 2 | Order Items | 893 | 9 | 0 | 0 | 0 |
| 3 | Payments | 387 | 7 | 0 | 0 | 0 |
| 4 | Restaurants | 24 | 7 | 0 | 0 | 0 |
# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
display(column_profile(customers_raw))
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | customer_id | object | 0 | 0.0 | 173 | [C0001, C0002, C0003, C0004, C0005] |
| 1 | full_name | object | 0 | 0.0 | 134 | [Eka Sari, Farah Kusuma, Dimas Sari, Wahyu San... |
| 2 | object | 0 | 0.0 | 173 | [ekasari1@example.com, farahkusuma2@example.co... | |
| 3 | phone_number | object | 0 | 0.0 | 173 | [085526299937, 081784596265, 084948484369, 084... |
| 4 | signup_date | object | 0 | 0.0 | 128 | [2025-06-12, 2025-08-20, 2025-09-15, 2025-08-2... |
| 5 | customer_zone | object | 0 | 0.0 | 5 | [South, East, Central, North, West] |
| 6 | customer_segment | object | 0 | 0.0 | 5 | [Loyal, New, Regular, NEW , LOYAL ] |
def category_review(df, column, top_n=30):
out = (
df[column]
.value_counts(dropna=False)
.rename_axis(column)
.reset_index(name="count")
)
out["pct"] = (out["count"] / len(df)).round(4)
return out.head(top_n)
# Contoh:
display(category_review(customers_raw, "customer_segment"))
display(category_review(orders_raw, "order_status"))
display(category_review(payments_raw, "payment_method"))
display(category_review(restaurants_raw, "price_tier"))
| customer_segment | count | pct | |
|---|---|---|---|
| 0 | Regular | 68 | 0.3908 |
| 1 | New | 56 | 0.3218 |
| 2 | Loyal | 48 | 0.2759 |
| 3 | NEW | 1 | 0.0057 |
| 4 | LOYAL | 1 | 0.0057 |
| order_status | count | pct | |
|---|---|---|---|
| 0 | completed | 303 | 0.8938 |
| 1 | cancelled | 34 | 0.1003 |
| 2 | COMPLETED | 2 | 0.0059 |
| payment_method | count | pct | |
|---|---|---|---|
| 0 | e_wallet | 98 | 0.2532 |
| 1 | credit_card | 97 | 0.2506 |
| 2 | bank_transfer | 95 | 0.2455 |
| 3 | cash | 95 | 0.2455 |
| 4 | BANK TRANSFER | 2 | 0.0052 |
| price_tier | count | pct | |
|---|---|---|---|
| 0 | Mid | 13 | 0.5417 |
| 1 | Premium | 7 | 0.2917 |
| 2 | Budget | 4 | 0.1667 |
def numeric_profile(df):
numeric_cols = df.select_dtypes(include="number").columns.tolist()
if not numeric_cols:
return pd.DataFrame()
return df[numeric_cols].describe().T
# Contoh:
display(numeric_profile(orders_raw))
display(numeric_profile(order_items_raw))
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| subtotal_amount | 339.0 | 124713.864307 | 70283.055034 | 11000.0 | 73500.0 | 110000.0 | 165000.0 | 612000.0 |
| delivery_fee | 339.0 | 12448.377581 | 3578.306849 | 8000.0 | 10000.0 | 12000.0 | 15000.0 | 18000.0 |
| discount_amount | 339.0 | 6474.926254 | 5861.451865 | 0.0 | 0.0 | 5000.0 | 10000.0 | 15000.0 |
| total_amount | 339.0 | 130687.315634 | 70499.294039 | 8000.0 | 80500.0 | 118000.0 | 167500.0 | 622000.0 |
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| line_number | 893.0 | 1.967525 | 0.948245 | 1.0 | 1.0 | 2.0 | 3.0 | 5.0 |
| unit_price | 893.0 | 24535.274356 | 15494.615688 | 9000.0 | 18000.0 | 22000.0 | 28000.0 | 405000.0 |
| line_discount | 893.0 | 1406.494961 | 1948.162964 | 0.0 | 0.0 | 0.0 | 2000.0 | 5000.0 |
| line_amount | 893.0 | 47334.826428 | 31480.275106 | 4000.0 | 23000.0 | 42000.0 | 64000.0 | 403000.0 |
Gunakan judgment:
- `FIX`
- `STANDARDIZE`
- `KEEP`
- `INVESTIGATE`
Jangan isi judgment sebelum verification evidence cukup.
FINDING_COLUMNS = [
"finding_id",
"dataset",
"field_key",
"finding_evidence",
"why_it_matters",
"verification_method",
"verification_evidence_result",
"judgment",
"reason_for_judgment",
"cleaning_rule_id",
]
findings_records = [
{
"finding_id": "F-01",
"dataset": "Customers",
"field_key": "customer_id",
"finding_evidence": "Duplicate customer record. customer_id = 'C0031' (Farah Saputra) muncul 2 kali pada baris index 30 dan index 173 dengan seluruh atribut identik (exact duplicate group).",
"why_it_matters": "Melanggar grain master customer (1 baris per pelanggan unik), menyebabkan overcounting total pelanggan dan duplikasi saat di-join dengan Orders.",
"verification_method": "Pengecekan duplikasi: customers_raw.duplicated(subset=['customer_id'], keep=False) dan customers_raw.duplicated(keep=False).",
"verification_evidence_result": "Ditemukan tepat 2 baris identik untuk customer_id C0031. 0 missing key.",
"judgment": "FIX",
"reason_for_judgment": "Baris identik merupakan redundant insert/duplicate ingest, aman di-deduplikasi dengan menyisakan satu instance pertama (drop exact duplicate).",
"cleaning_rule_id": "CR-01",
},
{
"finding_id": "F-02",
"dataset": "Customers",
"field_key": "customer_segment",
"finding_evidence": "Inkonsistensi nilai kategori akibat trailing whitespace dan variasi kapitalisasi ('NEW ' dan 'LOYAL ').",
"why_it_matters": "Segmentasi pelanggan terpecah menjadi kategori ganda saat aggregasi/grouping ('New' vs 'NEW ' dianggap entitas berbeda).",
"verification_method": "Inspeksi frekuensi kategori: customers_raw['customer_segment'].value_counts(dropna=False) dan trailing whitespace check.",
"verification_evidence_result": "Nilai unik: ['Loyal', 'New', 'Regular', 'NEW ', 'LOYAL ']. Ditemukan 2 varian dengan trailing space.",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Format kategori harus distandarisasi ke canonical category Title Case (.str.strip().str.title()) sesuai domain bisnis.",
"cleaning_rule_id": "CR-02",
},
{
"finding_id": "F-03",
"dataset": "Orders",
"field_key": "order_status",
"finding_evidence": "Inkonsistensi nilai status pesanan akibat trailing whitespace dan huruf besar ('COMPLETED ' sebanyak 2 transaksi).",
"why_it_matters": "Menghasilkan kategori ganda saat filter status pesanan atau kalkulasi completion rate transaksi.",
"verification_method": "Inspeksi frekuensi kategori: orders_raw['order_status'].value_counts(dropna=False).",
"verification_evidence_result": "Terdapat 2 baris bernilai 'COMPLETED ', 303 'completed', dan 34 'cancelled'.",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Distandarisasi ke bentuk canonical lowercase (.str.strip().str.lower()) menjadi 'completed'.",
"cleaning_rule_id": "CR-03",
},
{
"finding_id": "F-04",
"dataset": "Orders",
"field_key": "promo_code",
"finding_evidence": "Terdapat 189 nilai missing (NaN) pada kolom promo_code.",
"why_it_matters": "Nilai null perlu dipastikan apakah data hilang (system error) atau representasi sah transaksi tanpa voucher promo.",
"verification_method": "Filter orders_raw[orders_raw['promo_code'].isna()] dan cross-check nilai discount_amount.",
"verification_evidence_result": "Seluruh transaksi dengan promo_code = NaN memiliki discount_amount = 0. Merupakan transaksi organik tanpa promo.",
"judgment": "KEEP",
"reason_for_judgment": "Legitimate missing representation. Nilai null dipertahankan sesuai aslinya tanpa diubah sembarangan (memenuhi prinsip KEEP preservation V-15).",
"cleaning_rule_id": "",
},
{
"finding_id": "F-05",
"dataset": "Order Items",
"field_key": "quantity",
"finding_evidence": "Kolom quantity bertipe object karena terdapat string satuan ('3 pcs') di antara nilai numerik string ('1', '2', '3', '4').",
"why_it_matters": "Menghambat kalkulasi matematika langsung (agregasi kuantitas item, sum, perkalian formula) dan menyebabkan error operasional tipe data.",
"verification_method": "Pemeriksaan nilai unik: order_items_raw['quantity'].unique() dan regex check string teks non-digit.",
"verification_evidence_result": "Ditemukan nilai '3 pcs', sementara record lainnya adalah angka murni bertipe string.",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Ekstrak nilai angka numerik dan konversi tipe data kolom quantity menjadi integer (int64).",
"cleaning_rule_id": "CR-04",
},
{
"finding_id": "F-06",
"dataset": "Order Items",
"field_key": "item_category",
"finding_evidence": "Terdapat trailing whitespace dan huruf besar pada item_category ('DRINK ' dan 'MAIN ').",
"why_it_matters": "Agregasi penjualan per kategori menu terpecah menjadi kategori redundan.",
"verification_method": "Inspeksi frekuensi kategori: order_items_raw['item_category'].value_counts(dropna=False).",
"verification_evidence_result": "Ditemukan 1 baris 'DRINK ' dan 1 baris 'MAIN ' di samping kategori canonical 'Drink', 'Main', 'Snack', 'Dessert'.",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Standarisasi ke canonical Title Case (.str.strip().str.title()) agar konsisten dengan master kategori.",
"cleaning_rule_id": "CR-05",
},
{
"finding_id": "F-07",
"dataset": "Order Items",
"field_key": "line_amount",
"finding_evidence": "Formula mismatch pada 2 baris item: OI00131 (tercatat 20000, seharusnya 13000) dan OI00511 (tercatat 11000, seharusnya 16000).",
"why_it_matters": "Melanggar business rule item (line_amount = quantity * unit_price - line_discount) dan menyebabkan mismatch rekonsiliasi ke subtotal Orders.",
"verification_method": "Hitung expected_line_amount = quantity * unit_price - line_discount dan bandingkan terhadap line_amount aktual.",
"verification_evidence_result": "Ditemukan 2 baris formula mismatch: OI00131 (order O00047) dan OI00511 (order O00189). 891 baris lainnya cocok 100%.",
"judgment": "FIX",
"reason_for_judgment": "Rekalkulasi formula line_amount yang benar berdasarkan kuantitas, harga satuan, dan diskon item.",
"cleaning_rule_id": "CR-06",
},
{
"finding_id": "F-08",
"dataset": "Orders",
"field_key": "subtotal_amount, total_amount",
"finding_evidence": "Rekonsiliasi mismatch pada order O00201: subtotal di Orders tercatat 149.000, padahal total item detail adalah 139.000. Customer membayar 137.000.",
"why_it_matters": "Inkonsistensi antara nilai total header order dengan detail item pesanan dan bukti transaksi pembayaran riil.",
"verification_method": "Rekonsiliasi Orders ↔ Order Items: SUM(line_amount) O00201 = 139.000. Cross-check ke Payments (P00228): payment_amount = 137.000.",
"verification_evidence_result": "139.000 (subtotal item) + 8.000 (delivery) - 10.000 (discount) = 137.000 (cocok persis dengan pembayaran). Header Orders overstate 10.000.",
"judgment": "FIX",
"reason_for_judgment": "Koreksi subtotal_amount = 139000 dan total_amount = 137000 pada order O00201 agar selaras dengan item detail dan nominal bayar riil.",
"cleaning_rule_id": "CR-07",
},
{
"finding_id": "F-09",
"dataset": "Payments",
"field_key": "payment_method",
"finding_evidence": "Terdapat trailing whitespace dan huruf besar pada payment_method ('BANK TRANSFER ' sebanyak 2 transaksi).",
"why_it_matters": "Membagi metode pembayaran bank transfer menjadi dua kategori berbeda saat analisis payment channel share.",
"verification_method": "Inspeksi frekuensi kategori: payments_raw['payment_method'].value_counts(dropna=False).",
"verification_evidence_result": "Ditemukan 2 baris 'BANK TRANSFER ', sementara 95 baris lainnya menggunakan canonical snake_case 'bank_transfer'.",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Standarisasi ke canonical snake_case (.str.strip().str.lower().str.replace(' ', '_')) menjadi 'bank_transfer'.",
"cleaning_rule_id": "CR-08",
},
{
"finding_id": "F-10",
"dataset": "Payments",
"field_key": "order_id, attempt_number",
"finding_evidence": "Repeated order_id pada tabel Payments: 48 order memiliki 2 baris pembayaran (attempt_number 1 dan 2).",
"why_it_matters": "Berpotensi dicurigai sebagai duplicate key jika tidak memahami grain tabel pembayaran.",
"verification_method": "Groupby order_id, inspeksi payment_id, payment_status, dan attempt_number.",
"verification_evidence_result": "Seluruh payment_id 100% unik (387 unique). Transaksi berulang mencatat percobaan pembayaran gagal (failed attempt 1) lalu diulang berhasil (successful attempt 2).",
"judgment": "KEEP",
"reason_for_judgment": "Repeated order_id adalah sah secara bisnis sesuai grain pembayaran (1 baris per payment attempt). Tidak boleh dihapus (memenuhi V-05 & V-15).",
"cleaning_rule_id": "",
},
]
findings = pd.DataFrame(findings_records, columns=FINDING_COLUMNS)
display(findings)
| finding_id | dataset | field_key | finding_evidence | why_it_matters | verification_method | verification_evidence_result | judgment | reason_for_judgment | cleaning_rule_id | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | F-01 | Customers | customer_id | Duplicate customer record. customer_id = 'C003... | Melanggar grain master customer (1 baris per p... | Pengecekan duplikasi: customers_raw.duplicated... | Ditemukan tepat 2 baris identik untuk customer... | FIX | Baris identik merupakan redundant insert/dupli... | CR-01 |
| 1 | F-02 | Customers | customer_segment | Inkonsistensi nilai kategori akibat trailing w... | Segmentasi pelanggan terpecah menjadi kategori... | Inspeksi frekuensi kategori: customers_raw['cu... | Nilai unik: ['Loyal', 'New', 'Regular', 'NEW '... | STANDARDIZE | Format kategori harus distandarisasi ke canoni... | CR-02 |
| 2 | F-03 | Orders | order_status | Inkonsistensi nilai status pesanan akibat trai... | Menghasilkan kategori ganda saat filter status... | Inspeksi frekuensi kategori: orders_raw['order... | Terdapat 2 baris bernilai 'COMPLETED ', 303 'c... | STANDARDIZE | Distandarisasi ke bentuk canonical lowercase (... | CR-03 |
| 3 | F-04 | Orders | promo_code | Terdapat 189 nilai missing (NaN) pada kolom pr... | Nilai null perlu dipastikan apakah data hilang... | Filter orders_raw[orders_raw['promo_code'].isn... | Seluruh transaksi dengan promo_code = NaN memi... | KEEP | Legitimate missing representation. Nilai null ... | |
| 4 | F-05 | Order Items | quantity | Kolom quantity bertipe object karena terdapat ... | Menghambat kalkulasi matematika langsung (agre... | Pemeriksaan nilai unik: order_items_raw['quant... | Ditemukan nilai '3 pcs', sementara record lain... | STANDARDIZE | Ekstrak nilai angka numerik dan konversi tipe ... | CR-04 |
| 5 | F-06 | Order Items | item_category | Terdapat trailing whitespace dan huruf besar p... | Agregasi penjualan per kategori menu terpecah ... | Inspeksi frekuensi kategori: order_items_raw['... | Ditemukan 1 baris 'DRINK ' dan 1 baris 'MAIN '... | STANDARDIZE | Standarisasi ke canonical Title Case (.str.str... | CR-05 |
| 6 | F-07 | Order Items | line_amount | Formula mismatch pada 2 baris item: OI00131 (t... | Melanggar business rule item (line_amount = qu... | Hitung expected_line_amount = quantity * unit_... | Ditemukan 2 baris formula mismatch: OI00131 (o... | FIX | Rekalkulasi formula line_amount yang benar ber... | CR-06 |
| 7 | F-08 | Orders | subtotal_amount, total_amount | Rekonsiliasi mismatch pada order O00201: subto... | Inkonsistensi antara nilai total header order ... | Rekonsiliasi Orders ↔ Order Items: SUM(line_am... | 139.000 (subtotal item) + 8.000 (delivery) - 1... | FIX | Koreksi subtotal_amount = 139000 dan total_amo... | CR-07 |
| 8 | F-09 | Payments | payment_method | Terdapat trailing whitespace dan huruf besar p... | Membagi metode pembayaran bank transfer menjad... | Inspeksi frekuensi kategori: payments_raw['pay... | Ditemukan 2 baris 'BANK TRANSFER ', sementara ... | STANDARDIZE | Standarisasi ke canonical snake_case (.str.str... | CR-08 |
| 9 | F-10 | Payments | order_id, attempt_number | Repeated order_id pada tabel Payments: 48 orde... | Berpotensi dicurigai sebagai duplicate key jik... | Groupby order_id, inspeksi payment_id, payment... | Seluruh payment_id 100% unik (387 unique). Tra... | KEEP | Repeated order_id adalah sah secara bisnis ses... |
```python
new_finding = {
"finding_id": "F-01",
"dataset": "...",
"field_key": "...",
"finding_evidence": "...",
"why_it_matters": "...",
"verification_method": "...",
"verification_evidence_result": "...",
"judgment": "FIX / STANDARDIZE / KEEP / INVESTIGATE",
"reason_for_judgment": "...",
"cleaning_rule_id": "CR-01 atau kosong",
}
findings.loc[len(findings)] = new_finding
```
def fk_check(child_df, parent_df, child_key, parent_key, relationship_name):
child_keys = child_df[[child_key]].drop_duplicates()
parent_keys = parent_df[[parent_key]].drop_duplicates()
merged = child_keys.merge(
parent_keys,
left_on=child_key,
right_on=parent_key,
how="left",
indicator=True,
)
unmatched = merged.loc[merged["_merge"] == "left_only", child_key]
return {
"relationship": relationship_name,
"child_distinct_keys": len(child_keys),
"unmatched_count": len(unmatched),
"unmatched_sample": unmatched.head(10).tolist(),
}
if DATA_READY:
relationship_checks = pd.DataFrame([
fk_check(orders_raw, customers_raw, "customer_id", "customer_id", "Orders → Customers"),
fk_check(orders_raw, restaurants_raw, "restaurant_id", "restaurant_id", "Orders → Restaurants"),
fk_check(order_items_raw, orders_raw, "order_id", "order_id", "Order Items → Orders"),
fk_check(payments_raw, orders_raw, "order_id", "order_id", "Payments → Orders"),
])
display(relationship_checks)
| relationship | child_distinct_keys | unmatched_count | unmatched_sample | |
|---|---|---|---|---|
| 0 | Orders → Customers | 173 | 0 | [] |
| 1 | Orders → Restaurants | 24 | 0 | [] |
| 2 | Order Items → Orders | 339 | 0 | [] |
| 3 | Payments → Orders | 339 | 0 | [] |
### Order Items
`line_amount = quantity × unit_price − line_discount`
### Orders
`total_amount = subtotal_amount + delivery_fee − discount_amount`
Cell ini hanya membuat verification view. Tidak melakukan cleaning.
if DATA_READY:
item_verify = order_items_raw.copy()
item_verify["expected_line_amount"] = (
pd.to_numeric(item_verify["quantity"], errors="coerce")
* pd.to_numeric(item_verify["unit_price"], errors="coerce")
- pd.to_numeric(item_verify["line_discount"], errors="coerce")
)
item_verify["line_formula_match"] = (
pd.to_numeric(item_verify["line_amount"], errors="coerce")
== item_verify["expected_line_amount"]
)
display(
item_verify[
["order_item_id", "order_id", "quantity", "unit_price",
"line_discount", "line_amount", "expected_line_amount", "line_formula_match"]
].head()
)
order_verify = orders_raw.copy()
order_verify["expected_total_amount"] = (
pd.to_numeric(order_verify["subtotal_amount"], errors="coerce")
+ pd.to_numeric(order_verify["delivery_fee"], errors="coerce")
- pd.to_numeric(order_verify["discount_amount"], errors="coerce")
)
order_verify["order_formula_match"] = (
pd.to_numeric(order_verify["total_amount"], errors="coerce")
== order_verify["expected_total_amount"]
)
display(
order_verify[
["order_id", "subtotal_amount", "delivery_fee", "discount_amount",
"total_amount", "expected_total_amount", "order_formula_match"]
].head()
)
| order_item_id | order_id | quantity | unit_price | line_discount | line_amount | expected_line_amount | line_formula_match | |
|---|---|---|---|---|---|---|---|---|
| 0 | OI00001 | O00001 | 3 | 16000 | 2000 | 46000 | 46000.0 | True |
| 1 | OI00002 | O00001 | 3 | 22000 | 5000 | 61000 | 61000.0 | True |
| 2 | OI00003 | O00001 | 1 | 42000 | 0 | 42000 | 42000.0 | True |
| 3 | OI00004 | O00002 | 3 | 28000 | 5000 | 79000 | 79000.0 | True |
| 4 | OI00005 | O00002 | 2 | 28000 | 2000 | 54000 | 54000.0 | True |
| order_id | subtotal_amount | delivery_fee | discount_amount | total_amount | expected_total_amount | order_formula_match | |
|---|---|---|---|---|---|---|---|
| 0 | O00001 | 149000 | 18000 | 10000 | 157000 | 157000 | True |
| 1 | O00002 | 133000 | 12000 | 10000 | 135000 | 135000 | True |
| 2 | O00003 | 45000 | 18000 | 15000 | 48000 | 48000 | True |
| 3 | O00004 | 178000 | 12000 | 0 | 190000 | 190000 | True |
| 4 | O00005 | 248000 | 8000 | 15000 | 241000 | 241000 | True |
Lakukan reconciliation setelah memahami item-level validity. Jangan langsung mengoreksi Orders jika item detail sendiri belum diverifikasi.
if DATA_READY:
item_amount_numeric = pd.to_numeric(order_items_raw["line_amount"], errors="coerce")
item_agg_raw = (
order_items_raw.assign(_line_amount_numeric=item_amount_numeric)
.groupby("order_id", as_index=False)["_line_amount_numeric"]
.sum(min_count=1)
.rename(columns={"_line_amount_numeric": "sum_line_amount"})
)
order_item_recon_raw = orders_raw[
["order_id", "subtotal_amount"]
].merge(item_agg_raw, on="order_id", how="left")
order_item_recon_raw["subtotal_numeric"] = pd.to_numeric(
order_item_recon_raw["subtotal_amount"], errors="coerce"
)
order_item_recon_raw["reconcile_match"] = (
order_item_recon_raw["subtotal_numeric"]
== order_item_recon_raw["sum_line_amount"]
)
display(order_item_recon_raw.head())
print(
"Raw reconciliation mismatch count:",
int((~order_item_recon_raw["reconcile_match"]).sum())
)
| order_id | subtotal_amount | sum_line_amount | subtotal_numeric | reconcile_match | |
|---|---|---|---|---|---|
| 0 | O00001 | 149000 | 149000 | 149000 | True |
| 1 | O00002 | 133000 | 133000 | 133000 | True |
| 2 | O00003 | 45000 | 45000 | 45000 | True |
| 3 | O00004 | 178000 | 178000 | 178000 | True |
| 4 | O00005 | 248000 | 248000 | 248000 | True |
Raw reconciliation mismatch count: 3
Rulebook dibuat setelah verification dan sebelum coding.
Gunakan hanya `FIX` atau `STANDARDIZE` sebagai cleaning decision. Finding `KEEP` tidak boleh diubah hanya agar data terlihat lebih rapi.
RULE_COLUMNS = [
"rule_id",
"source_finding_id",
"dataset",
"trigger_condition",
"decision",
"cleaning_action",
"reason_evidence",
"implementation_note",
"validation_criteria",
"rule_status",
]
rules_records = [
{
"rule_id": "CR-01",
"source_finding_id": "F-01",
"dataset": "Customers",
"trigger_condition": "Duplicate record pada key customer_id = 'C0031'.",
"decision": "FIX",
"cleaning_action": "Hapus baris duplikat kedua menggunakan .drop_duplicates(subset=['customer_id'], keep='first').",
"reason_evidence": "Baris index 30 dan index 173 identik di semua kolom. Memenuhi grain 1 baris per pelanggan unik.",
"implementation_note": "Terapkan hanya pada customers_clean. Pertahankan instance pertama.",
"validation_criteria": "Total row berkurang dari 174 menjadi 173. Duplikat key = 0 (candidate_key_unique = True).",
"rule_status": "Approved",
},
{
"rule_id": "CR-02",
"source_finding_id": "F-02",
"dataset": "Customers",
"trigger_condition": "Trailing whitespace dan variasi kapitalisasi pada customer_segment ('NEW ', 'LOYAL ').",
"decision": "STANDARDIZE",
"cleaning_action": "Bersihkan spasi dan ubah ke Title Case menggunakan .str.strip().str.title().",
"reason_evidence": "Menyatukan kategori segmen terfragmentasi ke domain kanonikal ('Loyal', 'New', 'Regular').",
"implementation_note": "Terapkan pada seluruh kolom customer_segment di customers_clean.",
"validation_criteria": "Hanya ada 3 kategori valid: Regular (68), New (57), Loyal (48).",
"rule_status": "Approved",
},
{
"rule_id": "CR-03",
"source_finding_id": "F-03",
"dataset": "Orders",
"trigger_condition": "Trailing whitespace dan huruf besar pada order_status ('COMPLETED ').",
"decision": "STANDARDIZE",
"cleaning_action": "Bersihkan spasi dan ubah ke lowercase menggunakan .str.strip().str.lower().",
"reason_evidence": "Menyeragamkan status transaksi agar tidak terbagi menjadi kategori redundan.",
"implementation_note": "Terapkan pada orders_clean['order_status'].",
"validation_criteria": "Hanya ada 2 kategori valid: completed (305 transaksi) dan cancelled (34 transaksi).",
"rule_status": "Approved",
},
{
"rule_id": "CR-04",
"source_finding_id": "F-05",
"dataset": "Order Items",
"trigger_condition": "Terdapat string teks satuan ('3 pcs') pada kolom quantity.",
"decision": "STANDARDIZE",
"cleaning_action": "Ekstrak karakter digit numerik dan konversi tipe data kolom quantity menjadi integer (int64).",
"reason_evidence": "Memungkinkan operasi aritmetika dan kalkulasi formula pada baris item pesanan.",
"implementation_note": "order_items_clean['quantity'].astype(str).str.extract(r'(\\d+)')[0].astype(int).",
"validation_criteria": "Tidak ada string teks 'pcs'; tipe data menjadi integer numerik positif.",
"rule_status": "Approved",
},
{
"rule_id": "CR-05",
"source_finding_id": "F-06",
"dataset": "Order Items",
"trigger_condition": "Trailing whitespace dan huruf besar pada item_category ('DRINK ', 'MAIN ').",
"decision": "STANDARDIZE",
"cleaning_action": "Bersihkan spasi dan ubah ke Title Case menggunakan .str.strip().str.title().",
"reason_evidence": "Menyeragamkan kategori menu makanan dan minuman sesuai master kategori.",
"implementation_note": "Terapkan pada order_items_clean['item_category'].",
"validation_criteria": "Hanya terdapat 4 kategori kanonikal: Main (371), Drink (275), Snack (125), Dessert (122).",
"rule_status": "Approved",
},
{
"rule_id": "CR-06",
"source_finding_id": "F-07",
"dataset": "Order Items",
"trigger_condition": "Mismatch formula line_amount != quantity * unit_price - line_discount pada OI00131 dan OI00511.",
"decision": "FIX",
"cleaning_action": "Hitung ulang nilai kolom line_amount = quantity * unit_price - line_discount.",
"reason_evidence": "Error pencatatan formula line_amount pada data mentah. Memenuhi business rule item V-09.",
"implementation_note": "order_items_clean['line_amount'] = order_items_clean['quantity'] * order_items_clean['unit_price'] - order_items_clean['line_discount'].",
"validation_criteria": "100% baris item (893 baris) memenuhi formula item (0 mismatch).",
"rule_status": "Approved",
},
{
"rule_id": "CR-07",
"source_finding_id": "F-08",
"dataset": "Orders",
"trigger_condition": "Rekonsiliasi mismatch pada order O00201 (subtotal_amount = 149000 vs item detail 139000).",
"decision": "FIX",
"cleaning_action": "Koreksi subtotal_amount = 139000 dan total_amount = 137000 pada order O00201.",
"reason_evidence": "Item detail bernilai 139.000 dan customer membayar 137.000 (P00228). Subtotal Orders overstate 10.000.",
"implementation_note": "Update baris order O00201 pada orders_clean.",
"validation_criteria": "Rekonsiliasi Orders ↔ Order Items untuk seluruh 339 order cocok 100% (unmatched = 0).",
"rule_status": "Approved",
},
{
"rule_id": "CR-08",
"source_finding_id": "F-09",
"dataset": "Payments",
"trigger_condition": "Trailing whitespace dan huruf besar pada payment_method ('BANK TRANSFER ').",
"decision": "STANDARDIZE",
"cleaning_action": "Bersihkan spasi, lowercase, dan ubah spasi menjadi underscore ('bank_transfer').",
"reason_evidence": "Menyeragamkan penamaan metode pembayaran sesuai canonical snake_case values.",
"implementation_note": "payments_clean['payment_method'].str.strip().str.lower().str.replace(' ', '_').",
"validation_criteria": "Hanya ada 4 metode pembayaran: e_wallet (98), bank_transfer (97), credit_card (97), cash (95).",
"rule_status": "Approved",
},
]
cleaning_rules = pd.DataFrame(rules_records, columns=RULE_COLUMNS)
display(cleaning_rules)
| rule_id | source_finding_id | dataset | trigger_condition | decision | cleaning_action | reason_evidence | implementation_note | validation_criteria | rule_status | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | CR-01 | F-01 | Customers | Duplicate record pada key customer_id = 'C0031'. | FIX | Hapus baris duplikat kedua menggunakan .drop_d... | Baris index 30 dan index 173 identik di semua ... | Terapkan hanya pada customers_clean. Pertahank... | Total row berkurang dari 174 menjadi 173. Dupl... | Approved |
| 1 | CR-02 | F-02 | Customers | Trailing whitespace dan variasi kapitalisasi p... | STANDARDIZE | Bersihkan spasi dan ubah ke Title Case menggun... | Menyatukan kategori segmen terfragmentasi ke d... | Terapkan pada seluruh kolom customer_segment d... | Hanya ada 3 kategori valid: Regular (68), New ... | Approved |
| 2 | CR-03 | F-03 | Orders | Trailing whitespace dan huruf besar pada order... | STANDARDIZE | Bersihkan spasi dan ubah ke lowercase mengguna... | Menyeragamkan status transaksi agar tidak terb... | Terapkan pada orders_clean['order_status']. | Hanya ada 2 kategori valid: completed (305 tra... | Approved |
| 3 | CR-04 | F-05 | Order Items | Terdapat string teks satuan ('3 pcs') pada kol... | STANDARDIZE | Ekstrak karakter digit numerik dan konversi ti... | Memungkinkan operasi aritmetika dan kalkulasi ... | order_items_clean['quantity'].astype(str).str.... | Tidak ada string teks 'pcs'; tipe data menjadi... | Approved |
| 4 | CR-05 | F-06 | Order Items | Trailing whitespace dan huruf besar pada item_... | STANDARDIZE | Bersihkan spasi dan ubah ke Title Case menggun... | Menyeragamkan kategori menu makanan dan minuma... | Terapkan pada order_items_clean['item_category']. | Hanya terdapat 4 kategori kanonikal: Main (371... | Approved |
| 5 | CR-06 | F-07 | Order Items | Mismatch formula line_amount != quantity * uni... | FIX | Hitung ulang nilai kolom line_amount = quantit... | Error pencatatan formula line_amount pada data... | order_items_clean['line_amount'] = order_items... | 100% baris item (893 baris) memenuhi formula i... | Approved |
| 6 | CR-07 | F-08 | Orders | Rekonsiliasi mismatch pada order O00201 (subto... | FIX | Koreksi subtotal_amount = 139000 dan total_amo... | Item detail bernilai 139.000 dan customer memb... | Update baris order O00201 pada orders_clean. | Rekonsiliasi Orders ↔ Order Items untuk seluru... | Approved |
| 7 | CR-08 | F-09 | Payments | Trailing whitespace dan huruf besar pada payme... | STANDARDIZE | Bersihkan spasi, lowercase, dan ubah spasi men... | Menyeragamkan penamaan metode pembayaran sesua... | payments_clean['payment_method'].str.strip().s... | Hanya ada 4 metode pembayaran: e_wallet (98), ... | Approved |
```python
new_rule = {
"rule_id": "CR-01",
"source_finding_id": "F-01",
"dataset": "...",
"trigger_condition": "...",
"decision": "FIX / STANDARDIZE",
"cleaning_action": "...",
"reason_evidence": "...",
"implementation_note": "...",
"validation_criteria": "...",
"rule_status": "Approved",
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
```
if DATA_READY:
customers_clean = customers_raw.copy(deep=True)
orders_clean = orders_raw.copy(deep=True)
order_items_clean = order_items_raw.copy(deep=True)
payments_clean = payments_raw.copy(deep=True)
restaurants_clean = restaurants_raw.copy(deep=True)
print("Working copies created.")
Working copies created.
**Edit hanya cell Cleaning Code di bawah ini.**
Aturan:
- implementasikan hanya rule yang sudah kamu setujui di Working Paper;
- gunakan `customers_clean`, `orders_clean`, `order_items_clean`, `payments_clean`, atau `restaurants_clean`;
- jangan pernah overwrite object dengan suffix `_raw`;
- beri komentar `CR-xx` untuk setiap rule;
- setelah implementasi, catat before/after pada Cleaning Log.
Starter sengaja **tidak memberikan affected record ID atau jawaban** agar evidence tetap berasal dari variant kamu.
# ============================================================
# EDIT AREA 2 — APPROVED CLEANING CODE
# ============================================================
# Tambahkan code per Rule ID di bawah ini.
#
# Gunakan hanya dataframe *_clean.
# JANGAN ubah dataframe *_raw.
# CR-01 — Deduplikasi master customer (drop exact duplicate)
# Source Finding: F-01
# Evidence basis: customer_id = 'C0031' duplikat exact row (baris 30 dan 173).
# Validation criteria: Total baris berkurang dari 174 menjadi 173; duplicate key = 0.
customers_clean = customers_clean.drop_duplicates(subset=["customer_id"], keep="first").reset_index(drop=True)
# CR-02 — Standarisasi customer_segment
# Source Finding: F-02
# Evidence basis: Terdapat trailing whitespace dan kapitalisasi ('NEW ', 'LOYAL ').
# Validation criteria: Hanya 3 canonical values: 'Regular', 'New', 'Loyal'.
customers_clean["customer_segment"] = (
customers_clean["customer_segment"].astype(str).str.strip().str.title()
)
# CR-03 — Standarisasi order_status
# Source Finding: F-03
# Evidence basis: Terdapat trailing whitespace dan uppercase ('COMPLETED ').
# Validation criteria: Hanya 2 canonical values: 'completed' (305) dan 'cancelled' (34).
orders_clean["order_status"] = (
orders_clean["order_status"].astype(str).str.strip().str.lower()
)
# CR-04 — Standarisasi quantity pada Order Items
# Source Finding: F-05
# Evidence basis: Terdapat string satuan teks ('3 pcs') pada kolom quantity.
# Validation criteria: Tipe data integer numerik int64, 0 teks non-angka.
order_items_clean["quantity"] = (
order_items_clean["quantity"].astype(str).str.extract(r"(\d+)")[0].astype(int)
)
# CR-05 — Standarisasi item_category pada Order Items
# Source Finding: F-06
# Evidence basis: Terdapat trailing whitespace dan uppercase ('DRINK ', 'MAIN ').
# Validation criteria: Hanya 4 canonical values: 'Main', 'Drink', 'Snack', 'Dessert'.
order_items_clean["item_category"] = (
order_items_clean["item_category"].astype(str).str.strip().str.title()
)
# CR-06 — Fix formula line_amount pada Order Items
# Source Finding: F-07
# Evidence basis: Baris item OI00131 dan OI00511 formula mismatch terhadap harga, kuantitas, dan diskon.
# Validation criteria: 100% baris item (893 baris) memenuhi line_amount == quantity * unit_price - line_discount.
order_items_clean["line_amount"] = (
order_items_clean["quantity"] * order_items_clean["unit_price"] - order_items_clean["line_discount"]
)
# CR-07 — Koreksi subtotal_amount dan total_amount order O00201
# Source Finding: F-08
# Evidence basis: Rekonsiliasi O00201 overstate 10.000 (subtotal tercatat 149.000 vs item 139.000; payment 137.000).
# Validation criteria: Rekonsiliasi Orders ↔ Order Items cocok 100% (0 mismatch).
item_sum_201 = int(order_items_clean.loc[order_items_clean["order_id"] == "O00201", "line_amount"].sum())
orders_clean.loc[orders_clean["order_id"] == "O00201", "subtotal_amount"] = item_sum_201
orders_clean.loc[orders_clean["order_id"] == "O00201", "total_amount"] = (
item_sum_201
+ orders_clean.loc[orders_clean["order_id"] == "O00201", "delivery_fee"]
- orders_clean.loc[orders_clean["order_id"] == "O00201", "discount_amount"]
)
# CR-08 — Standarisasi payment_method
# Source Finding: F-09
# Evidence basis: Terdapat trailing whitespace dan uppercase ('BANK TRANSFER ').
# Validation criteria: Hanya 4 canonical snake_case values: 'e_wallet', 'bank_transfer', 'credit_card', 'cash'.
payments_clean["payment_method"] = (
payments_clean["payment_method"].astype(str).str.strip().str.lower().str.replace(" ", "_")
)
print("Approved cleaning rules CR-01 to CR-08 successfully executed.")
Approved cleaning rules CR-01 to CR-08 successfully executed.
CLEANING_LOG_COLUMNS = [
"log_id",
"rule_id",
"dataset",
"affected_records_scope",
"action_performed",
"before_evidence",
"after_evidence",
"rows_records_affected",
"result",
"notes",
]
cleaning_log_records = [
{
"log_id": "CL-01",
"rule_id": "CR-01",
"dataset": "Customers",
"affected_records_scope": "customer_id = 'C0031' (Farah Saputra, baris 30 & 173)",
"action_performed": "Deduplikasi record dengan menghapus instance kedua (.drop_duplicates(keep='first')).",
"before_evidence": "Total 174 baris, ditemukan 2 baris duplikat untuk C0031 (candidate_key_unique = False).",
"after_evidence": "Total 173 baris unik, 0 duplicate key (candidate_key_unique = True).",
"rows_records_affected": "1 baris duplikat dihapus",
"result": "Implemented",
"notes": "Grain 1 baris per customer terpenuhi.",
},
{
"log_id": "CL-02",
"rule_id": "CR-02",
"dataset": "Customers",
"affected_records_scope": "Kolom customer_segment (nilai 'NEW ' dan 'LOYAL ')",
"action_performed": "Strip trailing whitespace dan standarisasi ke Title Case (.str.strip().str.title()).",
"before_evidence": "5 variasi kategori: Regular (68), New (56), Loyal (47), NEW (1), LOYAL (1).",
"after_evidence": "3 kategori kanonikal: Regular (68), New (57), Loyal (48).",
"rows_records_affected": "2 baris distandarisasi",
"result": "Implemented",
"notes": "Domain kategori segmen konsisten.",
},
{
"log_id": "CL-03",
"rule_id": "CR-03",
"dataset": "Orders",
"affected_records_scope": "Kolom order_status (nilai 'COMPLETED ')",
"action_performed": "Strip trailing whitespace dan standarisasi ke lowercase (.str.strip().str.lower()).",
"before_evidence": "3 variasi kategori: completed (303), cancelled (34), COMPLETED (2).",
"after_evidence": "2 kategori kanonikal: completed (305), cancelled (34).",
"rows_records_affected": "2 baris distandarisasi",
"result": "Implemented",
"notes": "Domain kategori status order seragam.",
},
{
"log_id": "CL-04",
"rule_id": "CR-04",
"dataset": "Order Items",
"affected_records_scope": "Kolom quantity (record mengandung '3 pcs')",
"action_performed": "Ekstrak digit numerik dan konversi tipe data kolom quantity menjadi int64.",
"before_evidence": "Tipe data object, 2 baris bernilai string '3 pcs'.",
"after_evidence": "Tipe data int64, 0 teks satuan, nilai string terkonversi menjadi integer 3.",
"rows_records_affected": "2 baris dikonversi teksnya (893 baris bertipe integer)",
"result": "Implemented",
"notes": "Siap untuk kalkulasi aritmetika dan perkalian formula.",
},
{
"log_id": "CL-05",
"rule_id": "CR-05",
"dataset": "Order Items",
"affected_records_scope": "Kolom item_category (nilai 'DRINK ' dan 'MAIN ')",
"action_performed": "Strip trailing whitespace dan standarisasi ke Title Case (.str.strip().str.title()).",
"before_evidence": "6 variasi kategori: Main (370), Drink (274), Snack (125), Dessert (122), DRINK (1), MAIN (1).",
"after_evidence": "4 kategori kanonikal: Main (371), Drink (275), Snack (125), Dessert (122).",
"rows_records_affected": "2 baris distandarisasi",
"result": "Implemented",
"notes": "Kategori item menu tersinkronisasi bersih.",
},
{
"log_id": "CL-06",
"rule_id": "CR-06",
"dataset": "Order Items",
"affected_records_scope": "Baris item OI00131 (order O00047) dan OI00511 (order O00189)",
"action_performed": "Rekalkulasi nilai formula line_amount = quantity * unit_price - line_discount.",
"before_evidence": "OI00131 tercatat 20000 (harus 13000); OI00511 tercatat 11000 (harus 16000). Total 2 mismatch.",
"after_evidence": "OI00131 = 13000; OI00511 = 16000. Seluruh 893 baris item memenuhi formula (0 mismatch).",
"rows_records_affected": "2 baris dikoreksi nilainya",
"result": "Implemented",
"notes": "Business rule formula item V-09 PASS 100%.",
},
{
"log_id": "CL-07",
"rule_id": "CR-07",
"dataset": "Orders",
"affected_records_scope": "Order O00201 pada dataset Orders",
"action_performed": "Koreksi nilai subtotal_amount menjadi 139.000 dan total_amount menjadi 137.000.",
"before_evidence": "subtotal_amount = 149000, total_amount = 147000 (overstate 10.000 vs item 139k & payment 137k).",
"after_evidence": "subtotal_amount = 139000, total_amount = 137000 (cocok persis dengan detail item & payment).",
"rows_records_affected": "1 baris order dikoreksi",
"result": "Implemented",
"notes": "Rekonsiliasi Orders ↔ Order Items cocok 100% (0 mismatch).",
},
{
"log_id": "CL-08",
"rule_id": "CR-08",
"dataset": "Payments",
"affected_records_scope": "Kolom payment_method (nilai 'BANK TRANSFER ')",
"action_performed": "Strip trailing whitespace, lowercase, dan ubah spasi menjadi underscore ('bank_transfer').",
"before_evidence": "Terdapat 2 baris 'BANK TRANSFER ' di samping 95 baris 'bank_transfer'.",
"after_evidence": "Seluruh 97 transaksi bank transfer berformat kanonikal 'bank_transfer'.",
"rows_records_affected": "2 baris distandarisasi",
"result": "Implemented",
"notes": "Metode pembayaran kanonikal: e_wallet (98), bank_transfer (97), credit_card (97), cash (95).",
},
]
cleaning_log = pd.DataFrame(cleaning_log_records, columns=CLEANING_LOG_COLUMNS)
display(cleaning_log)
| log_id | rule_id | dataset | affected_records_scope | action_performed | before_evidence | after_evidence | rows_records_affected | result | notes | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | CL-01 | CR-01 | Customers | customer_id = 'C0031' (Farah Saputra, baris 30... | Deduplikasi record dengan menghapus instance k... | Total 174 baris, ditemukan 2 baris duplikat un... | Total 173 baris unik, 0 duplicate key (candida... | 1 baris duplikat dihapus | Implemented | Grain 1 baris per customer terpenuhi. |
| 1 | CL-02 | CR-02 | Customers | Kolom customer_segment (nilai 'NEW ' dan 'LOYA... | Strip trailing whitespace dan standarisasi ke ... | 5 variasi kategori: Regular (68), New (56), Lo... | 3 kategori kanonikal: Regular (68), New (57), ... | 2 baris distandarisasi | Implemented | Domain kategori segmen konsisten. |
| 2 | CL-03 | CR-03 | Orders | Kolom order_status (nilai 'COMPLETED ') | Strip trailing whitespace dan standarisasi ke ... | 3 variasi kategori: completed (303), cancelled... | 2 kategori kanonikal: completed (305), cancell... | 2 baris distandarisasi | Implemented | Domain kategori status order seragam. |
| 3 | CL-04 | CR-04 | Order Items | Kolom quantity (record mengandung '3 pcs') | Ekstrak digit numerik dan konversi tipe data k... | Tipe data object, 2 baris bernilai string '3 p... | Tipe data int64, 0 teks satuan, nilai string t... | 2 baris dikonversi teksnya (893 baris bertipe ... | Implemented | Siap untuk kalkulasi aritmetika dan perkalian ... |
| 4 | CL-05 | CR-05 | Order Items | Kolom item_category (nilai 'DRINK ' dan 'MAIN ') | Strip trailing whitespace dan standarisasi ke ... | 6 variasi kategori: Main (370), Drink (274), S... | 4 kategori kanonikal: Main (371), Drink (275),... | 2 baris distandarisasi | Implemented | Kategori item menu tersinkronisasi bersih. |
| 5 | CL-06 | CR-06 | Order Items | Baris item OI00131 (order O00047) dan OI00511 ... | Rekalkulasi nilai formula line_amount = quanti... | OI00131 tercatat 20000 (harus 13000); OI00511 ... | OI00131 = 13000; OI00511 = 16000. Seluruh 893 ... | 2 baris dikoreksi nilainya | Implemented | Business rule formula item V-09 PASS 100%. |
| 6 | CL-07 | CR-07 | Orders | Order O00201 pada dataset Orders | Koreksi nilai subtotal_amount menjadi 139.000 ... | subtotal_amount = 149000, total_amount = 14700... | subtotal_amount = 139000, total_amount = 13700... | 1 baris order dikoreksi | Implemented | Rekonsiliasi Orders ↔ Order Items cocok 100% (... |
| 7 | CL-08 | CR-08 | Payments | Kolom payment_method (nilai 'BANK TRANSFER ') | Strip trailing whitespace, lowercase, dan ubah... | Terdapat 2 baris 'BANK TRANSFER ' di samping 9... | Seluruh 97 transaksi bank transfer berformat k... | 2 baris distandarisasi | Implemented | Metode pembayaran kanonikal: e_wallet (98), ba... |
Validation harus membuktikan hasil cleaning memenuhi **15 check pada Working Paper**:
`V-01` sampai `V-15`
Starter menyediakan fungsi `record_check()`. Kamu tetap harus menentukan evidence dan hasil check berdasarkan variant kamu.
> `FINAL_GATE` tidak akan PASS jika check wajib belum lengkap.
validation_results = []
REQUIRED_VALIDATION_IDS = [f"V-{i:02d}" for i in range(1, 16)]
def record_check(check_id, level, dataset_relationship, question,
expected_criteria, actual_result, passed,
related_rule_id="", notes=""):
if check_id not in REQUIRED_VALIDATION_IDS:
print(f"WARNING — {check_id} bukan ID validation V-01 s.d. V-15.")
# Jika cell di-run ulang, replace check dengan ID yang sama agar tidak double.
validation_results[:] = [
row for row in validation_results if row["check_id"] != check_id
]
validation_results.append({
"check_id": check_id,
"validation_level": level,
"dataset_relationship": dataset_relationship,
"validation_question": question,
"expected_criteria": expected_criteria,
"actual_result_metric": actual_result,
"status": "PASS" if bool(passed) else "FAIL",
"related_rule_id": related_rule_id,
"notes": notes,
})
print("VALIDATION HARNESS READY")
print("Required IDs:", ", ".join(REQUIRED_VALIDATION_IDS))
VALIDATION HARNESS READY Required IDs: V-01, V-02, V-03, V-04, V-05, V-06, V-07, V-08, V-09, V-10, V-11, V-12, V-13, V-14, V-15
Gunakan area ini untuk mencatat check seperti `V-01` sampai `V-05` sesuai Working Paper.
Kamu boleh memakai kembali helper `key_check()` dari bagian sebelumnya. Jangan mengganti nilai menjadi PASS tanpa evidence.
if DATA_READY:
clean_tables = {
"Customers": customers_clean,
"Orders": orders_clean,
"Order Items": order_items_clean,
"Payments": payments_clean,
"Restaurants": restaurants_clean,
}
print("Clean dataset shapes:")
for name, df in clean_tables.items():
print(name, df.shape)
# ========================================================
# EDIT AREA 3A — V-01 s.d. V-05
# ========================================================
# V-01: Schema Check
schema_ok = True
schema_details = []
for label, required in REQUIRED_COLUMNS.items():
actual = set(clean_tables[label].columns)
missing = required - actual
if missing:
schema_ok = False
schema_details.append(f"{label}: {len(actual)} cols (0 missing)")
record_check(
"V-01", "Schema", "All clean datasets",
"Apakah seluruh required column masih tersedia?",
"Required columns available",
f"Seluruh 5 clean datasets memiliki required columns lengkap: {'; '.join(schema_details)}.",
schema_ok,
notes="Schema utuh pada seluruh 5 dataset.",
)
# V-02: Key Check Customers
cust_key = key_check(customers_clean, "customer_id", "Customers")
v2_passed = bool(cust_key.loc[0, "candidate_key_unique"])
record_check(
"V-02", "Key", "Customers",
"Apakah customer_id valid sesuai grain customer?",
"No null key; uniqueness sesuai grain",
f"Total {cust_key.loc[0, 'rows']} baris, unique key: {cust_key.loc[0, 'unique_key_count']}, duplicate: {cust_key.loc[0, 'duplicate_key_rows']}, null: {cust_key.loc[0, 'missing_key_rows']}.",
v2_passed,
related_rule_id="CR-01",
notes="Grain 1 baris per customer terpenuhi.",
)
# V-03: Key Check Orders
orders_key = key_check(orders_clean, "order_id", "Orders")
v3_passed = bool(orders_key.loc[0, "candidate_key_unique"])
record_check(
"V-03", "Key", "Orders",
"Apakah order_id valid sesuai grain order?",
"No null key; one row per order",
f"Total {orders_key.loc[0, 'rows']} baris, unique key: {orders_key.loc[0, 'unique_key_count']}, duplicate: {orders_key.loc[0, 'duplicate_key_rows']}, null: {orders_key.loc[0, 'missing_key_rows']}.",
v3_passed,
related_rule_id="CR-03",
notes="Grain 1 baris per order terpenuhi.",
)
# V-04: Key Check Order Items
oi_key = key_check(order_items_clean, "order_item_id", "Order Items")
v4_passed = bool(oi_key.loc[0, "candidate_key_unique"])
record_check(
"V-04", "Key", "Order Items",
"Apakah order_item_id / candidate business key valid?",
"No invalid duplicate key",
f"Total {oi_key.loc[0, 'rows']} baris, unique key: {oi_key.loc[0, 'unique_key_count']}, duplicate: {oi_key.loc[0, 'duplicate_key_rows']}, null: {oi_key.loc[0, 'missing_key_rows']}.",
v4_passed,
related_rule_id="CR-04",
notes="Surrogate key order_item_id unik 100%.",
)
# V-05: Key Check Payments
pay_key = key_check(payments_clean, "payment_id", "Payments")
repeated_orders_count = int(payments_clean["order_id"].duplicated().sum())
v5_passed = bool(pay_key.loc[0, "candidate_key_unique"]) and repeated_orders_count == 48
record_check(
"V-05", "Key", "Payments",
"Apakah payment_id unique tanpa menghapus repeated order_id yang valid?",
"payment_id unique; repeated order_id allowed by grain",
f"Payment ID unik: {pay_key.loc[0, 'unique_key_count']} dari {pay_key.loc[0, 'rows']} baris (0 duplicate). Repeated order_id sah: {repeated_orders_count} baris pembayaran multi-attempt.",
v5_passed,
related_rule_id="CR-08",
notes="Grain payment transaksi dipertahankan dengan benar.",
)
print("V-01 s.d. V-05 recorded successfully.")
else:
print("VALIDATION SKIPPED — DATA READY GATE belum PASS.")
Clean dataset shapes: Customers (173, 7) Orders (339, 10) Order Items (893, 9) Payments (387, 7) Restaurants (24, 7) V-01 s.d. V-05 recorded successfully.
Gunakan area ini untuk `V-06` sampai `V-14` sesuai Working Paper.
Manfaatkan helper yang sudah tersedia seperti `category_review()`, `fk_check()`, dan perhitungan formula. **Hitung ulang dari clean data**, bukan hanya melihat output raw.
if DATA_READY:
# ========================================================
# EDIT AREA 3B — V-06 s.d. V-14
# ========================================================
# V-06: Value consistency Customers
cust_seg = customers_clean["customer_segment"].value_counts()
expected_segs = {"Regular", "New", "Loyal"}
v6_passed = set(cust_seg.index) == expected_segs
record_check(
"V-06", "Value", "Customers / categories",
"Apakah category dan missing representation sudah konsisten?",
"No unexpected canonical-category issue",
f"Kategori customer_segment kanonikal: {dict(cust_seg)}. Tidak ada trailing whitespace / casing anomali.",
v6_passed,
related_rule_id="CR-02",
notes="Nilai kategori terstandarisasi Title Case.",
)
# V-07: Value consistency Orders
ord_stat = orders_clean["order_status"].value_counts()
expected_stat = {"completed", "cancelled"}
v7_passed = set(ord_stat.index) == expected_stat
record_check(
"V-07", "Value", "Orders / categories",
"Apakah order status dan representation sudah konsisten?",
"Only approved canonical values",
f"Kategori order_status kanonikal: {dict(ord_stat)}. Tidak ada whitespace atau casing non-standar.",
v7_passed,
related_rule_id="CR-03",
notes="Status order konsisten lowercase.",
)
# V-08: Value consistency Payments / Restaurants
pay_meth = payments_clean["payment_method"].value_counts()
expected_pay_meth = {"e_wallet", "bank_transfer", "credit_card", "cash"}
item_cat = order_items_clean["item_category"].value_counts()
expected_item_cat = {"Main", "Drink", "Snack", "Dessert"}
v8_passed = (set(pay_meth.index) == expected_pay_meth) and (set(item_cat.index) == expected_item_cat)
record_check(
"V-08", "Value", "Payments / Restaurants",
"Apakah category representation sudah konsisten?",
"Only approved canonical values",
f"Metode bayar kanonikal: {dict(pay_meth)}; Kategori item kanonikal: {dict(item_cat)}.",
v8_passed,
related_rule_id="CR-05, CR-08",
notes="Kategori payment method dan menu item tersinkronisasi bersih.",
)
# V-09: Business Rule Order Items formula
calc_line = order_items_clean["quantity"] * order_items_clean["unit_price"] - order_items_clean["line_discount"]
oi_diff = int((calc_line != order_items_clean["line_amount"]).sum())
v9_passed = oi_diff == 0
record_check(
"V-09", "Business Rule", "Order Items",
"Apakah line_amount memenuhi formula item?",
"quantity × unit_price - line_discount = line_amount",
f"Formula line_amount valid 100% pada seluruh {len(order_items_clean)} baris (0 mismatch).",
v9_passed,
related_rule_id="CR-06",
notes="Nilai formula pada OI00131 dan OI00511 telah dikoreksi.",
)
# V-10: Business Rule Orders formula
calc_tot = orders_clean["subtotal_amount"] + orders_clean["delivery_fee"] - orders_clean["discount_amount"]
ord_diff = int((calc_tot != orders_clean["total_amount"]).sum())
v10_passed = ord_diff == 0
record_check(
"V-10", "Business Rule", "Orders",
"Apakah total_amount memenuhi formula order?",
"subtotal + delivery_fee - discount_amount = total_amount",
f"Formula total_amount valid 100% pada seluruh {len(orders_clean)} baris (0 mismatch).",
v10_passed,
related_rule_id="CR-07",
notes="Total order sesuai penjumlahan subtotal + ongkir - diskon.",
)
# V-11: Cross-table Orders ↔ Order Items reconciliation
oi_sum = order_items_clean.groupby("order_id")["line_amount"].sum().reset_index()
oi_sum.rename(columns={"line_amount": "sum_line_amount"}, inplace=True)
reconcile = orders_clean[["order_id", "subtotal_amount"]].merge(oi_sum, on="order_id", how="left")
reconcile_diff = int((reconcile["subtotal_amount"] != reconcile["sum_line_amount"]).sum())
v11_passed = reconcile_diff == 0
record_check(
"V-11", "Cross-table", "Orders ↔ Order Items",
"Apakah subtotal order konsisten dengan detail item?",
"SUM(line_amount) = subtotal_amount",
f"Rekonsiliasi subtotal_amount vs SUM(line_amount) cocok 100% pada seluruh {len(orders_clean)} order (0 mismatch).",
v11_passed,
related_rule_id="CR-06, CR-07",
notes="Order O00201 telah diselaraskan dengan total item.",
)
# V-12: Cross-table Orders → Customers FK
fk_cust = fk_check(orders_clean, customers_clean, "customer_id", "customer_id", "Orders → Customers")
v12_passed = fk_cust["unmatched_count"] == 0
record_check(
"V-12", "Cross-table", "Orders → Customers",
"Apakah seluruh customer reference valid?",
"Unmatched customer = 0",
f"Unmatched customer_id: {fk_cust['unmatched_count']} dari {fk_cust['child_distinct_keys']} distinct customer di orders.",
v12_passed,
notes="Integritas referensial Orders → Customers valid 100%.",
)
# V-13: Cross-table Orders → Restaurants FK
fk_rest = fk_check(orders_clean, restaurants_clean, "restaurant_id", "restaurant_id", "Orders → Restaurants")
v13_passed = fk_rest["unmatched_count"] == 0
record_check(
"V-13", "Cross-table", "Orders → Restaurants",
"Apakah seluruh restaurant reference valid?",
"Unmatched restaurant = 0",
f"Unmatched restaurant_id: {fk_rest['unmatched_count']} dari {fk_rest['child_distinct_keys']} distinct restaurant di orders.",
v13_passed,
notes="Integritas referensial Orders → Restaurants valid 100%.",
)
# V-14: Cross-table Order Items / Payments → Orders FK
fk_oi_ord = fk_check(order_items_clean, orders_clean, "order_id", "order_id", "Order Items → Orders")
fk_pay_ord = fk_check(payments_clean, orders_clean, "order_id", "order_id", "Payments → Orders")
v14_passed = (fk_oi_ord["unmatched_count"] == 0) and (fk_pay_ord["unmatched_count"] == 0)
record_check(
"V-14", "Cross-table", "Order Items / Payments → Orders",
"Apakah seluruh order reference valid?",
"Orphan order reference = 0",
f"Unmatched order_id di Order Items: {fk_oi_ord['unmatched_count']}, di Payments: {fk_pay_ord['unmatched_count']}.",
v14_passed,
notes="Tidak ada transaksi orphan pada item maupun pembayaran.",
)
print("V-06 s.d. V-14 recorded successfully.")
else:
print("VALIDATION SKIPPED — DATA READY GATE belum PASS.")
V-06 s.d. V-14 recorded successfully.
Gunakan `V-15` untuk membuktikan bahwa finding yang kamu putuskan **KEEP** tidak ikut berubah akibat over-cleaning.
Jika kamu memiliki lebih dari satu KEEP finding, gunakan evidence yang cukup di Working Paper lalu rangkum hasilnya ke `V-15`.
# ============================================================
# EDIT AREA 3C — V-15 KEEP PRESERVATION
# ============================================================
null_promo_count = int(orders_clean["promo_code"].isna().sum())
repeated_orders_pay = int(payments_clean["order_id"].duplicated().sum())
v15_passed = (null_promo_count == 189) and (repeated_orders_pay == 48)
record_check(
"V-15", "Judgment", "KEEP findings",
"Apakah legitimate anomaly yang diputuskan KEEP masih dipertahankan?",
"KEEP findings remain unless new evidence justifies change",
f"189 missing promo_code (F-04) utuh sebagai transaksi organik tanpa promo; 48 repeated order_id di Payments (F-10) dipertahankan sebagai multi-attempt/retry sah. Over-cleaning = 0.",
v15_passed,
notes="Legitimate business phenomena dipertahankan 100%.",
)
print("V-15 KEEP Preservation recorded successfully.")
V-15 KEEP Preservation recorded successfully.
validation_df = pd.DataFrame(validation_results)
if validation_df.empty:
print("VALIDATION GATE: PENDING")
print("→ Belum ada validation evidence yang dicatat.")
else:
validation_df = validation_df.sort_values("check_id").reset_index(drop=True)
display(validation_df)
present_ids = set(validation_df["check_id"])
missing_validation_ids = [
x for x in REQUIRED_VALIDATION_IDS if x not in present_ids
]
print(validation_df["status"].value_counts(dropna=False))
if missing_validation_ids:
print("VALIDATION GATE: PENDING")
print("Missing required checks:", ", ".join(missing_validation_ids))
elif (validation_df["status"] == "FAIL").any():
print("VALIDATION GATE: FAIL")
print("→ Review FAIL sebelum finalisasi.")
else:
print("VALIDATION GATE: PASS")
| check_id | validation_level | dataset_relationship | validation_question | expected_criteria | actual_result_metric | status | related_rule_id | notes | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | V-01 | Schema | All clean datasets | Apakah seluruh required column masih tersedia? | Required columns available | Seluruh 5 clean datasets memiliki required col... | PASS | Schema utuh pada seluruh 5 dataset. | |
| 1 | V-02 | Key | Customers | Apakah customer_id valid sesuai grain customer? | No null key; uniqueness sesuai grain | Total 173 baris, unique key: 173, duplicate: 0... | PASS | CR-01 | Grain 1 baris per customer terpenuhi. |
| 2 | V-03 | Key | Orders | Apakah order_id valid sesuai grain order? | No null key; one row per order | Total 339 baris, unique key: 339, duplicate: 0... | PASS | CR-03 | Grain 1 baris per order terpenuhi. |
| 3 | V-04 | Key | Order Items | Apakah order_item_id / candidate business key ... | No invalid duplicate key | Total 893 baris, unique key: 893, duplicate: 0... | PASS | CR-04 | Surrogate key order_item_id unik 100%. |
| 4 | V-05 | Key | Payments | Apakah payment_id unique tanpa menghapus repea... | payment_id unique; repeated order_id allowed b... | Payment ID unik: 387 dari 387 baris (0 duplica... | PASS | CR-08 | Grain payment transaksi dipertahankan dengan b... |
| 5 | V-06 | Value | Customers / categories | Apakah category dan missing representation sud... | No unexpected canonical-category issue | Kategori customer_segment kanonikal: {'Regular... | PASS | CR-02 | Nilai kategori terstandarisasi Title Case. |
| 6 | V-07 | Value | Orders / categories | Apakah order status dan representation sudah k... | Only approved canonical values | Kategori order_status kanonikal: {'completed':... | PASS | CR-03 | Status order konsisten lowercase. |
| 7 | V-08 | Value | Payments / Restaurants | Apakah category representation sudah konsisten? | Only approved canonical values | Metode bayar kanonikal: {'e_wallet': np.int64(... | PASS | CR-05, CR-08 | Kategori payment method dan menu item tersinkr... |
| 8 | V-09 | Business Rule | Order Items | Apakah line_amount memenuhi formula item? | quantity × unit_price - line_discount = line_a... | Formula line_amount valid 100% pada seluruh 89... | PASS | CR-06 | Nilai formula pada OI00131 dan OI00511 telah d... |
| 9 | V-10 | Business Rule | Orders | Apakah total_amount memenuhi formula order? | subtotal + delivery_fee - discount_amount = to... | Formula total_amount valid 100% pada seluruh 3... | PASS | CR-07 | Total order sesuai penjumlahan subtotal + ongk... |
| 10 | V-11 | Cross-table | Orders ↔ Order Items | Apakah subtotal order konsisten dengan detail ... | SUM(line_amount) = subtotal_amount | Rekonsiliasi subtotal_amount vs SUM(line_amoun... | PASS | CR-06, CR-07 | Order O00201 telah diselaraskan dengan total i... |
| 11 | V-12 | Cross-table | Orders → Customers | Apakah seluruh customer reference valid? | Unmatched customer = 0 | Unmatched customer_id: 0 dari 173 distinct cus... | PASS | Integritas referensial Orders → Customers vali... | |
| 12 | V-13 | Cross-table | Orders → Restaurants | Apakah seluruh restaurant reference valid? | Unmatched restaurant = 0 | Unmatched restaurant_id: 0 dari 24 distinct re... | PASS | Integritas referensial Orders → Restaurants va... | |
| 13 | V-14 | Cross-table | Order Items / Payments → Orders | Apakah seluruh order reference valid? | Orphan order reference = 0 | Unmatched order_id di Order Items: 0, di Payme... | PASS | Tidak ada transaksi orphan pada item maupun pe... | |
| 14 | V-15 | Judgment | KEEP findings | Apakah legitimate anomaly yang diputuskan KEEP... | KEEP findings remain unless new evidence justi... | 189 missing promo_code (F-04) utuh sebagai tra... | PASS | Legitimate business phenomena dipertahankan 100%. |
status PASS 15 Name: count, dtype: int64 VALIDATION GATE: PASS
Cell ini membandingkan SHA-256 raw source saat pertama kali di-load dengan file raw saat ini.
Target: **tidak ada raw/source yang berubah**.
RAW_INTEGRITY_PASS = False
raw_integrity_rows = []
if not DATA_READY:
print("RAW INTEGRITY CHECK: SKIPPED")
print("→ DATA READY GATE belum PASS.")
else:
file_map_integrity = {
"Customers": RAW_DIR / EXPECTED_FILES["customers"],
"Orders": RAW_DIR / EXPECTED_FILES["orders"],
"Order Items": RAW_DIR / EXPECTED_FILES["order_items"],
"Payments": RAW_DIR / EXPECTED_FILES["payments"],
"Restaurants": RAW_DIR / EXPECTED_FILES["restaurants"],
}
for label, path in file_map_integrity.items():
current_hash = sha256_file(path)
baseline_hash = RAW_HASH_BASELINE.get(label)
raw_integrity_rows.append({
"dataset": label,
"unchanged": current_hash == baseline_hash,
})
raw_integrity_df = pd.DataFrame(raw_integrity_rows)
display(raw_integrity_df)
RAW_INTEGRITY_PASS = bool(raw_integrity_df["unchanged"].all())
if RAW_INTEGRITY_PASS:
print("RAW INTEGRITY CHECK: PASS")
else:
print("RAW INTEGRITY CHECK: FAIL")
print("→ Raw/source berubah. Kembalikan source asli sebelum finalisasi.")
| dataset | unchanged | |
|---|---|---|
| 0 | Customers | True |
| 1 | Orders | True |
| 2 | Order Items | True |
| 3 | Payments | True |
| 4 | Restaurants | True |
RAW INTEGRITY CHECK: PASS
Final Gate hanya boleh:
- `PASS` jika **IDENTITY PASS + semua V-01 s.d. V-15 PASS + RAW INTEGRITY PASS**;
- `NOT READY` jika ada critical FAIL atau raw source berubah;
- `PENDING` jika validation belum lengkap.
Jangan mengubah nilai `FINAL_GATE` secara manual.
def derive_final_gate(validation_df, raw_integrity_pass, identity_gate_pass):
if not identity_gate_pass:
return "PENDING", ["Identity belum lengkap."]
if validation_df.empty:
return "PENDING", ["Validation evidence masih kosong."]
present_ids = set(validation_df["check_id"])
missing_ids = [x for x in REQUIRED_VALIDATION_IDS if x not in present_ids]
required_rows = validation_df[
validation_df["check_id"].isin(REQUIRED_VALIDATION_IDS)
].copy()
if (required_rows["status"] == "FAIL").any():
failed_ids = required_rows.loc[
required_rows["status"] == "FAIL", "check_id"
].tolist()
return "NOT READY", [f"Validation FAIL: {', '.join(failed_ids)}"]
if not raw_integrity_pass:
return "NOT READY", ["Raw Integrity Check FAIL."]
if missing_ids:
return "PENDING", [f"Validation belum lengkap: {', '.join(missing_ids)}"]
all_pass = (
len(required_rows) == len(REQUIRED_VALIDATION_IDS)
and (required_rows["status"] == "PASS").all()
)
if all_pass:
return "PASS", []
return "PENDING", ["Review validation evidence."]
FINAL_GATE, FINAL_GATE_REASONS = derive_final_gate(
validation_df,
RAW_INTEGRITY_PASS,
IDENTITY_GATE_PASS,
)
print("PM1 DATA READINESS GATE:", FINAL_GATE)
if FINAL_GATE_REASONS:
for reason in FINAL_GATE_REASONS:
print("→", reason)
FINAL_JUDGMENT_RATIONALE = (
"Data SwiftBite (Variant SB-P11) telah melalui seluruh tahapan verifikasi kesiapan data dan dinyatakan ANALYTICS-READY (PASS). "
"Seluruh 15 validation checks (V-01 s.d. V-15) PASS 100%. Integritas file sumber asli terjaga tanpa perubahan (Raw Integrity PASS). "
"Grain dan keunikan kunci primer terverifikasi pada 5 dataset. Inkonsistensi format kategori dan spasi telah distandarisasi tanpa "
"mengubah makna data. Perhitungan matematis pada level item, order, serta rekonsiliasi cross-table orders vs order items tersinkronisasi 100%. "
"Anomali bisnis yang sah (missing promo_code organik dan multi-attempt retry payments) berhasil dilindungi dari over-cleaning. "
"Kelima berkas output clean telah diekspor dan siap dimanfaatkan untuk analisis bisnis, dashboarding, dan pemodelan prediktif lanjutan."
)
print("FINAL JUDGMENT RATIONALE:")
print(FINAL_JUDGMENT_RATIONALE)
PM1 DATA READINESS GATE: PASS FINAL JUDGMENT RATIONALE: Data SwiftBite (Variant SB-P11) telah melalui seluruh tahapan verifikasi kesiapan data dan dinyatakan ANALYTICS-READY (PASS). Seluruh 15 validation checks (V-01 s.d. V-15) PASS 100%. Integritas file sumber asli terjaga tanpa perubahan (Raw Integrity PASS). Grain dan keunikan kunci primer terverifikasi pada 5 dataset. Inkonsistensi format kategori dan spasi telah distandarisasi tanpa mengubah makna data. Perhitungan matematis pada level item, order, serta rekonsiliasi cross-table orders vs order items tersinkronisasi 100%. Anomali bisnis yang sah (missing promo_code organik dan multi-attempt retry payments) berhasil dilindungi dari over-cleaning. Kelima berkas output clean telah diekspor dan siap dimanfaatkan untuk analisis bisnis, dashboarding, dan pemodelan prediktif lanjutan.
Output:
- `customers_clean.csv`
- `orders_clean.csv`
- `order_items_clean.csv`
- `payments_clean.csv`
- `restaurants_clean.csv`
Jika Final Gate belum PASS, export akan **diblok otomatis**.
def export_clean_data():
if FINAL_GATE != "PASS":
print("CLEAN EXPORT BLOCKED")
print("→ FINAL_GATE belum PASS. Selesaikan validation terlebih dahulu.")
return False
customers_clean.to_csv(PROCESSED_DIR / "customers_clean.csv", index=False)
orders_clean.to_csv(PROCESSED_DIR / "orders_clean.csv", index=False)
order_items_clean.to_csv(PROCESSED_DIR / "order_items_clean.csv", index=False)
payments_clean.to_csv(PROCESSED_DIR / "payments_clean.csv", index=False)
restaurants_clean.to_csv(PROCESSED_DIR / "restaurants_clean.csv", index=False)
print("CLEAN PACKAGE SAVE: PASS")
print("Saved to:", PROCESSED_DIR)
return True
# Hapus tanda # pada baris berikut HANYA setelah FINAL_GATE: PASS
export_clean_data()
CLEAN PACKAGE SAVE: PASS Saved to: C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst\data\processed\pm1_swiftbite
True
Evidence boleh diekspor untuk review, tetapi **clean package final tetap hanya boleh disimpan saat FINAL_GATE: PASS**.
def export_evidence():
findings.to_csv(OUTPUT_DIR / "pm1_findings.csv", index=False)
cleaning_rules.to_csv(OUTPUT_DIR / "pm1_cleaning_rulebook.csv", index=False)
cleaning_log.to_csv(OUTPUT_DIR / "pm1_cleaning_log.csv", index=False)
validation_df.to_csv(OUTPUT_DIR / "pm1_validation_results.csv", index=False)
summary = {
"participant_name": PARTICIPANT_NAME,
"variant_id": VARIANT_ID,
"final_gate": FINAL_GATE,
"final_gate_reasons": FINAL_GATE_REASONS,
"raw_integrity_pass": RAW_INTEGRITY_PASS,
"final_judgment_rationale": FINAL_JUDGMENT_RATIONALE,
}
with open(OUTPUT_DIR / "pm1_readiness_summary.json", "w", encoding="utf-8") as f:
json.dump(summary, f, ensure_ascii=False, indent=2)
print("EVIDENCE EXPORT: PASS")
print("Saved to:", OUTPUT_DIR)
return True
# Hapus tanda # pada baris berikut setelah evidence sudah diisi:
export_evidence()
EVIDENCE EXPORT: PASS Saved to: C:\Users\Bening JPA\OneDrive\Documents\Bootcamp_Data_Analyst\outputs\pm1_swiftbite
True
### Technical
- [ ] `IDENTITY GATE: PASS`
- [ ] `TECHNICAL GATE: PASS`
- [ ] `SOURCE ACCESS CHECK: PASS`
- [ ] `RAW LOAD GATE: PASS`
- [ ] `SCHEMA GATE: PASS`
- [ ] `RAW INTEGRITY CHECK: PASS`
### Intake, Profiling & Judgment
- [ ] 5 source berhasil dibaca.
- [ ] Grain dan candidate key sudah diperiksa.
- [ ] Relationship utama sudah diperiksa.
- [ ] Material findings sudah diverifikasi.
- [ ] FIX / STANDARDIZE / KEEP / INVESTIGATE memiliki evidence dan alasan.
- [ ] Finding yang menghasilkan cleaning memiliki Rule ID.
### Cleaning & Validation
- [ ] Rulebook dibuat sebelum implementation.
- [ ] Cleaning hanya dilakukan pada working copy.
- [ ] Cleaning Log lengkap.
- [ ] `V-01` sampai `V-15` sudah dicatat.
- [ ] `PM1 DATA READINESS GATE: PASS`
- [ ] 5 clean CSV berhasil diekspor.
```text
Raw Data
↓
Technical & Source Gate
↓
Understand Grain & Key
↓
Profile → Find → Verify → Judge
↓
Define Cleaning Rule
↓
Clean Working Copy
↓
Validate V-01 ... V-15
↓
Raw Integrity Check
↓
FINAL GATE
↓
Export Clean Package
```