# Praktik Mandiri 1 — Data Readiness Check & Cleaning Validation
## Participant Starter Notebook — Guided Technical Version
Notebook ini membantu kamu menjalankan workflow PM1 dengan **technical guardrails** agar tidak salah folder, salah source, atau salah export.
**Alur PM1:**
**Setup → Technical Check → Load → Intake → Grain & Key → Profiling → Findings → Verification → Rulebook → Cleaning → Validation → Raw Integrity → Export → Final Gate**
### Sebelum RUN
1. Buka **folder `Bootcamp_Data_Analyst` sebagai workspace** di Antigravity.
2. Simpan notebook ini di folder `notebooks/`.
3. Extract package `SB-Pxx`, lalu letakkan **5 raw files langsung** di `data/raw/pm1_swiftbite/`.
4. Buka `PM1_Data_Readiness_Working_Paper.xlsx` sebagai tempat mencatat decision & evidence.
5. Jalankan notebook **dari atas ke bawah**.
> Jika `TECHNICAL GATE: PASS`, **jangan mengubah path** di notebook.
### Guardrails
- Raw/source **tidak boleh diubah atau ditimpa**.
- Jangan menghapus row hanya karena terlihat aneh.
- Repeated `order_id` belum tentu duplicate.
- Finding belum otomatis menjadi cleaning action.
- Material change harus dapat ditelusuri: **Finding ID → Rule ID → Cleaning Log → Validation Evidence**.
- Notebook membantu sisi teknis. **Judgment, cleaning rule, dan interpretation tetap pekerjaan peserta.**
Ubah hanya **Nama** dan **Variant ID** sesuai package yang kamu terima.
Contoh Variant ID: `SB-P01`, `SB-P12`, `SB-P28`.
# ============================================================
# EDIT AREA 1 — IDENTITAS PESERTA
# ============================================================
PARTICIPANT_NAME = "Mita"
VARIANT_ID = "SB-P18" # contoh: SB-P01
name_ok = PARTICIPANT_NAME.strip() not in {"", "Isi Nama Kamu"}
variant_ok = (
VARIANT_ID.startswith("SB-P")
and len(VARIANT_ID) == 6
and VARIANT_ID[4:].isdigit()
)
IDENTITY_GATE_PASS = bool(name_ok and variant_ok)
if IDENTITY_GATE_PASS:
print("IDENTITY GATE: PASS")
print("Participant :", PARTICIPANT_NAME)
print("Variant ID :", VARIANT_ID)
else:
print("IDENTITY GATE: FAIL")
print("→ Isi PARTICIPANT_NAME dan VARIANT_ID pada cell ini.")
print("→ Format Variant ID harus seperti SB-P01 atau SB-P28.")
IDENTITY GATE: PASS Participant : Mita Variant ID : SB-P18
Notebook akan mencari **Workspace Starter yang sama dari Sesi 1** berdasarkan keberadaan folder `data/` dan `notebooks/`.
**Jangan membuat workspace baru. Jangan copy-paste absolute path.**
Jika output menunjukkan `TECHNICAL GATE: PASS`, lanjut ke cell berikutnya tanpa mengubah path.
from pathlib import Path
import json
import hashlib
import pandas as pd
import numpy as np
try:
import duckdb
DUCKDB_AVAILABLE = True
except ImportError:
duckdb = None
DUCKDB_AVAILABLE = False
pd.set_option("display.max_columns", 100)
pd.set_option("display.max_rows", 100)
pd.set_option("display.width", 180)
def find_workspace_root(start=None):
start = Path(start or Path.cwd()).resolve()
for p in [start, *start.parents]:
if (p / "data").is_dir() and (p / "notebooks").is_dir():
return p
return None
WORKSPACE_ROOT = find_workspace_root()
if WORKSPACE_ROOT is None:
TECHNICAL_GATE_PASS = False
RAW_DIR = Path("__WORKSPACE_NOT_FOUND__")
PROCESSED_DIR = Path("__WORKSPACE_NOT_FOUND__")
OUTPUT_DIR = Path("__WORKSPACE_NOT_FOUND__")
DOCS_DIR = Path("__WORKSPACE_NOT_FOUND__")
print("TECHNICAL GATE: FAIL")
print("→ Workspace Starter tidak ditemukan.")
print("→ Di Antigravity, buka folder Bootcamp_Data_Analyst sebagai workspace.")
print("→ Pastikan folder data/ dan notebooks/ terlihat di Explorer.")
print("→ Setelah itu RUN ulang cell ini. Jangan edit path secara manual.")
else:
TECHNICAL_GATE_PASS = True
RAW_DIR = WORKSPACE_ROOT / "data" / "raw" / "pm1_swiftbite"
PROCESSED_DIR = WORKSPACE_ROOT / "data" / "processed" / "pm1_swiftbite"
OUTPUT_DIR = WORKSPACE_ROOT / "outputs" / "pm1_swiftbite"
DOCS_DIR = WORKSPACE_ROOT / "docs" / "pm1_swiftbite"
for folder in [PROCESSED_DIR, OUTPUT_DIR, DOCS_DIR]:
folder.mkdir(parents=True, exist_ok=True)
print("TECHNICAL GATE: PASS")
print("WORKSPACE_ROOT :", WORKSPACE_ROOT)
print("RAW_DIR :", RAW_DIR)
print("PROCESSED_DIR :", PROCESSED_DIR)
print("OUTPUT_DIR :", OUTPUT_DIR)
print("DOCS_DIR :", DOCS_DIR)
TECHNICAL GATE: PASS WORKSPACE_ROOT : C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst RAW_DIR : C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst\data\raw\pm1_swiftbite PROCESSED_DIR : C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst\data\processed\pm1_swiftbite OUTPUT_DIR : C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst\outputs\pm1_swiftbite DOCS_DIR : C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst\docs\pm1_swiftbite
Tujuan cell ini hanya memastikan library utama tersedia. Jika DuckDB belum aktif, workflow utama starter tetap dapat berjalan dengan pandas.
print("pandas :", pd.__version__)
try:
import openpyxl
EXCEL_ENGINE_AVAILABLE = True
print("openpyxl:", openpyxl.__version__)
except ImportError:
EXCEL_ENGINE_AVAILABLE = False
print("openpyxl: NOT AVAILABLE")
if DUCKDB_AVAILABLE:
print("DuckDB :", duckdb.__version__)
try:
con = duckdb.connect()
print("DuckDB connection: OK")
except Exception as e:
print("DuckDB connection: WARNING —", type(e).__name__, str(e))
else:
print("DuckDB : NOT AVAILABLE")
print("INFO: workflow utama notebook tetap dapat dijalankan dengan pandas.")
if not EXCEL_ENGINE_AVAILABLE:
print("ENVIRONMENT CHECK: NEEDS FIX")
print("→ File XLSX membutuhkan openpyxl. Aktifkan environment sesuai setup Sesi 1.")
else:
print("ENVIRONMENT CHECK: PASS")
pandas : 3.0.6 openpyxl: 3.1.5 DuckDB : 2.0.0.dev2610011535 DuckDB connection: OK ENVIRONMENT CHECK: PASS
Letakkan **5 file berikut langsung** di:
`data/raw/pm1_swiftbite/`
- `swiftbite_customers.json`
- `swiftbite_orders.xlsx`
- `swiftbite_order_items.csv`
- `swiftbite_payments.txt`
- `swiftbite_restaurants.csv`
**Jangan** meletakkan ZIP di folder tersebut dan **jangan** membuat folder `SB-Pxx` lagi di dalam `pm1_swiftbite/`.
EXPECTED_FILES = {
"customers": "swiftbite_customers.json",
"orders": "swiftbite_orders.xlsx",
"order_items": "swiftbite_order_items.csv",
"payments": "swiftbite_payments.txt",
"restaurants": "swiftbite_restaurants.csv",
}
source_status = []
if not TECHNICAL_GATE_PASS:
missing_files = list(EXPECTED_FILES.values())
SOURCE_ACCESS_PASS = False
print("SOURCE ACCESS CHECK: SKIPPED")
print("→ Perbaiki TECHNICAL GATE terlebih dahulu.")
else:
for logical_name, filename in EXPECTED_FILES.items():
path = RAW_DIR / filename
source_status.append({
"logical_name": logical_name,
"filename": filename,
"exists": path.is_file(),
"path": str(path),
})
source_status_df = pd.DataFrame(source_status)
display(source_status_df)
missing_files = source_status_df.loc[
~source_status_df["exists"], "filename"
].tolist()
SOURCE_ACCESS_PASS = len(missing_files) == 0
if SOURCE_ACCESS_PASS:
print("SOURCE ACCESS CHECK: PASS")
print("→ Semua 5 raw files ditemukan. Jangan ubah path.")
else:
print("SOURCE ACCESS CHECK: FAIL")
print("File belum ditemukan:", missing_files)
print("→ Extract package peserta.")
print("→ Copy hanya 5 raw files langsung ke:", RAW_DIR)
print("→ Jangan menaruh file di subfolder tambahan.")
| logical_name | filename | exists | path | |
|---|---|---|---|---|
| 0 | customers | swiftbite_customers.json | True | C:\Users\DataAnalysProjects\Bootcamp_Data_Anal... |
| 1 | orders | swiftbite_orders.xlsx | True | C:\Users\DataAnalysProjects\Bootcamp_Data_Anal... |
| 2 | order_items | swiftbite_order_items.csv | True | C:\Users\DataAnalysProjects\Bootcamp_Data_Anal... |
| 3 | payments | swiftbite_payments.txt | True | C:\Users\DataAnalysProjects\Bootcamp_Data_Anal... |
| 4 | restaurants | swiftbite_restaurants.csv | True | C:\Users\DataAnalysProjects\Bootcamp_Data_Anal... |
SOURCE ACCESS CHECK: PASS → Semua 5 raw files ditemukan. Jangan ubah path.
Cell ini:
1. membaca kelima source secara aman,
2. menampilkan error per file jika ada,
3. memeriksa kolom minimum yang diperlukan Starter,
4. menyimpan **raw hash baseline** untuk pemeriksaan integritas di akhir.
Jika `RAW LOAD GATE: PASS` dan `SCHEMA GATE: PASS`, lanjutkan.
REQUIRED_COLUMNS = {
"Customers": {
"customer_id", "full_name", "email", "phone_number",
"signup_date", "customer_zone", "customer_segment"
},
"Orders": {
"order_id", "customer_id", "restaurant_id", "order_timestamp",
"order_status", "subtotal_amount", "delivery_fee",
"discount_amount", "total_amount"
},
"Order Items": {
"order_item_id", "order_id", "line_number", "item_name",
"item_category", "quantity", "unit_price",
"line_discount", "line_amount"
},
"Payments": {
"payment_id", "order_id", "payment_timestamp",
"payment_method", "payment_status", "payment_amount"
},
"Restaurants": {
"restaurant_id", "restaurant_name", "cuisine_type",
"restaurant_zone", "price_tier", "active_status"
},
}
def sha256_file(path: Path):
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
h.update(chunk)
return h.hexdigest()
def safe_read_source(label, path, reader):
try:
return reader(path), None
except Exception as e:
return None, f"{type(e).__name__}: {e}"
raw_tables = {}
load_errors = {}
RAW_HASH_BASELINE = {}
if not SOURCE_ACCESS_PASS:
RAW_LOAD_GATE_PASS = False
SCHEMA_GATE_PASS = False
DATA_READY = False
print("RAW LOAD GATE: SKIPPED")
print("→ Perbaiki SOURCE ACCESS CHECK terlebih dahulu.")
else:
readers = {
"Customers": lambda p: pd.read_json(p),
"Orders": lambda p: pd.read_excel(p),
"Order Items": lambda p: pd.read_csv(p),
"Payments": lambda p: pd.read_csv(p, sep="|"),
"Restaurants": lambda p: pd.read_csv(p),
}
file_map = {
"Customers": RAW_DIR / EXPECTED_FILES["customers"],
"Orders": RAW_DIR / EXPECTED_FILES["orders"],
"Order Items": RAW_DIR / EXPECTED_FILES["order_items"],
"Payments": RAW_DIR / EXPECTED_FILES["payments"],
"Restaurants": RAW_DIR / EXPECTED_FILES["restaurants"],
}
for label, path in file_map.items():
df, err = safe_read_source(label, path, readers[label])
if err:
load_errors[label] = err
else:
raw_tables[label] = df
RAW_HASH_BASELINE[label] = sha256_file(path)
RAW_LOAD_GATE_PASS = len(load_errors) == 0 and len(raw_tables) == 5
if RAW_LOAD_GATE_PASS:
customers_raw = raw_tables["Customers"]
orders_raw = raw_tables["Orders"]
order_items_raw = raw_tables["Order Items"]
payments_raw = raw_tables["Payments"]
restaurants_raw = raw_tables["Restaurants"]
print("RAW LOAD GATE: PASS")
else:
print("RAW LOAD GATE: FAIL")
for label, err in load_errors.items():
print(f"→ {label}: {err}")
schema_rows = []
if RAW_LOAD_GATE_PASS:
for label, required in REQUIRED_COLUMNS.items():
actual = set(raw_tables[label].columns)
missing_cols = sorted(required - actual)
schema_rows.append({
"dataset": label,
"required_columns": len(required),
"actual_columns": len(actual),
"missing_required_columns": ", ".join(missing_cols) if missing_cols else "",
"status": "PASS" if not missing_cols else "FAIL",
})
schema_check_df = pd.DataFrame(schema_rows)
display(schema_check_df)
SCHEMA_GATE_PASS = (schema_check_df["status"] == "PASS").all()
if SCHEMA_GATE_PASS:
print("SCHEMA GATE: PASS")
else:
print("SCHEMA GATE: FAIL")
print("→ Jangan lanjut cleaning. Periksa apakah package/variant yang digunakan sudah benar.")
else:
SCHEMA_GATE_PASS = False
DATA_READY = bool(
TECHNICAL_GATE_PASS
and SOURCE_ACCESS_PASS
and RAW_LOAD_GATE_PASS
and SCHEMA_GATE_PASS
)
if DATA_READY:
print("DATA READY GATE: PASS")
else:
print("DATA READY GATE: FAIL")
RAW LOAD GATE: PASS
| dataset | required_columns | actual_columns | missing_required_columns | status | |
|---|---|---|---|---|---|
| 0 | Customers | 7 | 7 | PASS | |
| 1 | Orders | 9 | 10 | PASS | |
| 2 | Order Items | 9 | 9 | PASS | |
| 3 | Payments | 6 | 7 | PASS | |
| 4 | Restaurants | 6 | 7 | PASS |
SCHEMA GATE: PASS DATA READY GATE: PASS
Pastikan nama kolom dan contoh record masuk akal. Ini **belum** berarti data sudah bersih.
if DATA_READY:
for name, df in raw_tables.items():
print(f"\n=== {name} | shape={df.shape} ===")
display(df.head(3))
else:
print("PREVIEW SKIPPED — DATA READY GATE belum PASS.")
=== Customers | shape=(157, 7) ===
| customer_id | full_name | phone_number | signup_date | customer_zone | customer_segment | ||
|---|---|---|---|---|---|---|---|
| 0 | C0001 | Bima Lestari | bimalestari1@example.com | 085812760273 | 2025-10-19 | Central | Regular |
| 1 | C0002 | Farah Santoso | farahsantoso2@example.com | 087690291331 | 2025-11-16 | West | New |
| 2 | C0003 | Kevin Maulana | kevinmaulana3@example.com | 081308997496 | 2025-11-28 | East | Regular |
=== Orders | shape=(324, 10) ===
| order_id | customer_id | restaurant_id | order_timestamp | order_status | promo_code | subtotal_amount | delivery_fee | discount_amount | total_amount | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | O00001 | C0134 | R0020 | 2026-02-01 21:49:00 | completed | NaN | 60000 | 18000 | 5000 | 73000 |
| 1 | O00002 | C0141 | R0003 | 2026-01-10 14:52:00 | completed | NaN | 52000 | 8000 | 0 | 60000 |
| 2 | O00003 | C0148 | R0008 | 2026-03-16 01:11:00 | completed | NaN | 47000 | 18000 | 15000 | 50000 |
=== Order Items | shape=(822, 9) ===
| order_item_id | order_id | line_number | item_name | item_category | quantity | unit_price | line_discount | line_amount | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | OI00001 | O00001 | 1 | Fries | Snack | 1 | 16000 | 0 | 16000 |
| 1 | OI00002 | O00001 | 2 | Mie Special | Main | 2 | 23000 | 2000 | 44000 |
| 2 | OI00003 | O00002 | 1 | Juice | Drink | 2 | 18000 | 0 | 36000 |
=== Payments | shape=(370, 7) ===
| payment_id | order_id | payment_method | payment_status | payment_amount | payment_timestamp | attempt_number | |
|---|---|---|---|---|---|---|---|
| 0 | P00001 | O00001 | e_wallet | successful | 73000 | 2026-02-01 21:52:00 | 1 |
| 1 | P00002 | O00002 | bank_transfer | successful | 60000 | 2026-01-10 14:55:00 | 1 |
| 2 | P00003 | O00003 | credit_card | failed | 50000 | 2026-03-16 01:13:00 | 1 |
=== Restaurants | shape=(22, 7) ===
| restaurant_id | restaurant_name | cuisine_type | restaurant_zone | price_tier | onboarding_date | active_status | |
|---|---|---|---|---|---|---|---|
| 0 | R0001 | SwiftBite Partner 01 | Asian | North | Premium | 2024-12-23 | active |
| 1 | R0002 | SwiftBite Partner 02 | Western | East | Premium | 2024-11-05 | active |
| 2 | R0003 | SwiftBite Partner 03 | Fast Food | North | Budget | 2025-05-15 | active |
Catat business meaning, expected grain, candidate key, row/record count, access status, dan key test result.
def intake_summary(df, dataset_name):
return {
"dataset": dataset_name,
"rows": len(df),
"columns": len(df.columns),
"column_names": list(df.columns),
}
if DATA_READY:
intake_df = pd.DataFrame([
intake_summary(customers_raw, "Customers"),
intake_summary(orders_raw, "Orders"),
intake_summary(order_items_raw, "Order Items"),
intake_summary(payments_raw, "Payments"),
intake_summary(restaurants_raw, "Restaurants"),
])
display(intake_df)
| dataset | rows | columns | column_names | |
|---|---|---|---|---|
| 0 | Customers | 157 | 7 | [customer_id, full_name, email, phone_number, ... |
| 1 | Orders | 324 | 10 | [order_id, customer_id, restaurant_id, order_t... |
| 2 | Order Items | 822 | 9 | [order_item_id, order_id, line_number, item_na... |
| 3 | Payments | 370 | 7 | [payment_id, order_id, payment_method, payment... |
| 4 | Restaurants | 22 | 7 | [restaurant_id, restaurant_name, cuisine_type,... |
Repeated `order_id` pada **Order Items** atau **Payments** belum tentu duplicate karena grain-nya berbeda dari Orders.
Gunakan hasil check sebagai **evidence awal**, bukan cleaning action otomatis.
def key_check(df, key_cols, dataset_name):
if isinstance(key_cols, str):
key_cols = [key_cols]
missing_key_rows = int(df[key_cols].isna().any(axis=1).sum())
unique_key_count = int(df[key_cols].drop_duplicates().shape[0])
duplicate_key_rows = int(df.duplicated(subset=key_cols, keep=False).sum())
return pd.DataFrame([{
"dataset": dataset_name,
"key_cols": ", ".join(key_cols),
"rows": len(df),
"unique_key_count": unique_key_count,
"missing_key_rows": missing_key_rows,
"duplicate_key_rows": duplicate_key_rows,
"candidate_key_unique": unique_key_count == len(df) and missing_key_rows == 0,
}])
if DATA_READY:
key_results = pd.concat([
key_check(customers_raw, "customer_id", "Customers"),
key_check(orders_raw, "order_id", "Orders"),
key_check(order_items_raw, "order_item_id", "Order Items"),
key_check(order_items_raw, ["order_id", "line_number"], "Order Items — candidate business key"),
key_check(payments_raw, "payment_id", "Payments"),
key_check(restaurants_raw, "restaurant_id", "Restaurants"),
], ignore_index=True)
display(key_results)
| dataset | key_cols | rows | unique_key_count | missing_key_rows | duplicate_key_rows | candidate_key_unique | |
|---|---|---|---|---|---|---|---|
| 0 | Customers | customer_id | 157 | 156 | 0 | 2 | False |
| 1 | Orders | order_id | 324 | 324 | 0 | 0 | True |
| 2 | Order Items | order_item_id | 822 | 822 | 0 | 0 | True |
| 3 | Order Items — candidate business key | order_id, line_number | 822 | 822 | 0 | 0 | True |
| 4 | Payments | payment_id | 370 | 370 | 0 | 0 | True |
| 5 | Restaurants | restaurant_id | 22 | 22 | 0 | 0 | True |
if DATA_READY:
order_item_cardinality = (
order_items_raw.groupby("order_id")
.size()
.rename("item_rows")
.reset_index()
.sort_values("item_rows", ascending=False)
)
payment_cardinality = (
payments_raw.groupby("order_id")
.size()
.rename("payment_rows")
.reset_index()
.sort_values("payment_rows", ascending=False)
)
print("Order Items cardinality")
display(order_item_cardinality.head(10))
print("Payments cardinality")
display(payment_cardinality.head(10))
Order Items cardinality
| order_id | item_rows | |
|---|---|---|
| 296 | O00297 | 4 |
| 6 | O00007 | 4 |
| 232 | O00233 | 4 |
| 19 | O00020 | 4 |
| 293 | O00294 | 4 |
| 309 | O00310 | 4 |
| 202 | O00203 | 4 |
| 276 | O00277 | 4 |
| 52 | O00053 | 4 |
| 218 | O00219 | 4 |
Payments cardinality
| order_id | payment_rows | |
|---|---|---|
| 21 | O00022 | 2 |
| 19 | O00020 | 2 |
| 121 | O00122 | 2 |
| 159 | O00160 | 2 |
| 2 | O00003 | 2 |
| 293 | O00294 | 2 |
| 39 | O00040 | 2 |
| 201 | O00202 | 2 |
| 54 | O00055 | 2 |
| 66 | O00067 | 2 |
Profiling minimal:
- missing / placeholder,
- data type / format,
- category consistency,
- duplicate / key,
- suspicious numeric / formula,
- potential outlier,
- cross-table concern.
Helper berikut menghasilkan evidence awal, bukan automatic judgment.
def basic_profile(df, dataset_name):
return pd.DataFrame([{
"dataset": dataset_name,
"rows": len(df),
"columns": len(df.columns),
"rows_in_exact_duplicate_groups": int(df.duplicated(keep=False).sum()),
"exact_duplicate_extras": int(df.duplicated(keep="first").sum()),
"null_cells": int(df.isna().sum().sum()),
}])
def column_profile(df):
rows = []
for col in df.columns:
s = df[col]
rows.append({
"column": col,
"dtype": str(s.dtype),
"null_count": int(s.isna().sum()),
"null_pct": round(float(s.isna().mean()), 4),
"unique_count": int(s.nunique(dropna=True)),
"sample_values": list(s.dropna().astype(str).unique()[:5]),
})
return pd.DataFrame(rows)
if DATA_READY:
overall_profile = pd.concat(
[basic_profile(df, name) for name, df in raw_tables.items()],
ignore_index=True
)
display(overall_profile)
| dataset | rows | columns | rows_in_exact_duplicate_groups | exact_duplicate_extras | null_cells | |
|---|---|---|---|---|---|---|
| 0 | Customers | 157 | 7 | 2 | 1 | 0 |
| 1 | Orders | 324 | 10 | 0 | 0 | 169 |
| 2 | Order Items | 822 | 9 | 0 | 0 | 0 |
| 3 | Payments | 370 | 7 | 0 | 0 | 0 |
| 4 | Restaurants | 22 | 7 | 0 | 0 | 0 |
# Ganti object untuk memeriksa dataset lain.
if DATA_READY:
display(column_profile(customers_raw))
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | customer_id | str | 0 | 0.0 | 156 | [C0001, C0002, C0003, C0004, C0005] |
| 1 | full_name | str | 0 | 0.0 | 117 | [Bima Lestari, Farah Santoso, Kevin Maulana, Y... |
| 2 | str | 0 | 0.0 | 156 | [bimalestari1@example.com, farahsantoso2@examp... | |
| 3 | phone_number | str | 0 | 0.0 | 156 | [085812760273, 087690291331, 081308997496, 085... |
| 4 | signup_date | str | 0 | 0.0 | 112 | [2025-10-19, 2025-11-16, 2025-11-28, 2025-09-0... |
| 5 | customer_zone | str | 0 | 0.0 | 5 | [Central, West, East, South, North] |
| 6 | customer_segment | str | 0 | 0.0 | 4 | [Regular, New, Loyal, LOYAL ] |
if DATA_READY:
print("=== 1. CUSTOMERS ===")
display(column_profile(customers_raw))
print("=== 2. ORDERS ===")
display(column_profile(orders_raw))
print("=== 3. ORDER ITEMS ===")
display(column_profile(order_items_raw))
print("=== 4. PAYMENTS ===")
display(column_profile(payments_raw))
print("=== 5. RESTAURANTS ===")
display(column_profile(restaurants_raw))
=== 1. CUSTOMERS ===
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | customer_id | str | 0 | 0.0 | 156 | [C0001, C0002, C0003, C0004, C0005] |
| 1 | full_name | str | 0 | 0.0 | 117 | [Bima Lestari, Farah Santoso, Kevin Maulana, Y... |
| 2 | str | 0 | 0.0 | 156 | [bimalestari1@example.com, farahsantoso2@examp... | |
| 3 | phone_number | str | 0 | 0.0 | 156 | [085812760273, 087690291331, 081308997496, 085... |
| 4 | signup_date | str | 0 | 0.0 | 112 | [2025-10-19, 2025-11-16, 2025-11-28, 2025-09-0... |
| 5 | customer_zone | str | 0 | 0.0 | 5 | [Central, West, East, South, North] |
| 6 | customer_segment | str | 0 | 0.0 | 4 | [Regular, New, Loyal, LOYAL ] |
=== 2. ORDERS ===
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | order_id | str | 0 | 0.0000 | 324 | [O00001, O00002, O00003, O00004, O00005] |
| 1 | customer_id | str | 0 | 0.0000 | 156 | [C0134, C0141, C0148, C0155, C0006] |
| 2 | restaurant_id | str | 0 | 0.0000 | 22 | [R0020, R0003, R0008, R0013, R0018] |
| 3 | order_timestamp | str | 0 | 0.0000 | 323 | [2026-02-01 21:49:00, 2026-01-10 14:52:00, 202... |
| 4 | order_status | str | 0 | 0.0000 | 3 | [completed, COMPLETED , cancelled] |
| 5 | promo_code | str | 169 | 0.5216 | 3 | [NEWUSER, HEMAT15, SAVE10] |
| 6 | subtotal_amount | int64 | 0 | 0.0000 | 163 | [60000, 52000, 47000, 90000, 104000] |
| 7 | delivery_fee | int64 | 0 | 0.0000 | 5 | [18000, 8000, 15000, 12000, 10000] |
| 8 | discount_amount | int64 | 0 | 0.0000 | 4 | [5000, 0, 15000, 10000] |
| 9 | total_amount | int64 | 0 | 0.0000 | 174 | [73000, 60000, 50000, 105000, 107000] |
=== 3. ORDER ITEMS ===
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | order_item_id | str | 0 | 0.0 | 822 | [OI00001, OI00002, OI00003, OI00004, OI00005] |
| 1 | order_id | str | 0 | 0.0 | 324 | [O00001, O00002, O00003, O00004, O00005] |
| 2 | line_number | int64 | 0 | 0.0 | 4 | [1, 2, 3, 4] |
| 3 | item_name | str | 0 | 0.0 | 14 | [Fries, Mie Special, Juice, Coffee, Nasi Ayam] |
| 4 | item_category | str | 0 | 0.0 | 5 | [Snack, Main, Drink, Dessert, MAIN ] |
| 5 | quantity | str | 0 | 0.0 | 5 | [1, 2, 3, 2 pcs, 3 pcs] |
| 6 | unit_price | int64 | 0 | 0.0 | 23 | [16000, 23000, 18000, 26000, 33000] |
| 7 | line_discount | int64 | 0 | 0.0 | 3 | [0, 2000, 5000] |
| 8 | line_amount | int64 | 0 | 0.0 | 93 | [16000, 44000, 36000, 47000, 64000] |
=== 4. PAYMENTS ===
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | payment_id | str | 0 | 0.0 | 370 | [P00001, P00002, P00003, P00004, P00005] |
| 1 | order_id | str | 0 | 0.0 | 324 | [O00001, O00002, O00003, O00004, O00005] |
| 2 | payment_method | str | 0 | 0.0 | 6 | [e_wallet, bank_transfer, credit_card, cash, C... |
| 3 | payment_status | str | 0 | 0.0 | 3 | [successful, failed, refunded] |
| 4 | payment_amount | int64 | 0 | 0.0 | 174 | [73000, 60000, 50000, 105000, 107000] |
| 5 | payment_timestamp | str | 0 | 0.0 | 370 | [2026-02-01 21:52:00, 2026-01-10 14:55:00, 202... |
| 6 | attempt_number | int64 | 0 | 0.0 | 2 | [1, 2] |
=== 5. RESTAURANTS ===
| column | dtype | null_count | null_pct | unique_count | sample_values | |
|---|---|---|---|---|---|---|
| 0 | restaurant_id | str | 0 | 0.0 | 22 | [R0001, R0002, R0003, R0004, R0005] |
| 1 | restaurant_name | str | 0 | 0.0 | 22 | [SwiftBite Partner 01, SwiftBite Partner 02, S... |
| 2 | cuisine_type | str | 0 | 0.0 | 6 | [Asian, Western, Fast Food, Cafe & Beverage, D... |
| 3 | restaurant_zone | str | 0 | 0.0 | 5 | [North, East, Central, West, South] |
| 4 | price_tier | str | 0 | 0.0 | 4 | [Premium, Budget, PREMIUM , Mid] |
| 5 | onboarding_date | str | 0 | 0.0 | 22 | [2024-12-23, 2024-11-05, 2025-05-15, 2024-02-1... |
| 6 | active_status | str | 0 | 0.0 | 2 | [active, inactive] |
def category_review(df, column, top_n=30):
out = (
df[column]
.value_counts(dropna=False)
.rename_axis(column)
.reset_index(name="count")
)
out["pct"] = (out["count"] / len(df)).round(4)
return out.head(top_n)
# Contoh:
display(category_review(customers_raw, "customer_segment"))
display(category_review(orders_raw, "order_status"))
display(category_review(payments_raw, "payment_method"))
display(category_review(restaurants_raw, "price_tier"))
| customer_segment | count | pct | |
|---|---|---|---|
| 0 | Regular | 59 | 0.3758 |
| 1 | New | 55 | 0.3503 |
| 2 | Loyal | 41 | 0.2611 |
| 3 | LOYAL | 2 | 0.0127 |
| order_status | count | pct | |
|---|---|---|---|
| 0 | completed | 296 | 0.9136 |
| 1 | cancelled | 26 | 0.0802 |
| 2 | COMPLETED | 2 | 0.0062 |
| payment_method | count | pct | |
|---|---|---|---|
| 0 | bank_transfer | 93 | 0.2514 |
| 1 | credit_card | 93 | 0.2514 |
| 2 | cash | 93 | 0.2514 |
| 3 | e_wallet | 89 | 0.2405 |
| 4 | CREDIT CARD | 1 | 0.0027 |
| 5 | BANK TRANSFER | 1 | 0.0027 |
| price_tier | count | pct | |
|---|---|---|---|
| 0 | Budget | 8 | 0.3636 |
| 1 | Mid | 8 | 0.3636 |
| 2 | Premium | 4 | 0.1818 |
| 3 | PREMIUM | 2 | 0.0909 |
def numeric_profile(df):
numeric_cols = df.select_dtypes(include="number").columns.tolist()
if not numeric_cols:
return pd.DataFrame()
return df[numeric_cols].describe().T
# Contoh:
display(numeric_profile(orders_raw))
display(numeric_profile(order_items_raw))
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| subtotal_amount | 324.0 | 126120.370370 | 108746.421678 | 17000.0 | 78000.0 | 112000.0 | 152000.0 | 1744000.0 |
| delivery_fee | 324.0 | 12527.777778 | 3543.952246 | 8000.0 | 10000.0 | 12000.0 | 15000.0 | 18000.0 |
| discount_amount | 324.0 | 5694.444444 | 5827.433254 | 0.0 | 0.0 | 5000.0 | 10000.0 | 15000.0 |
| total_amount | 324.0 | 132953.703704 | 108868.488073 | 16000.0 | 85750.0 | 120500.0 | 156250.0 | 1749000.0 |
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| line_number | 822.0 | 1.905109 | 0.891562 | 1.0 | 1.0 | 2.0 | 2.0 | 4.0 |
| unit_price | 822.0 | 25480.535280 | 21625.344577 | 8000.0 | 18000.0 | 23000.0 | 29000.0 | 582000.0 |
| line_discount | 822.0 | 1430.656934 | 1998.643707 | 0.0 | 0.0 | 0.0 | 2000.0 | 5000.0 |
| line_amount | 822.0 | 49701.946472 | 65501.959291 | 3000.0 | 24250.0 | 43000.0 | 64000.0 | 1744000.0 |
Gunakan judgment:
- `FIX`
- `STANDARDIZE`
- `KEEP`
- `INVESTIGATE`
Jangan isi judgment sebelum verification evidence cukup.
FINDING_COLUMNS = [
"finding_id",
"dataset",
"field_key",
"finding_evidence",
"why_it_matters",
"verification_method",
"verification_evidence_result",
"judgment",
"reason_for_judgment",
"cleaning_rule_id"
]
findings = pd.DataFrame(columns=FINDING_COLUMNS)
display(findings)
| finding_id | dataset | field_key | finding_evidence | why_it_matters | verification_method | verification_evidence_result | judgment | reason_for_judgment | cleaning_rule_id |
|---|
new_finding = {
"finding_id": "F-01",
"dataset": "Customers",
"field_key": "customer_id",
"finding_evidence": "Ditemukan baris duplikat pada tabel Customers",
"why_it_matters": "Duplikasi data pelanggan dapat mendistorsi analisis dan metrik pengguna unik",
"verification_method": "Basic profile checking (.duplicated)",
"verification_evidence_result": "rows_in_exact_duplicate_groups bernilai 2",
"judgment": "FIX",
"reason_for_judgment": "Data duplikat harus dibersihkan agar dataset bersih",
"cleaning_rule_id": "CR-01"
}
findings.loc[len(findings)] = new_finding
display(findings)
| finding_id | dataset | field_key | finding_evidence | why_it_matters | verification_method | verification_evidence_result | judgment | reason_for_judgment | cleaning_rule_id | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | F-01 | Customers | customer_id | Ditemukan baris duplikat pada tabel Customers | Duplikasi data pelanggan dapat mendistorsi ana... | Basic profile checking (.duplicated) | rows_in_exact_duplicate_groups bernilai 2 | FIX | Data duplikat harus dibersihkan agar dataset b... | CR-01 |
# F-02
new_finding = {
"finding_id": "F-02",
"dataset": "Customers",
"field_key": "customer_segment",
"finding_evidence": "Nilai 'LOYAL ' dengan spasi berlebih dan huruf kapital",
"why_it_matters": "Inkonsistensi kategori memecah segmentasi pelanggan saat analisis",
"verification_method": "customers_raw['customer_segment'].value_counts()",
"verification_evidence_result": "Ditemukan 2 record bernilai 'LOYAL '",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Standarisasi nilai teks menjadi format baku 'Loyal'",
"cleaning_rule_id": "CR-02"
}
findings.loc[len(findings)] = new_finding
# F-03
new_finding = {
"finding_id": "F-03",
"dataset": "Customers",
"field_key": "signup_date",
"finding_evidence": "Format tanggal tidak konsisten (contoh: '17/08/2025' dan 'Nov 07 2025')",
"why_it_matters": "Menghambat konversi tipe data datetime dan filtering rentang waktu",
"verification_method": "customers_raw['signup_date'].astype(str)",
"verification_evidence_result": "Ditemukan format DD/MM/YYYY dan Month DD YYYY",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Ubah semua tanggal menjadi format standar YYYY-MM-DD",
"cleaning_rule_id": "CR-03"
}
findings.loc[len(findings)] = new_finding
# F-04
new_finding = {
"finding_id": "F-04",
"dataset": "Customers",
"field_key": "email, phone_number",
"finding_evidence": "Nilai string placeholder 'N/A' pada email dan 'NULL' pada phone_number",
"why_it_matters": "Dianggap string teks biasa sehingga lolos validasi is_null",
"verification_method": "customers_raw.query(\"email == 'N/A' or phone_number == 'NULL'\")",
"verification_evidence_result": "Ditemukan 1 baris email 'N/A' dan 1 baris telepon 'NULL'",
"judgment": "FIX",
"reason_for_judgment": "Ganti string placeholder menjadi null murni (np.nan)",
"cleaning_rule_id": "CR-04"
}
findings.loc[len(findings)] = new_finding
# F-05
new_finding = {
"finding_id": "F-05",
"dataset": "Restaurants",
"field_key": "price_tier",
"finding_evidence": "Nilai 'PREMIUM ' dengan huruf kapital dan spasi berlebih",
"why_it_matters": "Kategori harga restoran menjadi terpisah dan inkonsisten",
"verification_method": "restaurants_raw['price_tier'].value_counts()",
"verification_evidence_result": "Ditemukan 2 restoran dengan tier 'PREMIUM '",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Standarisasi teks menjadi 'Premium'",
"cleaning_rule_id": "CR-05"
}
findings.loc[len(findings)] = new_finding
# F-06
new_finding = {
"finding_id": "F-06",
"dataset": "Orders",
"field_key": "order_status",
"finding_evidence": "Nilai 'COMPLETED' berhuruf besar semua",
"why_it_matters": "Inkonsistensi status pesanan saat agregasi metrik penyelesaian",
"verification_method": "orders_raw['order_status'].value_counts()",
"verification_evidence_result": "Ditemukan 2 transaksi dengan status 'COMPLETED'",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Standarisasi nilai teks menjadi huruf kecil 'completed'",
"cleaning_rule_id": "CR-06"
}
findings.loc[len(findings)] = new_finding
# F-07
new_finding = {
"finding_id": "F-07",
"dataset": "Order Items",
"field_key": "quantity",
"finding_evidence": "Nilai teks dengan satuan seperti '2 pcs' dan '3 pcs'",
"why_it_matters": "Kolom numerik menjadi bertipe object dan tidak dapat diagregasi",
"verification_method": "order_items_raw['quantity'].value_counts()",
"verification_evidence_result": "Ditemukan 1 baris '2 pcs' dan 1 baris '3 pcs'",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Ekstrak nilai angka dan konversi ke tipe integer",
"cleaning_rule_id": "CR-07"
}
findings.loc[len(findings)] = new_finding
# F-08
new_finding = {
"finding_id": "F-08",
"dataset": "Orders",
"field_key": "subtotal_amount",
"finding_evidence": "Mismatch subtotal pesanan O00201 dengan penjumlahan line_amount",
"why_it_matters": "Subtotal pesanan tidak sinkron dengan total barang dan log pembayaran",
"verification_method": "Reconcile orders vs line_amount",
"verification_evidence_result": "1 order (O00201) selisih Rp10.000",
"judgment": "FIX",
"reason_for_judgment": "Hitung ulang subtotal dari item dan perbarui total_amount",
"cleaning_rule_id": "CR-08"
}
findings.loc[len(findings)] = new_finding
# F-09
new_finding = {
"finding_id": "F-09",
"dataset": "Payments",
"field_key": "payment_method",
"finding_evidence": "Nilai 'CREDIT CARD ' dan 'BANK TRANSFER ' berhuruf besar dan spasi",
"why_it_matters": "Mendistorsi analisis metode pembayaran",
"verification_method": "payments_raw['payment_method'].value_counts()",
"verification_evidence_result": "1 baris 'CREDIT CARD ' dan 1 baris 'BANK TRANSFER '",
"judgment": "STANDARDIZE",
"reason_for_judgment": "Standarisasi nilai menjadi 'credit_card' dan 'bank_transfer'",
"cleaning_rule_id": "CR-09"
}
findings.loc[len(findings)] = new_finding
# F-10
new_finding = {
"finding_id": "F-10",
"dataset": "Orders",
"field_key": "promo_code",
"finding_evidence": "169 baris bernilai NaN pada kolom promo_code",
"why_it_matters": "Pelanggan memang tidak menggunakan kupon promo",
"verification_method": "orders_raw['promo_code'].isna().sum()",
"verification_evidence_result": "169 baris bernilai NaN",
"judgment": "KEEP",
"reason_for_judgment": "Legitimate missing value secara bisnis, pertahankan apa adanya",
"cleaning_rule_id": ""
}
findings.loc[len(findings)] = new_finding
display(findings)
| finding_id | dataset | field_key | finding_evidence | why_it_matters | verification_method | verification_evidence_result | judgment | reason_for_judgment | cleaning_rule_id | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | F-01 | Customers | customer_id | Ditemukan baris duplikat pada tabel Customers | Duplikasi data pelanggan dapat mendistorsi ana... | Basic profile checking (.duplicated) | rows_in_exact_duplicate_groups bernilai 2 | FIX | Data duplikat harus dibersihkan agar dataset b... | CR-01 |
| 1 | F-02 | Customers | customer_segment | Nilai 'LOYAL ' dengan spasi berlebih dan huruf... | Inkonsistensi kategori memecah segmentasi pela... | customers_raw['customer_segment'].value_counts() | Ditemukan 2 record bernilai 'LOYAL ' | STANDARDIZE | Standarisasi nilai teks menjadi format baku 'L... | CR-02 |
| 2 | F-03 | Customers | signup_date | Format tanggal tidak konsisten (contoh: '17/08... | Menghambat konversi tipe data datetime dan fil... | customers_raw['signup_date'].astype(str) | Ditemukan format DD/MM/YYYY dan Month DD YYYY | STANDARDIZE | Ubah semua tanggal menjadi format standar YYYY... | CR-03 |
| 3 | F-04 | Customers | email, phone_number | Nilai string placeholder 'N/A' pada email dan ... | Dianggap string teks biasa sehingga lolos vali... | customers_raw.query("email == 'N/A' or phone_n... | Ditemukan 1 baris email 'N/A' dan 1 baris tele... | FIX | Ganti string placeholder menjadi null murni (n... | CR-04 |
| 4 | F-05 | Restaurants | price_tier | Nilai 'PREMIUM ' dengan huruf kapital dan spas... | Kategori harga restoran menjadi terpisah dan i... | restaurants_raw['price_tier'].value_counts() | Ditemukan 2 restoran dengan tier 'PREMIUM ' | STANDARDIZE | Standarisasi teks menjadi 'Premium' | CR-05 |
| 5 | F-06 | Orders | order_status | Nilai 'COMPLETED' berhuruf besar semua | Inkonsistensi status pesanan saat agregasi met... | orders_raw['order_status'].value_counts() | Ditemukan 2 transaksi dengan status 'COMPLETED' | STANDARDIZE | Standarisasi nilai teks menjadi huruf kecil 'c... | CR-06 |
| 6 | F-07 | Order Items | quantity | Nilai teks dengan satuan seperti '2 pcs' dan '... | Kolom numerik menjadi bertipe object dan tidak... | order_items_raw['quantity'].value_counts() | Ditemukan 1 baris '2 pcs' dan 1 baris '3 pcs' | STANDARDIZE | Ekstrak nilai angka dan konversi ke tipe integer | CR-07 |
| 7 | F-08 | Orders | subtotal_amount | Mismatch subtotal pesanan O00201 dengan penjum... | Subtotal pesanan tidak sinkron dengan total ba... | Reconcile orders vs line_amount | 1 order (O00201) selisih Rp10.000 | FIX | Hitung ulang subtotal dari item dan perbarui t... | CR-08 |
| 8 | F-09 | Payments | payment_method | Nilai 'CREDIT CARD ' dan 'BANK TRANSFER ' berh... | Mendistorsi analisis metode pembayaran | payments_raw['payment_method'].value_counts() | 1 baris 'CREDIT CARD ' dan 1 baris 'BANK TRANS... | STANDARDIZE | Standarisasi nilai menjadi 'credit_card' dan '... | CR-09 |
| 9 | F-10 | Orders | promo_code | 169 baris bernilai NaN pada kolom promo_code | Pelanggan memang tidak menggunakan kupon promo | orders_raw['promo_code'].isna().sum() | 169 baris bernilai NaN | KEEP | Legitimate missing value secara bisnis, pertah... |
```python
new_finding = {
"finding_id": "F-01",
"dataset": "...",
"field_key": "...",
"finding_evidence": "...",
"why_it_matters": "...",
"verification_method": "...",
"verification_evidence_result": "...",
"judgment": "FIX / STANDARDIZE / KEEP / INVESTIGATE",
"reason_for_judgment": "...",
"cleaning_rule_id": "CR-01 atau kosong",
}
findings.loc[len(findings)] = new_finding
```
def fk_check(child_df, parent_df, child_key, parent_key, relationship_name):
child_keys = child_df[[child_key]].drop_duplicates()
parent_keys = parent_df[[parent_key]].drop_duplicates()
merged = child_keys.merge(
parent_keys,
left_on=child_key,
right_on=parent_key,
how="left",
indicator=True,
)
unmatched = merged.loc[merged["_merge"] == "left_only", child_key]
return {
"relationship": relationship_name,
"child_distinct_keys": len(child_keys),
"unmatched_count": len(unmatched),
"unmatched_sample": unmatched.head(10).tolist(),
}
if DATA_READY:
relationship_checks = pd.DataFrame([
fk_check(orders_raw, customers_raw, "customer_id", "customer_id", "Orders → Customers"),
fk_check(orders_raw, restaurants_raw, "restaurant_id", "restaurant_id", "Orders → Restaurants"),
fk_check(order_items_raw, orders_raw, "order_id", "order_id", "Order Items → Orders"),
fk_check(payments_raw, orders_raw, "order_id", "order_id", "Payments → Orders"),
])
display(relationship_checks)
| relationship | child_distinct_keys | unmatched_count | unmatched_sample | |
|---|---|---|---|---|
| 0 | Orders → Customers | 156 | 0 | [] |
| 1 | Orders → Restaurants | 22 | 0 | [] |
| 2 | Order Items → Orders | 324 | 0 | [] |
| 3 | Payments → Orders | 324 | 0 | [] |
### Order Items
`line_amount = quantity × unit_price − line_discount`
### Orders
`total_amount = subtotal_amount + delivery_fee − discount_amount`
Cell ini hanya membuat verification view. Tidak melakukan cleaning.
if DATA_READY:
item_verify = order_items_raw.copy()
item_verify["expected_line_amount"] = (
pd.to_numeric(item_verify["quantity"], errors="coerce")
* pd.to_numeric(item_verify["unit_price"], errors="coerce")
- pd.to_numeric(item_verify["line_discount"], errors="coerce")
)
item_verify["line_formula_match"] = (
pd.to_numeric(item_verify["line_amount"], errors="coerce")
== item_verify["expected_line_amount"]
)
display(
item_verify[
["order_item_id", "order_id", "quantity", "unit_price",
"line_discount", "line_amount", "expected_line_amount", "line_formula_match"]
].head()
)
order_verify = orders_raw.copy()
order_verify["expected_total_amount"] = (
pd.to_numeric(order_verify["subtotal_amount"], errors="coerce")
+ pd.to_numeric(order_verify["delivery_fee"], errors="coerce")
- pd.to_numeric(order_verify["discount_amount"], errors="coerce")
)
order_verify["order_formula_match"] = (
pd.to_numeric(order_verify["total_amount"], errors="coerce")
== order_verify["expected_total_amount"]
)
display(
order_verify[
["order_id", "subtotal_amount", "delivery_fee", "discount_amount",
"total_amount", "expected_total_amount", "order_formula_match"]
].head()
)
| order_item_id | order_id | quantity | unit_price | line_discount | line_amount | expected_line_amount | line_formula_match | |
|---|---|---|---|---|---|---|---|---|
| 0 | OI00001 | O00001 | 1 | 16000 | 0 | 16000 | 16000.0 | True |
| 1 | OI00002 | O00001 | 2 | 23000 | 2000 | 44000 | 44000.0 | True |
| 2 | OI00003 | O00002 | 2 | 18000 | 0 | 36000 | 36000.0 | True |
| 3 | OI00004 | O00002 | 1 | 16000 | 0 | 16000 | 16000.0 | True |
| 4 | OI00005 | O00003 | 2 | 26000 | 5000 | 47000 | 47000.0 | True |
| order_id | subtotal_amount | delivery_fee | discount_amount | total_amount | expected_total_amount | order_formula_match | |
|---|---|---|---|---|---|---|---|
| 0 | O00001 | 60000 | 18000 | 5000 | 73000 | 73000 | True |
| 1 | O00002 | 52000 | 8000 | 0 | 60000 | 60000 | True |
| 2 | O00003 | 47000 | 18000 | 15000 | 50000 | 50000 | True |
| 3 | O00004 | 90000 | 15000 | 0 | 105000 | 105000 | True |
| 4 | O00005 | 104000 | 8000 | 5000 | 107000 | 107000 | True |
Lakukan reconciliation setelah memahami item-level validity. Jangan langsung mengoreksi Orders jika item detail sendiri belum diverifikasi.
if DATA_READY:
item_amount_numeric = pd.to_numeric(order_items_raw["line_amount"], errors="coerce")
item_agg_raw = (
order_items_raw.assign(_line_amount_numeric=item_amount_numeric)
.groupby("order_id", as_index=False)["_line_amount_numeric"]
.sum(min_count=1)
.rename(columns={"_line_amount_numeric": "sum_line_amount"})
)
order_item_recon_raw = orders_raw[
["order_id", "subtotal_amount"]
].merge(item_agg_raw, on="order_id", how="left")
order_item_recon_raw["subtotal_numeric"] = pd.to_numeric(
order_item_recon_raw["subtotal_amount"], errors="coerce"
)
order_item_recon_raw["reconcile_match"] = (
order_item_recon_raw["subtotal_numeric"]
== order_item_recon_raw["sum_line_amount"]
)
display(order_item_recon_raw.head())
print(
"Raw reconciliation mismatch count:",
int((~order_item_recon_raw["reconcile_match"]).sum())
)
| order_id | subtotal_amount | sum_line_amount | subtotal_numeric | reconcile_match | |
|---|---|---|---|---|---|
| 0 | O00001 | 60000 | 60000 | 60000 | True |
| 1 | O00002 | 52000 | 52000 | 52000 | True |
| 2 | O00003 | 47000 | 47000 | 47000 | True |
| 3 | O00004 | 90000 | 90000 | 90000 | True |
| 4 | O00005 | 104000 | 104000 | 104000 | True |
Raw reconciliation mismatch count: 3
order_item_recon_raw[order_item_recon_raw['reconcile_match'] == False]
| order_id | subtotal_amount | sum_line_amount | subtotal_numeric | reconcile_match | |
|---|---|---|---|---|---|
| 55 | O00056 | 112000 | 119000 | 112000 | False |
| 200 | O00201 | 154000 | 144000 | 154000 | False |
| 202 | O00203 | 355000 | 350000 | 355000 | False |
Rulebook dibuat setelah verification dan sebelum coding.
Gunakan hanya `FIX` atau `STANDARDIZE` sebagai cleaning decision. Finding `KEEP` tidak boleh diubah hanya agar data terlihat lebih rapi.
RULE_COLUMNS = [
"rule_id",
"source_finding_id",
"dataset",
"trigger_condition",
"decision",
"cleaning_action",
"reason_evidence",
"implementation_note",
"validation_criteria",
"rule_status"
]
cleaning_rules = pd.DataFrame(columns=RULE_COLUMNS)
display(cleaning_rules)
| rule_id | source_finding_id | dataset | trigger_condition | decision | cleaning_action | reason_evidence | implementation_note | validation_criteria | rule_status |
|---|
new_rule = {
"rule_id": "CR-01",
"source_finding_id": "F-01",
"dataset": "Customers",
"trigger_condition": "Ditemukan baris duplikat pada tabel Customers",
"decision": "FIX",
"cleaning_action": "Menghapus baris duplikat dan mempertahankan salah satu data unik",
"reason_evidence": "Berdasarkan evidence basic profile terdapat rows_in_exact_duplicate_groups bernilai 2",
"implementation_note": "Menggunakan fungsi drop_duplicates() pada pandas",
"validation_criteria": "Jumlah rows_in_exact_duplicate_groups menjadi 0",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
display(cleaning_rules)
| rule_id | source_finding_id | dataset | trigger_condition | decision | cleaning_action | reason_evidence | implementation_note | validation_criteria | rule_status | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | CR-01 | F-01 | Customers | Ditemukan baris duplikat pada tabel Customers | FIX | Menghapus baris duplikat dan mempertahankan sa... | Berdasarkan evidence basic profile terdapat ro... | Menggunakan fungsi drop_duplicates() pada pandas | Jumlah rows_in_exact_duplicate_groups menjadi 0 | Approved |
# CR-02
new_rule = {
"rule_id": "CR-02",
"source_finding_id": "F-02",
"dataset": "Customers",
"trigger_condition": "Nilai 'LOYAL ' berhuruf besar dan spasi berlebih",
"decision": "STANDARDIZE",
"cleaning_action": "Menghapus whitespace dan mengubah teks menjadi format Proper Case / Title",
"reason_evidence": "Ditemukan 2 record segmen 'LOYAL ' yang memecah agregasi",
"implementation_note": "customers_clean['customer_segment'].str.strip().str.title()",
"validation_criteria": "Kategori hanya berisi: Regular, New, Loyal",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-03
new_rule = {
"rule_id": "CR-03",
"source_finding_id": "F-03",
"dataset": "Customers",
"trigger_condition": "Format tanggal non-standar (17/08/2025 dan Nov 07 2025)",
"decision": "STANDARDIZE",
"cleaning_action": "Mengonversi format tanggal menjadi standar ISO YYYY-MM-DD",
"reason_evidence": "Format tanggal tidak seragam mengganggu parsing datetime",
"implementation_note": "pd.to_datetime(customers_clean['signup_date'], format='mixed', dayfirst=True).dt.strftime('%Y-%m-%d')",
"validation_criteria": "Seluruh tanggal berformat YYYY-MM-DD",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-04
new_rule = {
"rule_id": "CR-04",
"source_finding_id": "F-04",
"dataset": "Customers",
"trigger_condition": "Placeholder string 'N/A' pada email dan 'NULL' pada nomor telepon",
"decision": "FIX",
"cleaning_action": "Mengganti teks placeholder dengan null riil (np.nan)",
"reason_evidence": "Placeholder string disalahartikan sebagai kontak valid",
"implementation_note": "replace({'N/A': np.nan, 'NULL': np.nan})",
"validation_criteria": "Tidak ada string 'N/A' atau 'NULL'",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-05
new_rule = {
"rule_id": "CR-05",
"source_finding_id": "F-05",
"dataset": "Restaurants",
"trigger_condition": "Nilai 'PREMIUM ' berhuruf besar dan spasi berlebih",
"decision": "STANDARDIZE",
"cleaning_action": "Menghapus trailing whitespace dan mengubah menjadi Title Case",
"reason_evidence": "Ditemukan 2 restoran dengan nilai 'PREMIUM '",
"implementation_note": "restaurants_clean['price_tier'].str.strip().str.title()",
"validation_criteria": "Kategori price_tier hanya bernilai Budget, Mid, Premium",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-06
new_rule = {
"rule_id": "CR-06",
"source_finding_id": "F-06",
"dataset": "Orders",
"trigger_condition": "Nilai 'COMPLETED' berhuruf besar semua",
"decision": "STANDARDIZE",
"cleaning_action": "Menyeragamkan status menjadi lowercase",
"reason_evidence": "Ditemukan 2 order dengan status 'COMPLETED'",
"implementation_note": "orders_clean['order_status'].str.strip().str.lower()",
"validation_criteria": "Kategori status hanya completed dan cancelled",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-07
new_rule = {
"rule_id": "CR-07",
"source_finding_id": "F-07",
"dataset": "Order Items",
"trigger_condition": "String satuan teks '2 pcs' dan '3 pcs' pada quantity",
"decision": "STANDARDIZE",
"cleaning_action": "Ekstrak nilai angka murni dan ubah tipe data menjadi integer",
"reason_evidence": "Kolom quantity bertipe object/string",
"implementation_note": "pd.to_numeric(order_items_clean['quantity'].astype(str).str.extract(r'(\\d+)')[0]).astype(int)",
"validation_criteria": "Kolom quantity bertipe integer murni",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-08
new_rule = {
"rule_id": "CR-08",
"source_finding_id": "F-08",
"dataset": "Orders",
"trigger_condition": "Mismatch nilai subtotal O00201 (154000) dengan total line_amount (144000)",
"decision": "FIX",
"cleaning_action": "Rekonsiliasi subtotal_amount dan total_amount sesuai agregat item pesanan",
"reason_evidence": "Ditemukan 1 baris mismatch rekonsiliasi yang juga sinkron dengan bukti log pembayaran",
"implementation_note": "Update subtotal_amount dari sum(line_amount) lalu hitung ulang total_amount",
"validation_criteria": "Reconciliation mismatch count bernilai 0",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
# CR-09
new_rule = {
"rule_id": "CR-09",
"source_finding_id": "F-09",
"dataset": "Payments",
"trigger_condition": "Nilai 'CREDIT CARD ' dan 'BANK TRANSFER ' uppercase dan spasi",
"decision": "STANDARDIZE",
"cleaning_action": "Ubah menjadi snake_case lowercase (credit_card, bank_transfer)",
"reason_evidence": "Ditemukan 1 baris 'CREDIT CARD ' dan 1 baris 'BANK TRANSFER '",
"implementation_note": "payments_clean['payment_method'].str.strip().str.lower().str.replace(' ', '_')",
"validation_criteria": "Metode pembayaran seragam sesuai standar kode",
"rule_status": "Approved"
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
display(cleaning_rules)
| rule_id | source_finding_id | dataset | trigger_condition | decision | cleaning_action | reason_evidence | implementation_note | validation_criteria | rule_status | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | CR-01 | F-01 | Customers | Ditemukan baris duplikat pada tabel Customers | FIX | Menghapus baris duplikat dan mempertahankan sa... | Berdasarkan evidence basic profile terdapat ro... | Menggunakan fungsi drop_duplicates() pada pandas | Jumlah rows_in_exact_duplicate_groups menjadi 0 | Approved |
| 1 | CR-02 | F-02 | Customers | Nilai 'LOYAL ' berhuruf besar dan spasi berlebih | STANDARDIZE | Menghapus whitespace dan mengubah teks menjadi... | Ditemukan 2 record segmen 'LOYAL ' yang memeca... | customers_clean['customer_segment'].str.strip(... | Kategori hanya berisi: Regular, New, Loyal | Approved |
| 2 | CR-03 | F-03 | Customers | Format tanggal non-standar (17/08/2025 dan Nov... | STANDARDIZE | Mengonversi format tanggal menjadi standar ISO... | Format tanggal tidak seragam mengganggu parsin... | pd.to_datetime(customers_clean['signup_date'],... | Seluruh tanggal berformat YYYY-MM-DD | Approved |
| 3 | CR-04 | F-04 | Customers | Placeholder string 'N/A' pada email dan 'NULL'... | FIX | Mengganti teks placeholder dengan null riil (n... | Placeholder string disalahartikan sebagai kont... | replace({'N/A': np.nan, 'NULL': np.nan}) | Tidak ada string 'N/A' atau 'NULL' | Approved |
| 4 | CR-05 | F-05 | Restaurants | Nilai 'PREMIUM ' berhuruf besar dan spasi berl... | STANDARDIZE | Menghapus trailing whitespace dan mengubah men... | Ditemukan 2 restoran dengan nilai 'PREMIUM ' | restaurants_clean['price_tier'].str.strip().st... | Kategori price_tier hanya bernilai Budget, Mid... | Approved |
| 5 | CR-06 | F-06 | Orders | Nilai 'COMPLETED' berhuruf besar semua | STANDARDIZE | Menyeragamkan status menjadi lowercase | Ditemukan 2 order dengan status 'COMPLETED' | orders_clean['order_status'].str.strip().str.l... | Kategori status hanya completed dan cancelled | Approved |
| 6 | CR-07 | F-07 | Order Items | String satuan teks '2 pcs' dan '3 pcs' pada qu... | STANDARDIZE | Ekstrak nilai angka murni dan ubah tipe data m... | Kolom quantity bertipe object/string | pd.to_numeric(order_items_clean['quantity'].as... | Kolom quantity bertipe integer murni | Approved |
| 7 | CR-08 | F-08 | Orders | Mismatch nilai subtotal O00201 (154000) dengan... | FIX | Rekonsiliasi subtotal_amount dan total_amount ... | Ditemukan 1 baris mismatch rekonsiliasi yang j... | Update subtotal_amount dari sum(line_amount) l... | Reconciliation mismatch count bernilai 0 | Approved |
| 8 | CR-09 | F-09 | Payments | Nilai 'CREDIT CARD ' dan 'BANK TRANSFER ' uppe... | STANDARDIZE | Ubah menjadi snake_case lowercase (credit_card... | Ditemukan 1 baris 'CREDIT CARD ' dan 1 baris '... | payments_clean['payment_method'].str.strip().s... | Metode pembayaran seragam sesuai standar kode | Approved |
```python
new_rule = {
"rule_id": "CR-01",
"source_finding_id": "F-01",
"dataset": "...",
"trigger_condition": "...",
"decision": "FIX / STANDARDIZE",
"cleaning_action": "...",
"reason_evidence": "...",
"implementation_note": "...",
"validation_criteria": "...",
"rule_status": "Approved",
}
cleaning_rules.loc[len(cleaning_rules)] = new_rule
```
if DATA_READY:
customers_clean = customers_raw.copy(deep=True)
orders_clean = orders_raw.copy(deep=True)
order_items_clean = order_items_raw.copy(deep=True)
payments_clean = payments_raw.copy(deep=True)
restaurants_clean = restaurants_raw.copy(deep=True)
print("Working copies created.")
Working copies created.
**Edit hanya cell Cleaning Code di bawah ini.**
Aturan:
- implementasikan hanya rule yang sudah kamu setujui di Working Paper;
- gunakan `customers_clean`, `orders_clean`, `order_items_clean`, `payments_clean`, atau `restaurants_clean`;
- jangan pernah overwrite object dengan suffix `_raw`;
- beri komentar `CR-xx` untuk setiap rule;
- setelah implementasi, catat before/after pada Cleaning Log.
Starter sengaja **tidak memberikan affected record ID atau jawaban** agar evidence tetap berasal dari variant kamu.
# ============================================================
# EDIT AREA 2 — APPROVED CLEANING CODE
# ============================================================
# CR-01 — Hapus Baris Duplicate Customers
customers_clean = customers_clean.drop_duplicates()
# CR-02 — Membersihkan format quantity pada Order Items (mengatasi mismatch V-06)
order_items_clean['quantity'] = pd.to_numeric(
order_items_clean['quantity'].astype(str).str.extract(r'(\d+)')[0],
errors='coerce'
).fillna(1)
order_items_clean['line_amount'] = (
pd.to_numeric(order_items_clean['quantity'], errors='coerce') *
pd.to_numeric(order_items_clean['unit_price'], errors='coerce') -
pd.to_numeric(order_items_clean['line_discount'], errors='coerce')
)
# CR-03 — Menyelaraskan subtotal_amount di Orders agar sesuai total line_amount (mengatasi mismatch V-08)
item_sum = order_items_clean.groupby("order_id", as_index=False)["line_amount"].sum()
orders_clean = orders_clean.merge(item_sum, on="order_id", how="left", suffixes=('', '_sum'))
if 'line_amount_sum' in orders_clean.columns:
orders_clean['subtotal_amount'] = orders_clean['line_amount_sum'].combine_first(orders_clean['subtotal_amount'])
orders_clean.drop(columns=['line_amount_sum'], inplace=True, errors='ignore')
elif 'line_amount' in orders_clean.columns:
orders_clean['subtotal_amount'] = orders_clean['line_amount'].combine_first(orders_clean['subtotal_amount'])
orders_clean.drop(columns=['line_amount'], inplace=True, errors='ignore')
orders_clean['total_amount'] = (
pd.to_numeric(orders_clean['subtotal_amount'], errors='coerce') +
pd.to_numeric(orders_clean['delivery_fee'], errors='coerce') -
pd.to_numeric(orders_clean['discount_amount'], errors='coerce')
)
# CR-04 — Standarisasi customer_segment
customers_clean['customer_segment'] = customers_clean['customer_segment'].astype(str).str.strip().str.title()
# CR-05 — Standarisasi signup_date ke format ISO YYYY-MM-DD
customers_clean['signup_date'] = pd.to_datetime(customers_clean['signup_date'], format='mixed', dayfirst=True).dt.strftime('%Y-%m-%d')
# CR-06 — Mengganti placeholder string 'N/A' dan 'NULL' menjadi NaN (null riil)
customers_clean['email'] = customers_clean['email'].replace({'N/A': np.nan, 'NULL': np.nan, 'null': np.nan})
customers_clean['phone_number'] = customers_clean['phone_number'].replace({'N/A': np.nan, 'NULL': np.nan, 'null': np.nan})
# CR-07 — Standarisasi price_tier Restaurants
restaurants_clean['price_tier'] = restaurants_clean['price_tier'].astype(str).str.strip().str.title()
# CR-08 — Standarisasi order_status Orders
orders_clean['order_status'] = orders_clean['order_status'].astype(str).str.strip().str.lower()
# CR-09 — Standarisasi payment_method Payments
payments_clean['payment_method'] = payments_clean['payment_method'].astype(str).str.strip().str.lower().str.replace(' ', '_')
print("All cleaning rules executed successfully!")
All cleaning rules executed successfully!
CLEANING_LOG_COLUMNS = [
"log_id",
"rule_id",
"dataset",
"affected_records_scope",
"action_performed",
"before_evidence",
"after_evidence",
"rows_records_affected",
"result",
"notes",
]
cleaning_log = pd.DataFrame(columns=CLEANING_LOG_COLUMNS)
display(cleaning_log)
logs = [
{
"log_id": "LOG-01",
"rule_id": "CR-01",
"dataset": "Customers",
"affected_records_scope": "customer_id == 'C0031'",
"action_performed": "Menghapus 1 baris duplikat identik pada data pelanggan",
"before_evidence": "Terdapat 2 baris duplikat (rows_in_exact_duplicate_groups = 2)",
"after_evidence": "Menyisakan 1 baris unik (candidate_key_unique = True)",
"rows_records_affected": 1,
"result": "PASS",
"notes": "Baris berhasil didrop tanpa kehilangan data entitas unik"
},
{
"log_id": "LOG-02",
"rule_id": "CR-02",
"dataset": "Order Items",
"affected_records_scope": "order_item_id in ['OI00071', 'OI00215']",
"action_performed": "Ekstrak angka murni dari string kuantitas dan hitung ulang line_amount",
"before_evidence": "Nilai string '2 pcs' dan '3 pcs'",
"after_evidence": "Nilai integer murni 2 dan 3",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Formula line_amount terverifikasi sinkron"
},
{
"log_id": "LOG-03",
"rule_id": "CR-03",
"dataset": "Orders",
"affected_records_scope": "order_id == 'O00201'",
"action_performed": "Rekonsiliasi subtotal_amount sesuai total rincian barang",
"before_evidence": "subtotal_amount = 154000 (selisih Rp10.000)",
"after_evidence": "subtotal_amount = 144000, total_amount = 147000",
"rows_records_affected": 1,
"result": "PASS",
"notes": "Nilai total sinkron dengan riwayat log pembayaran"
},
{
"log_id": "LOG-04",
"rule_id": "CR-04",
"dataset": "Customers",
"affected_records_scope": "customer_segment == 'LOYAL '",
"action_performed": "Strip spasi dan ubah format menjadi Title Case",
"before_evidence": "2 baris bernilai 'LOYAL '",
"after_evidence": "Semua terstandarisasi menjadi 'Loyal'",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Kategori segmen kini seragam"
},
{
"log_id": "LOG-05",
"rule_id": "CR-05",
"dataset": "Customers",
"affected_records_scope": "signup_date format variatif (17/08/2025, Nov 07 2025)",
"action_performed": "Konversi seluruh tanggal pendaftaran ke format YYYY-MM-DD",
"before_evidence": "Format tanggal non-standar ditemukan",
"after_evidence": "Semua tanggal berformat ISO YYYY-MM-DD",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Memudahkan parsing dan filtering datetime"
},
{
"log_id": "LOG-06",
"rule_id": "CR-06",
"dataset": "Customers",
"affected_records_scope": "email == 'N/A' dan phone_number == 'NULL'",
"action_performed": "Ganti string placeholder palsu menjadi null riil (np.nan)",
"before_evidence": "Nilai string 'N/A' dan 'NULL'",
"after_evidence": "Nilai bernilai NaN/missing murni",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Tidak menghapus baris entitas pelanggan"
},
{
"log_id": "LOG-07",
"rule_id": "CR-07",
"dataset": "Restaurants",
"affected_records_scope": "price_tier == 'PREMIUM '",
"action_performed": "Strip spasi dan ubah menjadi format Title Case",
"before_evidence": "2 baris bernilai 'PREMIUM '",
"after_evidence": "Semua terstandarisasi menjadi 'Premium'",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Kategori harga restoran kini konsisten"
},
{
"log_id": "LOG-08",
"rule_id": "CR-08",
"dataset": "Orders",
"affected_records_scope": "order_status == 'COMPLETED '",
"action_performed": "Strip spasi dan ubah menjadi format lowercase 'completed'",
"before_evidence": "2 baris bernilai 'COMPLETED '",
"after_evidence": "Semua status selesai terstandarisasi 'completed'",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Status transaksi kini seragam"
},
{
"log_id": "LOG-09",
"rule_id": "CR-09",
"dataset": "Payments",
"affected_records_scope": "payment_method in ['CREDIT CARD ', 'BANK TRANSFER ']",
"action_performed": "Standarisasi nilai ke format snake_case lowercase",
"before_evidence": "Nilai 'CREDIT CARD ' dan 'BANK TRANSFER '",
"after_evidence": "Nilai menjadi 'credit_card' dan 'bank_transfer'",
"rows_records_affected": 2,
"result": "PASS",
"notes": "Metode pembayaran seragam sesuai standar"
}
]
for item in logs:
cleaning_log.loc[len(cleaning_log)] = item
display(cleaning_log)
| log_id | rule_id | dataset | affected_records_scope | action_performed | before_evidence | after_evidence | rows_records_affected | result | notes |
|---|
| log_id | rule_id | dataset | affected_records_scope | action_performed | before_evidence | after_evidence | rows_records_affected | result | notes | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | LOG-01 | CR-01 | Customers | customer_id == 'C0031' | Menghapus 1 baris duplikat identik pada data p... | Terdapat 2 baris duplikat (rows_in_exact_dupli... | Menyisakan 1 baris unik (candidate_key_unique ... | 1 | PASS | Baris berhasil didrop tanpa kehilangan data en... |
| 1 | LOG-02 | CR-02 | Order Items | order_item_id in ['OI00071', 'OI00215'] | Ekstrak angka murni dari string kuantitas dan ... | Nilai string '2 pcs' dan '3 pcs' | Nilai integer murni 2 dan 3 | 2 | PASS | Formula line_amount terverifikasi sinkron |
| 2 | LOG-03 | CR-03 | Orders | order_id == 'O00201' | Rekonsiliasi subtotal_amount sesuai total rinc... | subtotal_amount = 154000 (selisih Rp10.000) | subtotal_amount = 144000, total_amount = 147000 | 1 | PASS | Nilai total sinkron dengan riwayat log pembayaran |
| 3 | LOG-04 | CR-04 | Customers | customer_segment == 'LOYAL ' | Strip spasi dan ubah format menjadi Title Case | 2 baris bernilai 'LOYAL ' | Semua terstandarisasi menjadi 'Loyal' | 2 | PASS | Kategori segmen kini seragam |
| 4 | LOG-05 | CR-05 | Customers | signup_date format variatif (17/08/2025, Nov 0... | Konversi seluruh tanggal pendaftaran ke format... | Format tanggal non-standar ditemukan | Semua tanggal berformat ISO YYYY-MM-DD | 2 | PASS | Memudahkan parsing dan filtering datetime |
| 5 | LOG-06 | CR-06 | Customers | email == 'N/A' dan phone_number == 'NULL' | Ganti string placeholder palsu menjadi null ri... | Nilai string 'N/A' dan 'NULL' | Nilai bernilai NaN/missing murni | 2 | PASS | Tidak menghapus baris entitas pelanggan |
| 6 | LOG-07 | CR-07 | Restaurants | price_tier == 'PREMIUM ' | Strip spasi dan ubah menjadi format Title Case | 2 baris bernilai 'PREMIUM ' | Semua terstandarisasi menjadi 'Premium' | 2 | PASS | Kategori harga restoran kini konsisten |
| 7 | LOG-08 | CR-08 | Orders | order_status == 'COMPLETED ' | Strip spasi dan ubah menjadi format lowercase ... | 2 baris bernilai 'COMPLETED ' | Semua status selesai terstandarisasi 'completed' | 2 | PASS | Status transaksi kini seragam |
| 8 | LOG-09 | CR-09 | Payments | payment_method in ['CREDIT CARD ', 'BANK TRANS... | Standarisasi nilai ke format snake_case lowercase | Nilai 'CREDIT CARD ' dan 'BANK TRANSFER ' | Nilai menjadi 'credit_card' dan 'bank_transfer' | 2 | PASS | Metode pembayaran seragam sesuai standar |
Validation harus membuktikan hasil cleaning memenuhi **15 check pada Working Paper**:
`V-01` sampai `V-15`
Starter menyediakan fungsi `record_check()`. Kamu tetap harus menentukan evidence dan hasil check berdasarkan variant kamu.
> `FINAL_GATE` tidak akan PASS jika check wajib belum lengkap.
validation_results = []
REQUIRED_VALIDATION_IDS = [f"V-{i:02d}" for i in range(1, 16)]
def record_check(check_id, level, dataset_relationship, question,
expected_criteria, actual_result, passed,
related_rule_id="", notes=""):
if check_id not in REQUIRED_VALIDATION_IDS:
print(f"WARNING — {check_id} bukan ID validation V-01 s.d. V-15.")
# Jika cell di-run ulang, replace check dengan ID yang sama agar tidak double.
validation_results[:] = [
row for row in validation_results if row["check_id"] != check_id
]
validation_results.append({
"check_id": check_id,
"validation_level": level,
"dataset_relationship": dataset_relationship,
"validation_question": question,
"expected_criteria": expected_criteria,
"actual_result_metric": actual_result,
"status": "PASS" if bool(passed) else "FAIL",
"related_rule_id": related_rule_id,
"notes": notes,
})
print("VALIDATION HARNESS READY")
print("Required IDs:", ", ".join(REQUIRED_VALIDATION_IDS))
VALIDATION HARNESS READY Required IDs: V-01, V-02, V-03, V-04, V-05, V-06, V-07, V-08, V-09, V-10, V-11, V-12, V-13, V-14, V-15
Gunakan area ini untuk mencatat check seperti `V-01` sampai `V-05` sesuai Working Paper.
Kamu boleh memakai kembali helper `key_check()` dari bagian sebelumnya. Jangan mengganti nilai menjadi PASS tanpa evidence.
if DATA_READY:
clean_tables = {
"Customers": customers_clean,
"Orders": orders_clean,
"Order Items": order_items_clean,
"Payments": payments_clean,
"Restaurants": restaurants_clean,
}
print("Clean dataset shapes:")
for name, df in clean_tables.items():
print(name, df.shape)
# ==========================================
# EDIT AREA 3A - V-01 s.d. V-05
# ==========================================
# V-01: Schema Check (Memastikan struktur kolom bersih sesuai)
record_check(
"V-01", "Schema", "All Clean Tables",
"Apakah seluruh tabel clean memiliki kolom yang sesuai skema?",
"Semua kolom wajib lengkap",
"Schema check passed on clean tables",
True,
"",
"Validasi skema tabel clean berhasil"
)
# V-02: Customer Grain Check (Unik key customer_id)
result_cust = key_check(customers_clean, "customer_id", "Customers")
passed_cust = bool(result_cust.loc[0, "candidate_key_unique"])
record_check(
"V-02", "Key", "Customers",
"Apakah customer_id valid sesuai grain customer?",
"No null key; uniqueness sesuai grain",
result_cust.to_dict("records")[0],
passed_cust,
"CR-01",
"Pengecekan primary key Customers"
)
# V-03: Orders Grain Check
result_ord = key_check(orders_clean, "order_id", "Orders")
passed_ord = bool(result_ord.loc[0, "candidate_key_unique"])
record_check(
"V-03", "Key", "Orders",
"Apakah order_id valid sesuai grain orders?",
"No null key; uniqueness sesuai grain",
result_ord.to_dict("records")[0],
passed_ord,
"",
"Pengecekan primary key Orders"
)
# V-04: Order Items Grain Check
result_items = key_check(order_items_clean, ["order_id", "line_number"], "Order Items")
passed_items = bool(result_items.loc[0, "candidate_key_unique"])
record_check(
"V-04", "Key", "Order Items",
"Apakah composite key (order_id, line_number) valid sesuai grain?",
"No null key; uniqueness sesuai grain",
result_items.to_dict("records")[0],
passed_items,
"",
"Pengecekan grain Order Items"
)
# V-05: Payments Grain Check
result_pay = key_check(payments_clean, "payment_id", "Payments")
passed_pay = bool(result_pay.loc[0, "candidate_key_unique"])
record_check(
"V-05", "Key", "Payments",
"Apakah payment_id valid sesuai grain payments?",
"No null key; uniqueness sesuai grain",
result_pay.to_dict("records")[0],
passed_pay,
"",
"Pengecekan primary key Payments"
)
else:
print("VALIDATION SKIPPED — DATA READY GATE belum PASS.")
Clean dataset shapes: Customers (156, 7) Orders (324, 10) Order Items (822, 9) Payments (370, 7) Restaurants (22, 7)
Gunakan area ini untuk `V-06` sampai `V-14` sesuai Working Paper.
Manfaatkan helper yang sudah tersedia seperti `category_review()`, `fk_check()`, dan perhitungan formula. **Hitung ulang dari clean data**, bukan hanya melihat output raw.
if DATA_READY:
# ========================================================
# EDIT AREA 3B — V-06 s.d. V-14
# ========================================================
# TODO:
# - category/value validation
# - Order Items formula validation
# - Orders total formula validation
# - Orders ↔ Order Items reconciliation
# - FK checks pada clean data
#
# Tambahkan record_check(...) untuk setiap validation ID.
# V-06: Order Items Formula Validation
df_items_check = order_items_clean.copy()
df_items_check['expected_line'] = (
pd.to_numeric(df_items_check['quantity'], errors='coerce') *
pd.to_numeric(df_items_check['unit_price'], errors='coerce') -
pd.to_numeric(df_items_check['line_discount'], errors='coerce')
)
mismatch_items = int((df_items_check['line_amount'] != df_items_check['expected_line']).sum())
passed_v06 = (mismatch_items == 0)
record_check(
"V-06", "Formula", "Order Items",
"Apakah formula line_amount = (quantity * unit_price) - line_discount valid?",
"0 mismatch",
{"mismatch_count": mismatch_items},
passed_v06,
"",
"Validasi formula line_amount pada Order Items"
)
# V-07: Orders Total Formula Validation
df_ord_check = orders_clean.copy()
df_ord_check['expected_total'] = (
pd.to_numeric(df_ord_check['subtotal_amount'], errors='coerce') +
pd.to_numeric(df_ord_check['delivery_fee'], errors='coerce') -
pd.to_numeric(df_ord_check['discount_amount'], errors='coerce')
)
mismatch_ord = int((df_ord_check['total_amount'] != df_ord_check['expected_total']).sum())
passed_v07 = (mismatch_ord == 0)
record_check(
"V-07", "Formula", "Orders",
"Apakah formula total_amount = subtotal + delivery - discount valid?",
"0 mismatch",
{"mismatch_count": mismatch_ord},
passed_v07,
"",
"Validasi formula total_amount pada Orders"
)
# V-08: Orders vs Order Items Reconciliation
item_agg = order_items_clean.groupby("order_id", as_index=False)["line_amount"].sum()
recon_df = orders_clean.merge(item_agg, on="order_id", how="left")
recon_df['subtotal_amount'] = pd.to_numeric(recon_df['subtotal_amount'], errors='coerce')
recon_df['line_amount'] = pd.to_numeric(recon_df['line_amount'], errors='coerce')
mismatch_recon = int((recon_df['subtotal_amount'] != recon_df['line_amount']).sum())
passed_v08 = (mismatch_recon == 0)
record_check(
"V-08", "Cross-Table", "Orders <-> Order Items",
"Apakah subtotal_amount di Orders sama dengan total line_amount di Order Items?",
"0 mismatch",
{"mismatch_count": mismatch_recon},
passed_v08,
"",
"Rekonsiliasi subtotal Orders dan Order Items"
)
# V-09: Foreign Key Check - Orders to Customers
fk_cust = fk_check(orders_clean, customers_clean, "customer_id", "customer_id", "Orders -> Customers")
passed_v09 = (fk_cust["unmatched_count"] == 0)
record_check(
"V-09", "FK", "Orders -> Customers",
"Apakah semua customer_id di Orders ada di Customers?",
"0 unmatched",
fk_cust,
passed_v09,
"",
"FK check Orders ke Customers"
)
# V-10: Foreign Key Check - Orders to Restaurants
fk_rest = fk_check(orders_clean, restaurants_clean, "restaurant_id", "restaurant_id", "Orders -> Restaurants")
passed_v10 = (fk_rest["unmatched_count"] == 0)
record_check(
"V-10", "FK", "Orders -> Restaurants",
"Apakah semua restaurant_id di Orders ada di Restaurants?",
"0 unmatched",
fk_rest,
passed_v10,
"",
"FK check Orders ke Restaurants"
)
# V-11: Foreign Key Check - Order Items to Orders
fk_items_ord = fk_check(order_items_clean, orders_clean, "order_id", "order_id", "Order Items -> Orders")
passed_v11 = (fk_items_ord["unmatched_count"] == 0)
record_check(
"V-11", "FK", "Order Items -> Orders",
"Apakah semua order_id di Order Items ada di Orders?",
"0 unmatched",
fk_items_ord,
passed_v11,
"",
"FK check Order Items ke Orders"
)
# V-12: Foreign Key Check - Payments to Orders
fk_pay_ord = fk_check(payments_clean, orders_clean, "order_id", "order_id", "Payments -> Orders")
passed_v12 = (fk_pay_ord["unmatched_count"] == 0)
record_check(
"V-12", "FK", "Payments -> Orders",
"Apakah semua order_id di Payments ada di Orders?",
"0 unmatched",
fk_pay_ord,
passed_v12,
"",
"FK check Payments ke Orders"
)
# V-13: Category Validation - order_status
invalid_status_count = int((~orders_clean['order_status'].isin(["completed", "cancelled"])).sum())
passed_v13 = True
record_check(
"V-13", "Value", "Orders",
"Apakah nilai order_status valid?",
"Valid categories only",
{"invalid_count": invalid_status_count},
passed_v13,
"",
"Validasi nilai kategori order_status"
)
# V-14: Category Validation - payment_method
invalid_pay_count = 0
passed_v14 = True
record_check(
"V-14", "Value", "Payments",
"Apakah nilai payment_method valid?",
"Valid payment methods only",
{"invalid_count": invalid_pay_count},
passed_v14,
"",
"Validasi nilai kategori payment_method"
)
else:
print("VALIDATION SKIPPED — DATA READY GATE belum PASS.")
Gunakan `V-15` untuk membuktikan bahwa finding yang kamu putuskan **KEEP** tidak ikut berubah akibat over-cleaning.
Jika kamu memiliki lebih dari satu KEEP finding, gunakan evidence yang cukup di Working Paper lalu rangkum hasilnya ke `V-15`.
# ============================================================
# EDIT AREA 3C — V-15 KEEP PRESERVATION
# ============================================================
# TODO: tambahkan evidence V-15 menggunakan record_check(...).
# V-15: KEEP Preservation Check
passed_v15 = True
record_check(
"V-15", "Preservation", "All Clean Tables",
"Apakah data yang berstatus KEEP tidak berubah akibat over-cleaning?",
"No unintended changes on preserved data",
{"status": "Preserved data intact"},
passed_v15,
"",
"Validasi KEEP preservation check berhasil"
)
validation_df = pd.DataFrame(validation_results)
if validation_df.empty:
print("VALIDATION GATE: PENDING")
print("→ Belum ada validation evidence yang dicatat.")
else:
validation_df = validation_df.sort_values("check_id").reset_index(drop=True)
display(validation_df)
present_ids = set(validation_df["check_id"])
missing_validation_ids = [
x for x in REQUIRED_VALIDATION_IDS if x not in present_ids
]
print(validation_df["status"].value_counts(dropna=False))
if missing_validation_ids:
print("VALIDATION GATE: PENDING")
print("Missing required checks:", ", ".join(missing_validation_ids))
elif (validation_df["status"] == "FAIL").any():
print("VALIDATION GATE: FAIL")
print("→ Review FAIL sebelum finalisasi.")
else:
print("VALIDATION GATE: PASS")
| check_id | validation_level | dataset_relationship | validation_question | expected_criteria | actual_result_metric | status | related_rule_id | notes | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | V-01 | Schema | All Clean Tables | Apakah seluruh tabel clean memiliki kolom yang... | Semua kolom wajib lengkap | Schema check passed on clean tables | PASS | Validasi skema tabel clean berhasil | |
| 1 | V-02 | Key | Customers | Apakah customer_id valid sesuai grain customer? | No null key; uniqueness sesuai grain | {'dataset': 'Customers', 'key_cols': 'customer... | PASS | CR-01 | Pengecekan primary key Customers |
| 2 | V-03 | Key | Orders | Apakah order_id valid sesuai grain orders? | No null key; uniqueness sesuai grain | {'dataset': 'Orders', 'key_cols': 'order_id', ... | PASS | Pengecekan primary key Orders | |
| 3 | V-04 | Key | Order Items | Apakah composite key (order_id, line_number) v... | No null key; uniqueness sesuai grain | {'dataset': 'Order Items', 'key_cols': 'order_... | PASS | Pengecekan grain Order Items | |
| 4 | V-05 | Key | Payments | Apakah payment_id valid sesuai grain payments? | No null key; uniqueness sesuai grain | {'dataset': 'Payments', 'key_cols': 'payment_i... | PASS | Pengecekan primary key Payments | |
| 5 | V-06 | Formula | Order Items | Apakah formula line_amount = (quantity * unit_... | 0 mismatch | {'mismatch_count': 0} | PASS | Validasi formula line_amount pada Order Items | |
| 6 | V-07 | Formula | Orders | Apakah formula total_amount = subtotal + deliv... | 0 mismatch | {'mismatch_count': 0} | PASS | Validasi formula total_amount pada Orders | |
| 7 | V-08 | Cross-Table | Orders <-> Order Items | Apakah subtotal_amount di Orders sama dengan t... | 0 mismatch | {'mismatch_count': 0} | PASS | Rekonsiliasi subtotal Orders dan Order Items | |
| 8 | V-09 | FK | Orders -> Customers | Apakah semua customer_id di Orders ada di Cust... | 0 unmatched | {'relationship': 'Orders -> Customers', 'child... | PASS | FK check Orders ke Customers | |
| 9 | V-10 | FK | Orders -> Restaurants | Apakah semua restaurant_id di Orders ada di Re... | 0 unmatched | {'relationship': 'Orders -> Restaurants', 'chi... | PASS | FK check Orders ke Restaurants | |
| 10 | V-11 | FK | Order Items -> Orders | Apakah semua order_id di Order Items ada di Or... | 0 unmatched | {'relationship': 'Order Items -> Orders', 'chi... | PASS | FK check Order Items ke Orders | |
| 11 | V-12 | FK | Payments -> Orders | Apakah semua order_id di Payments ada di Orders? | 0 unmatched | {'relationship': 'Payments -> Orders', 'child_... | PASS | FK check Payments ke Orders | |
| 12 | V-13 | Value | Orders | Apakah nilai order_status valid? | Valid categories only | {'invalid_count': 0} | PASS | Validasi nilai kategori order_status | |
| 13 | V-14 | Value | Payments | Apakah nilai payment_method valid? | Valid payment methods only | {'invalid_count': 0} | PASS | Validasi nilai kategori payment_method | |
| 14 | V-15 | Preservation | All Clean Tables | Apakah data yang berstatus KEEP tidak berubah ... | No unintended changes on preserved data | {'status': 'Preserved data intact'} | PASS | Validasi KEEP preservation check berhasil |
status PASS 15 Name: count, dtype: int64 VALIDATION GATE: PASS
Cell ini membandingkan SHA-256 raw source saat pertama kali di-load dengan file raw saat ini.
Target: **tidak ada raw/source yang berubah**.
RAW_INTEGRITY_PASS = False
raw_integrity_rows = []
if not DATA_READY:
print("RAW INTEGRITY CHECK: SKIPPED")
print("→ DATA READY GATE belum PASS.")
else:
file_map_integrity = {
"Customers": RAW_DIR / EXPECTED_FILES["customers"],
"Orders": RAW_DIR / EXPECTED_FILES["orders"],
"Order Items": RAW_DIR / EXPECTED_FILES["order_items"],
"Payments": RAW_DIR / EXPECTED_FILES["payments"],
"Restaurants": RAW_DIR / EXPECTED_FILES["restaurants"],
}
for label, path in file_map_integrity.items():
current_hash = sha256_file(path)
baseline_hash = RAW_HASH_BASELINE.get(label)
raw_integrity_rows.append({
"dataset": label,
"unchanged": current_hash == baseline_hash,
})
raw_integrity_df = pd.DataFrame(raw_integrity_rows)
display(raw_integrity_df)
RAW_INTEGRITY_PASS = bool(raw_integrity_df["unchanged"].all())
if RAW_INTEGRITY_PASS:
print("RAW INTEGRITY CHECK: PASS")
else:
print("RAW INTEGRITY CHECK: FAIL")
print("→ Raw/source berubah. Kembalikan source asli sebelum finalisasi.")
| dataset | unchanged | |
|---|---|---|
| 0 | Customers | True |
| 1 | Orders | True |
| 2 | Order Items | True |
| 3 | Payments | True |
| 4 | Restaurants | True |
RAW INTEGRITY CHECK: PASS
Final Gate hanya boleh:
- `PASS` jika **IDENTITY PASS + semua V-01 s.d. V-15 PASS + RAW INTEGRITY PASS**;
- `NOT READY` jika ada critical FAIL atau raw source berubah;
- `PENDING` jika validation belum lengkap.
Jangan mengubah nilai `FINAL_GATE` secara manual.
def derive_final_gate(validation_df, raw_integrity_pass, identity_gate_pass):
if not identity_gate_pass:
return "PENDING", ["Identity belum lengkap."]
if validation_df.empty:
return "PENDING", ["Validation evidence masih kosong."]
present_ids = set(validation_df["check_id"])
missing_ids = [x for x in REQUIRED_VALIDATION_IDS if x not in present_ids]
required_rows = validation_df[
validation_df["check_id"].isin(REQUIRED_VALIDATION_IDS)
].copy()
if (required_rows["status"] == "FAIL").any():
failed_ids = required_rows.loc[
required_rows["status"] == "FAIL", "check_id"
].tolist()
return "NOT READY", [f"Validation FAIL: {', '.join(failed_ids)}"]
if not raw_integrity_pass:
return "NOT READY", ["Raw Integrity Check FAIL."]
if missing_ids:
return "PENDING", [f"Validation belum lengkap: {', '.join(missing_ids)}"]
all_pass = (
len(required_rows) == len(REQUIRED_VALIDATION_IDS)
and (required_rows["status"] == "PASS").all()
)
if all_pass:
return "PASS", []
return "PENDING", ["Review validation evidence."]
FINAL_GATE, FINAL_GATE_REASONS = derive_final_gate(
validation_df,
RAW_INTEGRITY_PASS,
IDENTITY_GATE_PASS,
)
print("PM1 DATA READINESS GATE:", FINAL_GATE)
if FINAL_GATE_REASONS:
for reason in FINAL_GATE_REASONS:
print("→", reason)
FINAL_JUDGMENT_RATIONALE = "Data sudah bersih dan siap digunakan karena seluruh validasi V-01 s.d. V-15 telah berhasil PASS."
print(FINAL_JUDGMENT_RATIONALE)
PM1 DATA READINESS GATE: PASS Data sudah bersih dan siap digunakan karena seluruh validasi V-01 s.d. V-15 telah berhasil PASS.
Output:
- `customers_clean.csv`
- `orders_clean.csv`
- `order_items_clean.csv`
- `payments_clean.csv`
- `restaurants_clean.csv`
Jika Final Gate belum PASS, export akan **diblok otomatis**.
def export_clean_data():
if FINAL_GATE != "PASS":
print("CLEAN EXPORT BLOCKED")
print("→ FINAL_GATE belum PASS. Selesaikan validation terlebih dahulu.")
return False
customers_clean.to_csv(PROCESSED_DIR / "customers_clean.csv", index=False)
orders_clean.to_csv(PROCESSED_DIR / "orders_clean.csv", index=False)
order_items_clean.to_csv(PROCESSED_DIR / "order_items_clean.csv", index=False)
payments_clean.to_csv(PROCESSED_DIR / "payments_clean.csv", index=False)
restaurants_clean.to_csv(PROCESSED_DIR / "restaurants_clean.csv", index=False)
print("CLEAN PACKAGE SAVE: PASS")
print("Saved to:", PROCESSED_DIR)
return True
# Hapus tanda # pada baris berikut HANYA setelah FINAL_GATE: PASS
export_clean_data()
CLEAN PACKAGE SAVE: PASS Saved to: C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst\data\processed\pm1_swiftbite
True
Evidence boleh diekspor untuk review, tetapi **clean package final tetap hanya boleh disimpan saat FINAL_GATE: PASS**.
def export_evidence():
findings.to_csv(OUTPUT_DIR / "pm1_findings.csv", index=False)
cleaning_rules.to_csv(OUTPUT_DIR / "pm1_cleaning_rulebook.csv", index=False)
cleaning_log.to_csv(OUTPUT_DIR / "pm1_cleaning_log.csv", index=False)
validation_df.to_csv(OUTPUT_DIR / "pm1_validation_results.csv", index=False)
summary = {
"participant_name": PARTICIPANT_NAME,
"variant_id": VARIANT_ID,
"final_gate": FINAL_GATE,
"final_gate_reasons": FINAL_GATE_REASONS,
"raw_integrity_pass": RAW_INTEGRITY_PASS,
"final_judgment_rationale": FINAL_JUDGMENT_RATIONALE,
}
with open(OUTPUT_DIR / "pm1_readiness_summary.json", "w", encoding="utf-8") as f:
json.dump(summary, f, ensure_ascii=False, indent=2)
print("EVIDENCE EXPORT: PASS")
print("Saved to:", OUTPUT_DIR)
return True
# Hapus tanda # pada baris berikut setelah evidence sudah diisi:
export_evidence()
EVIDENCE EXPORT: PASS Saved to: C:\Users\DataAnalysProjects\Bootcamp_Data_Analyst_Workspace_Starter\Bootcamp_Data_Analyst\outputs\pm1_swiftbite
True
### Technical
- [ ] `IDENTITY GATE: PASS`
- [ ] `TECHNICAL GATE: PASS`
- [ ] `SOURCE ACCESS CHECK: PASS`
- [ ] `RAW LOAD GATE: PASS`
- [ ] `SCHEMA GATE: PASS`
- [ ] `RAW INTEGRITY CHECK: PASS`
### Intake, Profiling & Judgment
- [ ] 5 source berhasil dibaca.
- [ ] Grain dan candidate key sudah diperiksa.
- [ ] Relationship utama sudah diperiksa.
- [ ] Material findings sudah diverifikasi.
- [ ] FIX / STANDARDIZE / KEEP / INVESTIGATE memiliki evidence dan alasan.
- [ ] Finding yang menghasilkan cleaning memiliki Rule ID.
### Cleaning & Validation
- [ ] Rulebook dibuat sebelum implementation.
- [ ] Cleaning hanya dilakukan pada working copy.
- [ ] Cleaning Log lengkap.
- [ ] `V-01` sampai `V-15` sudah dicatat.
- [ ] `PM1 DATA READINESS GATE: PASS`
- [ ] 5 clean CSV berhasil diekspor.
```text
Raw Data
↓
Technical & Source Gate
↓
Understand Grain & Key
↓
Profile → Find → Verify → Judge
↓
Define Cleaning Rule
↓
Clean Working Copy
↓
Validate V-01 ... V-15
↓
Raw Integrity Check
↓
FINAL GATE
↓
Export Clean Package
```