{ "cells": [ { "cell_type": "markdown", "id": "54aa00ab", "metadata": {}, "source": [ "# Phase 2 analysis\n", "\n", "This notebook follows the Phase 2 config organization (`p2a` to `p2e`) and maps each section directly to its config group.\n", "It separates three concerns:\n", "\n", "1. **Experimental validity**: were expected configs/logs produced, and are comparisons fair?\n", "2. **Evidence**: what do the 5-fold CV metrics support?\n", "3. **Decision**: which preprocessing choices should move into Phase 3?\n" ] }, { "cell_type": "markdown", "id": "734db3ee", "metadata": {}, "source": [ "## Questions\n", "\n", "| Section | Config group | Question | Required evidence |\n", "|---|---|---|---|\n", "| 2A | `p2a_*` | Shortcut analysis: normalization + source holdout | `p2a_t1_original`, `p2a_t2_real_norm`, `p2a_t3_holdout_*` |\n", "| 2B | `p2b_*` | Does 224 improve over 128? | `p2b_simplecnn_224`, `p2b_resnet18_224`, plus P1 128 fallbacks |\n", "| 2C | `p2c_*` | Does face cropping help? | `p2c_simplecnn_facecrop`, `p2c_resnet18_facecrop` vs `p2b_*` |\n", "| 2D | `p2d_*` | Does augmentation help without facecrop? | `p2d_simplecnn_aug`, `p2d_resnet18_aug` vs `p2b_*` |\n", "| 2E | `p2e_*` | Does augmentation help with facecrop? | `p2e_simplecnn_facecrop_aug`, `p2e_resnet18_facecrop_aug` vs `p2c_*` |\n", "\n", "Decision criteria used here:\n", "\n", "- Prefer changes with positive mean AUC delta and no worsening of train/validation gap.\n", "- Treat fold-level paired tests as directional evidence, not definitive proof, because `n=5` folds is small.\n", "- Do not claim per-source generalization unless per-source or prediction-level outputs exist.\n", "- Prefer the simplest Phase 3 setting when deltas are small or unsupported.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1f4c04b3", "metadata": {}, "outputs": [], "source": [ "from __future__ import annotations\n", "\n", "import json\n", "import math\n", "import os\n", "import sys\n", "from dataclasses import dataclass\n", "from pathlib import Path\n", "from typing import Any\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from scipy import stats\n", "\n", "try:\n", " from IPython.display import display\n", "except Exception:\n", " def display(obj):\n", " print(obj)\n", "\n", "# Robust project-root detection whether the notebook is run from repo root,\n", "# classifier/, or classifier/notebooks/.\n", "def find_project_root(start: Path | None = None) -> Path:\n", " start = (start or Path.cwd()).resolve()\n", " for candidate in [start, *start.parents]:\n", " if (candidate / \"classifier\" / \"v2.md\").exists() and (candidate / \"classifier\" / \"impl.md\").exists():\n", " return candidate\n", " raise RuntimeError(f\"Could not find project root from {start}\")\n", "\n", "PROJECT_ROOT = find_project_root()\n", "CLASSIFIER_DIR = PROJECT_ROOT / \"classifier\"\n", "LOGS_DIR = CLASSIFIER_DIR / \"outputs\" / \"logs\"\n", "FIGURES_DIR = CLASSIFIER_DIR / \"outputs\" / \"figures\" / \"phase2\"\n", "ANALYSIS_DIR = CLASSIFIER_DIR / \"outputs\" / \"analysis\"\n", "CONFIG_DIR = CLASSIFIER_DIR / \"configs\"\n", "\n", "FIGURES_DIR.mkdir(parents=True, exist_ok=True)\n", "ANALYSIS_DIR.mkdir(parents=True, exist_ok=True)\n", "\n", "if str(CLASSIFIER_DIR) not in sys.path:\n", " sys.path.insert(0, str(CLASSIFIER_DIR))\n", "\n", "sns.set_theme(style=\"whitegrid\", context=\"notebook\")\n", "plt.rcParams.update({\n", " \"figure.figsize\": (12, 7),\n", " \"axes.spines.top\": False,\n", " \"axes.spines.right\": False,\n", "})\n", "\n", "print(f\"Project root: {PROJECT_ROOT}\")\n", "print(f\"Logs: {LOGS_DIR}\")\n", "print(f\"Figures: {FIGURES_DIR}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "24830212", "metadata": {}, "outputs": [], "source": [ "@dataclass(frozen=True)\n", "class RunSpec:\n", " run: str\n", " label: str\n", " section: str\n", " model: str\n", " condition: str\n", " intended_role: str\n", " fallback_for: str | None = None\n", "\n", "RUN_SPECS = [\n", " # 2A: shortcut analysis (normalization + source holdout), ResNet18 only.\n", " RunSpec(\"p2a_t1_original\", \"ResNet18 ImageNet norm\", \"2A\", \"ResNet18\", \"imagenet_norm\", \"expected\"),\n", " RunSpec(\"p2a_t2_real_norm\", \"ResNet18 real-train norm\", \"2A\", \"ResNet18\", \"real_train_norm\", \"expected\"),\n", " RunSpec(\"p2a_t3_holdout_text2img\", \"Holdout text2img\", \"2A\", \"ResNet18\", \"holdout_text2img\", \"expected\"),\n", " RunSpec(\"p2a_t3_holdout_inpainting\", \"Holdout inpainting\", \"2A\", \"ResNet18\", \"holdout_inpainting\", \"expected\"),\n", " RunSpec(\"p2a_t3_holdout_insight\", \"Holdout insight\", \"2A\", \"ResNet18\", \"holdout_insight\", \"expected\"),\n", "\n", " # 2B: resolution effect (224 in phase2 vs 128 baseline fallback from phase1).\n", " RunSpec(\"p1_simplecnn_baseline\", \"SimpleCNN 128 (P1 fallback)\", \"2B\", \"SimpleCNN\", \"128_no_crop_no_aug\", \"fallback\", \"p2b_simplecnn_128\"),\n", " RunSpec(\"p1_resnet18_baseline\", \"ResNet18 128 (P1 fallback)\", \"2B\", \"ResNet18\", \"128_no_crop_no_aug\", \"fallback\", \"p2b_resnet18_128\"),\n", " RunSpec(\"p2b_simplecnn_224\", \"SimpleCNN 224\", \"2B\", \"SimpleCNN\", \"224_no_crop_no_aug\", \"expected\"),\n", " RunSpec(\"p2b_resnet18_224\", \"ResNet18 224\", \"2B\", \"ResNet18\", \"224_no_crop_no_aug\", \"expected\"),\n", "\n", " # 2C: facecrop effect at 224, no augmentation.\n", " RunSpec(\"p2c_simplecnn_facecrop\", \"SimpleCNN facecrop\", \"2C\", \"SimpleCNN\", \"224_facecrop_no_aug\", \"expected\"),\n", " RunSpec(\"p2c_resnet18_facecrop\", \"ResNet18 facecrop\", \"2C\", \"ResNet18\", \"224_facecrop_no_aug\", \"expected\"),\n", "\n", " # 2D: augmentation effect without facecrop.\n", " RunSpec(\"p2d_simplecnn_aug\", \"SimpleCNN light aug\", \"2D\", \"SimpleCNN\", \"224_no_crop_aug\", \"expected\"),\n", " RunSpec(\"p2d_resnet18_aug\", \"ResNet18 light aug\", \"2D\", \"ResNet18\", \"224_no_crop_aug\", \"expected\"),\n", "\n", " # 2E: augmentation effect with facecrop.\n", " RunSpec(\"p2e_simplecnn_facecrop_aug\", \"SimpleCNN facecrop + aug\", \"2E\", \"SimpleCNN\", \"224_facecrop_aug\", \"expected\"),\n", " RunSpec(\"p2e_resnet18_facecrop_aug\", \"ResNet18 facecrop + aug\", \"2E\", \"ResNet18\", \"224_facecrop_aug\", \"expected\"),\n", "]\n", "\n", "# Use these aliases when synthetic 128 run IDs are requested for 2B.\n", "RUN_ALIASES = {\n", " \"p2b_simplecnn_128\": \"p1_simplecnn_baseline\",\n", " \"p2b_resnet18_128\": \"p1_resnet18_baseline\",\n", "}\n", "\n", "PLANNED_COMPARISONS = [\n", " (\"2A\", \"ResNet18\", \"normalization\", \"p2a_t1_original\", \"p2a_t2_real_norm\", \"real_norm - imagenet_norm\"),\n", " (\"2A\", \"ResNet18\", \"source_holdout\", \"p2a_t1_original\", \"p2a_t3_holdout_text2img\", \"holdout text2img - all-source\"),\n", " (\"2A\", \"ResNet18\", \"source_holdout\", \"p2a_t1_original\", \"p2a_t3_holdout_inpainting\", \"holdout inpainting - all-source\"),\n", " (\"2A\", \"ResNet18\", \"source_holdout\", \"p2a_t1_original\", \"p2a_t3_holdout_insight\", \"holdout insight - all-source\"),\n", "\n", " (\"2B\", \"SimpleCNN\", \"resolution\", \"p2b_simplecnn_128\", \"p2b_simplecnn_224\", \"224 - 128\"),\n", " (\"2B\", \"ResNet18\", \"resolution\", \"p2b_resnet18_128\", \"p2b_resnet18_224\", \"224 - 128\"),\n", "\n", " (\"2C\", \"SimpleCNN\", \"facecrop\", \"p2b_simplecnn_224\", \"p2c_simplecnn_facecrop\", \"facecrop - no facecrop\"),\n", " (\"2C\", \"ResNet18\", \"facecrop\", \"p2b_resnet18_224\", \"p2c_resnet18_facecrop\", \"facecrop - no facecrop\"),\n", "\n", " (\"2D\", \"SimpleCNN\", \"augmentation\", \"p2b_simplecnn_224\", \"p2d_simplecnn_aug\", \"light aug - no aug\"),\n", " (\"2D\", \"ResNet18\", \"augmentation\", \"p2b_resnet18_224\", \"p2d_resnet18_aug\", \"light aug - no aug\"),\n", "\n", " (\"2E\", \"SimpleCNN\", \"facecrop + augmentation\", \"p2c_simplecnn_facecrop\", \"p2e_simplecnn_facecrop_aug\", \"facecrop+aug - facecrop\"),\n", " (\"2E\", \"ResNet18\", \"facecrop + augmentation\", \"p2c_resnet18_facecrop\", \"p2e_resnet18_facecrop_aug\", \"facecrop+aug - facecrop\"),\n", "]\n" ] }, { "cell_type": "markdown", "id": "6e2ccd27", "metadata": {}, "source": [ "## Evidence audit\n", "\n", "Before comparing numbers, check whether the planned artifacts exist. Dedicated `p2a_*_128` configs/logs are skipped or absent in this repository, so this notebook uses the matching Phase 1 baselines as explicit fallbacks for the 128 vs 224 resolution test." ] }, { "cell_type": "code", "execution_count": null, "id": "53356e8b", "metadata": {}, "outputs": [], "source": [ "def load_json(path: Path) -> dict[str, Any] | None:\n", " if not path.exists():\n", " return None\n", " with path.open() as f:\n", " return json.load(f)\n", "\n", "\n", "def config_path_for(run: str) -> Path | None:\n", " candidates = [\n", " CONFIG_DIR / \"phase2\" / f\"{run}.json\",\n", " CONFIG_DIR / \"phase2\" / f\"{run}.json.skip\",\n", " CONFIG_DIR / \"phase1\" / f\"{run}.json\",\n", " CONFIG_DIR / \"phase1\" / f\"{run}.json.skip\",\n", " ]\n", " return next((p for p in candidates if p.exists()), None)\n", "\n", "\n", "def log_path_for(run: str) -> Path:\n", " return LOGS_DIR / f\"{run}.json\"\n", "\n", "\n", "def resolve_run(run: str) -> str:\n", " return run if log_path_for(run).exists() else RUN_ALIASES.get(run, run)\n", "\n", "\n", "def load_results(run: str) -> dict[str, Any] | None:\n", " resolved = resolve_run(run)\n", " return load_json(log_path_for(resolved))\n", "\n", "\n", "def metric_values(results: dict[str, Any], metric: str = \"auc_roc\") -> np.ndarray:\n", " vals = []\n", " for fold in results.get(\"fold_results\", []):\n", " value = fold.get(\"test_metrics\", {}).get(metric)\n", " if value is not None:\n", " vals.append(float(value))\n", " return np.asarray(vals, dtype=float)\n", "\n", "\n", "def best_epoch_gap(fold: dict[str, Any], metric: str = \"auc\") -> float | None:\n", " hist = fold.get(\"history\", {})\n", " train_key = f\"train_{metric}\"\n", " val_key = f\"val_{metric}\"\n", " train = hist.get(train_key, [])\n", " val = hist.get(val_key, [])\n", " if not train or not val:\n", " return None\n", " idx = int(np.nanargmax(np.asarray(val, dtype=float)))\n", " return float(train[idx] - val[idx])\n", "\n", "\n", "def final_epoch_gap(fold: dict[str, Any], metric: str = \"auc\") -> float | None:\n", " hist = fold.get(\"history\", {})\n", " train = hist.get(f\"train_{metric}\", [])\n", " val = hist.get(f\"val_{metric}\", [])\n", " if not train or not val:\n", " return None\n", " return float(train[-1] - val[-1])\n", "\n", "\n", "def summarize_run(spec: RunSpec) -> dict[str, Any]:\n", " resolved = resolve_run(spec.run)\n", " results = load_results(spec.run)\n", " config_path = config_path_for(spec.run) or config_path_for(resolved)\n", " cfg = load_json(config_path) if config_path else None\n", "\n", " row = {\n", " \"section\": spec.section,\n", " \"run\": spec.run,\n", " \"resolved_run\": resolved,\n", " \"label\": spec.label,\n", " \"model\": spec.model,\n", " \"condition\": spec.condition,\n", " \"role\": spec.intended_role,\n", " \"fallback_for\": spec.fallback_for,\n", " \"config_path\": str(config_path.relative_to(PROJECT_ROOT)) if config_path else None,\n", " \"config_status\": \"present\" if config_path and config_path.suffix == \".json\" else (\"skipped\" if config_path else \"missing\"),\n", " \"log_status\": \"present\" if log_path_for(spec.run).exists() else (\"fallback\" if resolved != spec.run and log_path_for(resolved).exists() else \"missing\"),\n", " \"n_folds\": None,\n", " \"auc_mean\": np.nan,\n", " \"auc_std\": np.nan,\n", " \"acc_mean\": np.nan,\n", " \"f1_mean\": np.nan,\n", " \"gap_best_mean\": np.nan,\n", " \"gap_final_mean\": np.nan,\n", " \"image_size\": None,\n", " \"face_crop\": None,\n", " \"augment\": None,\n", " \"normalization\": None,\n", " \"train_sources\": None,\n", " \"eval_sources\": None,\n", " }\n", "\n", " if cfg:\n", " row.update({\n", " \"image_size\": cfg.get(\"image_size\"),\n", " \"face_crop\": cfg.get(\"face_crop\"),\n", " \"augment\": \"light\" if isinstance(cfg.get(\"augment\"), dict) else cfg.get(\"augment\"),\n", " \"normalization\": cfg.get(\"normalization\"),\n", " \"train_sources\": tuple(cfg.get(\"train_sources\", [])) or None,\n", " \"eval_sources\": tuple(cfg.get(\"eval_sources\", [])) or None,\n", " })\n", "\n", " if results:\n", " agg = results.get(\"aggregated_metrics\", {})\n", " row.update({\n", " \"n_folds\": results.get(\"n_folds\"),\n", " \"auc_mean\": agg.get(\"auc_roc\", {}).get(\"mean\", np.nan),\n", " \"auc_std\": agg.get(\"auc_roc\", {}).get(\"std\", np.nan),\n", " \"acc_mean\": agg.get(\"accuracy\", {}).get(\"mean\", np.nan),\n", " \"f1_mean\": agg.get(\"f1\", {}).get(\"mean\", np.nan),\n", " })\n", " best_gaps = [best_epoch_gap(f) for f in results.get(\"fold_results\", [])]\n", " final_gaps = [final_epoch_gap(f) for f in results.get(\"fold_results\", [])]\n", " best_gaps = [x for x in best_gaps if x is not None]\n", " final_gaps = [x for x in final_gaps if x is not None]\n", " row[\"gap_best_mean\"] = float(np.mean(best_gaps)) if best_gaps else np.nan\n", " row[\"gap_final_mean\"] = float(np.mean(final_gaps)) if final_gaps else np.nan\n", "\n", " return row\n", "\n", "runs_df = pd.DataFrame([summarize_run(spec) for spec in RUN_SPECS])\n", "\n", "# Prefer canonical rows for analysis: keep fallbacks only where expected rows are missing.\n", "canonical_runs_df = runs_df[runs_df[\"role\"] == \"expected\"].copy()\n", "for missing_run, fallback_run in RUN_ALIASES.items():\n", " mask = canonical_runs_df[\"run\"].eq(missing_run) & canonical_runs_df[\"log_status\"].eq(\"missing\")\n", " if mask.any():\n", " fallback = runs_df[runs_df[\"run\"].eq(fallback_run)].copy()\n", " if not fallback.empty:\n", " fallback.loc[:, \"run\"] = missing_run\n", " fallback.loc[:, \"label\"] = fallback.iloc[0][\"label\"].replace(\" (P1 fallback)\", \"\") + \" [P1 fallback]\"\n", " fallback.loc[:, \"role\"] = \"expected_via_fallback\"\n", " canonical_runs_df = pd.concat([canonical_runs_df[~mask], fallback], ignore_index=True)\n", "\n", "print(\"Artifact audit:\")\n", "display(runs_df[[\"section\", \"run\", \"resolved_run\", \"role\", \"config_status\", \"log_status\", \"n_folds\"]].sort_values([\"section\", \"run\"]))\n", "\n", "missing_expected = runs_df[(runs_df[\"role\"] == \"expected\") & (runs_df[\"log_status\"] == \"missing\")][\"run\"].tolist()\n", "print(f\"\\nExpected runs with no direct log: {missing_expected or 'none'}\")\n", "print(\"Fallbacks used:\", {k: v for k, v in RUN_ALIASES.items() if k in missing_expected})" ] }, { "cell_type": "code", "execution_count": null, "id": "b21a9faf", "metadata": {}, "outputs": [], "source": [ "# Protocol consistency audit from loaded logs/configs.\n", "protocol_fields = [\n", " \"cv_folds\", \"batch_size\", \"early_stopping_patience\", \"seed\", \"subsample\",\n", " \"lr\", \"weight_decay\", \"T_max\", \"epochs\",\n", "]\n", "\n", "protocol_rows = []\n", "for _, row in canonical_runs_df.iterrows():\n", " results = load_results(row[\"run\"])\n", " cfg = (results or {}).get(\"config\", {})\n", " protocol_rows.append({\"run\": row[\"run\"], **{k: cfg.get(k) for k in protocol_fields}})\n", "\n", "protocol_df = pd.DataFrame(protocol_rows)\n", "display(protocol_df)\n", "\n", "print(\"Field variability across loaded canonical runs:\")\n", "for field in protocol_fields:\n", " vals = sorted({str(v) for v in protocol_df[field].dropna().unique()})\n", " print(f\" {field:28s}: {vals}\")" ] }, { "cell_type": "markdown", "id": "6802bcd9", "metadata": {}, "source": [ "## Results table\n", "\n", "The table below is ranked by AUC and includes two gap estimates:\n", "\n", "- `gap_best_mean`: train AUC minus validation AUC at each fold's best validation epoch. This is closest to the saved best checkpoint.\n", "- `gap_final_mean`: train AUC minus validation AUC at the final epoch. This is useful for diagnosing late overfit but is less aligned with test evaluation." ] }, { "cell_type": "code", "execution_count": null, "id": "be1ec0ba", "metadata": {}, "outputs": [], "source": [ "analysis_df = canonical_runs_df[canonical_runs_df[\"log_status\"].isin([\"present\", \"fallback\"])].copy()\n", "analysis_df = analysis_df.sort_values(\"auc_mean\", ascending=False)\n", "\n", "cols = [\n", " \"section\", \"label\", \"run\", \"resolved_run\", \"model\", \"condition\", \"log_status\",\n", " \"auc_mean\", \"auc_std\", \"acc_mean\", \"f1_mean\", \"gap_best_mean\", \"gap_final_mean\",\n", "]\n", "\n", "display(\n", " analysis_df[cols]\n", " .style.format({\n", " \"auc_mean\": \"{:.4f}\",\n", " \"auc_std\": \"{:.4f}\",\n", " \"acc_mean\": \"{:.4f}\",\n", " \"f1_mean\": \"{:.4f}\",\n", " \"gap_best_mean\": \"{:+.4f}\",\n", " \"gap_final_mean\": \"{:+.4f}\",\n", " })\n", " .background_gradient(subset=[\"auc_mean\"], cmap=\"Greens\")\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "1e0d21c1", "metadata": {}, "outputs": [], "source": [ "def paired_comparison(section: str, model: str, question: str, before: str, after: str, contrast: str) -> dict[str, Any]:\n", " r0 = load_results(before)\n", " r1 = load_results(after)\n", " resolved_before = resolve_run(before)\n", " resolved_after = resolve_run(after)\n", " out = {\n", " \"section\": section,\n", " \"model\": model,\n", " \"question\": question,\n", " \"before\": before,\n", " \"after\": after,\n", " \"resolved_before\": resolved_before,\n", " \"resolved_after\": resolved_after,\n", " \"contrast\": contrast,\n", " \"status\": \"ok\" if r0 and r1 else \"missing\",\n", " \"n\": 0,\n", " \"before_auc\": np.nan,\n", " \"after_auc\": np.nan,\n", " \"delta_auc\": np.nan,\n", " \"delta_ci95\": np.nan,\n", " \"ttest_p\": np.nan,\n", " \"wilcoxon_p\": np.nan,\n", " \"cohen_dz\": np.nan,\n", " \"before_gap\": np.nan,\n", " \"after_gap\": np.nan,\n", " \"delta_gap\": np.nan,\n", " \"interpretation\": \"insufficient data\",\n", " \"caveat\": \"\",\n", " }\n", " if not (r0 and r1):\n", " return out\n", "\n", " v0 = metric_values(r0, \"auc_roc\")\n", " v1 = metric_values(r1, \"auc_roc\")\n", " n = min(len(v0), len(v1))\n", " v0, v1 = v0[:n], v1[:n]\n", " diff = v1 - v0\n", "\n", " out.update({\n", " \"n\": n,\n", " \"before_auc\": float(np.mean(v0)),\n", " \"after_auc\": float(np.mean(v1)),\n", " \"delta_auc\": float(np.mean(diff)),\n", " })\n", "\n", " if n >= 2:\n", " sd = float(np.std(diff, ddof=1))\n", " se = sd / math.sqrt(n) if sd > 0 else 0.0\n", " out[\"delta_ci95\"] = float(stats.t.ppf(0.975, df=n - 1) * se) if n > 1 else np.nan\n", " if sd > 0:\n", " out[\"cohen_dz\"] = float(np.mean(diff) / sd)\n", " out[\"ttest_p\"] = float(stats.ttest_rel(v1, v0).pvalue)\n", " if n >= 3 and not np.allclose(diff, 0):\n", " try:\n", " out[\"wilcoxon_p\"] = float(stats.wilcoxon(diff).pvalue)\n", " except ValueError:\n", " pass\n", "\n", " gaps0 = [best_epoch_gap(f) for f in r0.get(\"fold_results\", [])]\n", " gaps1 = [best_epoch_gap(f) for f in r1.get(\"fold_results\", [])]\n", " gaps0 = np.asarray([x for x in gaps0 if x is not None], dtype=float)\n", " gaps1 = np.asarray([x for x in gaps1 if x is not None], dtype=float)\n", " if len(gaps0) and len(gaps1):\n", " m = min(len(gaps0), len(gaps1))\n", " out[\"before_gap\"] = float(np.mean(gaps0[:m]))\n", " out[\"after_gap\"] = float(np.mean(gaps1[:m]))\n", " out[\"delta_gap\"] = float(np.mean(gaps1[:m] - gaps0[:m]))\n", "\n", " if question == \"source_holdout\":\n", " out[\"caveat\"] = \"Aggregate holdout-run AUC only; not held-out-source vs in-source AUC.\"\n", " if before != resolved_before or after != resolved_after:\n", " out[\"caveat\"] = (out[\"caveat\"] + \" \" if out[\"caveat\"] else \"\") + \"Uses Phase 1 fallback for missing p2a 128 log.\"\n", "\n", " if out[\"delta_auc\"] >= 0.01:\n", " out[\"interpretation\"] = \"meaningful improvement\"\n", " elif out[\"delta_auc\"] > 0.002:\n", " out[\"interpretation\"] = \"small improvement\"\n", " elif out[\"delta_auc\"] >= -0.002:\n", " out[\"interpretation\"] = \"negligible change\"\n", " elif out[\"delta_auc\"] > -0.01:\n", " out[\"interpretation\"] = \"small drop\"\n", " else:\n", " out[\"interpretation\"] = \"meaningful drop\"\n", " return out\n", "\n", "comparisons_df = pd.DataFrame([paired_comparison(*args) for args in PLANNED_COMPARISONS])\n", "\n", "# Benjamini-Hochberg correction across planned paired t-tests where available.\n", "valid_p = comparisons_df[\"ttest_p\"].notna()\n", "pvals = comparisons_df.loc[valid_p, \"ttest_p\"].to_numpy()\n", "qvals = np.full(len(comparisons_df), np.nan)\n", "if len(pvals):\n", " order = np.argsort(pvals)\n", " ranked = pvals[order]\n", " adjusted = np.empty_like(ranked)\n", " m = len(ranked)\n", " running = 1.0\n", " for i in range(m - 1, -1, -1):\n", " running = min(running, ranked[i] * m / (i + 1))\n", " adjusted[i] = running\n", " qvals[np.where(valid_p)[0][order]] = adjusted\n", "comparisons_df[\"bh_q\"] = qvals\n", "\n", "display(\n", " comparisons_df[[\n", " \"section\", \"model\", \"question\", \"contrast\", \"before_auc\", \"after_auc\", \"delta_auc\",\n", " \"delta_ci95\", \"ttest_p\", \"bh_q\", \"wilcoxon_p\", \"cohen_dz\", \"delta_gap\", \"interpretation\", \"caveat\",\n", " ]].style.format({\n", " \"before_auc\": \"{:.4f}\",\n", " \"after_auc\": \"{:.4f}\",\n", " \"delta_auc\": \"{:+.4f}\",\n", " \"delta_ci95\": \"\u00b1{:.4f}\",\n", " \"ttest_p\": \"{:.4f}\",\n", " \"bh_q\": \"{:.4f}\",\n", " \"wilcoxon_p\": \"{:.4f}\",\n", " \"cohen_dz\": \"{:+.2f}\",\n", " \"delta_gap\": \"{:+.4f}\",\n", " }).background_gradient(subset=[\"delta_auc\"], cmap=\"RdYlGn\", vmin=-0.06, vmax=0.06)\n", ")" ] }, { "cell_type": "markdown", "id": "f20e5262", "metadata": {}, "source": [ "## Visual summary\n", "\n", "Two plots are most useful for decision-making:\n", "\n", "- Ranking all conditions by AUC shows the best observed configurations but can overstate duplicated/near-identical runs.\n", "- Paired delta plot shows the controlled effect of each preprocessing change and exposes uncertainty." ] }, { "cell_type": "code", "execution_count": null, "id": "42882c6a", "metadata": {}, "outputs": [], "source": [ "plot_df = analysis_df.copy()\n", "plot_df[\"display_label\"] = plot_df[\"section\"] + \" | \" + plot_df[\"label\"]\n", "plot_df = plot_df.sort_values(\"auc_mean\", ascending=True)\n", "\n", "fig, ax = plt.subplots(figsize=(11, max(7, 0.35 * len(plot_df))))\n", "colors = {\"2A\": \"#4C78A8\", \"2B\": \"#F58518\", \"2C\": \"#54A24B\", \"2D\": \"#E45756\", \"2E\": \"#B279A2\"}\n", "ax.barh(\n", " plot_df[\"display_label\"],\n", " plot_df[\"auc_mean\"],\n", " xerr=plot_df[\"auc_std\"],\n", " color=[colors.get(s, \"#999999\") for s in plot_df[\"section\"]],\n", " alpha=0.85,\n", ")\n", "ax.set_xlim(0.65, 1.0)\n", "ax.set_xlabel(\"Mean AUC across CV folds\")\n", "ax.set_title(\"Phase 2 Conditions Ranked by AUC\")\n", "ax.axvline(0.95, color=\"black\", linewidth=1, linestyle=\"--\", alpha=0.4)\n", "for y, (_, row) in enumerate(plot_df.iterrows()):\n", " ax.text(row[\"auc_mean\"] + 0.004, y, f\"{row['auc_mean']:.4f}\", va=\"center\", fontsize=9)\n", "fig.tight_layout()\n", "fig.savefig(FIGURES_DIR / \"ranked_auc.png\", dpi=200, bbox_inches=\"tight\")\n", "plt.show()\n", "\n", "forest = comparisons_df.copy()\n", "forest[\"display\"] = forest[\"section\"] + \" \" + forest[\"model\"] + \" - \" + forest[\"contrast\"]\n", "forest = forest.iloc[::-1]\n", "fig, ax = plt.subplots(figsize=(11, max(6, 0.45 * len(forest))))\n", "y = np.arange(len(forest))\n", "ax.errorbar(\n", " forest[\"delta_auc\"], y,\n", " xerr=forest[\"delta_ci95\"],\n", " fmt=\"o\", color=\"#1F2937\", ecolor=\"#6B7280\", capsize=4,\n", ")\n", "ax.axvline(0, color=\"black\", linewidth=1)\n", "ax.axvspan(-0.002, 0.002, color=\"#9CA3AF\", alpha=0.18, label=\"negligible band\")\n", "ax.set_yticks(y)\n", "ax.set_yticklabels(forest[\"display\"])\n", "ax.set_xlabel(\"Delta AUC (after - before), paired by fold\")\n", "ax.set_title(\"Planned Phase 2 Effect Estimates\")\n", "ax.legend(loc=\"lower right\")\n", "fig.tight_layout()\n", "fig.savefig(FIGURES_DIR / \"planned_effects.png\", dpi=200, bbox_inches=\"tight\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e063cfc0", "metadata": {}, "source": [ "## 2A - Shortcut analysis\n", "\n", "Shortcut checks map to `p2a_*` configs:\n", "- `p2a_t1_original` vs `p2a_t2_real_norm` (normalization)\n", "- `p2a_t1_original` vs `p2a_t3_holdout_*` (source_holdout)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "910bd5bd", "metadata": {}, "outputs": [], "source": [ "def comparison_subset(section: str, question: str | None = None) -> pd.DataFrame:\n", " df = comparisons_df[comparisons_df[\"section\"].eq(section)].copy()\n", " if question:\n", " df = df[df[\"question\"].eq(question)]\n", " return df\n", "\n", "\n", "def print_comparison_readout(df: pd.DataFrame) -> None:\n", " for _, row in df.iterrows():\n", " print(f\"{row['section']} {row['model']} - {row['contrast']}\")\n", " print(f\" AUC: {row['before_auc']:.4f} -> {row['after_auc']:.4f} ({row['delta_auc']:+.4f})\")\n", " print(f\" paired t p={row['ttest_p']:.4f}, BH q={row['bh_q']:.4f}, CI95 delta=\u00b1{row['delta_ci95']:.4f}\")\n", " print(f\" gap delta: {row['delta_gap']:+.4f}; interpretation: {row['interpretation']}\")\n", " if row['caveat']:\n", " print(f\" caveat: {row['caveat']}\")\n", " print()\n", "\n", "print_comparison_readout(comparison_subset(\"2B\", \"resolution\"))\n", "\n", "res_plot = comparison_subset(\"2B\", \"resolution\")\n", "fig, ax = plt.subplots(figsize=(8, 5))\n", "for _, row in res_plot.iterrows():\n", " r0, r1 = load_results(row[\"before\"]), load_results(row[\"after\"])\n", " v0, v1 = metric_values(r0), metric_values(r1)\n", " x = [0, 1]\n", " for a, b in zip(v0, v1):\n", " ax.plot(x, [a, b], color=\"#9CA3AF\", alpha=0.7)\n", " ax.plot(x, [v0.mean(), v1.mean()], marker=\"o\", linewidth=3, label=row[\"model\"])\n", "ax.set_xticks([0, 1])\n", "ax.set_xticklabels([\"128\", \"224\"])\n", "ax.set_ylabel(\"AUC\")\n", "ax.set_title(\"2B Resolution: Fold-Paired AUC\")\n", "ax.legend()\n", "fig.tight_layout()\n", "fig.savefig(FIGURES_DIR / \"2b_resolution_paired.png\", dpi=200, bbox_inches=\"tight\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "530e8675", "metadata": {}, "source": [ "## 2B - Resolution impact\n", "\n", "This section compares 128 vs 224 using `p2b_*_224` and Phase 1 baselines as explicit 128 fallbacks.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "13304d38", "metadata": {}, "outputs": [], "source": [ "print_comparison_readout(comparison_subset(\"2C\", \"facecrop\"))\n", "\n", "face_df = canonical_runs_df[canonical_runs_df[\"section\"].eq(\"2C\")].copy()\n", "fig, axes = plt.subplots(1, 2, figsize=(12, 5), sharey=False)\n", "for ax, model in zip(axes, [\"SimpleCNN\", \"ResNet18\"]):\n", " sub = face_df[face_df[\"model\"].eq(model)].sort_values(\"face_crop\")\n", " ax.bar(sub[\"condition\"], sub[\"auc_mean\"], yerr=sub[\"auc_std\"], color=[\"#D97706\", \"#059669\"], alpha=0.85, capsize=5)\n", " ax.set_title(model)\n", " ax.set_ylim(0.70 if model == \"SimpleCNN\" else 0.94, 0.99)\n", " ax.set_ylabel(\"AUC\")\n", " ax.tick_params(axis=\"x\", rotation=20)\n", "fig.suptitle(\"2C Facecrop Impact\")\n", "fig.tight_layout()\n", "fig.savefig(FIGURES_DIR / \"2c_facecrop.png\", dpi=200, bbox_inches=\"tight\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "8702d10d", "metadata": {}, "source": [ "## 2C - Facecrop impact\n", "\n", "This section compares `p2c_*_facecrop` against the matching `p2b_*_224` no-facecrop baselines.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ec5e03ef", "metadata": {}, "outputs": [], "source": [ "print_comparison_readout(comparison_subset(\"2A\"))\n\n# Inspect whether logs contain the per-source data needed by v2.md.\nsource_audit = []\nfor run in [\"p2a_t1_original\", \"p2a_t3_holdout_text2img\", \"p2a_t3_holdout_inpainting\", \"p2a_t3_holdout_insight\"]:\n results = load_results(run)\n has_per_source = False\n has_records = False\n example_keys = []\n if results:\n for fold in results.get(\"fold_results\", []):\n tm = fold.get(\"test_metrics\", {})\n example_keys = sorted(tm.keys())\n has_per_source = has_per_source or any(k in tm for k in [\"per_source\", \"per_source_metrics\", \"pairwise_source_metrics\", \"source_metrics\", \"pair_metrics\"])\n has_records = has_records or any(k in fold for k in [\"records\", \"predictions\", \"test_records\"])\n source_audit.append({\n \"run\": run,\n \"has_per_source_metrics\": has_per_source,\n \"has_prediction_records\": has_records,\n \"test_metric_keys\": example_keys,\n })\nsource_audit_df = pd.DataFrame(source_audit)\ndisplay(source_audit_df)\n\nholdout_runs = [\"p2a_t1_original\", \"p2a_t3_holdout_text2img\", \"p2a_t3_holdout_inpainting\", \"p2a_t3_holdout_insight\"]\nholdout_df = canonical_runs_df[canonical_runs_df[\"run\"].isin(holdout_runs)].copy()\nholdout_df[\"delta_vs_all_source\"] = holdout_df[\"auc_mean\"] - float(holdout_df.loc[holdout_df[\"run\"].eq(\"p2a_t1_original\"), \"auc_mean\"].iloc[0])\n\nfig, ax = plt.subplots(figsize=(9, 5))\nax.bar(holdout_df[\"label\"], holdout_df[\"auc_mean\"], yerr=holdout_df[\"auc_std\"], color=\"#54A24B\", alpha=0.85, capsize=5)\nax.set_ylim(0.88, 0.99)\nax.set_ylabel(\"Aggregate AUC\")\nax.set_title(\"2C Source Holdout Proxy: Aggregate Test AUC\")\nax.tick_params(axis=\"x\", rotation=20)\nfor i, (_, row) in enumerate(holdout_df.iterrows()):\n ax.text(i, row[\"auc_mean\"] + 0.004, f\"{row['delta_vs_all_source']:+.3f}\", ha=\"center\", fontsize=9)\nfig.tight_layout()\nfig.savefig(FIGURES_DIR / \"2c_holdout_proxy.png\", dpi=200, bbox_inches=\"tight\")\nplt.show()\n\nprint(\"Geometry diagnostic evidence:\")\ngeometry_keys = []\nfor run in [\"p2a_t1_original\", \"p2a_t2_real_norm\"]:\n results = load_results(run)\n cfg = (results or {}).get(\"config\", {})\n geometry_keys.append({\n \"run\": run,\n \"config_geometry_condition\": cfg.get(\"geometry_condition\"),\n \"has_matched_geometry_metric\": any(\n \"geometry\" in str(k).lower() or \"matched\" in str(k).lower()\n for fold in (results or {}).get(\"fold_results\", [])\n for k in fold.get(\"test_metrics\", {}).keys()\n ),\n })\ndisplay(pd.DataFrame(geometry_keys))" ] }, { "cell_type": "markdown", "id": "2c3b8812", "metadata": {}, "source": [ "## 2D / 2E - Augmentation impact and test-set integrity\n", "\n", "The augmentation question has two parts:\n", "\n", "- Does light augmentation help at 224 without facecrop?\n", "- Does it help once facecrop is enabled?\n", "\n", "The implementation also needs to guarantee that validation/test evaluation is not stochastic. The preprocessing pipeline keeps stochastic operations behind `self.train`, so `train=False` disables them even if augmentation settings exist." ] }, { "cell_type": "code", "execution_count": null, "id": "f11c3257", "metadata": {}, "outputs": [], "source": [ "print(\"2D (p2d): augmentation without facecrop\")\n", "print_comparison_readout(comparison_subset(\"2D\", \"augmentation\"))\n", "print(\"2E (p2e): augmentation with facecrop\")\n", "print_comparison_readout(comparison_subset(\"2E\", \"facecrop + augmentation\"))\n", "\n", "aug_sections = comparisons_df[comparisons_df[\"section\"].isin([\"2D\", \"2E\"])].copy()\n", "fig, ax = plt.subplots(figsize=(9, 5))\n", "labels = aug_sections[\"section\"] + \" \" + aug_sections[\"model\"]\n", "ax.bar(labels, aug_sections[\"delta_auc\"], yerr=aug_sections[\"delta_ci95\"], color=[\"#E45756\" if d < 0 else \"#059669\" for d in aug_sections[\"delta_auc\"]], alpha=0.85, capsize=5)\n", "ax.axhline(0, color=\"black\", linewidth=1)\n", "ax.set_ylabel(\"Delta AUC from adding augmentation\")\n", "ax.set_title(\"Augmentation Effects Across Facecrop Conditions\")\n", "ax.tick_params(axis=\"x\", rotation=20)\n", "fig.tight_layout()\n", "fig.savefig(FIGURES_DIR / \"2d_2e_augmentation_effects.png\", dpi=200, bbox_inches=\"tight\")\n", "plt.show()\n", "\n", "# Static and behavioral audit of eval stochasticity.\n", "try:\n", " import inspect\n", " from src.preprocessing.pipeline import DFFImagePipeline\n", " from src.evaluation import evaluate as evaluate_module\n", "\n", " pipeline_src = inspect.getsource(DFFImagePipeline)\n", " build_transforms_src = inspect.getsource(evaluate_module.build_transforms)\n", " stochastic_guards = {\n", " \"flip_guarded_by_train\": \"if self.train and random.random() < self.hflip_p\" in pipeline_src,\n", " \"rotate_guarded_by_train\": \"if self.train and self.rotation_degrees > 0\" in pipeline_src,\n", " \"color_jitter_returns_when_not_train\": \"if not self.train:\" in pipeline_src,\n", " \"blur_guarded_by_train\": \"if self.train and random.random() < self.blur_p\" in pipeline_src,\n", " \"jpeg_guarded_by_train\": \"if self.train and random.random() < self.jpeg_p\" in pipeline_src,\n", " \"erase_guarded_by_train\": \"if self.train and random.random() < self.erase_p\" in pipeline_src,\n", " \"noise_guarded_by_train\": \"if self.train and random.random() < self.noise_p\" in pipeline_src,\n", " \"cv_transform_uses_train_flag\": \"get_transforms(train=train\" in build_transforms_src,\n", " }\n", " display(pd.DataFrame([stochastic_guards]).T.rename(columns={0: \"passes\"}))\n", "except Exception as exc:\n", " print(f\"Could not run transform audit: {exc}\")" ] }, { "cell_type": "markdown", "id": "02e47658", "metadata": {}, "source": [ "## Decision synthesis\n", "\n", "This section converts the evidence into Phase 3 settings. It intentionally distinguishes a recommendation from a claim:\n", "\n", "- Recommendation: choose the setting that is best supported for the next experiment.\n", "- Claim: what the current evidence proves. Some Phase 2C claims remain incomplete without per-source or matched-geometry outputs." ] }, { "cell_type": "code", "execution_count": null, "id": "7034443c", "metadata": {}, "outputs": [], "source": [ "def get_delta(question: str, model: str | None = None, section: str | None = None) -> pd.DataFrame:\n", " df = comparisons_df[comparisons_df[\"question\"].eq(question)].copy()\n", " if model:\n", " df = df[df[\"model\"].eq(model)]\n", " if section:\n", " df = df[df[\"section\"].eq(section)]\n", " return df\n", "\n", "resolution_resnet = get_delta(\"resolution\", \"ResNet18\").iloc[0]\n", "facecrop_resnet = get_delta(\"facecrop\", \"ResNet18\").iloc[0]\n", "facecrop_simple = get_delta(\"facecrop\", \"SimpleCNN\").iloc[0]\n", "aug_no_crop_resnet = get_delta(\"augmentation\", \"ResNet18\").iloc[0]\n", "aug_no_crop_simple = get_delta(\"augmentation\", \"SimpleCNN\").iloc[0]\n", "aug_crop_resnet = get_delta(\"facecrop + augmentation\", \"ResNet18\").iloc[0]\n", "aug_crop_simple = get_delta(\"facecrop + augmentation\", \"SimpleCNN\").iloc[0]\n", "norm = get_delta(\"normalization\", \"ResNet18\").iloc[0]\n", "\n", "recommendations = [\n", " {\n", " \"choice\": \"resolution\",\n", " \"recommendation\": \"224x224\",\n", " \"evidence\": f\"ResNet18 delta AUC {resolution_resnet.delta_auc:+.4f}; SimpleCNN does not determine Phase 3 capacity.\",\n", " \"confidence\": \"high\" if resolution_resnet.delta_auc > 0.02 else \"medium\",\n", " },\n", " {\n", " \"choice\": \"facecrop\",\n", " \"recommendation\": \"use facecrop\",\n", " \"evidence\": f\"Small positive deltas for both models: SimpleCNN {facecrop_simple.delta_auc:+.4f}, ResNet18 {facecrop_resnet.delta_auc:+.4f}.\",\n", " \"confidence\": \"medium\",\n", " },\n", " {\n", " \"choice\": \"augmentation\",\n", " \"recommendation\": \"do not use light augmentation for Phase 3 at 20% data\",\n", " \"evidence\": f\"SimpleCNN drops {aug_no_crop_simple.delta_auc:+.4f} without facecrop and {aug_crop_simple.delta_auc:+.4f} with facecrop; ResNet18 is neutral/slightly mixed ({aug_no_crop_resnet.delta_auc:+.4f}, {aug_crop_resnet.delta_auc:+.4f}).\",\n", " \"confidence\": \"high for SimpleCNN, medium for ResNet18\",\n", " },\n", " {\n", " \"choice\": \"normalization\",\n", " \"recommendation\": \"ImageNet normalization\",\n", " \"evidence\": f\"Real-train-only normalization delta AUC {norm.delta_auc:+.4f}; no useful gain and less standard for pretrained ResNet.\",\n", " \"confidence\": \"medium\",\n", " },\n", " {\n", " \"choice\": \"shortcut/source claims\",\n", " \"recommendation\": \"do not overclaim; add per-source or prediction exports before final report\",\n", " \"evidence\": \"Current CV logs lack held-out-source vs in-source AUC and matched-geometry test metrics.\",\n", " \"confidence\": \"high\",\n", " },\n", "]\n", "\n", "recommendations_df = pd.DataFrame(recommendations)\n", "display(recommendations_df)\n", "\n", "summary = {\n", " \"phase\": \"phase2\",\n", " \"source_documents\": [\"classifier/v2.md\", \"classifier/impl.md\"],\n", " \"artifact_counts\": {\n", " \"canonical_runs\": int(len(canonical_runs_df)),\n", " \"loaded_canonical_runs\": int(canonical_runs_df[\"log_status\"].isin([\"present\", \"fallback\"]).sum()),\n", " \"fallback_runs_used\": {k: v for k, v in RUN_ALIASES.items() if resolve_run(k) != k},\n", " },\n", " \"recommendations\": recommendations,\n", " \"planned_comparisons\": comparisons_df.replace({np.nan: None}).to_dict(orient=\"records\"),\n", " \"known_gaps\": [\n", " \"Dedicated p2a_*_128 logs are absent/skipped; Phase 1 baselines are used as fallbacks.\",\n", " \"Source holdout logs do not include prediction-level or per-source metrics, so held-out-source AUC vs in-source AUC cannot be computed.\",\n", " \"No matched-geometry evaluation metric is present in p2c logs, so geometry shortcut analysis is incomplete.\",\n", " ],\n", "}\n", "\n", "summary_path = ANALYSIS_DIR / \"phase2_analysis_summary.json\"\n", "with summary_path.open(\"w\") as f:\n", " json.dump(summary, f, indent=2)\n", "\n", "print(f\"Saved summary: {summary_path.relative_to(PROJECT_ROOT)}\")\n", "print(f\"Saved figures: {FIGURES_DIR.relative_to(PROJECT_ROOT)}\")" ] }, { "cell_type": "markdown", "id": "5a337f73", "metadata": {}, "source": [ "## Report-ready conclusion\n", "\n", "The strongest Phase 2 result is the resolution effect for ResNet18: moving to 224x224 substantially improves AUC under the controlled CV protocol. Face cropping gives a small positive effect and is reasonable to carry forward, especially because it aligns the model with face evidence rather than background context. Light augmentation is not supported at this 20% data setting: it strongly hurts SimpleCNN and provides no reliable gain for ResNet18, with or without face cropping. ImageNet normalization remains preferable because real-train-only normalization does not improve AUC and is less aligned with pretrained ResNet expectations.\n", "\n", "Recommended Phase 3 preprocessing: **224x224, facecrop enabled, no light augmentation, ImageNet normalization**.\n", "\n", "Limitations to fix before the final report: export prediction-level records or per-source pairwise metrics for source holdout, and add the matched-geometry evaluation required by the shortcut-analysis plan. Without those artifacts, Phase 2C can only support a limited shortcut analysis." ] } ], "metadata": { "kernelspec": { "display_name": "drl", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.13" } }, "nbformat": 4, "nbformat_minor": 5 }