Notebooks gerador com bom nome + README

generator/notebooks/_build.py

@@ -9,6 +9,7 @@ story structure: goal, what changed, evidence, decision, and conclusion.
 
 Real metric values are pulled from outputs/logs/*.json at build time and
 rendered into markdown headers and conclusions, so reports never drift from data.
+The generated filenames are numbered to make the intended reading order clear.
 """
 import json
 from pathlib import Path
@@ -17,6 +18,18 @@ ROOT = Path(__file__).resolve().parents[1]
 LOGS = ROOT / "outputs" / "logs"
 OUT  = ROOT / "notebooks"
 
+NOTEBOOK_SEQUENCE = {
+    "phase0": "01_baseline_sanity_check",
+    "phase1": "02_pipeline_selection",
+    "phase2": "03_gan_stability_progression",
+    "phase3": "04_vae_loss_progression",
+    "phase4": "05_ddpm_recipe_progression",
+    "phase5": "06_final_family_comparison",
+    "phase6": "07_final_sample_showcase",
+}
+
+OLD_GENERATED_PATTERNS = ["phase*.ipynb"]
+
 
 # notebook helpers
 def md(text):     return {"cell_type": "markdown", "metadata": {}, "source": text.splitlines(keepends=True)}
@@ -36,6 +49,12 @@ def write_nb(name, cells):
     path.write_text(json.dumps(nb, indent=1))
     print(f"  wrote {path.relative_to(ROOT)}")
 
+def remove_old_generated_notebooks():
+    for pattern in OLD_GENERATED_PATTERNS:
+        for path in OUT.glob(pattern):
+            path.unlink()
+            print(f"  removed {path.relative_to(ROOT)}")
+
 
 # log-derived facts, computed once and baked into markdown
 def load(name):
@@ -120,7 +139,7 @@ def build_phase0():
     p0 = {n: load(n) for n in ["p0_wgan", "p0_vae", "p0_ddpm", "p0_ddpm_small"]}
     cells = [
         md(f"""\
-# Phase 0 - Baseline Sanity Check
+# 01 - Baseline Sanity Check
 
 Phase 0 is the starting point of the generator story. It uses the raw, un-aligned
 images and very plain versions of each model family so we can confirm that the
@@ -146,7 +165,7 @@ FID was not logged in Phase 0. The evidence here is loss behavior plus saved
 sample grids.
 """),
         code(SHARED_IMPORTS),
-        md("## 1. Training loss curves\n\nThese curves check that the loops ran and produced stable logs. They are not enough to prove visual quality."),
+        md("## 1. Training loss curves\n\nThese curves check that the loops ran and produced stable logs. They are not enough to prove visual quality, but they are needed before interpreting samples: a broken optimization loop would make every later visual comparison meaningless.\n\n**What to look for:** the curves should move smoothly enough to show that each family is learning something. The limitation is that loss scale differs by family, so the curves compare stability, not final image quality."),
         code("""\
 runs = {n: load_log(n) for n in ["p0_wgan", "p0_vae", "p0_ddpm", "p0_ddpm_small"]}
 runs = {k: v for k, v in runs.items() if v}
@@ -181,7 +200,7 @@ axes[2].set_xlabel("Epoch"); axes[2].legend()
 
 plt.tight_layout(); plt.show()
 """),
-        md("## 2. Final sample grids\n\nThe final previews show the practical failure mode of the raw pipeline: the samples have some face-like structure, but identity, alignment, and detail are not under control. These PNGs are displayed exactly as saved, so older Phase 0 matrices keep their original layout instead of being forced into 4x4."),
+        md("## 2. Final sample grids\n\nThe final previews show the practical failure mode of the raw pipeline: the samples have some face-like structure, but identity, alignment, and detail are not under control. These PNGs are displayed exactly as saved, so older Phase 0 matrices keep their original layout instead of being forced into 4x4.\n\n**Why this matters:** this is the visual evidence that the first bottleneck is not only the model family. The data still contains too much pose, scale, and background variation for tiny baseline recipes."),
         code("""\
 last_epochs = {"p0_wgan": 200, "p0_vae": 100, "p0_ddpm": 200, "p0_ddpm_small": 100}
 
@@ -196,7 +215,7 @@ for ax, (name, ep) in zip(axes, last_epochs.items()):
     ax.axis("off")
 plt.tight_layout(); plt.show()
 """),
-        md("## 3. Progression - early vs late\n\nThe progression grids make the baseline failure visible over time. Later samples improve slightly, but the raw input distribution keeps the task too broad. The saved matrices are shown in their original layout."),
+        md("## 3. Progression - early vs late\n\nThe progression grids make the baseline failure visible over time. Later samples improve slightly, but the raw input distribution keeps the task too broad. The saved matrices are shown in their original layout.\n\n**How to read it:** if more epochs only turn noise into rough face-like blobs, the next decision should be pipeline cleanup rather than simply training the same recipe longer."),
         code("""\
 checkpoints = {
     "p0_wgan":       [50, 100, 200],
@@ -231,7 +250,7 @@ locks the pipeline before the project spends more compute on stronger recipes.
 establishes the baseline failure and motivates the move to aligned face crops.
 """),
     ]
-    write_nb("phase0_analysis", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase0"], cells)
 
 
 # PHASE 1 - Pipeline ablations with a DCGAN proxy
@@ -246,7 +265,7 @@ def build_phase1():
 
     cells = [
         md(f"""\
-# Phase 1 - Pipeline Selection
+# 02 - Pipeline Selection
 
 Phase 1 answers the data-handling question left open by the baseline. Instead
 of changing the model family, it uses a cheap DCGAN proxy and varies one
@@ -279,7 +298,7 @@ without any pipeline tuning, and it also collapsed. Phase 1 below uses the same
 with the data pipeline systematically varied; the architecture limitation is constant.
 """),
         code(SHARED_IMPORTS),
-        md("## 1. Load all experiment logs\n\nAll evidence in this notebook comes from the existing Phase 1 logs and sample folders."),
+        md("## 1. Load all experiment logs\n\nAll evidence in this notebook comes from the existing Phase 1 logs and sample folders. The cell is intentionally simple: it only inventories already saved experiments so the reader knows which pipeline ablations are being compared."),
         code("""\
 run_names = sorted(p.stem for p in LOGS.glob("p1*.json"))
 runs = {name: load_log(name) for name in run_names}
@@ -300,7 +319,7 @@ experiment_groups = {
                               "p1d_dcgan_combined": "Aligned + raw mixed"},
 }
 """),
-        md("## 2. FID comparison table\n\nThe table ranks the proxy runs. The values are useful within Phase 1, but they should not be compared directly with later FID protocols."),
+        md("## 2. FID comparison table\n\nThe table ranks the proxy runs. It is needed because the visual samples alone can be misleading: a run can look slightly better in one grid while still being worse across the saved distribution. The values are useful within Phase 1, but they should not be compared directly with later FID protocols."),
         code("""\
 rows = []
 for name in run_names:
@@ -332,7 +351,7 @@ ax.set_title("Phase 1 - FID across all pipeline ablations")
 ax.set_xticks(x); ax.set_xticklabels(labels, rotation=30, ha="right")
 ax.legend(); plt.tight_layout(); plt.show()
 """),
-        md("## 3. Controlled ablation results\n\nEach subplot holds the model approximately fixed and changes one pipeline factor. This is the decision evidence for the rest of the generator suite."),
+        md("## 3. Controlled ablation results\n\nEach subplot holds the model approximately fixed and changes one pipeline factor. This is the decision evidence for the rest of the generator suite: alignment, resolution, augmentation, and dataset mixing are treated as pipeline choices, not as disconnected experiments.\n\n**What to look for:** a useful pipeline change should lower FID consistently inside its ablation group, not only produce one nicer-looking example."),
         code("""\
 fig, axes = plt.subplots(2, 2, figsize=(14, 10))
 axes = axes.flatten()
@@ -356,6 +375,8 @@ plt.tight_layout(); plt.show()
 ## 4. Data pipeline visualization
 
 What each ablation actually changes, shown on the input data the model sees.
+These figures are not model outputs. They explain the input distribution that
+each model has to learn, which is why they sit next to the ablation results.
 """),
         code("""\
 import random
@@ -399,7 +420,7 @@ def show_unavailable(ax, message):
     ax.text(0.5, 0.5, message, ha="center", va="center", wrap=True, transform=ax.transAxes)
     ax.axis("off")
 """),
-        md("### 4A - Resolution\n\nSame raw image at 64x64 and 128x128. This is a paired comparison layout, so it keeps the original 2x4 format instead of being forced into a 4x4 sample grid."),
+        md("### 4A - Resolution\n\nSame raw image at 64x64 and 128x128. This is a paired comparison layout, so it keeps the original 2x4 format instead of being forced into a 4x4 sample grid.\n\n**Interpretation:** 128x128 carries more detail, but it also makes the proxy generator learn a harder distribution. The later decision favors 64x64 because stable face structure matters more than nominal resolution at this budget."),
         code("""\
 paths = sample_paths(RAW, k=4)
 fig, axes = plt.subplots(2, 4, figsize=(12, 6))
@@ -414,7 +435,7 @@ else:
 fig.suptitle("1A - Resolution: same image at two scales", fontsize=12, fontweight="bold")
 plt.tight_layout(); plt.show()
 """),
-        md("### 4B - Alignment\n\nRaw vs MTCNN-aligned 64x64 crops. This paired layout keeps the original 2x4 format so each raw image is directly above its aligned crop."),
+        md("### 4B - Alignment\n\nRaw vs MTCNN-aligned 64x64 crops. This paired layout keeps the original 2x4 format so each raw image is directly above its aligned crop.\n\n**Interpretation:** alignment removes background and scale variation before the generator spends capacity on it. This is why alignment becomes the strongest pipeline lever."),
         code("""\
 pairs = matched_pairs(k=4)
 fig, axes = plt.subplots(2, 4, figsize=(12, 6))
@@ -429,7 +450,7 @@ else:
 fig.suptitle("1B - Alignment: same source image, raw vs MTCNN-aligned", fontsize=12, fontweight="bold")
 plt.tight_layout(); plt.show()
 """),
-        md("### 4C - Augmentation\n\nOne aligned image, then deterministic examples of the saved augmentation idea. This keeps the original compact strip because the point is to compare transforms on one image, not to make a generated 4x4 sample matrix."),
+        md("### 4C - Augmentation\n\nOne aligned image, then deterministic examples of the saved augmentation idea. This keeps the original compact strip because the point is to compare transforms on one image, not to make a generated 4x4 sample matrix.\n\n**Interpretation:** augmentation can make the training distribution broader, but it can also blur already scarce structure. Phase 1 treats it as a pipeline setting to justify, not as an automatic improvement."),
         code("""\
 src = sample_paths(ALIGNED, k=1)
 if src:
@@ -454,7 +475,7 @@ else:
     fig.suptitle("1C - Augmentation", fontsize=12, fontweight="bold")
     plt.tight_layout(); plt.show()
 """),
-        md("### 4D - Dataset mixing\n\nMixing raw and aligned images asks one generator to model two different input distributions. This keeps the original paired 2x4 layout so the contrast is easy to read."),
+        md("### 4D - Dataset mixing\n\nMixing raw and aligned images asks one generator to model two different input distributions. This keeps the original paired 2x4 layout so the contrast is easy to read.\n\n**Interpretation:** mixing increases nuisance variation and makes the generator solve two problems at once. The later phases therefore inherit aligned-only data."),
         code("""\
 pairs = matched_pairs(k=4)
 fig, axes = plt.subplots(2, 4, figsize=(12, 6))
@@ -492,7 +513,7 @@ decision. Alignment is the main fix; Phase 2 can now focus on the GAN recipe
 instead of fighting raw-image variance.
 """),
     ]
-    write_nb("phase1_analysis", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase1"], cells)
 
 
 # PHASE 2 - GAN architecture and objective evolution
@@ -504,7 +525,7 @@ def build_phase2():
 
     cells = [
         md(f"""\
-# Phase 2 - GAN Progression
+# 03 - GAN Stability Progression
 
 Phase 2 keeps the Phase 1 pipeline fixed and changes the GAN recipe. This makes
 the question narrow: once the data is aligned, what model changes are needed to
@@ -591,7 +612,7 @@ df = pd.DataFrame(rows).sort_values("Best FID")
 df.style.format({"FID@25": "{:.1f}", "FID@50": "{:.1f}", "FID@100": "{:.1f}",
                  "Best FID": "{:.1f}", "Train (min)": "{:.1f}"})
 """),
-        md("## 3. FID curves - progression"),
+        md("## 3. FID curves - progression\n\nThis plot shows whether improvements happen gradually or as a step change. It is needed because the final FID table hides training dynamics: here the key story is that the 2.3 stability package changes the whole trajectory, while 2.1 and 2.2 remain collapsed."),
         code("""\
 fig, ax = plt.subplots(figsize=(10, 5))
 cmap = plt.cm.viridis
@@ -643,7 +664,7 @@ for ax, name in zip(axes, run_names):
     show_image_or_missing(ax, img_path, title)
 plt.tight_layout(); plt.show()
 """),
-        md("## 6. Progression - epoch 10 -> 50 -> 100"),
+        md("## 6. Progression - epoch 10 -> 50 -> 100\n\nThese panels connect time to visual quality. For the collapsed runs, the gray grids are still information: they show that more epochs did not fix the recipe. For the stabilized run, the same timeline shows recognizable faces emerging."),
         code("""\
 check_epochs = [10, 50, 100]
 for name in run_names:
@@ -659,7 +680,7 @@ for name in run_names:
     fig.suptitle(run_labels[name], fontsize=11, fontweight="bold")
     plt.tight_layout(); plt.show()
 """),
-        md("## 7. Pairwise comparison - what each step bought us"),
+        md("## 7. Pairwise comparison - what each step bought us\n\nEach pair isolates one decision. The purpose is to avoid saying simply that the final GAN is better: the comparison shows that Wasserstein loss alone is insufficient, the stability package is decisive, and 128x128 is premature under the saved compute budget."),
         code("""\
 transitions = [
     ("2.1 -> 2.2: BCE -> Wasserstein",      "p2_1_dcgan",         "p2_2_wgan"),
@@ -702,7 +723,7 @@ usable generator recipe, but it also shows that higher resolution is not helpful
 without enough training budget.
 """),
     ]
-    write_nb("phase2_analysis", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase2"], cells)
 
 
 # PHASE 3 - VAE composite-loss evolution
@@ -713,7 +734,7 @@ def build_phase3():
 
     cells = [
         md(f"""\
-# Phase 3 - VAE Progression
+# 04 - VAE Loss Progression
 
 Phase 3 studies the VAE family after the pipeline has been locked. The baseline
 VAE is fast and stable, but its MSE + KL objective tends to average away facial
@@ -775,7 +796,7 @@ df = pd.DataFrame(rows).sort_values("Best FID")
 df.style.format({"FID@50": "{:.1f}", "FID@100": "{:.1f}", "Best FID": "{:.1f}",
                  "Recon@100": "{:.4f}", "KL@100": "{:.2f}", "Train (min)": "{:.1f}"})
 """),
-        md("## 3. FID curves - progression"),
+        md("## 3. FID curves - progression\n\nThe curves show how each extra loss changes the generation trajectory, not just the final checkpoint. A useful VAE loss should improve prior-sample FID while preserving the stable behavior that makes VAEs attractive."),
         code("""\
 fig, ax = plt.subplots(figsize=(10, 5))
 cmap = plt.cm.plasma
@@ -805,7 +826,7 @@ axes[1].set_title("KL divergence");        axes[1].set_xlabel("Epoch"); axes[1].
 axes[2].set_title("Perceptual (VGG16)");   axes[2].set_xlabel("Epoch"); axes[2].legend(fontsize=8)
 plt.tight_layout(); plt.show()
 """),
-        md("## 5. Prior samples - epoch 100"),
+        md("## 5. Prior samples - epoch 100\n\nThese are generated samples from the latent prior, so they answer the true generator question: if we sample a random latent vector, do we get plausible faces? This is different from reconstruction quality."),
         code("""\
 fig, axes = plt.subplots(1, 3, figsize=(13, 4.5))
 for ax, name in zip(axes, run_names):
@@ -818,7 +839,7 @@ for ax, name in zip(axes, run_names):
 fig.suptitle("Prior samples (decoded from N(0, I))", fontsize=12, fontweight="bold")
 plt.tight_layout(); plt.show()
 """),
-        md("## 6. Reconstructions - epoch 100"),
+        md("## 6. Reconstructions - epoch 100\n\nReconstructions show whether the encoder-decoder still preserves real input structure. They are useful as a diagnostic, but they are not the final generation metric because reconstructing a known image is easier than sampling a new one."),
         code("""\
 fig, axes = plt.subplots(1, 3, figsize=(13, 4.5))
 for ax, name in zip(axes, run_names):
@@ -830,7 +851,7 @@ for ax, name in zip(axes, run_names):
 fig.suptitle("Reconstructions (real / decoded interleaved)", fontsize=12, fontweight="bold")
 plt.tight_layout(); plt.show()
 """),
-        md("## 7. Progression - epoch 10 -> 50 -> 100 (prior samples)"),
+        md("## 7. Progression - epoch 10 -> 50 -> 100 (prior samples)\n\nThe timeline shows how the sampled faces change as the loss stack trains. The limitation remains visible: the VAE becomes more structured and detailed, but it still tends toward smoother faces than GAN or DDPM samples."),
         code("""\
 check_epochs = [10, 50, 100]
 for name in run_names:
@@ -867,7 +888,7 @@ complementary losses, but even the selected recipe remains a speed-oriented
 family rather than the strongest quality candidate.
 """),
     ]
-    write_nb("phase3_analysis", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase3"], cells)
 
 
 # PHASE 4 - DDPM schedule, target, and width evolution
@@ -879,7 +900,7 @@ def build_phase4():
 
     cells = [
         md(f"""\
-# Phase 4 - DDPM Progression
+# 05 - DDPM Recipe Progression
 
 Phase 4 applies the same report logic to diffusion models. The pipeline is
 already fixed, so this notebook isolates the DDPM recipe: schedule, prediction
@@ -957,7 +978,7 @@ df = pd.DataFrame(rows).sort_values("Best FID")
 df.style.format({"FID@25": "{:.1f}", "FID@50": "{:.1f}", "FID@100": "{:.1f}",
                  "Best FID": "{:.1f}", "Loss@100": "{:.4f}", "Train (min)": "{:.1f}"})
 """),
-        md("## 3. FID curves - progression"),
+        md("## 3. FID curves - progression\n\nThe curves make the DDPM recipe evolution visible. The main evidence is not only that the wider final model wins, but that the big jump happens when the prediction target changes to v-prediction."),
         code("""\
 fig, ax = plt.subplots(figsize=(10, 5))
 cmap = plt.cm.cividis
@@ -983,7 +1004,7 @@ ax.set_xlabel("Epoch"); ax.set_ylabel("MSE on prediction target")
 ax.set_title("Loss (epsilon-MSE and v-MSE are not directly comparable)")
 ax.legend(); plt.tight_layout(); plt.show()
 """),
-        md("## 5. Sample grids - epoch 100"),
+        md("## 5. Sample grids - epoch 100\n\nThese grids show the qualitative side of the FID drop. They should be read as independent samples from each checkpoint, with attention to global face structure, texture noise, and artifact frequency."),
         code("""\
 fig, axes = plt.subplots(1, 4, figsize=(16, 4.5))
 for ax, name in zip(axes, run_names):
@@ -1054,7 +1075,7 @@ base_ch=192, and attention at 32/16/8 for the final comparison.
 into the strongest quality candidate for Phase 5.
 """),
     ]
-    write_nb("phase4_analysis", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase4"], cells)
 
 
 # PHASE 5 - Cross-family final comparison
@@ -1069,7 +1090,7 @@ def build_phase5():
 
     cells = [
         md(f"""\
-# Phase 5 - Final Comparison
+# 06 - Final Family Comparison
 
 Phase 5 is the project-level comparison. It loads the already trained best
 recipes from the GAN, VAE, and DDPM branches and compares their saved logs,
@@ -1136,7 +1157,7 @@ ax.set_title("Final comparison: quality vs training time")
 ax.grid(alpha=0.25)
 plt.tight_layout(); plt.show()
 """),
-        md("## 3. FID curves - all three families"),
+        md("## 3. FID curves - all three families\n\nThis plot puts the selected family recipes on one timeline. It is needed because the best final FID alone does not show convergence behavior: DDPM reaches the best quality, GAN remains close with less time, and VAE is fast but saturates at a higher FID."),
         code("""\
 fig, ax = plt.subplots(figsize=(10, 5))
 for fam, info in FAMILIES.items():
@@ -1222,6 +1243,11 @@ Smooth interpolation between two latent codes reveals whether the generator has
 learned a continuous manifold rather than a sparse memorisation. DDPM has no
 encoder, so this section is GAN/VAE only.
 
+The interpolation figures are not a ranking metric. They are included to make
+the learned representation easier to inspect: smooth transitions support the
+claim that the models learned a face manifold, while sudden jumps would suggest
+fragile or memorised structure.
+
 **Checkpoint loading note:** the cell below uses the same priority as
 `tools/sampling.py`: `final_ema` first, then `best_ema` as fallback. This avoids
 using a best-FID EMA snapshot that may have been saved very early for a
@@ -1340,14 +1366,14 @@ The final comparison supports DDPM as the best-quality generator and GAN as the
 best practical compromise.
 """),
     ]
-    write_nb("phase5_analysis", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase5"], cells)
 
 
 # PHASE 6 - Final selected sample showcase
 def build_phase6():
     cells = [
         md("""\
-# Phase 6 - Final Selected Samples
+# 07 - Final Sample Showcase
 
 This final notebook is a small showcase chapter. Phase 5 compared the model
 families quantitatively; this notebook selects the three strongest individual
@@ -1565,11 +1591,12 @@ large generated pool, so they should be used as final qualitative examples, not
 as a replacement for the full distribution-level metrics.
 """),
     ]
-    write_nb("phase6_final_showcase", cells)
+    write_nb(NOTEBOOK_SEQUENCE["phase6"], cells)
 
 
 if __name__ == "__main__":
     print("Building notebooks...")
+    remove_old_generated_notebooks()
     build_phase0()
     build_phase1()
     build_phase2()