Checkpoint 4
This commit is contained in:
Johnny Fernandes
+70
-54
@@ -1,21 +1,29 @@
|
||||
# Training pipeline
|
||||
|
||||
Behavior cloning of analytic herding teachers into a neural network
|
||||
policy that runs in Webots. PPO from scratch and PPO fine-tune of BC
|
||||
were tried earlier and are kept under `train_ppo.py` as experimental
|
||||
options, but the BC route alone is what we ship.
|
||||
Behavior cloning of analytic herding teachers into a neural-network
|
||||
policy that runs under LiDAR perception in Webots.
|
||||
|
||||
```
|
||||
sim demos (active-scan teacher on tracker output, K=4 frame stack)
|
||||
│
|
||||
▼
|
||||
bc_pretrain.py ──► runs/bc_v3 (deployed policy — beats Strömbom on n≥8)
|
||||
│
|
||||
▼ (optional: tools/auto_dagger.sh + tools/dagger_merge_train.py
|
||||
│ if sim-trained doesn't transfer cleanly to Webots)
|
||||
│
|
||||
runs/bc_dagger
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
herding_env.py — Gymnasium env (used for demo collection + eval)
|
||||
bc_pretrain.py — supervised MSE+cosine training of an SB3 MlpPolicy
|
||||
against (obs, action) demos
|
||||
herding_env.py — Gymnasium env (LiDAR raycast + tracker by default)
|
||||
bc_pretrain.py — MSE + cosine BC of (obs, action) demos into MlpPolicy
|
||||
eval.py — analytic teachers + BC policies, full n=1..10 grid
|
||||
parity_test.py — shape/determinism/baseline smoke test
|
||||
train_ppo.py — PPO trainer (experimental — see Appendix below)
|
||||
configs/ — PPO hyperparameter YAML
|
||||
runs/ — checkpoints (.gitignored)
|
||||
parity_test.py — shape / determinism / baseline smoke test
|
||||
runs/ — checkpoints (most are .gitignored; the deployed
|
||||
ones are whitelisted in the top-level .gitignore)
|
||||
```
|
||||
|
||||
## Setup
|
||||
@@ -31,66 +39,74 @@ rollout collection, not gradient compute.
|
||||
## The BC pipeline
|
||||
|
||||
```
|
||||
# 1. Generate demos from an analytic teacher.
|
||||
# --teacher: strombom (default), sequential, drive_only, hybrid, strombom_smooth
|
||||
# 1. Sim demos with the active-scan + Strömbom teacher under LiDAR
|
||||
# perception. K=4 frame stack so the MLP has temporal context.
|
||||
python -m tools.collect_demos --teacher strombom \
|
||||
--out demos.npz --seeds-per-n 30 --subsample 3
|
||||
--out demos_v3.npz --seeds-per-n 15 --subsample 3 --frame-stack 4
|
||||
|
||||
# 2. Behavior-clone the demos into an MLP policy.
|
||||
python -m training.bc_pretrain --demos demos.npz \
|
||||
--out runs/bc_flock --epochs 100 --net-arch 512,512
|
||||
# 2. Behavior-clone.
|
||||
python -m training.bc_pretrain --demos demos_v3.npz \
|
||||
--out runs/bc_v3 --epochs 60 --net-arch 512,512
|
||||
|
||||
# 3. Evaluate the resulting policy.
|
||||
python -m training.eval --policy runs/bc_flock \
|
||||
--max-flock 10 --max-steps 30000 --n-seeds 5
|
||||
# 3. Evaluate.
|
||||
python -m training.eval --policy runs/bc_v3 \
|
||||
--max-flock 10 --max-steps 8000 --n-seeds 5
|
||||
```
|
||||
|
||||
Wall time: ~10 min demos + ~5 min BC training + ~5 min eval.
|
||||
|
||||
`bc_pretrain.py` saves the **best-val_cos** snapshot, not the final
|
||||
epoch — multi-modal teachers (Strömbom's collect/drive switch) make
|
||||
training noisy and the last epoch is often worse than an earlier one.
|
||||
epoch — multi-modal teachers make training noisy and the last epoch is
|
||||
often worse than an earlier one.
|
||||
|
||||
## DAgger from Webots
|
||||
|
||||
Sim-only BC plateaus because the env's 2D raycast can't reproduce all
|
||||
the false-positive clusters Webots generates from real geometry. The
|
||||
fix is to collect (obs, teacher_action) pairs from inside Webots:
|
||||
|
||||
```
|
||||
# Headless DAgger collection: 5 flock sizes × 3 runs each.
|
||||
tools/auto_dagger.sh 3 60
|
||||
|
||||
# Merge with the sim baseline + retrain.
|
||||
python -m tools.dagger_merge_train --out runs/bc_dagger
|
||||
```
|
||||
|
||||
Iterate by re-running collection with the new student in the driver's
|
||||
seat:
|
||||
|
||||
```
|
||||
HERDING_POLICY_DIR=$PWD/training/runs/bc_dagger \
|
||||
HERDING_DAGGER_DRIVER=student \
|
||||
tools/auto_dagger.sh 3 60
|
||||
python -m tools.dagger_merge_train --out runs/bc_dagger_v2
|
||||
```
|
||||
|
||||
## Available analytic teachers
|
||||
|
||||
| Name | What it does | Best for |
|
||||
| Name | What it does | Notes |
|
||||
|---|---|---|
|
||||
| `strombom` | Canonical Strömbom — collect when flock is scattered, drive CoM otherwise | Tight-cohesion regime, n=1-10 |
|
||||
| `sequential` | Pick the sheep closest to the pen and drive only it | Loose-cohesion regime, n=1-10 |
|
||||
| `drive_only` | Strömbom drive without collect mode (continuous action) | Easier-to-BC alternative; less reliable than full Strömbom |
|
||||
| `hybrid` | Drive rearmost sheep when far, switch to closest near gate | Failed experiment, kept for write-up |
|
||||
| `strombom_smooth` | Sigmoid-blended Strömbom collect↔drive | Failed experiment |
|
||||
| `strombom` | Canonical Strömbom — collect when flock is scattered, drive CoM otherwise | Default; works well for n=1–10 under tight cohesion |
|
||||
| `sequential` | Pick the sheep closest to the pen and drive only it | Alternative; needs loose-cohesion regime |
|
||||
|
||||
## Evaluating the analytic teachers directly
|
||||
Both are wrapped at demo-collection time in
|
||||
`herding/active_scan.py:ActiveScanTeacher`, which adds an opening
|
||||
in-place rotation, walk-to-centre when the LiDAR sees nothing, and
|
||||
near-sheep speed modulation (the same modulation `herding/control.py`
|
||||
applies to every dog mode at inference).
|
||||
|
||||
## Evaluating analytic teachers directly
|
||||
|
||||
```
|
||||
python -m training.eval --policy strombom --max-flock 10 --max-steps 30000 --n-seeds 5
|
||||
python -m training.eval --policy sequential --max-flock 10 --max-steps 30000 --n-seeds 5
|
||||
python -m training.eval --policy strombom --max-flock 10 --max-steps 8000 --n-seeds 5
|
||||
python -m training.eval --policy sequential --max-flock 10 --max-steps 8000 --n-seeds 5
|
||||
```
|
||||
|
||||
## Webots inference
|
||||
|
||||
The Webots dog controller (`controllers/shepherd_dog/shepherd_dog.py`)
|
||||
loads a saved BC zip when launched in `rl` mode:
|
||||
|
||||
```
|
||||
HERDING_POLICY_DIR=$PWD/runs/bc_flock tools/run_webots.sh 10 rl
|
||||
tools/run_webots.sh 10 rl
|
||||
```
|
||||
|
||||
It auto-discovers a checkpoint named `policy.zip`, `best_model.zip`, or
|
||||
`final.zip` in the directory.
|
||||
|
||||
## Appendix — experimental PPO scripts
|
||||
|
||||
`train_ppo.py` contains the PPO/RL pipeline tried before BC:
|
||||
* PPO from scratch with curriculum learning over flock size + spawn area.
|
||||
* PPO fine-tune of a BC checkpoint.
|
||||
|
||||
Both ran into stability issues (PPO's exploration noise destroys BC
|
||||
weights faster than the reward signal can rebuild them; PPO from
|
||||
scratch never sees pen events often enough during random exploration to
|
||||
credit-assign the +500 done bonus).
|
||||
|
||||
The script is left in place because the abstractions are sound and the
|
||||
code is reusable for follow-up work (e.g. KL-regularised fine-tune
|
||||
with a frozen reference policy). Not part of the deliverable pipeline.
|
||||
The dog controller loads the highest-priority policy that exists
|
||||
(`bc_dagger_v2` → `bc_dagger` → `bc_v3`). Override with
|
||||
`HERDING_POLICY_DIR=…` if you want a specific checkpoint.
|
||||
|
||||
Reference in New Issue
Block a user