astrophotography/state/TODO.md

# TODO

> The current state of play. Keep it honest and current; this is what the next
> session reads to know what to do. Move items between sections as they progress.

---

## RESUME HERE - 2026-07-21, end of session

Three strands are live. Each has a clear next action.

### 1. Generic pipeline - MID-BUILD, this is the active work

Building a push-button pipeline that processes any session, not just LRGB from
one telescope. Branch `generic-pipeline`.

**Done:** `pipeline/session.py` - discovery, ingest, header-driven scanning,
filter grouping, palette detection (LRGB / RGB / SHO / HOO / MONO), manifest
writing. Tested against all five archived sessions.

**Done since:** `measure.py`, `register.py`, `colour.py`, `astrometry.py`.
Registration verified on the mixed bin1/bin2 case - upsampled bin2 colour
aligns to the bin1 luminance master within **0.04 px**. Colour renders LRGB,
RGB, SHO, HOO and mono.

**Next:** finish `astrometry.py` (see below), then `report.py` (per-session
METHODS.md) and `run.py` (the one-command orchestrator).

**PLATE SOLVE IS 2 OF 4.** NGC 5128 (by hand) and NGC 2030 solve; NGC 2070 and
NGC 6744 do not. What is already known, so the next session does not repeat it:

- **Not the plate scale.** Header and FOCALLEN/XPIXSZ agree on both failures,
  and the session that DOES solve has a header/computed discrepancy
  (0.527 vs 0.686) yet solves anyway - asterism matching is scale invariant,
  so scale only sets the catalogue cone radius.
- **Not the catalogue fetch.** Fixed: VizieR is queried first because Gaia's
  `launch_job_async` submits to a job queue and hung for ten minutes on 600
  rows while the network was healthy. Gaia synchronous is the fallback.
- **Not simple truncation.** Fixed: VizieR's `row_limit` truncates BEFORE
  sorting, so asking for 600 rows returned 600 arbitrary stars rather than the
  600 brightest. Now fetches up to 50000 and picks the brightest locally.
- **Partly saturation.** The detector rejects saturated cores, so the brightest
  DETECTIONS are not the brightest STARS while the catalogue's are, and the two
  top-N lists barely overlap. NGC 2030 only solved once a sliding magnitude
  window was tried, and it matched at `catalogue[15:135]` - skipping the 15
  brightest. That window search is in place and still is not enough for the
  other two.
- **Next things to try:** match on a magnitude-matched subset by estimating the
  image's own zero point first; loosen `aa.MIN_MATCHES_FRACTION`; raise
  astroalign's `max_control_points`; or fall back to a genuine blind solver
  (astrometry.net index files locally, or the nova API) for fields that resist
  the seeded approach. NGC 2070 is a 4 degree field at 3.5 arcsec/px with only
  305 detections, and NGC 6744 is a rich field - they may need different
  handling from each other.

**The blocker before any of that:** the disk is full (see below).

**The known hard part:** NGC 6744 mixes bin1 luminance at 4096x4096 with bin2
colour at 2048x2048. Registration must resample onto a common grid rather than
assume every frame shares a shape. Nothing else in the archive does this, so it
is easy to forget and it will fail loudly when it happens.

### 2. Three decisions waiting on the user

- **Where should the image archive live?** `C:` has ~2.6 GB free and the
  archive is 14 GB after extraction. ~2.4 GB is safely reclaimable (1.69 GB of
  extracted TIFFs the pipeline never reads, 0.74 GB of mirror-junk duplicates)
  but that only buys headroom, it does not fix it. The pipeline is location
  independent via `ASTRO_SESSION`, so moving the tree costs nothing but a copy.
  **Do not start batch processing until this is resolved - it will fail
  partway through.**
- **NGC 6744 already has outputs** from a different session on 2026-07-20, in
  `T59/NGC6744/20260719/stacked/` under a different naming convention
  (`NGC6744_LRGB_final.png`, `master_Blue_NGC6744_20260719.fit`). Re-process
  into the new structure, or leave it alone?
- **T08 NGC 2070 has one frame per filter.** No stacking, no outlier rejection
  possible. Worth an SHO composite anyway, or skip?

### 3. Drain campaign - runs itself, but needs checking daily

See `itelescope/CAMPAIGN.md`. Next checkpoints below under "iTelescope".

---

## The archive being processed

`Downloads\itelescope\` - five sessions, four telescopes, 2021 to 2026. Frame
counts below are AFTER removing the raw/calibrated duplication (see NOTES).

| Session | Palette | Frames | Note |
|---|---|---|---|
| T32 NGC 5128 | LRGB | 12/4/4/4 | matches the hand-processed run exactly: a working regression test |
| T59 NGC 6744 | LRGB | 15/7/7/7 | **L bin1 4096², RGB bin2 2048²** - mixed grids |
| T20 IC 1805 | Ha only | 14 | mono; **2 frames lost to corrupt archives** |
| T17 NGC 2030 | RGB, no L | 3/3/3 | only 6 min per channel |
| T08 NGC 2070 | SHO | 1/1/1 | one 60 s frame per filter |

---

## Done - 2026-07-21

### Repositories
- **Merged `itelescope` + `astro-pipeline` into `astrophotography`**, both
  histories preserved (45 commits; `git log --follow` traces files across the
  move). Old repos carry redirect notices in README and CLAUDE.md; **not
  archived**, because the campaign is live and other sessions work in
  `itelescope` most nights. Archive after 12 Aug.

### NGC 5128 processed end to end (the reference session)
Lives in `Downloads\NGC5128\20260721\`, organised into `raw/ calibrated/
stacks/ final/ renderings/ science/ intermediates/` with a `METHODS.md` and a
README in each subdirectory. Results:
- Plate solve: 237 stars, **0.27 arcsec residual**, 0.5376 arcsec/px
- Photometry: zero point G = -2.5 log10(flux) + 27.941, scatter 0.164 mag,
  limiting **G ~ 20.7** at SNR 5
- **Four deliverable images** in `final/`, from raw stack to science-informed,
  plus a close-up framed by the galaxy's measured extent
- **Globular clusters:** 289 candidates, 179 (62%) catalogue-confirmed, density
  falling **10.5x** outward. A real detection.
- **Surface photometry:** Sersic n = 4.27, Re = 400 arcsec, total G = 6.56
  which matches the catalogued value and validates the whole calibration chain.
  No shells, and the data could never have shown them (systematics-limited,
  3 mag short).
- **Transients/moving objects:** two clean nulls with measured limits.

### Corrections made (both were Claude's errors - do not repeat them)
- **The Cen A nucleus was never saturated.** A foreground star 69 arcsec from
  the nucleus was measured instead of the galaxy. A T32 booking made to "fix"
  it was cancelled before running, costing nothing. Always filter stars out
  before measuring a core.
- **NGC 6744 HAD already been processed** by another session on 20 Jul, and the
  user was told otherwise. Check for existing outputs before asserting none
  exist.

### Eclipse plan written
`observing/eclipse-2026-menorca/PLAN.md` - full field plan.

---

## Pending - pipeline

- Finish the stages: measure, register/stack, solve, compose, report, run.py.
- Then productionise: staged CLI, each stage resumable, packaged as an
  **Apptainer image**. **Slurm array jobs only for the expensive stages** -
  measured on NGC 5128, per-frame work is 2.9 s and all 24 frames take 1.1 min
  single-threaded, so a scheduler buys nothing on stacking. It pays for the
  Monte Carlo work (completeness and injection-recovery tests, difference
  imaging, shift-and-stack), which is where compute actually bit.
- Must handle beyond mono LRGB: narrowband palettes, one-shot colour with
  debayering, other observatories' header conventions, and full calibration
  from bias/dark/flat for sources that do not pre-calibrate.
- **Three findings from the first session are requirements, not options:**
  vet moving objects in DETECTOR coordinates; carry r50/psf into any catalogue
  cross-match; never fit a sky background to a field the target fills.
- Object-prefixed output naming should be derived from the OBJECT header, not
  typed in. On a cluster, `final-3-best.png` is ambiguous the moment there is a
  second dataset.
- Emit the alignment-only stack as a standard artefact of every run. It costs
  almost nothing and it is the reference every later stage can be diffed
  against when a processed result looks wrong.

## Pending - science (deliberately after the pipeline)

The analyses are per-object and need judgement; they are not push-button. A
globular cluster survey suits Centaurus A and is meaningless for the Heart
Nebula. Approach these one at a time once the imaging half runs unattended.

## Pending - observing

- **Total solar eclipse, 12 Aug 2026, Menorca.** Totality 1m12s at Ciutadella,
  Sun at **2 degrees**, azimuth 288, sunset 20:42.
  `observing/eclipse-2026-menorca/PLAN.md` has the full plan.
  - **ORDER THE CERTIFIED SOLAR FILTER.** Only item with no substitute.
  - Practise on a low sunset to calibrate exposures for the extreme extinction.
  - Outstanding: tailor the exposure table once the A7 body and lenses are known.

## Pending - iTelescope

- **CrA on T8** ran 11:45-14:45 UTC 22 Jul. Verify images under
  `/images/qisback/T8/` on t8.itelescope.net:8008, then put the actual bill from
  account/history.aspx into the ledger. **First T8 bill, so it fixes T8's
  effective rate** (card says 130/hr; trust nothing until observed).
- **T33 first-light test** on its third booking (749314, ran 14:05 UTC 21 Jul).
  If it failed again, stop rebooking: either walk it in (POST plan.asp per
  `itelescope/plans/README.md`) or drop the free-imaging objective.
- **Campaign:** 541 of 2,638 points spent, 2,097 remain, all to be gone by
  10 Aug with cancellation before the 12 Aug renewal (**user confirmation
  required for the cancel itself**). On current numbers the campaign
  **under-spends by ~800** - plan extra or longer runs, do not trim.
- Download all data from data.itelescope.net before the 10 Aug sweep.
- Automation: map ACP endpoints per scope; explore lookup.itelescope.online;
  enumerate more /DataService.svc methods.
- Fill in T74 when specs are published; reconcile retired scopes T9, T19, T31,
  T69; periodically re-export the specs sheet and diff against
  `itelescope/data/`.