laurence fbd1eb8ec5 Add the plate solve stage; it works on two sessions of four

astrometry.py solves the deepest master against Gaia and copies the WCS
into every master. Asterism matching is invariant to rotation and scale,
so camera angle never has to be guessed, but not to a mirror flip, so
both parities are tried.

Three defects were found and fixed by running it, and each is worth
recording because none would have been found by reading the code.

Gaia's launch_job_async submits to a job QUEUE and polls. On a query of
600 rows it hung for ten minutes while curl showed both the Gaia and
VizieR endpoints answering in under a second. VizieR is now queried
first - it serves the identical DR3 catalogue over a plain HTTP request -
with Gaia's SYNCHRONOUS endpoint as the fallback.

VizieR's row_limit truncates before sorting, so asking for the 600
brightest stars returned 600 arbitrary ones. Asterism matching only works
when both lists hold the same bright stars, so this silently produced no
match at all. It now fetches generously and picks the brightest locally.

The detector deliberately rejects saturated cores, which means the
brightest DETECTIONS are not the brightest STARS, while the catalogue's
are - so the two top-N lists can barely overlap. A sliding window down
the catalogue's magnitude ranking fixed NGC 2030, which matched at
catalogue[15:135], skipping the 15 brightest. That is the saturation
hypothesis confirmed rather than assumed.

NGC 2030 now solves on 243 stars at 0.31 arcsec residual. NGC 2070 and
NGC 6744 still do not, and TODO.md records what has already been ruled
out - scale, the fetch, truncation - so the next session starts from the
remaining candidates rather than repeating the elimination.

2026-07-21 21:27:33 +01:00

9.9 KiB

Raw Permalink Blame History

TODO

The current state of play. Keep it honest and current; this is what the next session reads to know what to do. Move items between sections as they progress.

RESUME HERE - 2026-07-21, end of session

Three strands are live. Each has a clear next action.

1. Generic pipeline - MID-BUILD, this is the active work

Building a push-button pipeline that processes any session, not just LRGB from one telescope. Branch generic-pipeline.

Done: pipeline/session.py - discovery, ingest, header-driven scanning, filter grouping, palette detection (LRGB / RGB / SHO / HOO / MONO), manifest writing. Tested against all five archived sessions.

Done since: measure.py, register.py, colour.py, astrometry.py. Registration verified on the mixed bin1/bin2 case - upsampled bin2 colour aligns to the bin1 luminance master within 0.04 px. Colour renders LRGB, RGB, SHO, HOO and mono.

Next: finish astrometry.py (see below), then report.py (per-session METHODS.md) and run.py (the one-command orchestrator).

PLATE SOLVE IS 2 OF 4. NGC 5128 (by hand) and NGC 2030 solve; NGC 2070 and NGC 6744 do not. What is already known, so the next session does not repeat it:

Not the plate scale. Header and FOCALLEN/XPIXSZ agree on both failures, and the session that DOES solve has a header/computed discrepancy (0.527 vs 0.686) yet solves anyway - asterism matching is scale invariant, so scale only sets the catalogue cone radius.
Not the catalogue fetch. Fixed: VizieR is queried first because Gaia's launch_job_async submits to a job queue and hung for ten minutes on 600 rows while the network was healthy. Gaia synchronous is the fallback.
Not simple truncation. Fixed: VizieR's row_limit truncates BEFORE sorting, so asking for 600 rows returned 600 arbitrary stars rather than the 600 brightest. Now fetches up to 50000 and picks the brightest locally.
Partly saturation. The detector rejects saturated cores, so the brightest DETECTIONS are not the brightest STARS while the catalogue's are, and the two top-N lists barely overlap. NGC 2030 only solved once a sliding magnitude window was tried, and it matched at catalogue[15:135] - skipping the 15 brightest. That window search is in place and still is not enough for the other two.
Next things to try: match on a magnitude-matched subset by estimating the image's own zero point first; loosen aa.MIN_MATCHES_FRACTION; raise astroalign's max_control_points; or fall back to a genuine blind solver (astrometry.net index files locally, or the nova API) for fields that resist the seeded approach. NGC 2070 is a 4 degree field at 3.5 arcsec/px with only 305 detections, and NGC 6744 is a rich field - they may need different handling from each other.

The blocker before any of that: the disk is full (see below).

The known hard part: NGC 6744 mixes bin1 luminance at 4096x4096 with bin2 colour at 2048x2048. Registration must resample onto a common grid rather than assume every frame shares a shape. Nothing else in the archive does this, so it is easy to forget and it will fail loudly when it happens.

2. Three decisions waiting on the user

Where should the image archive live? C: has ~2.6 GB free and the archive is 14 GB after extraction. ~2.4 GB is safely reclaimable (1.69 GB of extracted TIFFs the pipeline never reads, 0.74 GB of mirror-junk duplicates) but that only buys headroom, it does not fix it. The pipeline is location independent via ASTRO_SESSION, so moving the tree costs nothing but a copy. Do not start batch processing until this is resolved - it will fail partway through.
NGC 6744 already has outputs from a different session on 2026-07-20, in T59/NGC6744/20260719/stacked/ under a different naming convention (NGC6744_LRGB_final.png, master_Blue_NGC6744_20260719.fit). Re-process into the new structure, or leave it alone?
T08 NGC 2070 has one frame per filter. No stacking, no outlier rejection possible. Worth an SHO composite anyway, or skip?

3. Drain campaign - runs itself, but needs checking daily

See itelescope/CAMPAIGN.md. Next checkpoints below under "iTelescope".

The archive being processed

Downloads\itelescope\ - five sessions, four telescopes, 2021 to 2026. Frame counts below are AFTER removing the raw/calibrated duplication (see NOTES).

Session	Palette	Frames	Note
T32 NGC 5128	LRGB	12/4/4/4	matches the hand-processed run exactly: a working regression test
T59 NGC 6744	LRGB	15/7/7/7	L bin1 4096², RGB bin2 2048² - mixed grids
T20 IC 1805	Ha only	14	mono; 2 frames lost to corrupt archives
T17 NGC 2030	RGB, no L	3/3/3	only 6 min per channel
T08 NGC 2070	SHO	1/1/1	one 60 s frame per filter

Done - 2026-07-21

Repositories

Merged itelescope + astro-pipeline into astrophotography, both histories preserved (45 commits; git log --follow traces files across the move). Old repos carry redirect notices in README and CLAUDE.md; not archived, because the campaign is live and other sessions work in itelescope most nights. Archive after 12 Aug.

NGC 5128 processed end to end (the reference session)

Lives in Downloads\NGC5128\20260721\, organised into raw/ calibrated/ stacks/ final/ renderings/ science/ intermediates/ with a METHODS.md and a README in each subdirectory. Results:

Plate solve: 237 stars, 0.27 arcsec residual, 0.5376 arcsec/px
Photometry: zero point G = -2.5 log10(flux) + 27.941, scatter 0.164 mag, limiting G ~ 20.7 at SNR 5
Four deliverable images in final/, from raw stack to science-informed, plus a close-up framed by the galaxy's measured extent
Globular clusters: 289 candidates, 179 (62%) catalogue-confirmed, density falling 10.5x outward. A real detection.
Surface photometry: Sersic n = 4.27, Re = 400 arcsec, total G = 6.56 which matches the catalogued value and validates the whole calibration chain. No shells, and the data could never have shown them (systematics-limited, 3 mag short).
Transients/moving objects: two clean nulls with measured limits.

Corrections made (both were Claude's errors - do not repeat them)

The Cen A nucleus was never saturated. A foreground star 69 arcsec from the nucleus was measured instead of the galaxy. A T32 booking made to "fix" it was cancelled before running, costing nothing. Always filter stars out before measuring a core.
NGC 6744 HAD already been processed by another session on 20 Jul, and the user was told otherwise. Check for existing outputs before asserting none exist.

Eclipse plan written

observing/eclipse-2026-menorca/PLAN.md - full field plan.

Pending - pipeline

Finish the stages: measure, register/stack, solve, compose, report, run.py.
Then productionise: staged CLI, each stage resumable, packaged as an Apptainer image. Slurm array jobs only for the expensive stages - measured on NGC 5128, per-frame work is 2.9 s and all 24 frames take 1.1 min single-threaded, so a scheduler buys nothing on stacking. It pays for the Monte Carlo work (completeness and injection-recovery tests, difference imaging, shift-and-stack), which is where compute actually bit.
Must handle beyond mono LRGB: narrowband palettes, one-shot colour with debayering, other observatories' header conventions, and full calibration from bias/dark/flat for sources that do not pre-calibrate.
Three findings from the first session are requirements, not options: vet moving objects in DETECTOR coordinates; carry r50/psf into any catalogue cross-match; never fit a sky background to a field the target fills.
Object-prefixed output naming should be derived from the OBJECT header, not typed in. On a cluster, final-3-best.png is ambiguous the moment there is a second dataset.
Emit the alignment-only stack as a standard artefact of every run. It costs almost nothing and it is the reference every later stage can be diffed against when a processed result looks wrong.

Pending - science (deliberately after the pipeline)

The analyses are per-object and need judgement; they are not push-button. A globular cluster survey suits Centaurus A and is meaningless for the Heart Nebula. Approach these one at a time once the imaging half runs unattended.

Pending - observing

Total solar eclipse, 12 Aug 2026, Menorca. Totality 1m12s at Ciutadella, Sun at 2 degrees, azimuth 288, sunset 20:42. observing/eclipse-2026-menorca/PLAN.md has the full plan.
- ORDER THE CERTIFIED SOLAR FILTER. Only item with no substitute.
- Practise on a low sunset to calibrate exposures for the extreme extinction.
- Outstanding: tailor the exposure table once the A7 body and lenses are known.

Pending - iTelescope

CrA on T8 ran 11:45-14:45 UTC 22 Jul. Verify images under /images/qisback/T8/ on t8.itelescope.net:8008, then put the actual bill from account/history.aspx into the ledger. First T8 bill, so it fixes T8's effective rate (card says 130/hr; trust nothing until observed).
T33 first-light test on its third booking (749314, ran 14:05 UTC 21 Jul). If it failed again, stop rebooking: either walk it in (POST plan.asp per itelescope/plans/README.md) or drop the free-imaging objective.
Campaign: 541 of 2,638 points spent, 2,097 remain, all to be gone by 10 Aug with cancellation before the 12 Aug renewal (user confirmation required for the cancel itself). On current numbers the campaign under-spends by ~800 - plan extra or longer runs, do not trim.
Download all data from data.itelescope.net before the 10 Aug sweep.
Automation: map ACP endpoints per scope; explore lookup.itelescope.online; enumerate more /DataService.svc methods.
Fill in T74 when specs are published; reconcile retired scopes T9, T19, T31, T69; periodically re-export the specs sheet and diff against itelescope/data/.

9.9 KiB Raw Permalink Blame History