laurence/astrophotography
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Bring the state files up to date so a cold session can resume

Browse source 
Rule 2 of the workflow is that state lives in the repo rather than the
chat, and a long session had put most of the current position only in
the chat. This writes it down.

TODO.md gains a RESUME HERE block at the top, because the useful
question for the next session is not what has been done but what to do
next and what is blocking it. It names the three live strands, the three
decisions waiting on the user, and the one thing that will break the
batch run if it is forgotten: NGC 6744 mixes bin1 luminance at 4096x4096
with bin2 colour at 2048x2048, so registration has to resample onto a
common grid. Nothing else in the archive does that, which is exactly why
it will be forgotten. It also drops three duplicate "Done (recent)"
headings that had accumulated, and tabulates what the five archived
sessions actually contain.

NOTES.md gains the gotchas, all of which produced plausible wrong
answers rather than errors: every frame is delivered twice as calibrated
and raw, so extracting both silently doubles the stack; an interrupted
segmented download leaves DIRECTORIES named like files, which defeats
anything that trusts an extension; a re-mirror duplicates rather than
replaces; two T20 archives are genuinely corrupt; extraction triples the
disk footprint and filled the drive. Alongside them the analysis lessons
from NGC 5128, kept for the same reason - each is a way to be confidently
wrong.

DECISIONS.md records why the pipeline reads headers rather than
filenames, and why imaging is being made push-button while the science
stays hand-driven.

ARCHITECTURE.md described a documentation-only repository with no code,
which stopped being true when the pipeline merged in. It now covers all
three parts, the code-and-data separation the pipeline is built around,
and the on-disk session layout.

Both of this session's own errors are recorded in TODO.md as things not
to repeat: measuring a foreground star instead of the galaxy nucleus and
booking telescope time to fix it, and telling the user NGC 6744 had not
been processed when it had.
This commit is contained in:
laurence 2026-07-21 17:32:00 +01:00
parent 015bff47fc
commit 3f785136ed
4 changed files with 303 additions and 106 deletions
Download patch file Download diff file Expand all files Collapse all files

234
state/TODO.md
View file

@ -1,121 +1,163 @@
# TODO
> The current state of play. Keep it honest and current; this is what the next session
> reads to know what to do. Move items between sections as they progress.
> The current state of play. Keep it honest and current; this is what the next
> session reads to know what to do. Move items between sections as they progress.
## Done
---
- 2026-07-17: repo created on git.discworld.casa, Default Workflow scaffold copied in.
- 2026-07-17: specs sheet snapshotted to data/itelescope-telescopes.csv (24 scopes).
- 2026-07-17: TELESCOPES.md written: per-scope reviews for all 23 active telescopes
plus a choosing guide (T74 noted as in commissioning, no specs published).
- 2026-07-18: TARGETS.md written: southern targets invisible from the UK (dec tiers
from lat 51.5°N), matched to Q62/X07 scopes, seasonal booking calendar, and a
July-August starter campaign (T33 free, T71 narrowband, T73 galaxy, T8 LMC later).
## RESUME HERE - 2026-07-21, end of session
## Done (recent)
Three strands are live. Each has a clear next action.
- 2026-07-20: NGC 6744 on T59 SUCCEEDED (reservation 749227, txn 851148): full
LRGB dataset on the scope server (L 15x300s BIN1, RGB 7x300s each BIN2),
previews verified clean. Billed 381 pts (est was ~240: see the rate
correction in NOTES.md). Balance now 2,257. Ledger updated in CAMPAIGN.md.
### 1. Generic pipeline - MID-BUILD, this is the active work
## Done (recent)
Building a push-button pipeline that processes any session, not just LRGB from
one telescope. Branch `generic-pipeline`.
- 2026-07-21 ~08:55 UTC: free T33 47 Tuc retry booked as reservation 749314
(grid-verified), third attempt at the first-light test, runs 14:05-14:50 UTC
21 Jul, costs 0.
**Done:** `pipeline/session.py` - discovery, ingest, header-driven scanning,
filter grouping, palette detection (LRGB / RGB / SHO / HOO / MONO), manifest
writing. Tested against all five archived sessions.
## Done (recent)
**Next:** `measure` -> `register/stack` -> `solve` -> `compose` -> `report`,
then a `run.py` orchestrator with resumable stages.
- 2026-07-21 11:40 UTC: Cen A on T32 verified COMPLETE (billed 160, NGC5128
images on the scope server), and queue #2 (Corona Australis on T8) booked for
the night of 22 Jul, 21:45-00:45 SSO local, which brackets the target's
culmination instead of chasing it down the western sky. Rate model rebuilt
from two actual bills. Details in the CAMPAIGN.md session note.
**The blocker before any of that:** the disk is full (see below).
## In progress
**The known hard part:** NGC 6744 mixes bin1 luminance at 4096x4096 with bin2
colour at 2048x2048. Registration must resample onto a common grid rather than
assume every frame shares a shape. Nothing else in the archive does this, so it
is easy to forget and it will fail loudly when it happens.
- T33 first-light test on its THIRD booking (749314), which had NOT yet run as of
11:40 UTC 21 Jul: it starts 14:05 UTC. Verify after 14:50 UTC (images and logs
on t33.itelescope.net:8033 were still empty/stale at 11:40, as expected). The
two prior free reservations (749228, 749229) expired unused with reschedule=No;
if 749314 also fails to run, stop rebooking and either walk it in (POST
plan.asp per plans/README.md) or drop the free objective.
- The T32 HDR core set (plans/hdrcore-t32.txt) was booked and then CANCELLED on
2026-07-21 without running: the Cen A nucleus is not saturated, the earlier
claim having measured a foreground star. See the correction in CAMPAIGN.md.
Zero points spent. The plan file remains available on T32 if a genuinely
saturated target turns up.
- The 21 Jul Cen A data has been fully processed off-repo, in the user's
Downloads\NGC5128\20260721\stacked folder: masters, plate solution (0.27
arcsec on 237 stars), an LRGB composite, a globular cluster survey (289
candidates, 62% catalogue-confirmed, density falling 10.5x outward), surface
photometry (Sersic n = 4.27, Re = 400 arcsec, total G = 6.56, which matches
the catalogued brightness and so validates the photometric chain), and a
transient/moving-object search (both null, with measured limits). METHODS.md
there documents the lot. Nothing from that analysis needs to come back into
this repo, but two results affect FUTURE sessions and are recorded here:
**(a)** measure saturation properly before buying short subs, and **(b)** the
faint outer halo in that stack is limited by the sky-plane subtraction
(-17.9 ADU/px of real halo light absorbed), not by exposure time, so more
integration alone would not have gone deeper.
- CrA on T8 runs 11:45-14:45 UTC 22 Jul. Verify after 14:45 UTC: images under
/images/qisback/T8/ on t8.itelescope.net:8008, then the actual bill from
account/history.aspx into the ledger. This is the first T8 bill, so it also
fixes T8's effective rate (card says 130/hr; trust nothing until observed).
- DRAIN CAMPAIGN ACTIVE (see CAMPAIGN.md): spend the full balance by 10 Aug,
then cancel before the 12 Aug renewal (user confirmation required for the
cancel itself). 541 of 2,638 spent; 2,097 remain. The x1.6 estimate multiplier
is WITHDRAWN: billing is on plan imaging minutes, and per-scope rates must come
from actual bills (T59 127/hr, T32 80/hr observed; the NOTES rate card is
unreliable). On current numbers the campaign UNDER-spends by ~800: plan extra
or longer runs, do not trim. Reschedule flag never sticks (grid always shows
No): treat every reservation as one-shot and verify the morning after.
Download the NGC 6744 and NGC 5128 data from data.itelescope.net before the
10 Aug cancellation sweep. Update the CAMPAIGN.md ledger after every session
from account/history.aspx.
### 2. Three decisions waiting on the user
- **Where should the image archive live?** `C:` has ~2.6 GB free and the
archive is 14 GB after extraction. ~2.4 GB is safely reclaimable (1.69 GB of
extracted TIFFs the pipeline never reads, 0.74 GB of mirror-junk duplicates)
but that only buys headroom, it does not fix it. The pipeline is location
independent via `ASTRO_SESSION`, so moving the tree costs nothing but a copy.
**Do not start batch processing until this is resolved - it will fail
partway through.**
- **NGC 6744 already has outputs** from a different session on 2026-07-20, in
`T59/NGC6744/20260719/stacked/` under a different naming convention
(`NGC6744_LRGB_final.png`, `master_Blue_NGC6744_20260719.fit`). Re-process
into the new structure, or leave it alone?
- **T08 NGC 2070 has one frame per filter.** No stacking, no outlier rejection
possible. Worth an SHO composite anyway, or skip?
### 3. Drain campaign - runs itself, but needs checking daily
See `itelescope/CAMPAIGN.md`. Next checkpoints below under "iTelescope".
---
## The archive being processed
`Downloads\itelescope\` - five sessions, four telescopes, 2021 to 2026. Frame
counts below are AFTER removing the raw/calibrated duplication (see NOTES).
| Session | Palette | Frames | Note |
|---|---|---|---|
| T32 NGC 5128 | LRGB | 12/4/4/4 | matches the hand-processed run exactly: a working regression test |
| T59 NGC 6744 | LRGB | 15/7/7/7 | **L bin1 4096², RGB bin2 2048²** - mixed grids |
| T20 IC 1805 | Ha only | 14 | mono; **2 frames lost to corrupt archives** |
| T17 NGC 2030 | RGB, no L | 3/3/3 | only 6 min per channel |
| T08 NGC 2070 | SHO | 1/1/1 | one 60 s frame per filter |
---
## Done - 2026-07-21
### Repositories
- **Merged `itelescope` + `astro-pipeline` into `astrophotography`**, both
histories preserved (45 commits; `git log --follow` traces files across the
move). Old repos carry redirect notices in README and CLAUDE.md; **not
archived**, because the campaign is live and other sessions work in
`itelescope` most nights. Archive after 12 Aug.
### NGC 5128 processed end to end (the reference session)
Lives in `Downloads\NGC5128\20260721\`, organised into `raw/ calibrated/
stacks/ final/ renderings/ science/ intermediates/` with a `METHODS.md` and a
README in each subdirectory. Results:
- Plate solve: 237 stars, **0.27 arcsec residual**, 0.5376 arcsec/px
- Photometry: zero point G = -2.5 log10(flux) + 27.941, scatter 0.164 mag,
limiting **G ~ 20.7** at SNR 5
- **Four deliverable images** in `final/`, from raw stack to science-informed,
plus a close-up framed by the galaxy's measured extent
- **Globular clusters:** 289 candidates, 179 (62%) catalogue-confirmed, density
falling **10.5x** outward. A real detection.
- **Surface photometry:** Sersic n = 4.27, Re = 400 arcsec, total G = 6.56
which matches the catalogued value and validates the whole calibration chain.
No shells, and the data could never have shown them (systematics-limited,
3 mag short).
- **Transients/moving objects:** two clean nulls with measured limits.
### Corrections made (both were Claude's errors - do not repeat them)
- **The Cen A nucleus was never saturated.** A foreground star 69 arcsec from
the nucleus was measured instead of the galaxy. A T32 booking made to "fix"
it was cancelled before running, costing nothing. Always filter stars out
before measuring a core.
- **NGC 6744 HAD already been processed** by another session on 20 Jul, and the
user was told otherwise. Check for existing outputs before asserting none
exist.
### Eclipse plan written
`observing/eclipse-2026-menorca/PLAN.md` - full field plan.
---
## Pending - pipeline
- Productionise the processing code: a staged CLI
(ingest -> calibrate -> measure -> register -> stack -> solve -> compose ->
analyse), each stage resumable, packaged as an Apptainer image. Slurm array
jobs afterwards and ONLY for the expensive stages: measured on the NGC 5128
session, per-frame work is 2.9 s and all 24 frames take 1.1 min
single-threaded, so a scheduler buys nothing there. It pays for the Monte
Carlo work (completeness and injection-recovery tests, difference imaging,
shift-and-stack), which is where compute actually bit.
- Finish the stages: measure, register/stack, solve, compose, report, run.py.
- Then productionise: staged CLI, each stage resumable, packaged as an
**Apptainer image**. **Slurm array jobs only for the expensive stages** -
measured on NGC 5128, per-frame work is 2.9 s and all 24 frames take 1.1 min
single-threaded, so a scheduler buys nothing on stacking. It pays for the
Monte Carlo work (completeness and injection-recovery tests, difference
imaging, shift-and-stack), which is where compute actually bit.
- Must handle beyond mono LRGB: narrowband palettes, one-shot colour with
debayering, other observatories' header conventions, and full calibration
from bias/dark/flat for sources that do not pre-calibrate.
- Three findings from the first session are requirements, not options: vet
moving objects in DETECTOR coordinates; carry r50/psf into any catalogue
- **Three findings from the first session are requirements, not options:**
vet moving objects in DETECTOR coordinates; carry r50/psf into any catalogue
cross-match; never fit a sky background to a field the target fills.
- Object-prefixed output naming should be derived from the OBJECT header, not
typed in. On a cluster, `final-3-best.png` is ambiguous the moment there is a
second dataset.
- Emit the alignment-only stack as a standard artefact of every run. It costs
almost nothing and it is the reference every later stage can be diffed
against when a processed result looks wrong.
## Pending - science (deliberately after the pipeline)
The analyses are per-object and need judgement; they are not push-button. A
globular cluster survey suits Centaurus A and is meaningless for the Heart
Nebula. Approach these one at a time once the imaging half runs unattended.
## Pending - observing
- Total solar eclipse, 12 Aug 2026, Menorca (`observing/eclipse-2026-menorca/`).
Totality 1m12s at Ciutadella with the Sun at 2 degrees, azimuth 288.
**Order the certified solar filter - it is the only item with no substitute
and there are three weeks left.** Then practise on a low sunset to calibrate
exposures for the extreme atmospheric extinction.
- **Total solar eclipse, 12 Aug 2026, Menorca.** Totality 1m12s at Ciutadella,
Sun at **2 degrees**, azimuth 288, sunset 20:42.
`observing/eclipse-2026-menorca/PLAN.md` has the full plan.
- **ORDER THE CERTIFIED SOLAR FILTER.** Only item with no substitute.
- Practise on a low sunset to calibrate exposures for the extreme extinction.
- Outstanding: tailor the exposure table once the A7 body and lenses are known.
## Pending - iTelescope
- Automation: map the ACP web endpoints on a per-scope server (start with free
T33) for plan submission and status; explore lookup.itelescope.online (the new
planner); enumerate more /DataService.svc methods. Recon notes in NOTES.md.
- Campaign impact (status seen 2026-07-18): T71 and T73 are offline
(development/mount testing), as are T2, T17, T18, T20, T25, T30, T72, T74,
T75, T80. T8, T33, T32, T59 at Siding Spring are operational. Re-plan the
July-August campaign around T33 free time and Siding Spring scopes until the
Chile rebuild finishes; re-check status before any booking.
- Fill in T74 when iTelescope publishes its specs.
- Reconcile the retired scopes (T9, T19, T31, T69): the support article still lists
them but the maintained sheet does not; confirm status and note it.
- Optional: per-telescope detail pages on support.itelescope.net (links in
DECISIONS.md) hold photos and operational history if deeper entries are ever wanted.
- Periodic refresh: re-export the Google Sheet and diff against data/ to catch new or
changed systems.
- **CrA on T8** ran 11:45-14:45 UTC 22 Jul. Verify images under
`/images/qisback/T8/` on t8.itelescope.net:8008, then put the actual bill from
account/history.aspx into the ledger. **First T8 bill, so it fixes T8's
effective rate** (card says 130/hr; trust nothing until observed).
- **T33 first-light test** on its third booking (749314, ran 14:05 UTC 21 Jul).
If it failed again, stop rebooking: either walk it in (POST plan.asp per
`itelescope/plans/README.md`) or drop the free-imaging objective.
- **Campaign:** 541 of 2,638 points spent, 2,097 remain, all to be gone by
10 Aug with cancellation before the 12 Aug renewal (**user confirmation
required for the cancel itself**). On current numbers the campaign
**under-spends by ~800** - plan extra or longer runs, do not trim.
- Download all data from data.itelescope.net before the 10 Aug sweep.
- Automation: map ACP endpoints per scope; explore lookup.itelescope.online;
enumerate more /DataService.svc methods.
- Fill in T74 when specs are published; reconcile retired scopes T9, T19, T31,
T69; periodically re-export the specs sheet and diff against
`itelescope/data/`.