laurence 3f785136ed Bring the state files up to date so a cold session can resume

Rule 2 of the workflow is that state lives in the repo rather than the
chat, and a long session had put most of the current position only in
the chat. This writes it down.

TODO.md gains a RESUME HERE block at the top, because the useful
question for the next session is not what has been done but what to do
next and what is blocking it. It names the three live strands, the three
decisions waiting on the user, and the one thing that will break the
batch run if it is forgotten: NGC 6744 mixes bin1 luminance at 4096x4096
with bin2 colour at 2048x2048, so registration has to resample onto a
common grid. Nothing else in the archive does that, which is exactly why
it will be forgotten. It also drops three duplicate "Done (recent)"
headings that had accumulated, and tabulates what the five archived
sessions actually contain.

NOTES.md gains the gotchas, all of which produced plausible wrong
answers rather than errors: every frame is delivered twice as calibrated
and raw, so extracting both silently doubles the stack; an interrupted
segmented download leaves DIRECTORIES named like files, which defeats
anything that trusts an extension; a re-mirror duplicates rather than
replaces; two T20 archives are genuinely corrupt; extraction triples the
disk footprint and filled the drive. Alongside them the analysis lessons
from NGC 5128, kept for the same reason - each is a way to be confidently
wrong.

DECISIONS.md records why the pipeline reads headers rather than
filenames, and why imaging is being made push-button while the science
stays hand-driven.

ARCHITECTURE.md described a documentation-only repository with no code,
which stopped being true when the pipeline merged in. It now covers all
three parts, the code-and-data separation the pipeline is built around,
and the on-disk session layout.

Both of this session's own errors are recorded in TODO.md as things not
to repeat: measuring a foreground star instead of the galaxy nucleus and
booking telescope time to fix it, and telling the user NGC 6744 had not
been processed when it had.

2026-07-21 17:32:00 +01:00

3.5 KiB

Raw Blame History

Architecture

How the system is built and why. Update this when the structure changes; a change is not finished until this reflects it.

Three parts, which is why the repository has three top-level content directories. Until 2026-07-21 the first two lived in separate repositories.

itelescope/ - the network and the campaign

Documentation and data, no code.

TELESCOPES.md: the deliverable. Per-observatory sections, one entry per telescope (spec bullets then a review paragraph), a "choosing a telescope" table, and general observations. Compiled from the sources below.
TARGETS.md: southern targets invisible from the UK, matched to scopes and to the time of year.
CAMPAIGN.md: the points drain campaign - ledger, bookings, queue, rate model. Updated after every session from the account's transaction history.
data/itelescope-telescopes.csv: verbatim CSV export of iTelescope's maintained specs sheet. The source of truth for numbers; re-exported to refresh (see state/NOTES.md).
plans/: ACP observing plans, one per booked run, uploaded to the scope.

pipeline/ - the processing code

Python 3.12. The design rule is that code and data are separate: nothing here knows an absolute path. A session is located at runtime through the ASTRO_SESSION environment variable.

layout.py maps a filename to the subdirectory it belongs in, using the same rules the session directories are organised with. A script asks for master-Red.fit or _stars.npz and gets the right path without any call site knowing the structure. Unrecognised names resolve to the session root, which is visible and correctable rather than silently wrong.
session.py is the entry point for anything generic. It discovers sessions, ingests and uncompresses them, reads every frame's header, groups frames by filter, and derives the colour scheme (LRGB / RGB / SHO / HOO / MONO) from what is actually present. Header-driven, never filename-driven - see DECISIONS.
restructure.py reorganises a flat session directory into the standard layout. Idempotent, dry run by default.
The remaining scripts are the NGC 5128 session as it was actually run: unzip -> analyse -> stack -> solve -> depth -> compose -> hdr/enhance/ starless/annotate -> final -> closeup -> triptych, plus the analysis families gc-* (globular clusters), sb_* (surface photometry) and mo_* (moving objects and transients). They are kept as provenance for published results and are being generalised stage by stage.

Session layout on disk (not in this repo)

<session>/
  METHODS.md            what was done to this data and what was found
  raw/                  exactly what the telescope delivered
  calibrated/           uncompressed calibrated subs
  stacks/masters/       per-filter registered, combined, plate-solved
  stacks/original/      alignment-only baseline, no other processing
  final/                the deliverable renderings
  renderings/           other finished images
  science/figures|catalogues|data|notes
  intermediates/        caches; deletable, regenerated by a re-run

observing/ - in-person plans

Field plans worked out from real numbers, revised as a date approaches. The reasoning is the part worth keeping, so plans record why a decision was made, not only what was decided.

The workflow itself

CLAUDE.md + docs/: the Default Workflow, copied in so the project is self-contained. state/: project memory - objective, current work, decisions, notes.

3.5 KiB Raw Blame History