laurence/astrophotography
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Generic pipeline stage 1, and state files a cold session can resume from #1

Merged
laurence merged 2 commits from generic-pipeline into main 2026-07-21 17:32:28 +01:00
Conversation 0 Commits 2 Files changed 5 +656 -106
laurence commented 2026-07-21 17:32:27 +01:00
Owner
Copy link

Two things: stage 1 of the generic pipeline, and resumable state

Stage 1 - session discovery and ingest

The existing scripts parse filenames, expecting calibrated-T32-...-Luminance-BIN2-W-300-001.fit. That works for one telescope, one binning and one exposure length. Three of the four archived sessions have no luminance channel at all, so the approach could not survive contact with the archive.

pipeline/session.py reads headers instead — FILTER, EXPTIME, XBINNING, OBJECT, DATE-OBS — groups frames by filter, and derives the colour scheme from what is present: LRGB, RGB without luminance, SHO, HOO or mono.

Three defects were found by running it against the real archive rather than by reasoning about it:

Problem Symptom Fix
Every frame delivered twice (calibrated- and raw-) NGC 5128 scanned as 24 luminance frames when it has 12 Extract calibrated archives only; CALSTAT as a header-level safety net
Interrupted segmented download leaves directories named like files 28 directories called something.fit; anything trusting an extension opens a directory as a frame Skipped explicitly at discovery, ingest and scan
A re-mirror duplicates rather than replaces ~1 GB restored to session roots already filed into raw/ Byte-identical copies removed as they are refiled

Two T20 archives are genuinely corrupt and cannot be extracted — real data loss, not a bug. That session has 14 of its 16 frames, and the warning is printed rather than swallowed.

NGC 5128 now scans as exactly 12/4/4/4, matching the session processed by hand, which makes it a usable regression test for everything that follows.

Resumable state

A long session had left most of the current position in the chat rather than the repo.

  • TODO.md gains a RESUME HERE block: the three live strands, the three decisions waiting on the user, and the one thing that will break the batch run if forgotten — NGC 6744 mixes bin1 luminance at 4096² with bin2 colour at 2048², so registration must resample onto a common grid. Nothing else in the archive does that, which is precisely why it will be forgotten.
  • NOTES.md gains the gotchas, all of which produced plausible wrong answers rather than errors, plus the analysis lessons from NGC 5128.
  • DECISIONS.md records why the pipeline is header-driven, and why imaging is push-button while the science stays hand-driven.
  • ARCHITECTURE.md described a documentation-only repository with no code, which stopped being true at the merge.

Both of this session's own errors are recorded as things not to repeat: measuring a foreground star instead of the galaxy nucleus and booking telescope time to fix it, and telling the user NGC 6744 had not been processed when it had.

Not done

Batch processing is blocked on disk space — C: has ~2.6 GB free against a 14 GB archive. Recorded in TODO rather than worked around.

## Two things: stage 1 of the generic pipeline, and resumable state ### Stage 1 - session discovery and ingest The existing scripts parse filenames, expecting `calibrated-T32-...-Luminance-BIN2-W-300-001.fit`. That works for one telescope, one binning and one exposure length. **Three of the four archived sessions have no luminance channel at all**, so the approach could not survive contact with the archive. `pipeline/session.py` reads headers instead — FILTER, EXPTIME, XBINNING, OBJECT, DATE-OBS — groups frames by filter, and derives the colour scheme from what is present: LRGB, RGB without luminance, SHO, HOO or mono. **Three defects were found by running it against the real archive rather than by reasoning about it:** | Problem | Symptom | Fix | |---|---|---| | Every frame delivered twice (`calibrated-` and `raw-`) | NGC 5128 scanned as **24** luminance frames when it has 12 | Extract calibrated archives only; `CALSTAT` as a header-level safety net | | Interrupted segmented download leaves **directories named like files** | 28 directories called `something.fit`; anything trusting an extension opens a directory as a frame | Skipped explicitly at discovery, ingest and scan | | A re-mirror duplicates rather than replaces | ~1 GB restored to session roots already filed into `raw/` | Byte-identical copies removed as they are refiled | Two T20 archives are genuinely corrupt and cannot be extracted — real data loss, not a bug. That session has 14 of its 16 frames, and the warning is printed rather than swallowed. **NGC 5128 now scans as exactly 12/4/4/4, matching the session processed by hand**, which makes it a usable regression test for everything that follows. ### Resumable state A long session had left most of the current position in the chat rather than the repo. - **TODO.md** gains a `RESUME HERE` block: the three live strands, the three decisions waiting on the user, and the one thing that will break the batch run if forgotten — NGC 6744 mixes bin1 luminance at 4096² with bin2 colour at 2048², so registration must resample onto a common grid. Nothing else in the archive does that, which is precisely why it will be forgotten. - **NOTES.md** gains the gotchas, all of which produced *plausible wrong answers* rather than errors, plus the analysis lessons from NGC 5128. - **DECISIONS.md** records why the pipeline is header-driven, and why imaging is push-button while the science stays hand-driven. - **ARCHITECTURE.md** described a documentation-only repository with no code, which stopped being true at the merge. Both of this session's own errors are recorded as things not to repeat: measuring a foreground star instead of the galaxy nucleus and booking telescope time to fix it, and telling the user NGC 6744 had not been processed when it had. ### Not done Batch processing is **blocked on disk space** — C: has ~2.6 GB free against a 14 GB archive. Recorded in TODO rather than worked around.
laurence added 2 commits 2026-07-21 17:32:27 +01:00 
First stage of the generic pipeline. The existing scripts parse
filenames, expecting calibrated-T32-...-Luminance-BIN2-W-300-001.fit,
which works for exactly one telescope, one binning and one exposure
length. Three of the four archived sessions have no luminance channel at
all, so that approach cannot process them.

session.py opens each frame and reads FILTER, EXPTIME, XBINNING, OBJECT
and DATE-OBS instead. It groups frames by filter, works out which colour
scheme the session supports (LRGB, RGB, SHO, HOO or mono), and writes a
manifest. Filter aliases are mapped, so a header saying H-alpha, Halpha
or HA all resolve to Ha.

Three real defects were found by running it against the archive rather
than by reasoning about it:

Frame counts came out doubled. iTelescope ships every frame twice, as
'calibrated-*' with bias, dark and flat applied and 'raw-*' without, and
extracting both put two copies of each frame into the stack. Only the
calibrated archives are extracted now, with CALSTAT used as a header
level safety net for sessions that were already extracted the old way.
NGC 5128 now scans as 12/4/4/4, exactly matching the session processed
by hand, which makes it a usable regression test.

A re-mirror over the organised tree produced duplicates and mirror
damage. An interrupted segmented download leaves DIRECTORIES named after
the file being fetched - a directory called something.fit - plus
.cyberducksegment parts. Anything that trusts a file extension will try
to open a directory as a frame. Those are now skipped explicitly at
discovery, ingest and scan, and byte-identical copies that a re-mirror
puts back are removed as they are refiled.

Two T20 archives are corrupt and cannot be extracted. That is real data
loss, not a bug: the session has 14 of its 16 frames and the warning is
printed rather than swallowed.

Not yet done: NGC 6744 mixes bin1 luminance at 4096x4096 with bin2
colour at 2048x2048, so registration will need to resample onto a common
grid rather than assuming every frame shares a shape.
Rule 2 of the workflow is that state lives in the repo rather than the
chat, and a long session had put most of the current position only in
the chat. This writes it down.

TODO.md gains a RESUME HERE block at the top, because the useful
question for the next session is not what has been done but what to do
next and what is blocking it. It names the three live strands, the three
decisions waiting on the user, and the one thing that will break the
batch run if it is forgotten: NGC 6744 mixes bin1 luminance at 4096x4096
with bin2 colour at 2048x2048, so registration has to resample onto a
common grid. Nothing else in the archive does that, which is exactly why
it will be forgotten. It also drops three duplicate "Done (recent)"
headings that had accumulated, and tabulates what the five archived
sessions actually contain.

NOTES.md gains the gotchas, all of which produced plausible wrong
answers rather than errors: every frame is delivered twice as calibrated
and raw, so extracting both silently doubles the stack; an interrupted
segmented download leaves DIRECTORIES named like files, which defeats
anything that trusts an extension; a re-mirror duplicates rather than
replaces; two T20 archives are genuinely corrupt; extraction triples the
disk footprint and filled the drive. Alongside them the analysis lessons
from NGC 5128, kept for the same reason - each is a way to be confidently
wrong.

DECISIONS.md records why the pipeline reads headers rather than
filenames, and why imaging is being made push-button while the science
stays hand-driven.

ARCHITECTURE.md described a documentation-only repository with no code,
which stopped being true when the pipeline merged in. It now covers all
three parts, the code-and-data separation the pipeline is built around,
and the on-disk session layout.

Both of this session's own errors are recorded in TODO.md as things not
to repeat: measuring a foreground star instead of the galaxy nucleus and
booking telescope time to fix it, and telling the user NGC 6744 had not
been processed when it had.
laurence merged commit f32411970f into main 2026-07-21 17:32:28 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: laurence/astrophotography#1
No description provided.
Delete branch "generic-pipeline"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?