laurence
59d6e30712
The mechanical merge preserved both histories but left two repositories sitting side by side rather than one repository. This is the part that makes it whole. A top-level README explains the three strands - the telescope network and campaign, the processing code, and in-person observing plans - and says plainly that image data does not live here, because that is the first question anyone will have when they find a pipeline with no pixels. state/PROJECT.md was still describing a markdown-only reference project whose scope explicitly EXCLUDED automating bookings and image processing. Both of those are now most of what the project does, so the objective, scope and key facts are rewritten to match reality. state/DECISIONS.md records why the merge happened - the two repos were split by chronology rather than design, and the seam was already leaking, with the campaign's TODO citing results held in the other repo and the pipeline's README explaining a campaign it did not contain. It also records that the image data deliberately stays out of the repository. state/TODO.md gains the pipeline and observing work that previously had nowhere to live, including the measurement that changes the pipeline plan: per-frame processing is 2.9 seconds and the whole 24-frame session is 1.1 minutes single-threaded, so a scheduler earns nothing on stacking and should be pointed at the Monte Carlo stages instead. Internal links fixed for the new paths: pipeline/README.md referred to session-scripts/ throughout, and itelescope/README.md pointed at a state/ directory that is now one level up. The .gitignore conflict between the two repositories is resolved by combining them, with image formats ignored globally except under docs/, where documentation figures are committed deliberately.
95 lines
4.1 KiB
Markdown
95 lines
4.1 KiB
Markdown
# pipeline
|
|
|
|
Processing and analysis code for remote-telescope imaging sessions, starting
|
|
with iTelescope data from the [itelescope](https://git.discworld.casa/laurence/itelescope)
|
|
drain campaign.
|
|
|
|
The code lives here. **The data does not** - image sessions stay on disk (or
|
|
wherever they are archived) and are addressed by an environment variable, so a
|
|
session directory contains only pixels, results and a description of what was
|
|
done to them.
|
|
|
|
## What is here now
|
|
|
|
`pipeline/` - the 50 scripts that processed the NGC 5128 session of
|
|
2026-07-21, exactly as they were run, plus the shared `layout.py` that tells
|
|
them where files live. This is a working record rather than a finished product:
|
|
the scripts were written in sequence as the work went along, several of them by
|
|
parallel agents, and they show it. They are kept because they are the honest
|
|
provenance of a set of published results, and because the productionised
|
|
pipeline should be able to reproduce those results exactly.
|
|
|
|
`pipeline/restructure.py` - reorganises a flat session directory into the
|
|
named layout below. Idempotent, dry run by default.
|
|
|
|
## Pointing the scripts at a session
|
|
|
|
```
|
|
set ASTRO_SESSION=D:\astro\NGC5128\20260721 # Windows
|
|
export ASTRO_SESSION=/data/astro/NGC5128/20260721 # POSIX
|
|
python pipeline/layout.py # prints the resolved layout
|
|
```
|
|
|
|
`layout.py` maps a **filename** to its subdirectory, so a script asks for
|
|
`master-Red.fit` or `_stars.npz` and gets the right path without knowing the
|
|
directory structure:
|
|
|
|
| Directory | Holds |
|
|
|---|---|
|
|
| `raw/` | exactly what the telescope delivered: archives and their preview jpegs |
|
|
| `calibrated/` | uncompressed calibrated subs |
|
|
| `stacks/masters/` | per-filter registered, plate-solved masters |
|
|
| `stacks/original/` | alignment-only baseline stacks, no other processing |
|
|
| `final/` | the deliverable renderings |
|
|
| `renderings/` | other finished images |
|
|
| `science/figures/` | analysis plots |
|
|
| `science/catalogues/` | measured tables (CSV) |
|
|
| `science/data/` | models, masks, derived quantities |
|
|
| `science/notes/` | analysis write-ups |
|
|
| `intermediates/` | caches a re-run can regenerate |
|
|
|
|
Every session directory also carries its own `METHODS.md` describing what was
|
|
done to that data and what was found - written for a reader who was not there.
|
|
|
|
## Running order
|
|
|
|
The scripts are named for their stage and run in this order:
|
|
|
|
```
|
|
unzip.py -> analyse.py -> stack.py -> solve.py -> depth.py
|
|
-> compose.py -> hdr.py / enhance.py / starless.py / annotate.py
|
|
-> final.py -> closeup.py -> triptych.py
|
|
```
|
|
|
|
The analysis families are independent of each other and of the renderings:
|
|
`gc-*` (globular clusters), `sb_*` (surface photometry), `mo_*` (moving objects
|
|
and transients).
|
|
|
|
## Requirements
|
|
|
|
Python 3.12 with numpy, scipy, astropy, scikit-image, sep, astroalign,
|
|
photutils, astroquery, matplotlib, tifffile, Pillow.
|
|
|
|
## Where this is going
|
|
|
|
The next piece of work is a scheduler-driven pipeline: a staged CLI
|
|
(`ingest -> calibrate -> measure -> register -> stack -> solve -> compose ->
|
|
analyse`) with each stage resumable, packaged as an Apptainer image and driven
|
|
by Slurm array jobs. Targets beyond mono LRGB: narrowband palettes, one-shot
|
|
colour with debayering, other observatories' header conventions, and full
|
|
calibration from bias/dark/flat for sources that do not pre-calibrate.
|
|
|
|
Three findings from the first session are requirements for that build, not
|
|
optional extras:
|
|
|
|
1. **Vet moving-object candidates in detector coordinates.** Registration holds
|
|
the sky still, so it drags detector-fixed defects across the frame on
|
|
perfectly straight, constant-rate tracks. Hot pixels are better-behaved
|
|
asteroids than real asteroids. This one cut took 141 confident spurious
|
|
detections to zero.
|
|
2. **Carry `r50/psf` through to any catalogue cross-match.** Comparing an
|
|
aperture magnitude of a resolved source against a point-source catalogue
|
|
like Gaia is meaningless, and looks exactly like a 2.8 magnitude outburst.
|
|
3. **Never fit a sky background to a field the target fills.** A plane fitted
|
|
around a large galaxy absorbs its halo - measured at -17.9 ADU/px here.
|
|
Fit the background and a source model together.
|