laurence
5286a2e81b
The scripts that processed the NGC 5128 session of 2026-07-21 previously lived inside the data directory and addressed it with absolute paths. Code and data are now separated: the code lives here, and a session is located at runtime through the ASTRO_SESSION environment variable. layout.py is what makes that work. It maps a FILENAME to the subdirectory that file belongs in, using the same rules the session directories are organised with, so a script can go on asking for 'master-Red.fit' or '_stars.npz' without any call site knowing the directory structure. Anything unrecognised resolves to the session root, which is visible and correctable rather than silently wrong. restructure.py reorganises a flat session directory into that layout. It is idempotent and dry-run by default. The 50 session scripts are kept as they were run rather than tidied into a library. They were written in sequence as the work went along, several of them by parallel agents, and they show it - but they are the honest provenance of a published set of results, and the productionised pipeline should be able to reproduce those results exactly. Verified before committing: all 51 files compile without warnings, and verify_core.py, closeup.py and triptych.py were run end to end against the reorganised session, correctly finding inputs across calibrated/, stacks/masters/ and final/ and writing outputs back to the right places.
95 lines
4.2 KiB
Markdown
95 lines
4.2 KiB
Markdown
# astro-pipeline
|
|
|
|
Processing and analysis code for remote-telescope imaging sessions, starting
|
|
with iTelescope data from the [itelescope](https://git.discworld.casa/laurence/itelescope)
|
|
drain campaign.
|
|
|
|
The code lives here. **The data does not** - image sessions stay on disk (or
|
|
wherever they are archived) and are addressed by an environment variable, so a
|
|
session directory contains only pixels, results and a description of what was
|
|
done to them.
|
|
|
|
## What is here now
|
|
|
|
`session-scripts/` - the 50 scripts that processed the NGC 5128 session of
|
|
2026-07-21, exactly as they were run, plus the shared `layout.py` that tells
|
|
them where files live. This is a working record rather than a finished product:
|
|
the scripts were written in sequence as the work went along, several of them by
|
|
parallel agents, and they show it. They are kept because they are the honest
|
|
provenance of a set of published results, and because the productionised
|
|
pipeline should be able to reproduce those results exactly.
|
|
|
|
`session-scripts/restructure.py` - reorganises a flat session directory into the
|
|
named layout below. Idempotent, dry run by default.
|
|
|
|
## Pointing the scripts at a session
|
|
|
|
```
|
|
set ASTRO_SESSION=D:\astro\NGC5128\20260721 # Windows
|
|
export ASTRO_SESSION=/data/astro/NGC5128/20260721 # POSIX
|
|
python session-scripts/layout.py # prints the resolved layout
|
|
```
|
|
|
|
`layout.py` maps a **filename** to its subdirectory, so a script asks for
|
|
`master-Red.fit` or `_stars.npz` and gets the right path without knowing the
|
|
directory structure:
|
|
|
|
| Directory | Holds |
|
|
|---|---|
|
|
| `raw/` | exactly what the telescope delivered: archives and their preview jpegs |
|
|
| `calibrated/` | uncompressed calibrated subs |
|
|
| `stacks/masters/` | per-filter registered, plate-solved masters |
|
|
| `stacks/original/` | alignment-only baseline stacks, no other processing |
|
|
| `final/` | the deliverable renderings |
|
|
| `renderings/` | other finished images |
|
|
| `science/figures/` | analysis plots |
|
|
| `science/catalogues/` | measured tables (CSV) |
|
|
| `science/data/` | models, masks, derived quantities |
|
|
| `science/notes/` | analysis write-ups |
|
|
| `intermediates/` | caches a re-run can regenerate |
|
|
|
|
Every session directory also carries its own `METHODS.md` describing what was
|
|
done to that data and what was found - written for a reader who was not there.
|
|
|
|
## Running order
|
|
|
|
The scripts are named for their stage and run in this order:
|
|
|
|
```
|
|
unzip.py -> analyse.py -> stack.py -> solve.py -> depth.py
|
|
-> compose.py -> hdr.py / enhance.py / starless.py / annotate.py
|
|
-> final.py -> closeup.py -> triptych.py
|
|
```
|
|
|
|
The analysis families are independent of each other and of the renderings:
|
|
`gc-*` (globular clusters), `sb_*` (surface photometry), `mo_*` (moving objects
|
|
and transients).
|
|
|
|
## Requirements
|
|
|
|
Python 3.12 with numpy, scipy, astropy, scikit-image, sep, astroalign,
|
|
photutils, astroquery, matplotlib, tifffile, Pillow.
|
|
|
|
## Where this is going
|
|
|
|
The next piece of work is a scheduler-driven pipeline: a staged CLI
|
|
(`ingest -> calibrate -> measure -> register -> stack -> solve -> compose ->
|
|
analyse`) with each stage resumable, packaged as an Apptainer image and driven
|
|
by Slurm array jobs. Targets beyond mono LRGB: narrowband palettes, one-shot
|
|
colour with debayering, other observatories' header conventions, and full
|
|
calibration from bias/dark/flat for sources that do not pre-calibrate.
|
|
|
|
Three findings from the first session are requirements for that build, not
|
|
optional extras:
|
|
|
|
1. **Vet moving-object candidates in detector coordinates.** Registration holds
|
|
the sky still, so it drags detector-fixed defects across the frame on
|
|
perfectly straight, constant-rate tracks. Hot pixels are better-behaved
|
|
asteroids than real asteroids. This one cut took 141 confident spurious
|
|
detections to zero.
|
|
2. **Carry `r50/psf` through to any catalogue cross-match.** Comparing an
|
|
aperture magnitude of a resolved source against a point-source catalogue
|
|
like Gaia is meaningless, and looks exactly like a 2.8 magnitude outburst.
|
|
3. **Never fit a sky background to a field the target fills.** A plane fitted
|
|
around a large galaxy absorbs its halo - measured at -17.9 ADU/px here.
|
|
Fit the background and a source model together.
|