laurence
38f7a81395
run.py is the one command the whole exercise was aimed at:
python run.py --all PATH
It finds every session beneath a directory and takes each through
ingest, measure, register, solve, colour and report. Stages are skipped
when their output already exists, so an interrupted run continues rather
than restarting, and --force overrides that.
A failure in one stage of one session does not stop the others. In a
batch of twenty the useful outcome is nineteen results and one clear
error, not nothing - so failures are caught, recorded and summarised at
the end, and the plate solve in particular is allowed to fail without
taking the images down with it. An unsolved session still produces a
perfectly good picture; it just cannot produce positions.
report.py writes each session its own METHODS.md, generated from what
the stages actually returned rather than from what they were supposed to
return. That matters for the honest parts: a session whose plate solve
failed says so in its own documentation, and the limitations section is
derived from the data - too few frames for outlier rejection, an
integration too short to recover, a missing luminance channel, seeing
that caps the achievable detail. A hand-written note is written once and
then rots; this is rebuilt on every run.
|
||
|---|---|---|
| .. | ||
| analyse.py | ||
| annotate.py | ||
| astrometry.py | ||
| closeup.py | ||
| colour.py | ||
| compose.py | ||
| depth.py | ||
| enhance.py | ||
| final.py | ||
| gaia_colours.py | ||
| gc-bwtrial.py | ||
| gc-classify.py | ||
| gc-complete-inner.py | ||
| gc-complete.py | ||
| gc-detect.py | ||
| gc-fetch-vizier.py | ||
| gc-gaia-astrom2.py | ||
| gc-plots.py | ||
| gc-validate.py | ||
| hdr.py | ||
| layout.py | ||
| measure.py | ||
| mo_cavs.py | ||
| mo_check.py | ||
| mo_common.py | ||
| mo_detect.py | ||
| mo_fig_moving.py | ||
| mo_fig_transient.py | ||
| mo_finalise.py | ||
| mo_gccheck.py | ||
| mo_link.py | ||
| mo_mpc.py | ||
| mo_sensitivity.py | ||
| mo_shiftstack.py | ||
| mo_transient.py | ||
| mo_vet.py | ||
| original.py | ||
| README.md | ||
| register.py | ||
| rename.py | ||
| report.py | ||
| restructure.py | ||
| run.py | ||
| sb_common.py | ||
| sb_dust.py | ||
| sb_iso.py | ||
| sb_limits.py | ||
| sb_model.py | ||
| sb_prep.py | ||
| sb_profile.py | ||
| sb_render_tail.py | ||
| sb_residual.py | ||
| session.py | ||
| solve.py | ||
| stack.py | ||
| starless.py | ||
| subdir_readmes.py | ||
| triptych.py | ||
| unzip.py | ||
| verify_core.py | ||
pipeline
Processing and analysis code for remote-telescope imaging sessions, starting with iTelescope data from the itelescope drain campaign.
The code lives here. The data does not - image sessions stay on disk (or wherever they are archived) and are addressed by an environment variable, so a session directory contains only pixels, results and a description of what was done to them.
What is here now
pipeline/ - the 50 scripts that processed the NGC 5128 session of
2026-07-21, exactly as they were run, plus the shared layout.py that tells
them where files live. This is a working record rather than a finished product:
the scripts were written in sequence as the work went along, several of them by
parallel agents, and they show it. They are kept because they are the honest
provenance of a set of published results, and because the productionised
pipeline should be able to reproduce those results exactly.
pipeline/restructure.py - reorganises a flat session directory into the
named layout below. Idempotent, dry run by default.
Pointing the scripts at a session
set ASTRO_SESSION=D:\astro\NGC5128\20260721 # Windows
export ASTRO_SESSION=/data/astro/NGC5128/20260721 # POSIX
python pipeline/layout.py # prints the resolved layout
layout.py maps a filename to its subdirectory, so a script asks for
master-Red.fit or _stars.npz and gets the right path without knowing the
directory structure:
| Directory | Holds |
|---|---|
raw/ |
exactly what the telescope delivered: archives and their preview jpegs |
calibrated/ |
uncompressed calibrated subs |
stacks/masters/ |
per-filter registered, plate-solved masters |
stacks/original/ |
alignment-only baseline stacks, no other processing |
final/ |
the deliverable renderings |
renderings/ |
other finished images |
science/figures/ |
analysis plots |
science/catalogues/ |
measured tables (CSV) |
science/data/ |
models, masks, derived quantities |
science/notes/ |
analysis write-ups |
intermediates/ |
caches a re-run can regenerate |
Every session directory also carries its own METHODS.md describing what was
done to that data and what was found - written for a reader who was not there.
Running order
The scripts are named for their stage and run in this order:
unzip.py -> analyse.py -> stack.py -> solve.py -> depth.py
-> compose.py -> hdr.py / enhance.py / starless.py / annotate.py
-> final.py -> closeup.py -> triptych.py
The analysis families are independent of each other and of the renderings:
gc-* (globular clusters), sb_* (surface photometry), mo_* (moving objects
and transients).
Requirements
Python 3.12 with numpy, scipy, astropy, scikit-image, sep, astroalign, photutils, astroquery, matplotlib, tifffile, Pillow.
Where this is going
The next piece of work is a scheduler-driven pipeline: a staged CLI
(ingest -> calibrate -> measure -> register -> stack -> solve -> compose -> analyse) with each stage resumable, packaged as an Apptainer image and driven
by Slurm array jobs. Targets beyond mono LRGB: narrowband palettes, one-shot
colour with debayering, other observatories' header conventions, and full
calibration from bias/dark/flat for sources that do not pre-calibrate.
Three findings from the first session are requirements for that build, not optional extras:
- Vet moving-object candidates in detector coordinates. Registration holds the sky still, so it drags detector-fixed defects across the frame on perfectly straight, constant-rate tracks. Hot pixels are better-behaved asteroids than real asteroids. This one cut took 141 confident spurious detections to zero.
- Carry
r50/psfthrough to any catalogue cross-match. Comparing an aperture magnitude of a resolved source against a point-source catalogue like Gaia is meaningless, and looks exactly like a 2.8 magnitude outburst. - Never fit a sky background to a field the target fills. A plane fitted around a large galaxy absorbs its halo - measured at -17.9 ADU/px here. Fit the background and a source model together.