astrometry.py solves the deepest master against Gaia and copies the WCS
into every master. Asterism matching is invariant to rotation and scale,
so camera angle never has to be guessed, but not to a mirror flip, so
both parities are tried.
Three defects were found and fixed by running it, and each is worth
recording because none would have been found by reading the code.
Gaia's launch_job_async submits to a job QUEUE and polls. On a query of
600 rows it hung for ten minutes while curl showed both the Gaia and
VizieR endpoints answering in under a second. VizieR is now queried
first - it serves the identical DR3 catalogue over a plain HTTP request -
with Gaia's SYNCHRONOUS endpoint as the fallback.
VizieR's row_limit truncates before sorting, so asking for the 600
brightest stars returned 600 arbitrary ones. Asterism matching only works
when both lists hold the same bright stars, so this silently produced no
match at all. It now fetches generously and picks the brightest locally.
The detector deliberately rejects saturated cores, which means the
brightest DETECTIONS are not the brightest STARS, while the catalogue's
are - so the two top-N lists can barely overlap. A sliding window down
the catalogue's magnitude ranking fixed NGC 2030, which matched at
catalogue[15:135], skipping the 15 brightest. That is the saturation
hypothesis confirmed rather than assumed.
NGC 2030 now solves on 243 stars at 0.31 arcsec residual. NGC 2070 and
NGC 6744 still do not, and TODO.md records what has already been ruled
out - scale, the fetch, truncation - so the next session starts from the
remaining candidates rather than repeating the elimination.
colour.py assembles a viewable image from whatever a session actually
has: LRGB, RGB with a synthetic luminance, SHO in the Hubble palette,
HOO, or a single filter as greyscale. Nothing is assumed about which
filters exist, which is the whole point - three of the four archived
sessions have no luminance channel.
The background fit takes the lesson from Centaurus A, where a plane
fitted around a large galaxy absorbed 17.9 ADU/px of its halo. The
excluded region is now sized from the data: the ellipse grows until it
contains most of the flux above sky, so it suits a galaxy filling the
frame and a small nebula equally, and only a plane is ever fitted, never
a flexible surface. One mask is derived from the deepest channel and
applied to all of them, so the fit cannot shift the colour balance.
Broadband and narrowband need opposite stretches, which cost a round to
discover. Broadband channels share the luminance's midtone, preserving
the real brightness ratios that keep star colours honest. Narrowband
cannot: the lines differ enormously in strength, so a shared stretch
renders the entire nebula in whichever colour Ha was mapped to. Each
narrowband channel is now stretched to its own sky target instead.
Green suppression is applied to broadband only. On a narrowband palette
it would fight the palette, green being a deliberate channel assignment
rather than an artefact.
Measured rather than eyeballed, because the SHO render looked wrong by
eye and was not: both palettes put the sky at 0.098 against a 0.10
target and neutral to within 0.004. What differs is the data. NGC 2030's
brightest pixels reach 0.80 to 0.87, the Tarantula's only 0.23 to 0.33,
because that session is a single 60 second frame per filter - three
minutes in total. A correct pipeline renders nearly empty data as a
nearly empty frame.
measure.py produces the numbers used to choose a registration reference
and to weight the stack - sky, noise, seeing - and caches each frame's
star list, because registration and the plate solve both need it and
detection costs far more than reading a small array back. Frames are
opened one at a time; a 4096x4096 float32 frame is 67 MB and a session
holds ninety of them.
register.py aligns everything onto a single reference and combines per
filter. One reference for ALL filters, not one per filter, which is what
makes the masters pixel-aligned so the colour composite needs no further
registration. The reference is the sharpest frame of the filter with the
most signal, because the reference sets the output grid and a poor choice
costs resolution everywhere, permanently.
The case worth the care is NGC 6744: luminance at bin1 4096x4096 and
colour at bin2 2048x2048, so frames share neither shape nor pixel scale.
Asterism matching already returns a similarity transform including scale,
so the maths was never the problem; the trap is assuming the warp output
is the same shape as its input, which would write a bin2 frame onto a
bin2 grid that silently fails to line up with a bin1 reference. Every
warp is now given the reference shape explicitly.
Verified rather than assumed: after stacking, the upsampled bin2 colour
masters align to the bin1 luminance master to within 0.04 pixels, across
about 245 matched stars per channel.
Sigma clipping is skipped below three frames, where there is nothing to
reject against and clipping would only discard signal - which matters
because one session has a single frame per filter.
Tested on three sessions covering the awkward shapes: RGB without
luminance, SHO with one frame per filter, and the mixed bin1/bin2 LRGB.
First stage of the generic pipeline. The existing scripts parse
filenames, expecting calibrated-T32-...-Luminance-BIN2-W-300-001.fit,
which works for exactly one telescope, one binning and one exposure
length. Three of the four archived sessions have no luminance channel at
all, so that approach cannot process them.
session.py opens each frame and reads FILTER, EXPTIME, XBINNING, OBJECT
and DATE-OBS instead. It groups frames by filter, works out which colour
scheme the session supports (LRGB, RGB, SHO, HOO or mono), and writes a
manifest. Filter aliases are mapped, so a header saying H-alpha, Halpha
or HA all resolve to Ha.
Three real defects were found by running it against the archive rather
than by reasoning about it:
Frame counts came out doubled. iTelescope ships every frame twice, as
'calibrated-*' with bias, dark and flat applied and 'raw-*' without, and
extracting both put two copies of each frame into the stack. Only the
calibrated archives are extracted now, with CALSTAT used as a header
level safety net for sessions that were already extracted the old way.
NGC 5128 now scans as 12/4/4/4, exactly matching the session processed
by hand, which makes it a usable regression test.
A re-mirror over the organised tree produced duplicates and mirror
damage. An interrupted segmented download leaves DIRECTORIES named after
the file being fetched - a directory called something.fit - plus
.cyberducksegment parts. Anything that trusts a file extension will try
to open a directory as a frame. Those are now skipped explicitly at
discovery, ingest and scan, and byte-identical copies that a re-mirror
puts back are removed as they are refiled.
Two T20 archives are corrupt and cannot be extracted. That is real data
loss, not a bug: the session has 14 of its 16 frames and the warning is
printed rather than swallowed.
Not yet done: NGC 6744 mixes bin1 luminance at 4096x4096 with bin2
colour at 2048x2048, so registration will need to resample onto a common
grid rather than assuming every frame shares a shape.
The mechanical merge preserved both histories but left two repositories
sitting side by side rather than one repository. This is the part that
makes it whole.
A top-level README explains the three strands - the telescope network
and campaign, the processing code, and in-person observing plans - and
says plainly that image data does not live here, because that is the
first question anyone will have when they find a pipeline with no
pixels.
state/PROJECT.md was still describing a markdown-only reference project
whose scope explicitly EXCLUDED automating bookings and image
processing. Both of those are now most of what the project does, so the
objective, scope and key facts are rewritten to match reality.
state/DECISIONS.md records why the merge happened - the two repos were
split by chronology rather than design, and the seam was already
leaking, with the campaign's TODO citing results held in the other repo
and the pipeline's README explaining a campaign it did not contain. It
also records that the image data deliberately stays out of the
repository.
state/TODO.md gains the pipeline and observing work that previously had
nowhere to live, including the measurement that changes the pipeline
plan: per-frame processing is 2.9 seconds and the whole 24-frame session
is 1.1 minutes single-threaded, so a scheduler earns nothing on stacking
and should be pointed at the Monte Carlo stages instead.
Internal links fixed for the new paths: pipeline/README.md referred to
session-scripts/ throughout, and itelescope/README.md pointed at a
state/ directory that is now one level up.
The .gitignore conflict between the two repositories is resolved by
combining them, with image formats ignored globally except under docs/,
where documentation figures are committed deliberately.
Preparing to merge this repository into a combined astrophotography
repo. session-scripts/ becomes pipeline/ because the scripts import
layout.py from their own directory and must stay together, and because
'pipeline' says what it is rather than how it came about. observing/
stays at the top level: observing plans are not processing code.
