Three fixes and one refusal, all found by working on the wide-field
narrowband session that would not solve.
The detection floor was a fixed 12 pixels, which quietly assumed a well
sampled star. At 0.53 arcsec/px with 5.9 px seeing that is right; at 3.5
arcsec/px the stars are undersampled at 1.75 px FWHM and cover a handful
of pixels each, so the floor discarded nearly every real star and kept
blends and galaxies instead. It now scales with the measured seeing, and
that session went from 305 usable detections to 600.
Matching now happens over the frame's inscribed circle rather than a cone
reaching its corners. On a 4 degree field the old cone covered 33 square
degrees of sky against 16 of image, so half the catalogue was not in the
picture at all.
Where the header records a roll angle - and iTelescope's does - the
orientation no longer has to be recovered from scratch. Rotation, scale
and parity are applied directly and only the residual pointing error is
searched, by histogramming every detection-to-catalogue offset and taking
the peak. Asterism matching remains as the fallback for headers that say
nothing about orientation.
The refusal matters most. With catalogue depth slices added, the wide
field produced a "solution" of 25 stars at 2.50 px residual, claiming a
centre half a degree from the pointing. A genuine solve on that field
matches hundreds of stars and refines to well under a pixel. Accepting it
would have silently corrupted every position in the science catalogue, so
a solution must now reach 40 stars AND 1.5 px or it is discarded and the
session is reported as unsolved. A wrong WCS is worse than no WCS.
NGC 2030 improves from 0.31 to 0.12 arcsec on 123 stars under the
pointing-based match. NGC 2070 is honestly unsolved.
The four-image set worked out on Centaurus A is now generated for any
target, because its value is the comparison: the same data at four levels
of treatment, so a viewer can see what processing did and did not add.
colour.py's assembly is driven by flags - gradient, neutralise, denoise,
saturation, hdr, protect-compact - so the baseline and the fully
corrected version come from ONE code path and the only differences
between them are the ones named.
Image 3 applies what the measurements justify rather than a house style.
Each item is there because measuring the first session caught the
conventional version getting something wrong: a plane fit that had
absorbed 17.9 ADU/px of galaxy halo, a core flattened by a white point
set by field stars, deconvolution ringing around every bright star, and
denoising erasing faint compact sources that turned out to be globular
clusters. Compact sources are now explicitly protected from smoothing -
2374 of them on NGC 2030.
The close-up revealed a real design error, caught by its own assertion.
Forcing a square crop cannot contain a target wider than the frame is
tall, which is the normal case for a nebula in a wide field, and the
assertion fired rather than silently cutting the subject in half. Crops
are no longer square, and when a target genuinely fills the field the
close-up is skipped with that said plainly - re-saving image 3 under a
name claiming to be a close-up would be worse than producing nothing.
science.py adds the measurements that generalise to any target:
photometric calibration from the field's own Gaia stars, the limiting
magnitude actually reached, a source catalogue with calibrated
magnitudes, an annotated field placed by the plate solution, and a radial
surface-brightness profile. Object-specific analyses stay hand-driven,
because a cluster survey suits a galaxy and is meaningless for a nebula.
All of it depends on astrometry, so an unsolved session gets no science
and says so instead of quietly producing less. NGC 2030 calibrates to a
zero point of 24.794 with 0.202 mag scatter on 917 stars, 3470 sources,
limiting G of 18.2.
run.py is the one command the whole exercise was aimed at:
python run.py --all PATH
It finds every session beneath a directory and takes each through
ingest, measure, register, solve, colour and report. Stages are skipped
when their output already exists, so an interrupted run continues rather
than restarting, and --force overrides that.
A failure in one stage of one session does not stop the others. In a
batch of twenty the useful outcome is nineteen results and one clear
error, not nothing - so failures are caught, recorded and summarised at
the end, and the plate solve in particular is allowed to fail without
taking the images down with it. An unsolved session still produces a
perfectly good picture; it just cannot produce positions.
report.py writes each session its own METHODS.md, generated from what
the stages actually returned rather than from what they were supposed to
return. That matters for the honest parts: a session whose plate solve
failed says so in its own documentation, and the limitations section is
derived from the data - too few frames for outlier rejection, an
integration too short to recover, a missing luminance channel, seeing
that caps the achievable detail. A hand-written note is written once and
then rots; this is rebuilt on every run.
astrometry.py solves the deepest master against Gaia and copies the WCS
into every master. Asterism matching is invariant to rotation and scale,
so camera angle never has to be guessed, but not to a mirror flip, so
both parities are tried.
Three defects were found and fixed by running it, and each is worth
recording because none would have been found by reading the code.
Gaia's launch_job_async submits to a job QUEUE and polls. On a query of
600 rows it hung for ten minutes while curl showed both the Gaia and
VizieR endpoints answering in under a second. VizieR is now queried
first - it serves the identical DR3 catalogue over a plain HTTP request -
with Gaia's SYNCHRONOUS endpoint as the fallback.
VizieR's row_limit truncates before sorting, so asking for the 600
brightest stars returned 600 arbitrary ones. Asterism matching only works
when both lists hold the same bright stars, so this silently produced no
match at all. It now fetches generously and picks the brightest locally.
The detector deliberately rejects saturated cores, which means the
brightest DETECTIONS are not the brightest STARS, while the catalogue's
are - so the two top-N lists can barely overlap. A sliding window down
the catalogue's magnitude ranking fixed NGC 2030, which matched at
catalogue[15:135], skipping the 15 brightest. That is the saturation
hypothesis confirmed rather than assumed.
NGC 2030 now solves on 243 stars at 0.31 arcsec residual. NGC 2070 and
NGC 6744 still do not, and TODO.md records what has already been ruled
out - scale, the fetch, truncation - so the next session starts from the
remaining candidates rather than repeating the elimination.
colour.py assembles a viewable image from whatever a session actually
has: LRGB, RGB with a synthetic luminance, SHO in the Hubble palette,
HOO, or a single filter as greyscale. Nothing is assumed about which
filters exist, which is the whole point - three of the four archived
sessions have no luminance channel.
The background fit takes the lesson from Centaurus A, where a plane
fitted around a large galaxy absorbed 17.9 ADU/px of its halo. The
excluded region is now sized from the data: the ellipse grows until it
contains most of the flux above sky, so it suits a galaxy filling the
frame and a small nebula equally, and only a plane is ever fitted, never
a flexible surface. One mask is derived from the deepest channel and
applied to all of them, so the fit cannot shift the colour balance.
Broadband and narrowband need opposite stretches, which cost a round to
discover. Broadband channels share the luminance's midtone, preserving
the real brightness ratios that keep star colours honest. Narrowband
cannot: the lines differ enormously in strength, so a shared stretch
renders the entire nebula in whichever colour Ha was mapped to. Each
narrowband channel is now stretched to its own sky target instead.
Green suppression is applied to broadband only. On a narrowband palette
it would fight the palette, green being a deliberate channel assignment
rather than an artefact.
Measured rather than eyeballed, because the SHO render looked wrong by
eye and was not: both palettes put the sky at 0.098 against a 0.10
target and neutral to within 0.004. What differs is the data. NGC 2030's
brightest pixels reach 0.80 to 0.87, the Tarantula's only 0.23 to 0.33,
because that session is a single 60 second frame per filter - three
minutes in total. A correct pipeline renders nearly empty data as a
nearly empty frame.
measure.py produces the numbers used to choose a registration reference
and to weight the stack - sky, noise, seeing - and caches each frame's
star list, because registration and the plate solve both need it and
detection costs far more than reading a small array back. Frames are
opened one at a time; a 4096x4096 float32 frame is 67 MB and a session
holds ninety of them.
register.py aligns everything onto a single reference and combines per
filter. One reference for ALL filters, not one per filter, which is what
makes the masters pixel-aligned so the colour composite needs no further
registration. The reference is the sharpest frame of the filter with the
most signal, because the reference sets the output grid and a poor choice
costs resolution everywhere, permanently.
The case worth the care is NGC 6744: luminance at bin1 4096x4096 and
colour at bin2 2048x2048, so frames share neither shape nor pixel scale.
Asterism matching already returns a similarity transform including scale,
so the maths was never the problem; the trap is assuming the warp output
is the same shape as its input, which would write a bin2 frame onto a
bin2 grid that silently fails to line up with a bin1 reference. Every
warp is now given the reference shape explicitly.
Verified rather than assumed: after stacking, the upsampled bin2 colour
masters align to the bin1 luminance master to within 0.04 pixels, across
about 245 matched stars per channel.
Sigma clipping is skipped below three frames, where there is nothing to
reject against and clipping would only discard signal - which matters
because one session has a single frame per filter.
Tested on three sessions covering the awkward shapes: RGB without
luminance, SHO with one frame per filter, and the mixed bin1/bin2 LRGB.
First stage of the generic pipeline. The existing scripts parse
filenames, expecting calibrated-T32-...-Luminance-BIN2-W-300-001.fit,
which works for exactly one telescope, one binning and one exposure
length. Three of the four archived sessions have no luminance channel at
all, so that approach cannot process them.
session.py opens each frame and reads FILTER, EXPTIME, XBINNING, OBJECT
and DATE-OBS instead. It groups frames by filter, works out which colour
scheme the session supports (LRGB, RGB, SHO, HOO or mono), and writes a
manifest. Filter aliases are mapped, so a header saying H-alpha, Halpha
or HA all resolve to Ha.
Three real defects were found by running it against the archive rather
than by reasoning about it:
Frame counts came out doubled. iTelescope ships every frame twice, as
'calibrated-*' with bias, dark and flat applied and 'raw-*' without, and
extracting both put two copies of each frame into the stack. Only the
calibrated archives are extracted now, with CALSTAT used as a header
level safety net for sessions that were already extracted the old way.
NGC 5128 now scans as 12/4/4/4, exactly matching the session processed
by hand, which makes it a usable regression test.
A re-mirror over the organised tree produced duplicates and mirror
damage. An interrupted segmented download leaves DIRECTORIES named after
the file being fetched - a directory called something.fit - plus
.cyberducksegment parts. Anything that trusts a file extension will try
to open a directory as a frame. Those are now skipped explicitly at
discovery, ingest and scan, and byte-identical copies that a re-mirror
puts back are removed as they are refiled.
Two T20 archives are corrupt and cannot be extracted. That is real data
loss, not a bug: the session has 14 of its 16 frames and the warning is
printed rather than swallowed.
Not yet done: NGC 6744 mixes bin1 luminance at 4096x4096 with bin2
colour at 2048x2048, so registration will need to resample onto a common
grid rather than assuming every frame shares a shape.
The mechanical merge preserved both histories but left two repositories
sitting side by side rather than one repository. This is the part that
makes it whole.
A top-level README explains the three strands - the telescope network
and campaign, the processing code, and in-person observing plans - and
says plainly that image data does not live here, because that is the
first question anyone will have when they find a pipeline with no
pixels.
state/PROJECT.md was still describing a markdown-only reference project
whose scope explicitly EXCLUDED automating bookings and image
processing. Both of those are now most of what the project does, so the
objective, scope and key facts are rewritten to match reality.
state/DECISIONS.md records why the merge happened - the two repos were
split by chronology rather than design, and the seam was already
leaking, with the campaign's TODO citing results held in the other repo
and the pipeline's README explaining a campaign it did not contain. It
also records that the image data deliberately stays out of the
repository.
state/TODO.md gains the pipeline and observing work that previously had
nowhere to live, including the measurement that changes the pipeline
plan: per-frame processing is 2.9 seconds and the whole 24-frame session
is 1.1 minutes single-threaded, so a scheduler earns nothing on stacking
and should be pointed at the Monte Carlo stages instead.
Internal links fixed for the new paths: pipeline/README.md referred to
session-scripts/ throughout, and itelescope/README.md pointed at a
state/ directory that is now one level up.
The .gitignore conflict between the two repositories is resolved by
combining them, with image formats ignored globally except under docs/,
where documentation figures are committed deliberately.
Preparing to merge this repository into a combined astrophotography
repo. session-scripts/ becomes pipeline/ because the scripts import
layout.py from their own directory and must stay together, and because
'pipeline' says what it is rather than how it came about. observing/
stays at the top level: observing plans are not processing code.
