Make the merged repository coherent: README, state, and internal links

The mechanical merge preserved both histories but left two repositories
sitting side by side rather than one repository. This is the part that
makes it whole.

A top-level README explains the three strands - the telescope network
and campaign, the processing code, and in-person observing plans - and
says plainly that image data does not live here, because that is the
first question anyone will have when they find a pipeline with no
pixels.

state/PROJECT.md was still describing a markdown-only reference project
whose scope explicitly EXCLUDED automating bookings and image
processing. Both of those are now most of what the project does, so the
objective, scope and key facts are rewritten to match reality.

state/DECISIONS.md records why the merge happened - the two repos were
split by chronology rather than design, and the seam was already
leaking, with the campaign's TODO citing results held in the other repo
and the pipeline's README explaining a campaign it did not contain. It
also records that the image data deliberately stays out of the
repository.

state/TODO.md gains the pipeline and observing work that previously had
nowhere to live, including the measurement that changes the pipeline
plan: per-frame processing is 2.9 seconds and the whole 24-frame session
is 1.1 minutes single-threaded, so a scheduler earns nothing on stacking
and should be pointed at the Monte Carlo stages instead.

Internal links fixed for the new paths: pipeline/README.md referred to
session-scripts/ throughout, and itelescope/README.md pointed at a
state/ directory that is now one level up.

The .gitignore conflict between the two repositories is resolved by
combining them, with image formats ignored globally except under docs/,
where documentation figures are committed deliberately.
This commit is contained in:
laurence 2026-07-21 17:16:20 +01:00
parent 87387064c5
commit 59d6e30712
6 changed files with 160 additions and 28 deletions

62
README.md Normal file
View file

@ -0,0 +1,62 @@
# astrophotography
Everything to do with taking, processing and understanding astronomical images:
the remote telescopes, the code that turns their frames into pictures and
measurements, and the plans for observing in person.
Formed by merging the `itelescope` and `astro-pipeline` repositories, with the
history of both preserved.
## Layout
| Directory | What it is |
|---|---|
| **[itelescope/](itelescope/)** | The iTelescope.net remote telescope network: a review of every scope, a southern-target guide, the observing plans, and the points drain campaign |
| **[pipeline/](pipeline/)** | The processing and analysis code. Calibration, stacking, plate solving, rendering and the science analyses |
| **[observing/](observing/)** | Plans for observing in person. Currently the total solar eclipse of 12 August 2026 from Menorca |
| **[state/](state/)** | Project state under the Default Workflow: objective, current work, decisions, working notes |
| **[docs/](docs/)** | The Default Workflow itself: branching, commits, documentation policy, cost control |
## Start here
- Planning or reviewing a remote imaging run: **[itelescope/CAMPAIGN.md](itelescope/CAMPAIGN.md)** and **[itelescope/TELESCOPES.md](itelescope/TELESCOPES.md)**
- Processing a session's data: **[pipeline/README.md](pipeline/README.md)**
- What is happening right now: **[state/TODO.md](state/TODO.md)**
- Why something was done a particular way: **[state/DECISIONS.md](state/DECISIONS.md)**
## Where the image data lives
**Not in this repository.** A single calibrated frame is 61 MB and a session
runs to several gigabytes, so sessions stay on disk and the code finds them
through the `ASTRO_SESSION` environment variable:
```
set ASTRO_SESSION=...\NGC5128\20260721
python pipeline/stack.py
```
Each session directory carries its own `METHODS.md` describing what was done to
that data and what was found, written for a reader who was not there. The code
lives here; the pixels and the account of them live with the data.
## Things learned the hard way
Recorded because each cost real time or real money, and because each is a
requirement for anything built next rather than a curiosity:
- **Measure whether a core is saturated before buying time to fix it.** A
foreground star 69 arcsec from Centaurus A's nucleus was mistaken for the
galaxy, and a telescope booking was made to solve a problem that did not
exist. Filter the stars out of the measurement first.
- **Vet moving-object candidates in detector coordinates.** Registration holds
the sky still, so it drags sensor defects across the frame on perfectly
straight, constant-rate tracks. Hot pixels are better-behaved asteroids than
real asteroids. One cut took 141 confident spurious detections to zero.
- **Never compare an aperture magnitude against a point-source catalogue
without checking the source is a point.** A resolved object looks exactly like
a 2.8 magnitude outburst.
- **Never fit a sky background to a field the target fills.** A plane fitted
around a large galaxy eats its halo, measured at -17.9 ADU/px. Fit the
background and a source model together.
- **A photographic ND filter is not a solar filter.** It passes the infrared
that carries the heat.

View file

@ -10,7 +10,8 @@ book for a given target.
to scopes and to the time of year to book them.
- **[data/itelescope-telescopes.csv](data/itelescope-telescopes.csv)**: snapshot of
the network's maintained specs sheet (source of truth for the numbers).
- **state/**: project state under the Default Workflow.
- **[../state/](../state/)**: project state under the Default Workflow,
covering this repository as a whole.
Sources: the iTelescope support article
([telescope summary](https://support.itelescope.net/support/solutions/articles/247371))

View file

@ -1,4 +1,4 @@
# astro-pipeline
# pipeline
Processing and analysis code for remote-telescope imaging sessions, starting
with iTelescope data from the [itelescope](https://git.discworld.casa/laurence/itelescope)
@ -11,7 +11,7 @@ done to them.
## What is here now
`session-scripts/` - the 50 scripts that processed the NGC 5128 session of
`pipeline/` - the 50 scripts that processed the NGC 5128 session of
2026-07-21, exactly as they were run, plus the shared `layout.py` that tells
them where files live. This is a working record rather than a finished product:
the scripts were written in sequence as the work went along, several of them by
@ -19,21 +19,15 @@ parallel agents, and they show it. They are kept because they are the honest
provenance of a set of published results, and because the productionised
pipeline should be able to reproduce those results exactly.
`session-scripts/restructure.py` - reorganises a flat session directory into the
`pipeline/restructure.py` - reorganises a flat session directory into the
named layout below. Idempotent, dry run by default.
`observing/` - plans for observing sessions that are not remote-telescope runs.
Currently the total solar eclipse of 12 August 2026, seen from Menorca. These
live here rather than in a notes app because they are worked out from real
numbers, they get revised as the date approaches, and the reasoning behind each
decision is worth keeping.
## Pointing the scripts at a session
```
set ASTRO_SESSION=D:\astro\NGC5128\20260721 # Windows
export ASTRO_SESSION=/data/astro/NGC5128/20260721 # POSIX
python session-scripts/layout.py # prints the resolved layout
python pipeline/layout.py # prints the resolved layout
```
`layout.py` maps a **filename** to its subdirectory, so a script asks for

View file

@ -3,6 +3,38 @@
> A dated, append only log of decisions and their rationale. Newest at the top. Never
> rewrite past entries; if a decision is reversed, add a new entry that says so.
## 2026-07-21: merge itelescope and astro-pipeline into one repository
The two were split by accident of chronology rather than by design. `itelescope`
began as a reference to the telescope network and grew into the campaign that
books and runs sessions; `astro-pipeline` was created three hours before this
merge to hold the processing code once it was moved out of the image directory.
They describe two halves of one activity, and the seam was already leaking: the
campaign's TODO referred to processing results held in the other repo, and the
processing code's README explained a campaign it did not contain.
Merged with **both histories preserved**, by relocating each repository's files
into a subdirectory in its own history first and then merging the unrelated
histories. 44 commits, and `git log --follow` still traces a file across the
move.
Layout: `itelescope/` for the network and campaign, `pipeline/` for the code,
`observing/` for in-person plans, with the Default Workflow files (CLAUDE.md,
docs/, state/) at the top level governing the whole repository.
**The old repositories are not deleted.** The drain campaign is live until
10 Aug with sessions running most nights, and other sessions may be working in
`itelescope` at the moment of the merge. Both carry a notice pointing here.
Archive them once the campaign closes, not before.
## 2026-07-21: image data does not live in the repository
A single calibrated frame is 61 MB and a session runs to several gigabytes.
Sessions stay on disk, organised into named subdirectories, and the code finds
them through the `ASTRO_SESSION` environment variable. Each session carries its
own `METHODS.md` describing what was done and what was found, so the account of
the data travels with the data while the code travels separately.
## 2026-07-17: follow the Google Sheet, not the support article
The support article (https://support.itelescope.net/support/solutions/articles/247371)

View file

@ -1,29 +1,41 @@
# Project: itelescope
# Project: astrophotography
> The anchor document. A session that reads only this, TODO.md and DECISIONS.md should
> understand what the project is and what to do next. Keep it current.
## Objective
A reference and review of the iTelescope.net remote telescope network: what each
telescope is, what it is good at, and which one to book for a given kind of target.
Take astronomical images, process them properly, and understand what is in them.
Three strands, all in this repository since the 2026-07-21 merge of the
`itelescope` and `astro-pipeline` repositories:
Since 2026-07-18 the active objective is the drain campaign (CAMPAIGN.md): spend
the account's 2,638 points on southern-sky imaging by 10 Aug 2026, download all
data, then cancel the membership before the 12 Aug renewal. Cancellation needs
explicit user confirmation on the day.
1. **Remote imaging** (`itelescope/`) - a review of the iTelescope.net network
and which scope to book for a given target. The active objective is the drain
campaign (`itelescope/CAMPAIGN.md`): spend the account's remaining points on
southern-sky imaging by 10 Aug 2026, download all data, then cancel before the
12 Aug renewal. **Cancellation needs explicit user confirmation on the day.**
2. **Processing and analysis** (`pipeline/`) - the code that turns frames into
calibrated, plate-solved images and into measurements. Proven on the NGC 5128
session of 2026-07-21; being productionised into a staged, containerised
pipeline that can run under a scheduler.
3. **Observing in person** (`observing/`) - currently the total solar eclipse of
12 Aug 2026, seen from Menorca.
## Scope
- In scope: reviews and spec tables for every telescope on the iTelescope network,
grouped by observatory; source data snapshots; guidance on choosing a scope.
- Out of scope: automating bookings, image processing, anything requiring the
iTelescope login (the launchpad at go.itelescope.net is authenticated).
- In scope: the telescope network reference; booking and running remote sessions;
calibration, stacking, plate solving, rendering and analysis code; observing
plans for trips.
- Out of scope: the image data itself. Sessions stay on disk and are located
through `ASTRO_SESSION`; a single frame is 61 MB and a session is several GB.
Each session directory carries its own METHODS.md.
- Previously out of scope, now central: automating bookings and image
processing. Both proved necessary and both are done.
## Audience
Laurence, when planning imaging or photometry runs on iTelescope, and anyone else
choosing a telescope on the network.
Laurence, when planning or processing imaging runs, and anyone else choosing a
telescope on the network or wanting to reproduce a result.
## Description
@ -42,6 +54,12 @@ assessments plus a "which scope for what" guide.
## Key facts
- Trunk branch: main
- Forge / remote: https://git.discworld.casa/laurence/itelescope
- Runtime / stack: markdown only; source data in data/ as CSV
- How to run it: nothing to run; read TELESCOPES.md
- Forge / remote: https://git.discworld.casa/laurence/astrophotography
(merged 2026-07-21 from `itelescope` + `astro-pipeline`; both histories intact)
- Runtime / stack: markdown for the reference and campaign; Python 3.12 for the
pipeline (numpy, scipy, astropy, scikit-image, sep, astroalign, photutils,
astroquery, matplotlib)
- How to run it: read `itelescope/TELESCOPES.md` to choose a scope; set
`ASTRO_SESSION` and run `pipeline/*.py` to process a session
- Image data is NOT in this repo: it lives on disk per session, each with its
own METHODS.md

View file

@ -76,7 +76,32 @@
10 Aug cancellation sweep. Update the CAMPAIGN.md ledger after every session
from account/history.aspx.
## Pending
## Pending - pipeline
- Productionise the processing code: a staged CLI
(ingest -> calibrate -> measure -> register -> stack -> solve -> compose ->
analyse), each stage resumable, packaged as an Apptainer image. Slurm array
jobs afterwards and ONLY for the expensive stages: measured on the NGC 5128
session, per-frame work is 2.9 s and all 24 frames take 1.1 min
single-threaded, so a scheduler buys nothing there. It pays for the Monte
Carlo work (completeness and injection-recovery tests, difference imaging,
shift-and-stack), which is where compute actually bit.
- Must handle beyond mono LRGB: narrowband palettes, one-shot colour with
debayering, other observatories' header conventions, and full calibration
from bias/dark/flat for sources that do not pre-calibrate.
- Three findings from the first session are requirements, not options: vet
moving objects in DETECTOR coordinates; carry r50/psf into any catalogue
cross-match; never fit a sky background to a field the target fills.
## Pending - observing
- Total solar eclipse, 12 Aug 2026, Menorca (`observing/eclipse-2026-menorca/`).
Totality 1m12s at Ciutadella with the Sun at 2 degrees, azimuth 288.
**Order the certified solar filter - it is the only item with no substitute
and there are three weeks left.** Then practise on a low sunset to calibrate
exposures for the extreme atmospheric extinction.
## Pending - iTelescope
- Automation: map the ACP web endpoints on a per-scope server (start with free
T33) for plan submission and status; explore lookup.itelescope.online (the new