diff --git a/README.md b/README.md new file mode 100644 index 0000000..4a4d2d0 --- /dev/null +++ b/README.md @@ -0,0 +1,62 @@ +# astrophotography + +Everything to do with taking, processing and understanding astronomical images: +the remote telescopes, the code that turns their frames into pictures and +measurements, and the plans for observing in person. + +Formed by merging the `itelescope` and `astro-pipeline` repositories, with the +history of both preserved. + +## Layout + +| Directory | What it is | +|---|---| +| **[itelescope/](itelescope/)** | The iTelescope.net remote telescope network: a review of every scope, a southern-target guide, the observing plans, and the points drain campaign | +| **[pipeline/](pipeline/)** | The processing and analysis code. Calibration, stacking, plate solving, rendering and the science analyses | +| **[observing/](observing/)** | Plans for observing in person. Currently the total solar eclipse of 12 August 2026 from Menorca | +| **[state/](state/)** | Project state under the Default Workflow: objective, current work, decisions, working notes | +| **[docs/](docs/)** | The Default Workflow itself: branching, commits, documentation policy, cost control | + +## Start here + +- Planning or reviewing a remote imaging run: **[itelescope/CAMPAIGN.md](itelescope/CAMPAIGN.md)** and **[itelescope/TELESCOPES.md](itelescope/TELESCOPES.md)** +- Processing a session's data: **[pipeline/README.md](pipeline/README.md)** +- What is happening right now: **[state/TODO.md](state/TODO.md)** +- Why something was done a particular way: **[state/DECISIONS.md](state/DECISIONS.md)** + +## Where the image data lives + +**Not in this repository.** A single calibrated frame is 61 MB and a session +runs to several gigabytes, so sessions stay on disk and the code finds them +through the `ASTRO_SESSION` environment variable: + +``` +set ASTRO_SESSION=...\NGC5128\20260721 +python pipeline/stack.py +``` + +Each session directory carries its own `METHODS.md` describing what was done to +that data and what was found, written for a reader who was not there. The code +lives here; the pixels and the account of them live with the data. + +## Things learned the hard way + +Recorded because each cost real time or real money, and because each is a +requirement for anything built next rather than a curiosity: + +- **Measure whether a core is saturated before buying time to fix it.** A + foreground star 69 arcsec from Centaurus A's nucleus was mistaken for the + galaxy, and a telescope booking was made to solve a problem that did not + exist. Filter the stars out of the measurement first. +- **Vet moving-object candidates in detector coordinates.** Registration holds + the sky still, so it drags sensor defects across the frame on perfectly + straight, constant-rate tracks. Hot pixels are better-behaved asteroids than + real asteroids. One cut took 141 confident spurious detections to zero. +- **Never compare an aperture magnitude against a point-source catalogue + without checking the source is a point.** A resolved object looks exactly like + a 2.8 magnitude outburst. +- **Never fit a sky background to a field the target fills.** A plane fitted + around a large galaxy eats its halo, measured at -17.9 ADU/px. Fit the + background and a source model together. +- **A photographic ND filter is not a solar filter.** It passes the infrared + that carries the heat. diff --git a/itelescope/README.md b/itelescope/README.md index 1007f33..60d1a09 100644 --- a/itelescope/README.md +++ b/itelescope/README.md @@ -10,7 +10,8 @@ book for a given target. to scopes and to the time of year to book them. - **[data/itelescope-telescopes.csv](data/itelescope-telescopes.csv)**: snapshot of the network's maintained specs sheet (source of truth for the numbers). -- **state/**: project state under the Default Workflow. +- **[../state/](../state/)**: project state under the Default Workflow, + covering this repository as a whole. Sources: the iTelescope support article ([telescope summary](https://support.itelescope.net/support/solutions/articles/247371)) diff --git a/pipeline/README.md b/pipeline/README.md index 78b1e14..eb95b72 100644 --- a/pipeline/README.md +++ b/pipeline/README.md @@ -1,4 +1,4 @@ -# astro-pipeline +# pipeline Processing and analysis code for remote-telescope imaging sessions, starting with iTelescope data from the [itelescope](https://git.discworld.casa/laurence/itelescope) @@ -11,7 +11,7 @@ done to them. ## What is here now -`session-scripts/` - the 50 scripts that processed the NGC 5128 session of +`pipeline/` - the 50 scripts that processed the NGC 5128 session of 2026-07-21, exactly as they were run, plus the shared `layout.py` that tells them where files live. This is a working record rather than a finished product: the scripts were written in sequence as the work went along, several of them by @@ -19,21 +19,15 @@ parallel agents, and they show it. They are kept because they are the honest provenance of a set of published results, and because the productionised pipeline should be able to reproduce those results exactly. -`session-scripts/restructure.py` - reorganises a flat session directory into the +`pipeline/restructure.py` - reorganises a flat session directory into the named layout below. Idempotent, dry run by default. -`observing/` - plans for observing sessions that are not remote-telescope runs. -Currently the total solar eclipse of 12 August 2026, seen from Menorca. These -live here rather than in a notes app because they are worked out from real -numbers, they get revised as the date approaches, and the reasoning behind each -decision is worth keeping. - ## Pointing the scripts at a session ``` set ASTRO_SESSION=D:\astro\NGC5128\20260721 # Windows export ASTRO_SESSION=/data/astro/NGC5128/20260721 # POSIX -python session-scripts/layout.py # prints the resolved layout +python pipeline/layout.py # prints the resolved layout ``` `layout.py` maps a **filename** to its subdirectory, so a script asks for diff --git a/state/DECISIONS.md b/state/DECISIONS.md index a461b78..25eb138 100644 --- a/state/DECISIONS.md +++ b/state/DECISIONS.md @@ -3,6 +3,38 @@ > A dated, append only log of decisions and their rationale. Newest at the top. Never > rewrite past entries; if a decision is reversed, add a new entry that says so. +## 2026-07-21: merge itelescope and astro-pipeline into one repository + +The two were split by accident of chronology rather than by design. `itelescope` +began as a reference to the telescope network and grew into the campaign that +books and runs sessions; `astro-pipeline` was created three hours before this +merge to hold the processing code once it was moved out of the image directory. +They describe two halves of one activity, and the seam was already leaking: the +campaign's TODO referred to processing results held in the other repo, and the +processing code's README explained a campaign it did not contain. + +Merged with **both histories preserved**, by relocating each repository's files +into a subdirectory in its own history first and then merging the unrelated +histories. 44 commits, and `git log --follow` still traces a file across the +move. + +Layout: `itelescope/` for the network and campaign, `pipeline/` for the code, +`observing/` for in-person plans, with the Default Workflow files (CLAUDE.md, +docs/, state/) at the top level governing the whole repository. + +**The old repositories are not deleted.** The drain campaign is live until +10 Aug with sessions running most nights, and other sessions may be working in +`itelescope` at the moment of the merge. Both carry a notice pointing here. +Archive them once the campaign closes, not before. + +## 2026-07-21: image data does not live in the repository + +A single calibrated frame is 61 MB and a session runs to several gigabytes. +Sessions stay on disk, organised into named subdirectories, and the code finds +them through the `ASTRO_SESSION` environment variable. Each session carries its +own `METHODS.md` describing what was done and what was found, so the account of +the data travels with the data while the code travels separately. + ## 2026-07-17: follow the Google Sheet, not the support article The support article (https://support.itelescope.net/support/solutions/articles/247371) diff --git a/state/PROJECT.md b/state/PROJECT.md index 316170d..4bc4473 100644 --- a/state/PROJECT.md +++ b/state/PROJECT.md @@ -1,29 +1,41 @@ -# Project: itelescope +# Project: astrophotography > The anchor document. A session that reads only this, TODO.md and DECISIONS.md should > understand what the project is and what to do next. Keep it current. ## Objective -A reference and review of the iTelescope.net remote telescope network: what each -telescope is, what it is good at, and which one to book for a given kind of target. +Take astronomical images, process them properly, and understand what is in them. +Three strands, all in this repository since the 2026-07-21 merge of the +`itelescope` and `astro-pipeline` repositories: -Since 2026-07-18 the active objective is the drain campaign (CAMPAIGN.md): spend -the account's 2,638 points on southern-sky imaging by 10 Aug 2026, download all -data, then cancel the membership before the 12 Aug renewal. Cancellation needs -explicit user confirmation on the day. +1. **Remote imaging** (`itelescope/`) - a review of the iTelescope.net network + and which scope to book for a given target. The active objective is the drain + campaign (`itelescope/CAMPAIGN.md`): spend the account's remaining points on + southern-sky imaging by 10 Aug 2026, download all data, then cancel before the + 12 Aug renewal. **Cancellation needs explicit user confirmation on the day.** +2. **Processing and analysis** (`pipeline/`) - the code that turns frames into + calibrated, plate-solved images and into measurements. Proven on the NGC 5128 + session of 2026-07-21; being productionised into a staged, containerised + pipeline that can run under a scheduler. +3. **Observing in person** (`observing/`) - currently the total solar eclipse of + 12 Aug 2026, seen from Menorca. ## Scope -- In scope: reviews and spec tables for every telescope on the iTelescope network, - grouped by observatory; source data snapshots; guidance on choosing a scope. -- Out of scope: automating bookings, image processing, anything requiring the - iTelescope login (the launchpad at go.itelescope.net is authenticated). +- In scope: the telescope network reference; booking and running remote sessions; + calibration, stacking, plate solving, rendering and analysis code; observing + plans for trips. +- Out of scope: the image data itself. Sessions stay on disk and are located + through `ASTRO_SESSION`; a single frame is 61 MB and a session is several GB. + Each session directory carries its own METHODS.md. +- Previously out of scope, now central: automating bookings and image + processing. Both proved necessary and both are done. ## Audience -Laurence, when planning imaging or photometry runs on iTelescope, and anyone else -choosing a telescope on the network. +Laurence, when planning or processing imaging runs, and anyone else choosing a +telescope on the network or wanting to reproduce a result. ## Description @@ -42,6 +54,12 @@ assessments plus a "which scope for what" guide. ## Key facts - Trunk branch: main -- Forge / remote: https://git.discworld.casa/laurence/itelescope -- Runtime / stack: markdown only; source data in data/ as CSV -- How to run it: nothing to run; read TELESCOPES.md +- Forge / remote: https://git.discworld.casa/laurence/astrophotography + (merged 2026-07-21 from `itelescope` + `astro-pipeline`; both histories intact) +- Runtime / stack: markdown for the reference and campaign; Python 3.12 for the + pipeline (numpy, scipy, astropy, scikit-image, sep, astroalign, photutils, + astroquery, matplotlib) +- How to run it: read `itelescope/TELESCOPES.md` to choose a scope; set + `ASTRO_SESSION` and run `pipeline/*.py` to process a session +- Image data is NOT in this repo: it lives on disk per session, each with its + own METHODS.md diff --git a/state/TODO.md b/state/TODO.md index 13a5890..5645984 100644 --- a/state/TODO.md +++ b/state/TODO.md @@ -76,7 +76,32 @@ 10 Aug cancellation sweep. Update the CAMPAIGN.md ledger after every session from account/history.aspx. -## Pending +## Pending - pipeline + +- Productionise the processing code: a staged CLI + (ingest -> calibrate -> measure -> register -> stack -> solve -> compose -> + analyse), each stage resumable, packaged as an Apptainer image. Slurm array + jobs afterwards and ONLY for the expensive stages: measured on the NGC 5128 + session, per-frame work is 2.9 s and all 24 frames take 1.1 min + single-threaded, so a scheduler buys nothing there. It pays for the Monte + Carlo work (completeness and injection-recovery tests, difference imaging, + shift-and-stack), which is where compute actually bit. +- Must handle beyond mono LRGB: narrowband palettes, one-shot colour with + debayering, other observatories' header conventions, and full calibration + from bias/dark/flat for sources that do not pre-calibrate. +- Three findings from the first session are requirements, not options: vet + moving objects in DETECTOR coordinates; carry r50/psf into any catalogue + cross-match; never fit a sky background to a field the target fills. + +## Pending - observing + +- Total solar eclipse, 12 Aug 2026, Menorca (`observing/eclipse-2026-menorca/`). + Totality 1m12s at Ciutadella with the Sun at 2 degrees, azimuth 288. + **Order the certified solar filter - it is the only item with no substitute + and there are three weeks left.** Then practise on a low sunset to calibrate + exposures for the extreme atmospheric extinction. + +## Pending - iTelescope - Automation: map the ACP web endpoints on a per-scope server (start with free T33) for plan submission and status; explore lookup.itelescope.online (the new