Make the merged repository coherent: README, state, and internal links
The mechanical merge preserved both histories but left two repositories sitting side by side rather than one repository. This is the part that makes it whole. A top-level README explains the three strands - the telescope network and campaign, the processing code, and in-person observing plans - and says plainly that image data does not live here, because that is the first question anyone will have when they find a pipeline with no pixels. state/PROJECT.md was still describing a markdown-only reference project whose scope explicitly EXCLUDED automating bookings and image processing. Both of those are now most of what the project does, so the objective, scope and key facts are rewritten to match reality. state/DECISIONS.md records why the merge happened - the two repos were split by chronology rather than design, and the seam was already leaking, with the campaign's TODO citing results held in the other repo and the pipeline's README explaining a campaign it did not contain. It also records that the image data deliberately stays out of the repository. state/TODO.md gains the pipeline and observing work that previously had nowhere to live, including the measurement that changes the pipeline plan: per-frame processing is 2.9 seconds and the whole 24-frame session is 1.1 minutes single-threaded, so a scheduler earns nothing on stacking and should be pointed at the Monte Carlo stages instead. Internal links fixed for the new paths: pipeline/README.md referred to session-scripts/ throughout, and itelescope/README.md pointed at a state/ directory that is now one level up. The .gitignore conflict between the two repositories is resolved by combining them, with image formats ignored globally except under docs/, where documentation figures are committed deliberately.
This commit is contained in:
parent
87387064c5
commit
59d6e30712
6 changed files with 160 additions and 28 deletions
62
README.md
Normal file
62
README.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
# astrophotography
|
||||
|
||||
Everything to do with taking, processing and understanding astronomical images:
|
||||
the remote telescopes, the code that turns their frames into pictures and
|
||||
measurements, and the plans for observing in person.
|
||||
|
||||
Formed by merging the `itelescope` and `astro-pipeline` repositories, with the
|
||||
history of both preserved.
|
||||
|
||||
## Layout
|
||||
|
||||
| Directory | What it is |
|
||||
|---|---|
|
||||
| **[itelescope/](itelescope/)** | The iTelescope.net remote telescope network: a review of every scope, a southern-target guide, the observing plans, and the points drain campaign |
|
||||
| **[pipeline/](pipeline/)** | The processing and analysis code. Calibration, stacking, plate solving, rendering and the science analyses |
|
||||
| **[observing/](observing/)** | Plans for observing in person. Currently the total solar eclipse of 12 August 2026 from Menorca |
|
||||
| **[state/](state/)** | Project state under the Default Workflow: objective, current work, decisions, working notes |
|
||||
| **[docs/](docs/)** | The Default Workflow itself: branching, commits, documentation policy, cost control |
|
||||
|
||||
## Start here
|
||||
|
||||
- Planning or reviewing a remote imaging run: **[itelescope/CAMPAIGN.md](itelescope/CAMPAIGN.md)** and **[itelescope/TELESCOPES.md](itelescope/TELESCOPES.md)**
|
||||
- Processing a session's data: **[pipeline/README.md](pipeline/README.md)**
|
||||
- What is happening right now: **[state/TODO.md](state/TODO.md)**
|
||||
- Why something was done a particular way: **[state/DECISIONS.md](state/DECISIONS.md)**
|
||||
|
||||
## Where the image data lives
|
||||
|
||||
**Not in this repository.** A single calibrated frame is 61 MB and a session
|
||||
runs to several gigabytes, so sessions stay on disk and the code finds them
|
||||
through the `ASTRO_SESSION` environment variable:
|
||||
|
||||
```
|
||||
set ASTRO_SESSION=...\NGC5128\20260721
|
||||
python pipeline/stack.py
|
||||
```
|
||||
|
||||
Each session directory carries its own `METHODS.md` describing what was done to
|
||||
that data and what was found, written for a reader who was not there. The code
|
||||
lives here; the pixels and the account of them live with the data.
|
||||
|
||||
## Things learned the hard way
|
||||
|
||||
Recorded because each cost real time or real money, and because each is a
|
||||
requirement for anything built next rather than a curiosity:
|
||||
|
||||
- **Measure whether a core is saturated before buying time to fix it.** A
|
||||
foreground star 69 arcsec from Centaurus A's nucleus was mistaken for the
|
||||
galaxy, and a telescope booking was made to solve a problem that did not
|
||||
exist. Filter the stars out of the measurement first.
|
||||
- **Vet moving-object candidates in detector coordinates.** Registration holds
|
||||
the sky still, so it drags sensor defects across the frame on perfectly
|
||||
straight, constant-rate tracks. Hot pixels are better-behaved asteroids than
|
||||
real asteroids. One cut took 141 confident spurious detections to zero.
|
||||
- **Never compare an aperture magnitude against a point-source catalogue
|
||||
without checking the source is a point.** A resolved object looks exactly like
|
||||
a 2.8 magnitude outburst.
|
||||
- **Never fit a sky background to a field the target fills.** A plane fitted
|
||||
around a large galaxy eats its halo, measured at -17.9 ADU/px. Fit the
|
||||
background and a source model together.
|
||||
- **A photographic ND filter is not a solar filter.** It passes the infrared
|
||||
that carries the heat.
|
||||
|
|
@ -10,7 +10,8 @@ book for a given target.
|
|||
to scopes and to the time of year to book them.
|
||||
- **[data/itelescope-telescopes.csv](data/itelescope-telescopes.csv)**: snapshot of
|
||||
the network's maintained specs sheet (source of truth for the numbers).
|
||||
- **state/**: project state under the Default Workflow.
|
||||
- **[../state/](../state/)**: project state under the Default Workflow,
|
||||
covering this repository as a whole.
|
||||
|
||||
Sources: the iTelescope support article
|
||||
([telescope summary](https://support.itelescope.net/support/solutions/articles/247371))
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# astro-pipeline
|
||||
# pipeline
|
||||
|
||||
Processing and analysis code for remote-telescope imaging sessions, starting
|
||||
with iTelescope data from the [itelescope](https://git.discworld.casa/laurence/itelescope)
|
||||
|
|
@ -11,7 +11,7 @@ done to them.
|
|||
|
||||
## What is here now
|
||||
|
||||
`session-scripts/` - the 50 scripts that processed the NGC 5128 session of
|
||||
`pipeline/` - the 50 scripts that processed the NGC 5128 session of
|
||||
2026-07-21, exactly as they were run, plus the shared `layout.py` that tells
|
||||
them where files live. This is a working record rather than a finished product:
|
||||
the scripts were written in sequence as the work went along, several of them by
|
||||
|
|
@ -19,21 +19,15 @@ parallel agents, and they show it. They are kept because they are the honest
|
|||
provenance of a set of published results, and because the productionised
|
||||
pipeline should be able to reproduce those results exactly.
|
||||
|
||||
`session-scripts/restructure.py` - reorganises a flat session directory into the
|
||||
`pipeline/restructure.py` - reorganises a flat session directory into the
|
||||
named layout below. Idempotent, dry run by default.
|
||||
|
||||
`observing/` - plans for observing sessions that are not remote-telescope runs.
|
||||
Currently the total solar eclipse of 12 August 2026, seen from Menorca. These
|
||||
live here rather than in a notes app because they are worked out from real
|
||||
numbers, they get revised as the date approaches, and the reasoning behind each
|
||||
decision is worth keeping.
|
||||
|
||||
## Pointing the scripts at a session
|
||||
|
||||
```
|
||||
set ASTRO_SESSION=D:\astro\NGC5128\20260721 # Windows
|
||||
export ASTRO_SESSION=/data/astro/NGC5128/20260721 # POSIX
|
||||
python session-scripts/layout.py # prints the resolved layout
|
||||
python pipeline/layout.py # prints the resolved layout
|
||||
```
|
||||
|
||||
`layout.py` maps a **filename** to its subdirectory, so a script asks for
|
||||
|
|
|
|||
|
|
@ -3,6 +3,38 @@
|
|||
> A dated, append only log of decisions and their rationale. Newest at the top. Never
|
||||
> rewrite past entries; if a decision is reversed, add a new entry that says so.
|
||||
|
||||
## 2026-07-21: merge itelescope and astro-pipeline into one repository
|
||||
|
||||
The two were split by accident of chronology rather than by design. `itelescope`
|
||||
began as a reference to the telescope network and grew into the campaign that
|
||||
books and runs sessions; `astro-pipeline` was created three hours before this
|
||||
merge to hold the processing code once it was moved out of the image directory.
|
||||
They describe two halves of one activity, and the seam was already leaking: the
|
||||
campaign's TODO referred to processing results held in the other repo, and the
|
||||
processing code's README explained a campaign it did not contain.
|
||||
|
||||
Merged with **both histories preserved**, by relocating each repository's files
|
||||
into a subdirectory in its own history first and then merging the unrelated
|
||||
histories. 44 commits, and `git log --follow` still traces a file across the
|
||||
move.
|
||||
|
||||
Layout: `itelescope/` for the network and campaign, `pipeline/` for the code,
|
||||
`observing/` for in-person plans, with the Default Workflow files (CLAUDE.md,
|
||||
docs/, state/) at the top level governing the whole repository.
|
||||
|
||||
**The old repositories are not deleted.** The drain campaign is live until
|
||||
10 Aug with sessions running most nights, and other sessions may be working in
|
||||
`itelescope` at the moment of the merge. Both carry a notice pointing here.
|
||||
Archive them once the campaign closes, not before.
|
||||
|
||||
## 2026-07-21: image data does not live in the repository
|
||||
|
||||
A single calibrated frame is 61 MB and a session runs to several gigabytes.
|
||||
Sessions stay on disk, organised into named subdirectories, and the code finds
|
||||
them through the `ASTRO_SESSION` environment variable. Each session carries its
|
||||
own `METHODS.md` describing what was done and what was found, so the account of
|
||||
the data travels with the data while the code travels separately.
|
||||
|
||||
## 2026-07-17: follow the Google Sheet, not the support article
|
||||
|
||||
The support article (https://support.itelescope.net/support/solutions/articles/247371)
|
||||
|
|
|
|||
|
|
@ -1,29 +1,41 @@
|
|||
# Project: itelescope
|
||||
# Project: astrophotography
|
||||
|
||||
> The anchor document. A session that reads only this, TODO.md and DECISIONS.md should
|
||||
> understand what the project is and what to do next. Keep it current.
|
||||
|
||||
## Objective
|
||||
|
||||
A reference and review of the iTelescope.net remote telescope network: what each
|
||||
telescope is, what it is good at, and which one to book for a given kind of target.
|
||||
Take astronomical images, process them properly, and understand what is in them.
|
||||
Three strands, all in this repository since the 2026-07-21 merge of the
|
||||
`itelescope` and `astro-pipeline` repositories:
|
||||
|
||||
Since 2026-07-18 the active objective is the drain campaign (CAMPAIGN.md): spend
|
||||
the account's 2,638 points on southern-sky imaging by 10 Aug 2026, download all
|
||||
data, then cancel the membership before the 12 Aug renewal. Cancellation needs
|
||||
explicit user confirmation on the day.
|
||||
1. **Remote imaging** (`itelescope/`) - a review of the iTelescope.net network
|
||||
and which scope to book for a given target. The active objective is the drain
|
||||
campaign (`itelescope/CAMPAIGN.md`): spend the account's remaining points on
|
||||
southern-sky imaging by 10 Aug 2026, download all data, then cancel before the
|
||||
12 Aug renewal. **Cancellation needs explicit user confirmation on the day.**
|
||||
2. **Processing and analysis** (`pipeline/`) - the code that turns frames into
|
||||
calibrated, plate-solved images and into measurements. Proven on the NGC 5128
|
||||
session of 2026-07-21; being productionised into a staged, containerised
|
||||
pipeline that can run under a scheduler.
|
||||
3. **Observing in person** (`observing/`) - currently the total solar eclipse of
|
||||
12 Aug 2026, seen from Menorca.
|
||||
|
||||
## Scope
|
||||
|
||||
- In scope: reviews and spec tables for every telescope on the iTelescope network,
|
||||
grouped by observatory; source data snapshots; guidance on choosing a scope.
|
||||
- Out of scope: automating bookings, image processing, anything requiring the
|
||||
iTelescope login (the launchpad at go.itelescope.net is authenticated).
|
||||
- In scope: the telescope network reference; booking and running remote sessions;
|
||||
calibration, stacking, plate solving, rendering and analysis code; observing
|
||||
plans for trips.
|
||||
- Out of scope: the image data itself. Sessions stay on disk and are located
|
||||
through `ASTRO_SESSION`; a single frame is 61 MB and a session is several GB.
|
||||
Each session directory carries its own METHODS.md.
|
||||
- Previously out of scope, now central: automating bookings and image
|
||||
processing. Both proved necessary and both are done.
|
||||
|
||||
## Audience
|
||||
|
||||
Laurence, when planning imaging or photometry runs on iTelescope, and anyone else
|
||||
choosing a telescope on the network.
|
||||
Laurence, when planning or processing imaging runs, and anyone else choosing a
|
||||
telescope on the network or wanting to reproduce a result.
|
||||
|
||||
## Description
|
||||
|
||||
|
|
@ -42,6 +54,12 @@ assessments plus a "which scope for what" guide.
|
|||
## Key facts
|
||||
|
||||
- Trunk branch: main
|
||||
- Forge / remote: https://git.discworld.casa/laurence/itelescope
|
||||
- Runtime / stack: markdown only; source data in data/ as CSV
|
||||
- How to run it: nothing to run; read TELESCOPES.md
|
||||
- Forge / remote: https://git.discworld.casa/laurence/astrophotography
|
||||
(merged 2026-07-21 from `itelescope` + `astro-pipeline`; both histories intact)
|
||||
- Runtime / stack: markdown for the reference and campaign; Python 3.12 for the
|
||||
pipeline (numpy, scipy, astropy, scikit-image, sep, astroalign, photutils,
|
||||
astroquery, matplotlib)
|
||||
- How to run it: read `itelescope/TELESCOPES.md` to choose a scope; set
|
||||
`ASTRO_SESSION` and run `pipeline/*.py` to process a session
|
||||
- Image data is NOT in this repo: it lives on disk per session, each with its
|
||||
own METHODS.md
|
||||
|
|
|
|||
|
|
@ -76,7 +76,32 @@
|
|||
10 Aug cancellation sweep. Update the CAMPAIGN.md ledger after every session
|
||||
from account/history.aspx.
|
||||
|
||||
## Pending
|
||||
## Pending - pipeline
|
||||
|
||||
- Productionise the processing code: a staged CLI
|
||||
(ingest -> calibrate -> measure -> register -> stack -> solve -> compose ->
|
||||
analyse), each stage resumable, packaged as an Apptainer image. Slurm array
|
||||
jobs afterwards and ONLY for the expensive stages: measured on the NGC 5128
|
||||
session, per-frame work is 2.9 s and all 24 frames take 1.1 min
|
||||
single-threaded, so a scheduler buys nothing there. It pays for the Monte
|
||||
Carlo work (completeness and injection-recovery tests, difference imaging,
|
||||
shift-and-stack), which is where compute actually bit.
|
||||
- Must handle beyond mono LRGB: narrowband palettes, one-shot colour with
|
||||
debayering, other observatories' header conventions, and full calibration
|
||||
from bias/dark/flat for sources that do not pre-calibrate.
|
||||
- Three findings from the first session are requirements, not options: vet
|
||||
moving objects in DETECTOR coordinates; carry r50/psf into any catalogue
|
||||
cross-match; never fit a sky background to a field the target fills.
|
||||
|
||||
## Pending - observing
|
||||
|
||||
- Total solar eclipse, 12 Aug 2026, Menorca (`observing/eclipse-2026-menorca/`).
|
||||
Totality 1m12s at Ciutadella with the Sun at 2 degrees, azimuth 288.
|
||||
**Order the certified solar filter - it is the only item with no substitute
|
||||
and there are three weeks left.** Then practise on a low sunset to calibrate
|
||||
exposures for the extreme atmospheric extinction.
|
||||
|
||||
## Pending - iTelescope
|
||||
|
||||
- Automation: map the ACP web endpoints on a per-scope server (start with free
|
||||
T33) for plan submission and status; explore lookup.itelescope.online (the new
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue