laurence
3f785136ed
Rule 2 of the workflow is that state lives in the repo rather than the chat, and a long session had put most of the current position only in the chat. This writes it down. TODO.md gains a RESUME HERE block at the top, because the useful question for the next session is not what has been done but what to do next and what is blocking it. It names the three live strands, the three decisions waiting on the user, and the one thing that will break the batch run if it is forgotten: NGC 6744 mixes bin1 luminance at 4096x4096 with bin2 colour at 2048x2048, so registration has to resample onto a common grid. Nothing else in the archive does that, which is exactly why it will be forgotten. It also drops three duplicate "Done (recent)" headings that had accumulated, and tabulates what the five archived sessions actually contain. NOTES.md gains the gotchas, all of which produced plausible wrong answers rather than errors: every frame is delivered twice as calibrated and raw, so extracting both silently doubles the stack; an interrupted segmented download leaves DIRECTORIES named like files, which defeats anything that trusts an extension; a re-mirror duplicates rather than replaces; two T20 archives are genuinely corrupt; extraction triples the disk footprint and filled the drive. Alongside them the analysis lessons from NGC 5128, kept for the same reason - each is a way to be confidently wrong. DECISIONS.md records why the pipeline reads headers rather than filenames, and why imaging is being made push-button while the science stays hand-driven. ARCHITECTURE.md described a documentation-only repository with no code, which stopped being true when the pipeline merged in. It now covers all three parts, the code-and-data separation the pipeline is built around, and the on-disk session layout. Both of this session's own errors are recorded in TODO.md as things not to repeat: measuring a foreground star instead of the galaxy nucleus and booking telescope time to fix it, and telling the user NGC 6744 had not been processed when it had.
149 lines
8.4 KiB
Markdown
149 lines
8.4 KiB
Markdown
# Notes
|
|
|
|
> Working notes, gotchas, environment quirks, and dead ends to avoid. Free form. The
|
|
> point is to save a future session from rediscovering something the hard way.
|
|
|
|
- Refresh the specs snapshot with:
|
|
`curl -sL "https://docs.google.com/spreadsheets/d/1jZWkkjewOuyNC9YzQ8y2d0pO1e4T7EBeysmQMPBVSOk/export?format=csv" -o data/itelescope-telescopes.csv`
|
|
- The CSV has multi-line cells (quoted); line count is not row count. Telescope rows
|
|
start `T<number>,`; observatory header rows have an empty first column.
|
|
- Sensors are 16-bit ADC unless noted (T20's ASI2400C is 14-bit).
|
|
- Free scopes (30 min/day): T68 (Utah), T33 (Siding Spring).
|
|
- Minimum elevation limits come from the support article, not the sheet.
|
|
|
|
## Automation recon (2026-07-18, all verified live)
|
|
|
|
Credentials live OUTSIDE this repo (local claudetemp, never commit them).
|
|
|
|
- go.itelescope.net is ASP.NET WebForms. Login: GET /login.aspx, POST
|
|
UsernameTextBox/PasswordTextBox plus __VIEWSTATE, __VIEWSTATEGENERATOR,
|
|
__EVENTVALIDATION, LoginButton=Login; session cookie then works everywhere.
|
|
- JSON service at /DataService.svc (session-cookie auth): GetAccountStatus
|
|
(balance, plan, renewal), GetMyAvailablePlans (per-plan, per-scope
|
|
AccessDiscount), ApplyPromoCode (POST). More methods likely; enumerate from
|
|
inline $.ajax calls on each page (Scripts/Common.js has none).
|
|
- Home page /default.aspx carries live network status text (per-scope
|
|
offline/online reasons) plus points balance: cheap to poll.
|
|
- Reservation/Overview.aspx links every telescope's own control server:
|
|
https://tNN.itelescope.online/ or http://tNN.itelescope.net:80NN/. These are
|
|
ACP web servers behind HTTP auth accepting the same iTelescope credentials
|
|
(verified read-only on T33: index.asp is a TiddlyWiki info page, min elevation
|
|
20 degrees, 900 s max exposure). Session control and plan submission happen
|
|
here; ACP endpoint mapping is the next step.
|
|
- Reservation/Planner.aspx now redirects to https://lookup.itelescope.online
|
|
(new planning tool, not yet explored).
|
|
- Weather per site: Weather/Weather.aspx?site=UDRO|SSO|DSC|SRO|AC|EYE.
|
|
- Account: Plan-40 (GBP 32.00 per 28 days), renewal 12 Aug 2026, subscribed
|
|
since Apr 2019, balance 2,638 points at recon time.
|
|
|
|
## Rate card (2026-07-18, points per imaging hour, billed per minute)
|
|
|
|
Extracted from MyAccount/UpdatePlan.aspx: each scope row carries
|
|
data-default-rate (dark-of-moon base); effective rate = base x (1 -
|
|
AccessDiscount from DataService GetMyAvailablePlans) x (1 - moon discount,
|
|
server-side table keyed on illumination, not extracted yet). Plan-40 discount
|
|
is 10% on most scopes, 20% on a few, 0% on the newest (T25, T26, T73, T74).
|
|
|
|
Plan-40 rates before moon discount: T2 50, T5 68, T8 130, T10 146, T11 161,
|
|
T14 97, T17 135, T18 81, T20 89, T21 76, T24 186, T25 322, T26 394, T30 161,
|
|
T32 146, T33 0 (free), T59 80, T68 0 (free), T70 119, T71 106, T72 202,
|
|
T73 224, T74 295, T75 202, T80 119.
|
|
|
|
Standouts: T59 at 80/hr is the value big scope (20" CDK, square 37 arcmin
|
|
field, 900 s subs); T5 at 68/hr the cheapest paid; T26 at 394/hr the dearest.
|
|
Billing is per minute of imaging (exposure) time, so overhead is not billed.
|
|
|
|
**RATE CORRECTION (2026-07-20, from the first real bill):** the paragraph above
|
|
underestimates. T59 session 851148: 180 min of exposure, wall clock 00:05:22 to
|
|
03:29:56 (3h24m34s), history "Time Used" 3h24m32s = WALL time, billed 381 points
|
|
at 0% moon discount. That is ~112 pts/hr of wall time or ~127 pts/hr of exposure
|
|
time, vs the 80/hr the card predicted. So either Time Used (wall clock) is what
|
|
gets billed, or the card's data-default-rate is stale. Consequence: estimate
|
|
sessions at rate x reserved-slot-length (slots run ~1.15-1.5x imaging time), i.e.
|
|
multiply old exposure-based estimates by ~1.6 until more bills confirm the model.
|
|
|
|
**RATE CORRECTION 2 (2026-07-21, second bill; SUPERSEDES the x1.6 rule above).**
|
|
T32 session 851344: 120 min of exposure, Time Used 2h25m55s, 25% moon discount,
|
|
billed 160. 160 = 2.00 h x 80 exactly, and the T59 bill is 381 = 3.00 h x 127
|
|
exactly. Both land on the plan's IMAGING minutes to the point, and neither lands
|
|
on wall time. So the wall-clock theory is dead and the original "billed per
|
|
minute of exposure" documentation was right; what is wrong is the card's
|
|
per-scope numbers. Observed effective rates: **T59 127/hr** (card said 80),
|
|
**T32 80/hr even after a 25% moon discount**, implying a base near 107/hr (card
|
|
said 146). The pair looks transposed, but two samples cannot prove it. Rules:
|
|
do NOT apply any blanket multiplier; derive each scope's rate from its first
|
|
real bill; until then treat a card rate as +/-60%; and compute estimates from
|
|
the plan's exposure total, never from the reserved slot length.
|
|
Paid imaging needs no reservation when a scope is idle: same walk-in ACP flow
|
|
as the free scopes, points just deduct. Reservations (to guarantee a slot) go
|
|
through lookup.itelescope.online, which is Cloudflare-protected: Playwright
|
|
needed.
|
|
|
|
|
|
## Processing the downloaded archive (2026-07-21)
|
|
|
|
Gotchas found by running code against the real archive rather than reasoning
|
|
about it. Each one produced a plausible-looking wrong answer, not an error.
|
|
|
|
- **Every frame is delivered twice.** iTelescope ships `calibrated-*` (bias,
|
|
dark and flat applied) and `raw-*` (not). Extracting both puts two copies of
|
|
every frame into the stack, and the counts look merely generous rather than
|
|
wrong: NGC 5128 scanned as 24 luminance frames when it has 12. `CALSTAT` in
|
|
the header is the reliable discriminator - calibrated frames carry it, raw
|
|
frames do not. Only the calibrated archives are extracted; the raw archives
|
|
stay zipped as the untouched originals.
|
|
|
|
- **An interrupted segmented download leaves DIRECTORIES named like files.**
|
|
Cyberduck creates a directory called `something.fit` while fetching, with
|
|
`.cyberducksegment` parts inside. Anything that trusts a file extension will
|
|
try to open a directory as a frame. After a re-mirror over an already
|
|
organised tree there were 28 of them. Discovery, ingest and scan all skip
|
|
them explicitly.
|
|
|
|
- **A re-mirror duplicates rather than replaces.** Mirroring on top of an
|
|
organised tree restored ~1 GB of files to session roots that had already been
|
|
filed into `raw/`. Ingest now removes byte-identical copies as it refiles.
|
|
|
|
- **Two T20 archives are genuinely corrupt** and cannot be extracted
|
|
(`ic1805-...-Ha-...-015` and `-016`). That is real data loss, not a bug: the
|
|
session has 14 of 16 frames. The warning is printed rather than swallowed.
|
|
|
|
- **Extraction triples the disk footprint.** The archive went from 6.1 GB of
|
|
zips to 14 GB extracted, which filled the drive. The TIFFs are 1.69 GB of
|
|
that and the pipeline never reads them - it only uses the FITS.
|
|
|
|
- **Channels do not always share a pixel grid.** NGC 6744 was shot with
|
|
luminance at bin1 (4096x4096) and colour at bin2 (2048x2048). Registration
|
|
has to resample onto a common grid; code that assumes a single frame shape
|
|
will fail on this session and only this session.
|
|
|
|
## Processing lessons from NGC 5128 (2026-07-21)
|
|
|
|
Kept because each is a way to get a confident wrong answer:
|
|
|
|
- **Measure a core with the stars filtered out.** Taking the maximum pixel in a
|
|
box around the frame centre found a foreground star, not the galaxy, and led
|
|
to booking telescope time to fix a saturation problem that did not exist.
|
|
|
|
- **Registration makes sensor defects look like perfect asteroids.** Holding
|
|
the sky still drags anything detector-fixed across the frame on a dead
|
|
straight, constant-rate, constant-brightness track. Hot pixels pass every
|
|
standard quality cut. Requiring the motion to be real in DETECTOR coordinates
|
|
took 141 confident detections to zero.
|
|
|
|
- **Never compare an aperture magnitude to a point-source catalogue without
|
|
checking the source is a point.** A resolved object looks exactly like a
|
|
2.8 magnitude outburst. Carry r50/psf through to the cross-match.
|
|
|
|
- **Never fit a sky background to a field the target fills.** A plane fitted
|
|
around a large galaxy absorbs its halo, measured at -17.9 ADU/px. Fit the
|
|
background and a source model together.
|
|
|
|
- **"No Gaia counterpart" is not a star-rejection test** at extragalactic
|
|
distances: half of Centaurus A's catalogued globular clusters have a Gaia
|
|
source, because a cluster at 3.8 Mpc is a point source. Reject on parallax
|
|
and proper motion instead.
|
|
|
|
- **A coarse background mesh fails in the direction that flatters you.** It
|
|
destroys detection near a bright galaxy while leaving the outer field intact,
|
|
manufacturing a falsely flat radial profile.
|