deploy: self-update re-exec guard in deploy.sh
When deploy.sh itself changes in the commit being pulled, the bash process is still running the OLD script from memory — git reset --hard updated the file on disk but the in-memory instructions are stale. This bit the 2026-04-09 Dalidou deploy: the old pre-build-sha Step 2 ran against fresh source, so the container started with ATOCORE_BUILD_SHA="unknown" instead of the real commit. Manual re-run fixed it, but the class of bug will re-emerge every time deploy.sh itself changes. Fix (Step 1.5): - After git reset --hard, sha1 the running script ($0) and the on-disk copy at $APP_DIR/deploy/dalidou/deploy.sh - If they differ, export ATOCORE_DEPLOY_REEXECED=1 and exec into the fresh copy so Step 2 onward runs under the new script - The sentinel env var prevents recursion - Skipped in dry-run mode, when $0 isn't readable, or when the on-disk script doesn't exist yet Docs (docs/dalidou-deployment.md): - New "The deploy.sh self-update race" troubleshooting section explaining the root cause, the Step 1.5 mechanism, what the log output looks like, and how to opt out Verified syntax and dry-run. 219/219 tests still passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -160,6 +160,45 @@ else
|
||||
run "git clone --branch '$BRANCH' '$GIT_REMOTE' '$APP_DIR'"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Step 1.5: self-update re-exec guard
|
||||
# ---------------------------------------------------------------------
|
||||
#
|
||||
# When deploy.sh itself changes in the commit we just pulled, the bash
|
||||
# process running this script is still executing the OLD deploy.sh
|
||||
# from memory — git reset --hard updated the file on disk but our
|
||||
# in-memory instructions are stale. That's exactly how the first
|
||||
# 2026-04-09 Dalidou deploy silently wrote "unknown" build_sha: old
|
||||
# Step 2 logic ran against fresh source. Detect the mismatch and
|
||||
# re-exec into the fresh copy so every post-update run exercises the
|
||||
# new script.
|
||||
#
|
||||
# Guard rails:
|
||||
# - Only runs when $APP_DIR exists, holds a git checkout, and a
|
||||
# deploy.sh exists there (i.e. after Step 1 succeeded).
|
||||
# - Uses a sentinel env var ATOCORE_DEPLOY_REEXECED=1 to make sure
|
||||
# we only re-exec once, never recurse.
|
||||
# - Skipped in dry-run mode (no mutation).
|
||||
# - Skipped if $0 isn't a readable file (bash -c pipe inputs, etc.).
|
||||
|
||||
if [ "$DRY_RUN" != "1" ] \
|
||||
&& [ -z "${ATOCORE_DEPLOY_REEXECED:-}" ] \
|
||||
&& [ -r "$0" ] \
|
||||
&& [ -f "$APP_DIR/deploy/dalidou/deploy.sh" ]; then
|
||||
ON_DISK_HASH="$(sha1sum "$APP_DIR/deploy/dalidou/deploy.sh" 2>/dev/null | awk '{print $1}')"
|
||||
RUNNING_HASH="$(sha1sum "$0" 2>/dev/null | awk '{print $1}')"
|
||||
if [ -n "$ON_DISK_HASH" ] \
|
||||
&& [ -n "$RUNNING_HASH" ] \
|
||||
&& [ "$ON_DISK_HASH" != "$RUNNING_HASH" ]; then
|
||||
log "Step 1.5: deploy.sh changed in the pulled commit; re-exec'ing"
|
||||
log " running script hash: $RUNNING_HASH"
|
||||
log " on-disk script hash: $ON_DISK_HASH"
|
||||
log " re-exec -> $APP_DIR/deploy/dalidou/deploy.sh"
|
||||
export ATOCORE_DEPLOY_REEXECED=1
|
||||
exec bash "$APP_DIR/deploy/dalidou/deploy.sh" "$@"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Step 2: capture build provenance to pass to the container
|
||||
# ---------------------------------------------------------------------
|
||||
|
||||
@@ -140,6 +140,36 @@ If you see `{"status": "unavailable", "fail_open": true}` from the
|
||||
client, the first thing to check is whether the base URL resolves
|
||||
from where you're running the client.
|
||||
|
||||
### The deploy.sh self-update race
|
||||
|
||||
When `deploy.sh` itself changes in the commit being pulled, the
|
||||
first run after the update is still executing the *old* script from
|
||||
the bash process's in-memory copy. `git reset --hard` updates the
|
||||
file on disk, but the running bash has already loaded the
|
||||
instructions. On 2026-04-09 this silently shipped an "unknown"
|
||||
`build_sha` because the old Step 2 (which predated env-var export)
|
||||
ran against fresh source.
|
||||
|
||||
`deploy.sh` now detects this: Step 1.5 compares the sha1 of `$0`
|
||||
(the running script) against the sha1 of
|
||||
`$APP_DIR/deploy/dalidou/deploy.sh` (the on-disk copy) after the
|
||||
git reset. If they differ, it sets `ATOCORE_DEPLOY_REEXECED=1` and
|
||||
`exec`s the fresh copy so the rest of the deploy runs under the new
|
||||
script. The sentinel env var prevents infinite recursion.
|
||||
|
||||
You'll see this in the logs as:
|
||||
|
||||
```text
|
||||
==> Step 1.5: deploy.sh changed in the pulled commit; re-exec'ing
|
||||
==> running script hash: <old>
|
||||
==> on-disk script hash: <new>
|
||||
==> re-exec -> /srv/storage/atocore/app/deploy/dalidou/deploy.sh
|
||||
```
|
||||
|
||||
To opt out (debugging, for example), pre-set
|
||||
`ATOCORE_DEPLOY_REEXECED=1` before invoking `deploy.sh` and the
|
||||
self-update guard will be skipped.
|
||||
|
||||
### Deployment drift detection
|
||||
|
||||
`/health` reports drift signals at three increasing levels of
|
||||
|
||||
Reference in New Issue
Block a user