Harden against OOM: activate mod_wsgi daemon mode + maximum-requests + swap + memory alert (refs Gluejar/regluit#1078) - #44
Merged
Conversation
… (refs Gluejar/regluit#1078) Root cause of the 2026-06-22 unglue.it outage: a mod_wsgi worker grew to ~2.8GB and the OOM-killer fired; with NO swap the host thrashed into a hard wedge (sshd couldn't fork, apache stopped) and needed an AWS reboot. Three layers of mitigation: 1. apache.conf.j2: add maximum-requests (default 500) + inactivity-timeout (300) to WSGIDaemonProcess so daemon workers recycle and a leak/large response can't grow unbounded. This is the real fix; the */20 apache-restart cron was only a band-aid (kept for now; can retire once this bakes). 2. swap.yml: add a {{ swap_file_size_mb|default(4096) }}MB swapfile + low vm.swappiness (10) as a pressure cushion so spikes degrade instead of wedging. 3. monitoring.yml + mem_alert.sh: cron emails admins when MemAvailable drops below threshold (default 12%) — early warning before an OOM, with top consumers. All tunable via vars; ansible.posix.sysctl confirmed available; syntax-check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011dumTGwdDfpJJMGC4ThisJ
…e-restart cron Forensics on the 2026-06-22 OOM (Gluejar/regluit#1078): the */20 'service apache2 restart' cron did NOT reap mod_wsgi daemon processes (systemd left-over-process), so restarts spawned fresh workers while bloated old ones survived -> ~13.3GB across ~6 orphaned workers -> OOM (no swap). Remove that cron (state: absent); proper recycling now via WSGIDaemonProcess maximum-requests. Also: setup-prod.yml include_role -> import_role and tags on swap/apache-config/ monitoring/cron-cleanup tasks, so the hardening can be applied as a minimal subset: ansible-playbook -i hosts setup-prod.yml --tags oom-hardening [--check --diff] (verified via --list-tasks; full untagged run still lists all 77 tasks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011dumTGwdDfpJJMGC4ThisJ
…ap + safer alert Codex caught that WSGIDaemonProcess was defined but UNUSED (no WSGIProcessGroup), so the app ran EMBEDDED in apache mpm children and maximum-requests was inert — verified on the box (no process-group anywhere; embedded children already back to ~3GB hours post-reboot). The real fix: - apache.conf.j2: add 'WSGIProcessGroup regluit' + 'process-group=regluit' on WSGIScriptAlias so the app runs in the 2 managed daemon processes where maximum-requests actually recycles them. Add graceful-timeout=30. - swap.yml: make idempotent — guard mkswap with a blkid swap-signature check and swapon with a /proc swapon --show check (the previous string-match failed_when was fragile). - mem_alert.sh: set the dedup flag ONLY after a successful sendmail (was touching it unconditionally -> a transient mail failure would suppress all future alerts); also validate MemAvailable, not just MemTotal. syntax-check + bash -n clean; --tags oom-hardening selects the expected tasks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011dumTGwdDfpJJMGC4ThisJ
rdhyee
changed the title
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident: unglue.it OOM-wedged 2026-06-22; recovered via AWS reboot. Root cause pinned by log forensics and a Codex review that caught a showstopper in the first draft.
The real root cause (corrected after Codex review)
WSGIDaemonProcess regluitwas defined but never used — there was noWSGIProcessGroup, so Django ran embedded in the apache mpm_event children, which never recycle (MaxConnectionsPerChild=0). The*/20 service apache2 restartcron didn't reap them (systemd left-over processes), so workers piled up to ~13.3 GB across ~6 processes → OOM (zero swap). Verified on the box: noprocess-groupanywhere; embedded children back to ~3 GB hours after the reboot.maximum-requestsalone would have been inert (it only governs daemon processes).Fixes
WSGIProcessGroup regluit+process-group=regluitonWSGIScriptAliasso the app runs in the 2 managed daemon processes — wheremaximum-requests=500(+inactivity-timeout=300,graceful-timeout=30) actually recycles them and frees memory cleanly.apache-restartcron (state: absent) — redundant + harmful.vm.swappiness=10— cushion (box had none). Idempotent (blkid/swapon --showguards).Codex review — all blockers addressed
maximum-requestsrecycles with no restarts.Apply (tagged, minimal — no full re-provision)
Verify after:
curl200;psshows 2(wsgi:regluit)daemon procs + small mpm children; swap active; old cron gone.Underlying leak — queued separately
maximum-requestscaps the symptom; a worker reaching ~3 GB is still abnormal. Endpoint memory profiling (the suspected unbounded querysets on/free,/bypub,/pid, OPDS) is tracked in Gluejar/regluit#1189. Bot load in #1173.🤖 Generated with Claude Code