Skip to content

Target-based scaling instance churn increased ~30–100x since 2026-08-18 with no corresponding code/config change (Flex Consumption plan, Node.js) #11943

Description

Investigative information

Timestamp: 2026-08-18T15:30:51Z (onset), ongoing through 2026-08-25 (further escalation); representative example below at 2026-08-24T12:00:00Z
Function App version: 4.1052.300.26370 (FunctionsExtensionVersion=~4)
Function App name: zoho-functions
Function name(s): Timer A (15-min schedule), Timer B (hourly), Timer C (4h schedule with internal catch-up/due-check logic) — see Invocation IDs below
Invocation ID (all for 2026-08-24T12:00:00Z, same scheduled minute, three separate instances):
Timer A: f64c0039-93cf-4c6d-9747-6c6b8fc1b8b9
Timer B: ec58b4f4-c238-4594-b8fc-65dafb572c0e
Timer C: db5259cc-96bf-4bf2-b8d6-8065d0fb654c
Region: Germany West Central
Hosting plan: Flex Consumption, Instance memory 512MB, Always-ready instances = 0 for all triggers, on-demand maximum instance count 100.

Repro steps

No deterministic repro steps — this is a spontaneous, sustained change in production platform behavior, not triggered by a specific code path. Pattern: multiple timer-triggered functions scheduled for the same minute (e.g. 12:00 UTC) each get dispatched to their own separate instance (distinct HostInstanceId, not sharing a process), and each instance is drained and its worker process terminated shortly after completing its single invocation.

Expected behavior

Based on our own baseline (2026-08-01 to 08-16): occasional instance recycling (DrainMode mode enabled, category Microsoft.Azure.WebJobs.Host.DrainModeManager) at 1–4 occurrences/day, with drain cycles rarely resulting in a full Language Worker Process exited (worker process typically reused/reset internally instead).

Actual behavior

Since 2026-08-18, DrainMode mode enabled frequency increased ~30–100x and has stayed elevated:

  • 2026-08-01 to 08-16 (baseline): 1–4/day
  • 2026-08-18: 92/day
  • 2026-08-19 to 08-24: 238–320/day
  • 2026-08-25: 186/day (partial day)

Additionally, starting 2026-08-25, the proportion of drain cycles ending in a full Language Worker Process exited jumped sharply:

  • 2026-08-24: 320 DrainMode events, 24 worker exits → 7.5% exit-to-drain ratio
  • 2026-08-25 (partial): 186 DrainMode events, 153 worker exits → 82.3% exit-to-drain ratio

We confirmed via a join of all 153 exit events on 2026-08-25 against DrainMode mode enabled (same cloud_RoleInstance, within 5 minutes prior) that 100% were preceded by an orderly drain sequence — no unresponsive-process kills, just far more of them completing all the way to process termination. The 2026-08-25 escalation began between 21:00–23:00 UTC on 08-24 (3 → 11 exits/hour) and reached full elevated levels by 00:00 UTC on 08-25 (18/hour).

Example instance lifecycle for the three invocations listed above (2026-08-24, 12:00 UTC): Timer A runs 18.3s, completes successfully, then ~19s later its instance receives DrainMode mode enabled, cleanly stops all listeners (~100ms), Job host stopped. Timer C completes a trivial 446ms no-op, sits idle 38s, then is drained the same way, ending in Language Worker Process exited. Pid=64. Timer B completes successfully (9.5s), drained ~30s later via the same clean sequence. No exception, no timeout, no unresponsiveness in any case.

We ruled out on our side: our own deployments/merges (none in the relevant windows, or on unrelated feature branches not deployed to production), configuration/Activity Log changes, Azure Service Health incidents, host runtime version changes (single consistent version across 2093 host starts, 08-17 to 08-24), and memory pressure (Max Memory working set constantly ~1GB against configured 512MB for 30+ days with no change around either onset date — pre-existing, unrelated to the timing). A manual restart of the Function App (2026-08-19) did not reduce the churn frequency afterward.

Known workarounds

None found. Manual Function App restart did not help.

Related information

Programming language: Node.js 22, @azure/functions v4 programming model
Bindings used: Timer trigger (NCRONTAB schedules), also affects HTTP-triggered functions on the same app (elevated error rate on one HTTP function from Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content, consistent with connections aborting during the same instance churn)
Question for the maintainers: is a ~30–100x increase in target-based-scaling instance churn, and the jump from ~7.5% to ~82% in drain cycles ending in full process termination, expected under some condition on Flex Consumption with Always-ready instances = 0, or does this indicate a regression in the scaling controller?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions