Before submitting
Affected area
Other / not user-facing
Layer
Infrastructure / CI
Summary
Around 5-10 times per day, we're getting a 60 second downtime where no ECS tasks are live. Reported by https://community.doenet.org/t/503-and-504-errors/154.
Context
With our current load, our backend ECS tasks are getting sigkilled 5-10 times per day due to running out of memory. The memory issue isn't a Node/Javascript "Out of heap memory" issue, it's a container memory issues - something outside of Node is causing this.
I've verified this by looking through our load balancer logs, the ECS tasks events, and some ECS logs.
When we're running only one ECS task (which is most of the time), this causes a 60 second downtime of the backend.
This is probably a Prisma issue, since the version we're currently using runs a somewhat heavy Rust binary outside of the V8 engine. We should upgrade to a later version of Prisma which doesn't use Rust.
Before submitting
Affected area
Other / not user-facing
Layer
Infrastructure / CI
Summary
Around 5-10 times per day, we're getting a 60 second downtime where no ECS tasks are live. Reported by https://community.doenet.org/t/503-and-504-errors/154.
Context
With our current load, our backend ECS tasks are getting sigkilled 5-10 times per day due to running out of memory. The memory issue isn't a Node/Javascript "Out of heap memory" issue, it's a container memory issues - something outside of Node is causing this.
I've verified this by looking through our load balancer logs, the ECS tasks events, and some ECS logs.
When we're running only one ECS task (which is most of the time), this causes a 60 second downtime of the backend.
This is probably a Prisma issue, since the version we're currently using runs a somewhat heavy Rust binary outside of the V8 engine. We should upgrade to a later version of Prisma which doesn't use Rust.