Add asynchronous task execution

[mxsrc]

This adds support for asynchronous background tasks. FastAPI's native support for these is limited: The builtin background tasks are anonymous background workers. They have no identity that can be used to communicate ongoing process, and they are lost on restarts.

Instead, this changeset introduces the use of celery, which is the go-to for these tasks in the Python ecosystem. The deployment includes an additional worker container that picks up tasks. The broker and result backend use the existing postgres database, alternatives are available.

To confirm the new approach works as intended, the resizing is converted to use celery tasks. Future changes will convert other potentially long-running tasks, in particular anything concerning branch lifecycle management, and ideally monitoring and recurring tasks like backup cleanup and scheduled snapshot creation.

The resize operation now starts individual resizes for each of the affected resources, awaits all results and writes the new values to the database. The ID of the ongoing resize task is stored into the database entity, and the RESIZING status is derived from this. The task itself cleans this up upon completion.

[boddumanohar]

Overall this is a very solid re-implementation of resize functionality.

2 things:

• when the database resize is complete we are setting the size as input request size. Not the value from the PVC.
• Earlier we check for FileSystemResizeSuccessful event on the PVC only then mark the resize as complete. Although this state is no longer relevant with simplyblock/simplyblock-csi#295. The field .capacity.storage tells that Block device is resized. But doesn't give visibility over the status of Filesystem Resize. My idea with PR: simplyblock/autoscaling#31 was to emit an event so that filesystem resize is complete was complete. Which the Vela controller can listen on. I am still testing the changes. Alternatively, the controller can also set annotation that resize status is complete.

[mxsrc]

Earlier we check for FileSystemResizeSuccessful event on the PVC only then mark the resize as complete. Although this state is no longer relevant with simplyblock/simplyblock-csi#295. The field .capacity.storage tells that Block device is resized. But doesn't give visibility over the status of Filesystem Resize. My idea with PR: simplyblock/autoscaling#31 was to emit an event so that filesystem resize is complete was complete. Which the Vela controller can listen on. I am still testing the changes. Alternatively, the controller can also set annotation that resize status is complete.

My understanding was that the event we now emit in the autoscaler matches the one we are looking for right now. So in that case things should work just as before, but I might be wrong about this?

[boddumanohar]

Yes things in this PR are just as the way it is in the latest dev version of controller. But right now vela-controller has no visibility over whether database's filesystem resize is complete or not. Earlier we had this visibility because of the bug in simplyblock/simplyblock-csi#295. Meaning both latest dev and the changes in this PR has no visible over filesystem resize status.

But the current implementation doesn't look for events, it only looks for status.capacity.storage. So we might need to add an additional check to also make sure that filesystem is resized. I can take this as a backlog item and fix it in an other PR. if that's okay for you.

[boddumanohar]

I just checked the current branches in the PR deployment. Although the tests have passed, All the branches are stuck in resizing state.

vela-pr667-01kmfs4jwv90xcp3ga7ftg3rfd   Active   20h
vela-pr667-01kmg9e8ryn55t61r3s92684tm   Active   15h
vela-pr667-01kmg9qen0wfv1p8cwhhap1n2x   Active   15h
vela-pr667-01kmgajr6pqvdevt4c0q4y9c3c   Active   15h