Scalable updaters

[kepricon]

Hi @slinlee @alexamakarov @xneyder
I'd like to start to discuss Scalable Updaters here.

I think we already have the necessary modules for making scalble from ticket, PR, design and It's a good time to understand the current implementation better and discuss how to make it happen.

for starters, I have a couple of questions about our implementations.

1. We have a health check endpoint for puctuator and updaters from add health endpoint to all running modules #3034.
   Have we used this endpoint from somewhere such as pathmind-admin? @xneyder

2. under the current circumstances(single punctuator and single updater), if puctuator and updaters are dead or stuck for some reason, users can't get updates from webapp.

• In an abnormal situation, puctuator and updaters can be dead
• if progress updates went wrong(memory issue or so) or took a lot of time(big csv files or so)
  these cases can affect users that they can't get updates from webapp.
  Is this the right understanding? @alexamakarov

3. What is our TTL setting for punctuator queue?
   It seems like we use TTL for avoiding duplicates from Allow scaling to multiple updaters #1540 (comment)
   Is this sqs-visibility-timeout? and how much time we set for TTL? @alexamakarov

Let's set a plan(how to monitor puctuator and updaters, how to deploy multiple updaters, dynamic provisioning is available?) later in this thread.

[kepricon]

it's moved to #3257