"14 UNAVAILABLE: failed to connect to all addresses" exception is thrown by language worker

[alrod]

On restarting a worker language channel (worker crash or timeout) we need to check for grpc server healthiness and shutdown the host itself if the grpc server is unhealthy.

CRI1
CRI2
https://stackoverflow.com/questions/59823424/grpc-14-unavailable-failed-to-connect-to-all-addresses

[pragnagopa]

Documentation on Channel State API: https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md#channel-state-api

@fabiocav - please find an owner.

[TeplrGuy]

@pragnagopa @fabiocav can we please get an ETA on this? Even an estimation will suffice. Thanks a lot team.

[fabiocav]

@TeplrGuy this has been assigned to sprint 114. We'll continue to update the issue as we make progress.

[kshyju]

@alrod Looked into the functions logs for the error mentioned in the attached CRIs (14 UNAVAILABLE: failed to connect to all addresses) and I can see that this error is reported from the node.js language worker. Queried logs for the last 3 days in CUS and all the entries are coming from node.js worker.

Channel is a connection abstraction on the client side. A channel instance is needed on the client side to establish a connection to a grpc host/server so that a grpc client/stub instance can be created for further communication to the server. On a node.js client. the channel state check should be done using getConnectivityState or watchConnectivityState APIs. (Link to docs)

I think the next action item here is to investigate the node.js language worker implementation to see why it is getting the connectivity error. I did a quick scan on the node.js worker repo and I do not see the above-mentioned APIs are being used. I tried to repro this error locally with a node.js language worker (v14.16.0), but was unsuccessful in doing so(this could be a race condition issue).

Transferring this to node.js worker repo for next steps.

[alrod]

Reopening the issue, this fix will help to recover function host from "failed to connect to all addresses" grpc error:
Azure/azure-functions-host#7979

We still can not reproduce "14 UNAVAILABLE: failed to connect to all addresses" error but the fix mentioned above will improve automatic recovering after the error.

[alrod]

Fixing race during language worker start:
Azure/azure-functions-host@5fe7711