Missing account history data in ElasticSearch if node is restarted while syncing

[abitmore]

Bug Description

The elasticsearch plugin does not send data in memory to ElasticSearch when the node is shutting down. If the node is still syncing (30+ seconds behind the current head block time), after it is restarted, it will resume from a potential much later block number.

Related:

• If the node is already in sync, the elasticsearch plugin sends data to ElasticSearch on every block, so no issue.
• The missing data can be fixed with a replay with an appropriate starting block number.
• This issue does not exist in the es_objects plugin (version test-6.1.2).

blocksights/blocksights-open-explorer#7 might be caused by this issue.

This might be the answer to #2549 (comment)

... it is strange that the error didn't pop up on the first sync, but popped up on replay.

#2493 is also somewhat related.

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)
  • ElasticSearch History plugin

Host Environment
Please provide details about the host environment. Much of this information can be found running: witness_node --version.

• BitShares Version: test-6.1.2

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation