Trainer topic subscription idempotency.

[harrykeightley]

The trainer subscription can fire the same training event multiple times, which would start different training run throughs on cloud run (wasting resources, overwriting models etc.).

This is not a priority to fix at this point, but something to think about.

Copying over a reply below from Nick.

Maybe I just put it as a flag on the firestore Model that training has begun?

If firebase has atomic transactions this might work, otherwise you're just going to have the same run condition in a different place. The ideal scenario is to have idempotent training jobs, where the end state isn't affected by multiple runs (either because it realised that a training job with the given inputs has already occurred and short-circuits, or it does the training again in a non-destructive way - both of these typically involve hash comparisons between inputs/outputs). This isn't easy to pull off, and probably isn't a priority at the moment, but something to think about.

Originally posted by @nicklambourne in #89 (comment)