Updated OOD_WORKFLOW_SYNC_KEY usage in project manager tutorial

[harshit-soora]

https://osc.github.io/ood-documentation-test/develop/

Added details on possible use of OOD_WORKFLOW_SYNC_KEY, not restricted to examples provided.
Feature was added in PR OSC/ondemand#5348

Let me know if you can think of any other use-cases, I will add those too.

[harshit-soora]

@Bubballoo3 Please help to review this

[Bubballoo3]

I think this will work better if we weave it into the existing tutorial rather than add a section beneath it. First things first, we need to update the existing tutorial to use the OOD_WORKFLOW_SYNC_KEY instead of the COUNT variable. We can move your first section explaining the basic premise/behavior at the top of that section (where we currently have 'We also add a COUNT variable so that we can control when data is overwritten, an essential consideration if you plan to have more than one instance of a workflow run simultaneously.'). We will also need to add the step of enabling the sync key to that section.

I think it would also be helpful to demonstrate that the sync key overrides the value set by the launcher, which is useful for debugging. That is, we configure an environment variable for OOD_WORKFLOW_SYNC_KEY in both launchers, with a default value like 'test'. That way we can run each launcher independently to ensure that they work as expected. When they run in the workflow, that 'test' value gets overridden with the actual synchronization value. I think that process is super important to understand for debugging, so it would be great to include there.

I think the tempdir and lockfile examples are alright, but could be covered more briefly given that they are independent from the tangible example used in the rest of the tutorial. Maybe we can add a section like 'Alternative synchronization approaches' and then cover those briefly without presenting any code (so that all the code contributes to the main example).

Finally, I really like the conclusion of 'The Project Manager is designed to be flexible, so if the examples above aren't applicable to your needs, you can always design your own systems and conventions that fit your specific situation.' so let's keep that as the closing line.

[Bubballoo3]

Now that I think about it a little more, I am not sure that the lock file example you provide is actually the best way to do that. Since dependent launchers won't run if their parent job fails, it would be better to promote a pattern where a script checks its own output before finishing, and fails if something has gone wrong. That's a minor point, but it might be safer to just cover tmpdirs to show that you can name directories with the sync key in addition to files (which will be demonstrated in the main tutorial content). Since we already show files being named with the key, I would rather avoid the question of what people end up using those synchronized files for (leave it up to them).