Resolve confusion about "batching" support

[markmc]

Capturing some observations from #157

1. Pipeline has "batching" support - it can shard the dataset and spawn an instance of the pipeline for each shard - batch_num_workers and batch_size
2. LLMBlock has "batching" support - it can request multiple chat completions from the OpenAI server using the n argument - num_instructions_to_generate

In ilab we disable (1) with llama-cpp by passing batch_size=None - see instructlab/instructlab#346

In LLMBlock we disable (2) with llama-cpp with the server_supports_batched which checks whether the n argument works

Resolve:

• Do we want to call both of these "batching"
• Do we want to different ways of handling backend-specific capabilities?
• Should the library be trying to probe the backend for its capabilities, or should the library user give it information about the backend?
• server_supports_batched should be a property on PipelineContext, not something we set on the OpenAI client object

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.

[bbrowning]

This has confused me more than once - some tech debt to tackle here for sure.

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant!