feat(dataset): Enable versioned access and S3 credential caching

[briangreunke]

Summary

This PR introduces the ability to load datasets by a specific version. It also implements S3 credential caching within the DatasetManager to optimize performance by reducing redundant API calls for temporary S3 credentials. Additionally, it includes minor refactoring for clarity and consistency across the dataset loading workflow.

Changes

Features

• Versioned Dataset Loading:
  • The ApiClient.get_dataset_access_credentials method now accepts a version parameter, allowing for the retrieval of S3 credentials specific to a dataset version.
  • The DatasetDownloadRequest model has been updated to require a version string, ensuring that all download requests are version-aware. The default "latest" is now handled at the call site within DatasetManager.get_remote_load_uri.
  • DatasetManager.get_remote_load_uri and DatasetManager.get_s3_config have been modified to pass the requested dataset version to the API, enabling version-specific URI generation and credential retrieval.
  • The load_dataset function in dreadnode/dataset.py now leverages these version-aware DatasetManager methods to support loading datasets by their specified version.
• S3 Credential Caching:
  • The DatasetManager.get_remote_load_uri method now stores the pafs.S3FileSystem object and its associated credentials' expiration time (_credentials_expiry). This lays the groundwork for reusing an active S3 filesystem, reducing the overhead of repeatedly fetching new temporary S3 credentials from the API.

Refactoring & Improvements

• Parameter Renaming: The fsm parameter in dreadnode/dataset.py::load_dataset and its call site in dreadnode/main.py::Dreadnode.load_dataset_from_disk has been renamed to dataset_manager for improved clarity and consistency.
• Logging Adjustment: The log message for automatic version bumps in _ensure_version_bump has been changed from print_info to print_warning to better reflect the significance of data changes triggering a new version.
• API Parameter Handling: The ApiClient.download_dataset method has been slightly refactored to pass the version query

---

Generated Summary:

• Updated get_dataset_access_credentials method in ApiClient to now require a version parameter.
• Modified parameter handling for dataset access credentials API request to include the version explicitly in the request.
• Changed DatasetDownloadRequest model to require a version string instead of allowing None.
• Refactored various instances of DatasetManager to reflect the new naming convention for the parameter from fsm to dataset_manager for clarity.
• Changed logging statement for version bump from print_info to print_warning to better reflect the significance of the version change.
• Adjusted the get_remote_load_uri to handle version defaulting internally, ensuring backward compatibility with existing code.
• Overall, these changes improve clarity and enforce the requirement of a version when accessing dataset credentials, which may impact consumer code interacting with these methods.

This summary was generated with ❤️ by rigging