Skip to content

Conversation

@hanouticelina
Copy link
Contributor

This PR enables structured output (NativeOutput) for Hugging Face models and auto selects the best inference provider based on capabilities.

Changes Made

  • pydantic_ai/models/huggingface.py

    • Added response_format parameter to chat completions for native JSON schema output
    • Maps OutputObjectDefinition to Hugging Face client object when output_mode='native'
    • Supports json_object response format for prompted output
  • pydantic_ai/providers/huggingface.py

    • Added HF Inference Providers model API integration (router.huggingface.co/v1/models) to fetch model capabilities
    • New select_provider() function that picks providers based on:
      1. Both tool calling and structured output.
      2. Either capability
      3. Any live provider
    • model_profile() now dynamically sets supports_tools, supports_json_schema_output, supports_json_object_output from these data.
  • Added a test in tests/models/test_huggingface.py and tests/providers/test_huggingface.py. DISCLAIMER: most of the tests are ai-generated.

cc @DouweM for a review 🙏

base_profile = provider_to_profile[model_prefix](model_suffix)

# fetch model capabilities
router_info = _get_router_info(model_name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I think it breaks expectations too much for the sync Provider.model_profile method to be doing a HTTP request 😬 We could make it a bit less problematic by moving the request to a thread using _utils.run_in_executor but I'm still uncomfortable with this to the point where I can't see myself merging this...

Overriding self._client below is also problematic because these providers are often reused across multiple model instances.

Is there any way we could achieve your goals by waiting to do this capabilities checking / provider selection until Model.request?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants