Add Azure Analysis Services model refresh support - #71350
Conversation
dabla
left a comment
There was a problem hiding this comment.
The PR adds a well-structured set of components (Hook, Operator, Sensor, Trigger) for Azure Analysis Services model refreshes. The architecture is sound: the trigger correctly offloads synchronous HTTP calls to asyncio.to_thread, deferrable mode is properly implemented, and the test suite is thorough.
Two items must be fixed before merge:
-
_get_base_urlallowsuserinfoand port injection (CWE-918). The host validation explicitly guards against path/query/fragment but silently acceptsuser@host(userinfo) andhost:portinputs, which can redirect the AAS bearer token to an attacker-controlled server. Addingor parsed_host.userinfo or parsed_host.portto the guard closes the gap. A test case should also be added. -
requestsmust not be added as a dependency.httpxis already provided by the provider's transitive dependency onkiota-httpand is the established HTTP client in this provider. The tworequests.get/requests.postcalls are straightforward to replace with thehttpxsync API — see the inline comment for the exact diff.
Drafted-by: Claude Sonnet 4.6 (claude-sonnet-4.6); reviewed by @dabla before posting
| # It was added in https://github.com/apache/airflow/pull/47990/files | ||
| # maybe this should be set from upstream | ||
| "msal-extensions>=1.3.0", | ||
| "requests>=2.32.0,<3", |
There was a problem hiding this comment.
[blocker] requests must not be added as a new dependency -- httpx is already a transitive dependency of the provider via kiota-http and is the established HTTP client across the provider.
Beyond the dependency concern, switching to httpx.AsyncClient eliminates the asyncio.to_thread workaround in the trigger entirely: httpx has native async support, so get_refresh_status and trigger_refresh can be made proper async methods on the hook, and the trigger can await them directly.
pyproject.toml -- remove the added line:
- "requests>=2.32.0,<3",hooks/analysis_services.py -- swap to httpx.AsyncClient and make the methods async:
-import requests
+import httpx- def get_refresh_status(self, server_name: str, database: str, refresh_id: str) -> dict:
+ async def get_refresh_status(self, server_name: str, database: str, refresh_id: str) -> dict:
try:
- response = requests.get(refresh_url, headers=self._get_headers(), timeout=self.request_timeout)
+ async with httpx.AsyncClient() as client:
+ response = await client.get(refresh_url, headers=self._get_headers(), timeout=self.request_timeout)
response.raise_for_status()
- except requests.RequestException as error:
+ except httpx.HTTPError as error:
raise AzureAnalysisServicesRefreshError(...) from error- def trigger_refresh(self, server_name: str, database: str, refresh_type: str) -> dict:
+ async def trigger_refresh(self, server_name: str, database: str, refresh_type: str) -> dict:
try:
- response = requests.post(url, json={"Type": refresh_type}, headers=self._get_headers(), timeout=self.request_timeout)
+ async with httpx.AsyncClient() as client:
+ response = await client.post(url, json={"Type": refresh_type}, headers=self._get_headers(), timeout=self.request_timeout)
response.raise_for_status()
- except requests.RequestException as error:
+ except httpx.HTTPError as error:
raise AzureAnalysisServicesRefreshError(...) from errortriggers/analysis_services.py -- drop asyncio.to_thread and await directly:
-import asyncio
...
- result = await asyncio.to_thread(self.hook.get_refresh_status, ...)
+ result = await self.hook.get_refresh_status(...)The response.raise_for_status(), response.json(), response.headers, and response.text APIs are identical in httpx. Test mocks will need updating accordingly (mock.AsyncMock(spec=httpx.Response)).
Drafted-by: Claude Sonnet 4.6 (claude-sonnet-4.6); reviewed by @dabla before posting
There was a problem hiding this comment.
Oh, if httpx can be used instead of requests, then there's indeed no need to add a new dependency. Let me try to refactor it. 👀
There was a problem hiding this comment.
Refactor done, and passed the e2e test:
I kept asyncio.to_thread() because get_refresh_status() and ClientSecretCredential.get_token() are synchronous and are also used by the Operator, Sensor, and Hook polling paths. Offloading the call keeps the triggerer event loop non-blocking without duplicating the Hook API, and follows the existing pattern in triggers/batch.py.
There was a problem hiding this comment.
Maybe you're not aware but you have also an async variant of ClientSecretCredential, but if we would go that route that would mean we probably need a dedicated async hook, so would leave it with asyncio.to_thread as you mentioned. Could be an idea for a follow up PR in the future.
There was a problem hiding this comment.
@aaron-y-chen I'm just wondering if the operator couldn't be async/deferred only like the MSGraphAsyncOperator? That would mean the hook could be purely written as a native hook, polling and blocking a worker isn't the best approach anyway, hence why the MSGraphAsyncOperator is purely deferred as well. If so, that would mean the code would become even more simpler, especially on the operator side and the hook would be purely async and no more blocking polling in the worker but forcing usage of trigger mechanism, which is the purpose of it, triggers are great at polling without blocking the workers. WDYT?
Thank you for the requested changes, beside above remark, the PR is looking good an clean to me, but before merging I wanted to address above point and see what your opinion is about it?
There was a problem hiding this comment.
Wow, this is my first time learning that there's a purely async operator in Airflow. After doing some survey, I think AAS refresh is a good fit for this approach since it avoids occupying a worker slot while polling. It's a great idea to refactor it to be async-only now that I see the benefits. Please give me some time to implement and verify it :)
There was a problem hiding this comment.
It's a choice, for MSGraph it was an obvious choice as the Python SDK was async only, so it made sense. Here we can choose, hence why I brought it up. Still I think polling should always be done in deferred mode and thus via a trigger to avoid blocking the worker. As this is a new hook/operator, I think it's an opportunity to make that deliberate choice as well.
There was a problem hiding this comment.
I see. After thinking it over, I still believe we can go with the async-only approach since it's a good fit for the AAS refresh scenario.
closes: #51377
Summary
azure_analysis_servicesconnection type using Microsoft Entra service-principal client-secret authentication.Testing
Run the system-test Dag in Breeze against a live Azure Analysis Services instance:
calculaterefreshesThe Dag runs a fire-and-forget Operator, a deferrable-enabled Sensor, and a deferrable-enabled Operator. All three task instances complete successfully on their first try.
Verify both refresh IDs returned by Airflow through the Azure Analysis Services REST API:
437aa86c-…-8169ecalculatesucceededd9f4974e-…-9020calculatesucceededThe Azure screenshot confirms the D1 resource, West US location, and compatibility-level-1200 model.
Was generative AI tooling used to co-author this PR?
Generated-by: [GPT 5.6-sol] following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.