Skip to content

Commit fc945a4

Browse files
aaron-y-chenaaron-y-chen
committed
Add Azure Analysis Services model refresh support
1 parent f9b72d8 commit fc945a4

17 files changed

Lines changed: 2040 additions & 0 deletions

File tree

‎providers/microsoft/azure/README.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ PIP package Version required
8686
``microsoft-kiota-abstractions`` ``>=1.9.4,<2.0.0``
8787
``microsoft-kiota-authentication-azure`` ``>=1.9.4,<2.0.0``
8888
``msal-extensions`` ``>=1.3.0``
89+
``requests`` ``>=2.32.0,<3``
8990
========================================== ===================
9091

9192
Optional cross provider package dependencies

‎providers/microsoft/azure/docs/connections/analysis_services.rst‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
or more contributor license agreements. See the NOTICE file
3+
distributed with this work for additional information
4+
regarding copyright ownership. The ASF licenses this file
5+
to you under the Apache License, Version 2.0 (the
6+
"License"); you may not use this file except in compliance
7+
with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
software distributed under the License is distributed on an
13+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
KIND, either express or implied. See the License for the
15+
specific language governing permissions and limitations
16+
under the License.
17+
18+
.. _howto/connection:azure_analysis_services:
19+
20+
Microsoft Azure Analysis Services Connection
21+
============================================
22+
23+
The Microsoft Azure Analysis Services connection type enables model refresh operations through the
24+
`asynchronous refresh REST API <https://learn.microsoft.com/en-us/analysis-services/azure-analysis-services/analysis-services-async-refresh>`__.
25+
26+
The :class:`~airflow.providers.microsoft.azure.hooks.analysis_services.AzureAnalysisServicesHook`,
27+
:class:`~airflow.providers.microsoft.azure.operators.analysis_services.AzureAnalysisServicesRefreshOperator`,
28+
and :class:`~airflow.providers.microsoft.azure.sensors.analysis_services.AzureAnalysisServicesSensor`
29+
use this connection.
30+
31+
Default Connection ID
32+
---------------------
33+
34+
The default connection ID is ``azure_analysis_services_default``.
35+
36+
Configuring the Connection
37+
--------------------------
38+
39+
Region Endpoint
40+
Specify the rollout endpoint in ``host``, for example ``westus.asazure.windows.net``. Do not include
41+
``https://``, the Analysis Services server name, or a path. You can find this endpoint in the server's
42+
full name, such as ``asazure://westus.asazure.windows.net/example-server``.
43+
44+
Client ID
45+
Specify the Microsoft Entra service principal application (client) ID in ``login``.
46+
47+
Client Secret
48+
Specify the service principal client secret in ``password``.
49+
50+
Tenant ID
51+
Specify the Microsoft Entra tenant ID in the ``tenantId`` extra field.
52+
53+
Azure Analysis Services currently requires the service principal to be a server administrator for
54+
asynchronous refresh REST API calls. Add it to the server administrator role using the format
55+
``app:<client-id>@<tenant-id>``. See `Add a service principal to the server administrator role
56+
<https://learn.microsoft.com/en-us/analysis-services/azure-analysis-services/analysis-services-addservprinc-admins>`__.
57+
58+
This connection supports service principal client-secret authentication. Managed identity authentication is
59+
not supported because Azure Analysis Services does not support managed identities for these operations.
60+
61+
.. spelling:word-list::
62+
63+
rollout

‎providers/microsoft/azure/docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,7 @@ PIP package Version required
140140
``microsoft-kiota-abstractions`` ``>=1.9.4,<2.0.0``
141141
``microsoft-kiota-authentication-azure`` ``>=1.9.4,<2.0.0``
142142
``msal-extensions`` ``>=1.3.0``
143+
``requests`` ``>=2.32.0,<3``
143144
========================================== ===================
144145

145146
Optional cross provider package dependencies

‎providers/microsoft/azure/docs/operators/analysis_services.rst‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
or more contributor license agreements. See the NOTICE file
3+
distributed with this work for additional information
4+
regarding copyright ownership. The ASF licenses this file
5+
to you under the Apache License, Version 2.0 (the
6+
"License"); you may not use this file except in compliance
7+
with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
software distributed under the License is distributed on an
13+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
KIND, either express or implied. See the License for the
15+
specific language governing permissions and limitations
16+
under the License.
17+
18+
Microsoft Azure Analysis Services Operators
19+
===========================================
20+
21+
Use these components to start and monitor asynchronous model refresh operations in Microsoft Azure Analysis
22+
Services. Configure an :ref:`Azure Analysis Services connection <howto/connection:azure_analysis_services>`
23+
before using them.
24+
25+
Prerequisite Tasks
26+
------------------
27+
28+
.. include:: /operators/_partials/prerequisite_tasks.rst
29+
30+
.. _howto/operator:AzureAnalysisServicesRefreshOperator:
31+
32+
Start a model refresh
33+
---------------------
34+
35+
Use :class:`~airflow.providers.microsoft.azure.operators.analysis_services.AzureAnalysisServicesRefreshOperator`
36+
with ``wait_for_termination=False`` to start a refresh and immediately return its refresh ID.
37+
38+
.. exampleinclude:: /../tests/system/microsoft/azure/example_azure_analysis_services.py
39+
:language: python
40+
:dedent: 4
41+
:start-after: [START howto_operator_azure_analysis_services_refresh]
42+
:end-before: [END howto_operator_azure_analysis_services_refresh]
43+
44+
By default, ``wait_for_termination=True`` and ``deferrable=False``, so the operator waits synchronously for
45+
the refresh to finish.
46+
47+
Wait with a deferrable operator
48+
-------------------------------
49+
50+
Set ``deferrable=True`` to release the worker slot while the triggerer polls the refresh status. The
51+
``timeout`` parameter limits the entire wait, while ``request_timeout`` limits each REST request.
52+
53+
.. exampleinclude:: /../tests/system/microsoft/azure/example_azure_analysis_services.py
54+
:language: python
55+
:dedent: 4
56+
:start-after: [START howto_operator_azure_analysis_services_deferrable]
57+
:end-before: [END howto_operator_azure_analysis_services_deferrable]
58+
59+
.. _howto/sensor:AzureAnalysisServicesSensor:
60+
61+
Wait with a sensor
62+
------------------
63+
64+
The operator return value can be passed directly to
65+
:class:`~airflow.providers.microsoft.azure.sensors.analysis_services.AzureAnalysisServicesSensor`.
66+
This separates starting the refresh from waiting for it.
67+
68+
.. exampleinclude:: /../tests/system/microsoft/azure/example_azure_analysis_services.py
69+
:language: python
70+
:dedent: 4
71+
:start-after: [START howto_sensor_azure_analysis_services_refresh]
72+
:end-before: [END howto_sensor_azure_analysis_services_refresh]
73+
74+
Azure Analysis Services accepts only one active refresh for a model. A second request returns HTTP 409, so
75+
serialize refresh tasks that target the same model, for example with task dependencies or an Airflow pool.
76+
77+
Reference
78+
---------
79+
80+
For more information, see `Asynchronous refresh with the REST API
81+
<https://learn.microsoft.com/en-us/analysis-services/azure-analysis-services/analysis-services-async-refresh>`__.

‎providers/microsoft/azure/provider.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,12 @@ versions:
122122
- 1.0.0
123123

124124
integrations:
125+
- integration-name: Microsoft Azure Analysis Services
126+
external-doc-url: https://learn.microsoft.com/en-us/analysis-services/azure-analysis-services/
127+
how-to-guide:
128+
- /docs/apache-airflow-providers-microsoft-azure/operators/analysis_services.rst
129+
logo: /docs/integration-logos/Microsoft-Azure.png
130+
tags: [azure]
125131
- integration-name: Microsoft Azure Batch
126132
external-doc-url: https://azure.microsoft.com/en-us/services/batch/
127133
how-to-guide:
@@ -213,6 +219,9 @@ integrations:
213219
tags: [azure]
214220

215221
operators:
222+
- integration-name: Microsoft Azure Analysis Services
223+
python-modules:
224+
- airflow.providers.microsoft.azure.operators.analysis_services
216225
- integration-name: Microsoft Azure Compute
217226
python-modules:
218227
- airflow.providers.microsoft.azure.operators.compute
@@ -254,6 +263,9 @@ operators:
254263
- airflow.providers.microsoft.azure.operators.powerbi
255264

256265
sensors:
266+
- integration-name: Microsoft Azure Analysis Services
267+
python-modules:
268+
- airflow.providers.microsoft.azure.sensors.analysis_services
257269
- integration-name: Microsoft Azure Compute
258270
python-modules:
259271
- airflow.providers.microsoft.azure.sensors.compute
@@ -275,6 +287,9 @@ filesystems:
275287
- airflow.providers.microsoft.azure.fs.msgraph
276288

277289
hooks:
290+
- integration-name: Microsoft Azure Analysis Services
291+
python-modules:
292+
- airflow.providers.microsoft.azure.hooks.analysis_services
278293
- integration-name: Microsoft Azure Compute
279294
python-modules:
280295
- airflow.providers.microsoft.azure.hooks.compute
@@ -327,6 +342,9 @@ hooks:
327342
- airflow.providers.microsoft.azure.hooks.powerbi
328343

329344
triggers:
345+
- integration-name: Microsoft Azure Analysis Services
346+
python-modules:
347+
- airflow.providers.microsoft.azure.triggers.analysis_services
330348
- integration-name: Microsoft Azure Batch
331349
python-modules:
332350
- airflow.providers.microsoft.azure.triggers.batch
@@ -387,6 +405,22 @@ transfers:
387405
python-module: airflow.providers.microsoft.azure.transfers.gcs_to_wasb
388406

389407
connection-types:
408+
- hook-class-name: airflow.providers.microsoft.azure.hooks.analysis_services.AzureAnalysisServicesHook
409+
hook-name: "Azure Analysis Services"
410+
connection-type: azure_analysis_services
411+
ui-field-behaviour:
412+
hidden-fields: ["schema", "port", "extra"]
413+
relabeling:
414+
host: Region Endpoint
415+
login: Client ID
416+
password: Client Secret
417+
placeholders:
418+
host: westus.asazure.windows.net
419+
conn-fields:
420+
tenantId:
421+
label: Tenant ID
422+
schema:
423+
type: ["string", "null"]
390424
- hook-class-name: airflow.providers.microsoft.azure.hooks.base_azure.AzureBaseHook
391425
hook-name: "Azure"
392426
connection-type: azure

‎providers/microsoft/azure/pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ dependencies = [
9999
# It was added in https://github.com/apache/airflow/pull/47990/files
100100
# maybe this should be set from upstream
101101
"msal-extensions>=1.3.0",
102+
"requests>=2.32.0,<3",
102103
]
103104

104105
# The optional dependencies should be modified in place in the generated file

‎providers/microsoft/azure/src/airflow/providers/microsoft/azure/get_provider_info.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,15 @@ def get_provider_info():
2727
"name": "Microsoft Azure",
2828
"description": "`Microsoft Azure <https://azure.microsoft.com/>`__\n",
2929
"integrations": [
30+
{
31+
"integration-name": "Microsoft Azure Analysis Services",
32+
"external-doc-url": "https://learn.microsoft.com/en-us/analysis-services/azure-analysis-services/",
33+
"how-to-guide": [
34+
"/docs/apache-airflow-providers-microsoft-azure/operators/analysis_services.rst"
35+
],
36+
"logo": "/docs/integration-logos/Microsoft-Azure.png",
37+
"tags": ["azure"],
38+
},
3039
{
3140
"integration-name": "Microsoft Azure Batch",
3241
"external-doc-url": "https://azure.microsoft.com/en-us/services/batch/",
@@ -149,6 +158,10 @@ def get_provider_info():
149158
},
150159
],
151160
"operators": [
161+
{
162+
"integration-name": "Microsoft Azure Analysis Services",
163+
"python-modules": ["airflow.providers.microsoft.azure.operators.analysis_services"],
164+
},
152165
{
153166
"integration-name": "Microsoft Azure Compute",
154167
"python-modules": ["airflow.providers.microsoft.azure.operators.compute"],
@@ -203,6 +216,10 @@ def get_provider_info():
203216
},
204217
],
205218
"sensors": [
219+
{
220+
"integration-name": "Microsoft Azure Analysis Services",
221+
"python-modules": ["airflow.providers.microsoft.azure.sensors.analysis_services"],
222+
},
206223
{
207224
"integration-name": "Microsoft Azure Compute",
208225
"python-modules": ["airflow.providers.microsoft.azure.sensors.compute"],
@@ -229,6 +246,10 @@ def get_provider_info():
229246
"airflow.providers.microsoft.azure.fs.msgraph",
230247
],
231248
"hooks": [
249+
{
250+
"integration-name": "Microsoft Azure Analysis Services",
251+
"python-modules": ["airflow.providers.microsoft.azure.hooks.analysis_services"],
252+
},
232253
{
233254
"integration-name": "Microsoft Azure Compute",
234255
"python-modules": ["airflow.providers.microsoft.azure.hooks.compute"],
@@ -299,6 +320,10 @@ def get_provider_info():
299320
},
300321
],
301322
"triggers": [
323+
{
324+
"integration-name": "Microsoft Azure Analysis Services",
325+
"python-modules": ["airflow.providers.microsoft.azure.triggers.analysis_services"],
326+
},
302327
{
303328
"integration-name": "Microsoft Azure Batch",
304329
"python-modules": ["airflow.providers.microsoft.azure.triggers.batch"],
@@ -379,6 +404,21 @@ def get_provider_info():
379404
},
380405
],
381406
"connection-types": [
407+
{
408+
"hook-class-name": "airflow.providers.microsoft.azure.hooks.analysis_services.AzureAnalysisServicesHook",
409+
"hook-name": "Azure Analysis Services",
410+
"connection-type": "azure_analysis_services",
411+
"ui-field-behaviour": {
412+
"hidden-fields": ["schema", "port", "extra"],
413+
"relabeling": {
414+
"host": "Region Endpoint",
415+
"login": "Client ID",
416+
"password": "Client Secret",
417+
},
418+
"placeholders": {"host": "westus.asazure.windows.net"},
419+
},
420+
"conn-fields": {"tenantId": {"label": "Tenant ID", "schema": {"type": ["string", "null"]}}},
421+
},
382422
{
383423
"hook-class-name": "airflow.providers.microsoft.azure.hooks.base_azure.AzureBaseHook",
384424
"hook-name": "Azure",

0 commit comments

Comments
 (0)