Use platform ?sql param for SQL SELECT

Author: tpellissier

README.md

@@ -2,7 +2,7 @@
 
 A minimal Python SDK to use Microsoft Dataverse as a database for Azure AI Foundry–style apps.
 
-- Read (SQL) — Execute read-only T‑SQL via the McpExecuteSqlQuery Custom API. Returns `list[dict]`.
+- Read (SQL) — Execute constrained read-only SQL via the Dataverse Web API `?sql=` parameter. Returns `list[dict]`.
 - OData CRUD — Thin wrappers over Dataverse Web API (create/get/update/delete).
 - Bulk create — Pass a list of records to `create(...)` to invoke the bound `CreateMultiple` action; returns `list[str]` of GUIDs. If `@odata.type` is absent the SDK resolves the logical name from metadata (cached).
 - Retrieve multiple (paging) — Generator-based `get_multiple(...)` that yields pages, supports `$top` and Prefer: `odata.maxpagesize` (`page_size`).
@@ -13,7 +13,7 @@ A minimal Python SDK to use Microsoft Dataverse as a database for Azure AI Found
 ## Features
 
 - Simple `DataverseClient` facade for CRUD, SQL (read-only), and table metadata.
-- SQL-over-API: T-SQL routed through Custom API endpoint (no ODBC / TDS driver required).
+- SQL-over-API: Constrained T-SQL (single SELECT with limited WHERE/TOP/ORDER BY) via native Web API `?sql=` parameter.
 - Table metadata ops: create simple custom tables with primitive columns (string/int/decimal/float/datetime/bool) and delete them.
 - Bulk create via `CreateMultiple` (collection-bound) by passing `list[dict]` to `create(entity_set, payloads)`; returns list of created IDs.
 - Retrieve multiple with server-driven paging: `get_multiple(...)` yields lists (pages) following `@odata.nextLink`. Control total via `$top` and per-page via `page_size` (Prefer: `odata.maxpagesize`).
@@ -33,12 +33,12 @@ Create and activate a Python 3.13+ environment, then install dependencies:
 python -m pip install -r requirements.txt
 ```
 
-Direct TDS via ODBC is not used; SQL reads are executed via the Custom API over OData.
+Direct TDS via ODBC is not used; SQL reads are executed via the Web API using the `?sql=` query parameter.
 
 ## Configuration Notes
 
 - For Web API (OData), tokens target your Dataverse org URL scope: https://yourorg.crm.dynamics.com/.default. The SDK requests this scope from the provided TokenCredential.
-- For complete functionalities, please use one of the PREPROD BAP environments, otherwise McpExecuteSqlQuery might not work.
+	(Preprod environments may surface newest SQL subset capabilities sooner than production.)
 
 ### Configuration (DataverseConfig)
 
@@ -48,7 +48,7 @@ Pass a `DataverseConfig` or rely on sane defaults:
 from dataverse_sdk import DataverseClient
 from dataverse_sdk.config import DataverseConfig
 
-cfg = DataverseConfig()  # defaults: language_code=1033, sql_api_name="McpExecuteSqlQuery"
+cfg = DataverseConfig()  # defaults: language_code=1033
 client = DataverseClient(base_url="https://yourorg.crm.dynamics.com", config=cfg)
 
 # Optional HTTP tunables (timeouts/retries)
@@ -68,7 +68,7 @@ The quickstart demonstrates:
 - Creating, reading, updating, and deleting records (OData)
 - Bulk create (CreateMultiple) to insert many records in one call
 - Retrieve multiple with paging (contrasting `$top` vs `page_size`)
-- Executing a read-only SQL query
+- Executing a read-only SQL query (Web API `?sql=`)
 
 ## Examples
 
@@ -103,7 +103,7 @@ updated = client.update("accounts", account_id, {"telephone1": "555-0199"})
 # Delete
 client.delete("accounts", account_id)
 
-# SQL (read-only) via Custom API
+# SQL (read-only) via Web API `?sql=`
 rows = client.query_sql("SELECT TOP 3 accountid, name FROM account ORDER BY createdon DESC")
 for r in rows:
 	print(r.get("accountid"), r.get("name"))
@@ -239,7 +239,7 @@ Notes:
 - Passing a list of payloads to `create` triggers bulk create and returns `list[str]` of IDs.
 - Use `get_multiple` for paging through result sets; prefer `select` to limit columns.
 - For CRUD methods that take a record id, pass the GUID string (36-char hyphenated). Parentheses around the GUID are accepted but not required.
-- SQL is routed through the Custom API named in `DataverseConfig.sql_api_name` (default: `McpExecuteSqlQuery`).
+* SQL queries are executed directly against entity set endpoints using the `?sql=` parameter. Supported subset only (single SELECT, optional WHERE/TOP/ORDER BY, alias). Unsupported constructs will be rejected by the service.
 
 ### Pandas helpers
 

examples/quickstart.py

@@ -286,15 +286,17 @@ def print_line_summaries(label: str, summaries: list[dict]) -> None:
 except Exception as e:
 	print(f"Update/verify failed: {e}")
 	sys.exit(1)
-# 4) Query records via SQL Custom API
-print("Query (SQL via Custom API):")
+# 4) Query records via SQL (Web API ?sql=)
+print("Query (SQL via Web API ?sql=):")
 try:
 	import time
 	pause("Execute SQL Query")
 
 	def _run_query():
-		log_call(f"client.query_sql(\"SELECT TOP 2 * FROM {logical} ORDER BY {attr_prefix}_amount DESC\")")
-		return client.query_sql(f"SELECT TOP 2 * FROM {logical} ORDER BY {attr_prefix}_amount DESC")
+		cols = f"{id_key}, {code_key}, {amount_key}, {when_key}"
+		query = f"SELECT TOP 2 {cols} FROM {logical} ORDER BY {attr_prefix}_amount DESC"
+		log_call(f"client.query_sql(\"{query}\") (Web API ?sql=)")
+		return client.query_sql(query)
 
 	def _retry_if(ex: Exception) -> bool:
 		msg = str(ex) if ex else ""
@@ -317,7 +319,7 @@ def _retry_if(ex: Exception) -> bool:
 		)
 	print_line_summaries("TDS record summaries (top 2 by amount):", tds_summaries)
 except Exception as e:
-	print(f"SQL via Custom API failed: {e}")
+	print(f"SQL query failed: {e}")
 
 # Pause between SQL query and retrieve-multiple demos
 pause("Retrieve multiple (OData paging demos)")

examples/quickstart_pandas.py

@@ -183,8 +183,8 @@ def backoff_retry(op, *, delays=(0, 2, 5, 10, 20), retry_http_statuses=(400, 403
 	print(f"Update/verify failed: {e}")
 	sys.exit(1)
 
-# 4) Query records via SQL Custom API
-print("(Pandas) Query (SQL via Custom API):")
+# 4) Query records via SQL (Web API ?sql=)
+print("(Pandas) Query (SQL via Web API ?sql=):")
 try:
 	# Try singular logical name first, then plural entity set, with short backoff
 	import time
@@ -196,7 +196,9 @@ def backoff_retry(op, *, delays=(0, 2, 5, 10, 20), retry_http_statuses=(400, 403
 	df_rows = None
 	for name in candidates:
 		def _run_query():
-			return PANDAS.query_sql_df(f"SELECT TOP 3 * FROM {name} ORDER BY createdon DESC")
+			id_key = f"{logical}id"
+			cols = f"{id_key}, {attr_prefix}_code, {attr_prefix}_amount, {attr_prefix}_when"
+			return PANDAS.query_sql_df(f"SELECT TOP 3 {cols} FROM {name} ORDER BY {attr_prefix}_amount DESC")
 		def _retry_if(ex: Exception) -> bool:
 			msg = str(ex) if ex else ""
 			return ("Invalid table name" in msg) or ("Invalid object name" in msg)
@@ -211,7 +213,7 @@ def _retry_if(ex: Exception) -> bool:
 except SystemExit:
 	pass
 except Exception as e:
-	print(f"SQL via Custom API failed: {e}")
+	print(f"SQL query failed: {e}")
 
 # 5) Delete record
 print("(Pandas) Delete (OData via Pandas wrapper):")

src/dataverse_sdk/client.py

@@ -164,19 +164,23 @@ def get_multiple(
             page_size=page_size,
         )
 
-    # SQL via Custom API
+    # SQL via Web API sql parameter
     def query_sql(self, tsql: str):
-        """Execute a read-only SQL query via the configured Custom API.
+        """Execute a read-only SQL query using the Dataverse Web API `?sql=` capability.
+
+        The query must follow the currently supported subset: single SELECT with optional WHERE,
+        TOP (integer), ORDER BY (columns only), and simple alias after FROM. Example:
+            ``SELECT TOP 3 accountid, name FROM account ORDER BY name DESC``
 
         Parameters
         ----------
         tsql : str
-            A SELECT-only T-SQL statement (e.g., ``"SELECT TOP 3 * FROM account"``).
+            Supported single SELECT statement.
 
         Returns
         -------
         list[dict]
-            Rows as a list of dictionaries.
+            Result rows (empty list if none).
         """
         return self._get_odata().query_sql(tsql)
 

src/dataverse_sdk/config.py

@@ -7,7 +7,6 @@
 @dataclass(frozen=True)
 class DataverseConfig:
     language_code: int = 1033
-    sql_api_name: str = "McpExecuteSqlQuery"
 
     # Optional HTTP tuning (not yet wired everywhere; reserved for future use)
     http_retries: Optional[int] = None
@@ -19,7 +18,6 @@ def from_env(cls) -> "DataverseConfig":
         # Environment-free defaults
         return cls(
             language_code=1033,
-            sql_api_name="McpExecuteSqlQuery",
             http_retries=None,
             http_backoff=None,
             http_timeout=None,

src/dataverse_sdk/odata.py

@@ -29,6 +29,8 @@ def __init__(self, auth, base_url: str, config=None) -> None:
         )
         # Cache: entity set name -> logical name (resolved via metadata lookup)
         self._entityset_logical_cache = {}
+        # Cache: logical name -> entity set name (reverse lookup for SQL endpoint)
+        self._logical_to_entityset_cache: dict[str, str] = {}
 
     def _headers(self) -> Dict[str, str]:
         """Build standard OData headers with bearer auth."""
@@ -318,41 +320,105 @@ def _do_request(url: str, *, params: Optional[Dict[str, Any]] = None) -> Dict[st
 
     # --------------------------- SQL Custom API -------------------------
     def query_sql(self, tsql: str) -> list[dict[str, Any]]:
-        """Execute a read-only T-SQL query via the configured Custom API.
+        """Execute a read-only SQL query using the Dataverse Web API `?sql=` capability.
+
+        The platform supports a constrained subset of SQL SELECT statements directly on entity set endpoints:
+            GET /{entity_set}?sql=<encoded select statement>
+
+        This client extracts the logical table name from the query, resolves the corresponding
+        entity set name (cached) and invokes the Web API using the `sql` query parameter.
 
         Parameters
         ----------
         tsql : str
-            SELECT-style Dataverse-supported T-SQL (read-only).
+            Single SELECT statement within supported subset.
 
         Returns
         -------
         list[dict]
-            Rows materialised as list of dictionaries (empty list if no rows).
+            Result rows (empty list if none).
 
         Raises
         ------
+        ValueError
+            If the SQL is empty or malformed, or if the table logical name cannot be determined.
         RuntimeError
-            If the Custom API response is missing the expected ``queryresult`` property or type is unexpected.
+            If metadata lookup for the logical name fails.
         """
-        payload = {"querytext": tsql}
-        headers = self._headers()
-        api_name = self.config.sql_api_name
-        url = f"{self.api}/{api_name}"
-        r = self._request("post", url, headers=headers, json=payload)
+        if not isinstance(tsql, str) or not tsql.strip():
+            raise ValueError("tsql must be a non-empty string")
+        sql = tsql.strip()
+
+        # Naive parse: find token after FROM (ignore brackets); stop at whitespace/newline
+        # Example: SELECT name FROM account AS a WHERE a.name LIKE 'Acme%'
+        m = re.search(r"from\s+([a-zA-Z0-9_]+)\b", sql, flags=re.IGNORECASE)
+        if not m:
+            raise ValueError("Unable to determine table logical name from SQL (expected 'FROM <logical>').")
+        logical_candidate = m.group(1)
+        logical = logical_candidate.lower()
+
+        entity_set = self._entity_set_from_logical(logical)
+        # Issue GET /{entity_set}?sql=<query>
+        headers = self._headers().copy()
+        url = f"{self.api}/{entity_set}"
+        params = {"sql": sql}
+        r = self._request("get", url, headers=headers, params=params)
+        try:
+            r.raise_for_status()
+        except Exception as e:
+            # Attach response snippet to aid debugging unsupported SQL patterns
+            resp_text = None
+            try:
+                resp_text = r.text[:500] if getattr(r, 'text', None) else None
+            except Exception:
+                pass
+            detail = f" SQL query failed (status={getattr(r, 'status_code', '?')}): {resp_text}" if resp_text else ""
+            raise RuntimeError(str(e) + detail) from e
+        try:
+            body = r.json()
+        except ValueError:
+            return []
+        if isinstance(body, dict):
+            value = body.get("value")
+            if isinstance(value, list):
+                # Ensure dict rows only
+                return [row for row in value if isinstance(row, dict)]
+        # Fallbacks: if body itself is a list
+        if isinstance(body, list):
+            return [row for row in body if isinstance(row, dict)]
+        return []
+
+    # ---------------------- Entity set resolution -----------------------
+    def _entity_set_from_logical(self, logical: str) -> str:
+        """Resolve entity set name (plural) from a logical (singular) name using metadata.
+
+        Caches results for subsequent SQL queries.
+        """
+        if not logical:
+            raise ValueError("logical name required")
+        cached = self._logical_to_entityset_cache.get(logical)
+        if cached:
+            return cached
+        url = f"{self.api}/EntityDefinitions"
+        logical_escaped = self._escape_odata_quotes(logical)
+        params = {
+            "$select": "LogicalName,EntitySetName",
+            "$filter": f"LogicalName eq '{logical_escaped}'",
+        }
+        r = self._request("get", url, headers=self._headers(), params=params)
         r.raise_for_status()
-        data = r.json()
-        if "queryresult" not in data:
-            raise RuntimeError(f"{api_name} response missing 'queryresult'.")
-        q = data["queryresult"]
-        if q is None:
-            parsed = []
-        elif isinstance(q, str):
-            s = q.strip()
-            parsed = [] if not s else json.loads(s)
-        else:
-            raise RuntimeError(f"Unexpected queryresult type: {type(q)}")
-        return parsed
+        try:
+            body = r.json()
+            items = body.get("value", []) if isinstance(body, dict) else []
+        except ValueError:
+            items = []
+        if not items:
+            raise RuntimeError(f"Unable to resolve entity set for logical name '{logical}'.")
+        es = items[0].get("EntitySetName")
+        if not es:
+            raise RuntimeError(f"Metadata response missing EntitySetName for logical '{logical}'.")
+        self._logical_to_entityset_cache[logical] = es
+        return es
 
     # ---------------------- Table metadata helpers ----------------------
     def _label(self, text: str) -> Dict[str, Any]:

src/dataverse_sdk/odata_pandas_wrappers.py

@@ -14,7 +14,7 @@
   DataFrame summarizing success/failure.
 * get_ids: fetches a set of ids returning a DataFrame of the merged JSON
   objects (outer union of keys). Missing keys are NaN.
-* query_sql_df: runs a SQL query via Custom API and returns the result rows as
+* query_sql_df: runs a SQL query via the Web API `?sql=` parameter and returns the result rows as
   a DataFrame (empty DataFrame if no rows).
 
 Edge cases & behaviors:
@@ -140,8 +140,9 @@ def get_ids(self, entity_set: str, ids: Sequence[str] | pd.Series | pd.Index, se
 
     # --------------------------- Query SQL -------------------------------
     def query_sql_df(self, tsql: str) -> pd.DataFrame:
-        """Execute a SQL query via Custom API and return a DataFrame.
+        """Execute a SQL query via the Dataverse Web API `?sql=` parameter and return a DataFrame.
 
+        The statement must adhere to the supported subset (single SELECT, optional WHERE/TOP/ORDER BY, no joins).
         Empty result -> empty DataFrame (columns inferred only if rows present).
         """
         rows: Any = self._c.query_sql(tsql)