data quality checks for variant data type

[STEFANOVIVAS]

It is possible to apply data quality checks to columns with variant data type? The reference guide only mention Struct, Map Type and Array Type. In case it's not, could we implement it?

[ghanse]

It works!

from databricks.labs.dqx.engine import DQEngine
from databricks.labs.dqx.rule import DQRowRule
from databricks.labs.dqx import check_funcs
from databricks.sdk import WorkspaceClient 

df = (
  spark
  .range(5)
  .selectExpr(
    "id",
    "to_variant_object(map('key1', '', 'key2', 'b')) as example_var",
  )
)

checks = [
  DQRowRule(
    name="is_missing_variant_key1",
    criticality="error",
    check_func=check_funcs.is_not_empty,
    column="variant_get(example_var, '$.key1', 'string')"
  ),
]

engine = DQEngine(WorkspaceClient())
display(engine.apply_checks(df, checks))

[ghanse]

This should also work for rules defined using configuration/YAML.

You can set column to be a PySpark function (e.g. try_variant_get) when defining the rules in code.

[STEFANOVIVAS]

Great! It works with yaml files too! Thanks