Map Natural Language DQ Rules to the AI-Assisted Quality Checks Generation

[hyqel-razali]

Hi peeps,

Forgive me if this is missed out as I'm not entirely sure how to source it out but I have difficulties trying to map each of my 1000 AI-Assisted Quality Checks Generation to the NLP DQ rules that I feed generator.generate_dq_rules_ai_assisted(user_input=user_input)

Is there a way that we can somehow have a parameter within the yml file that tracks that so its easier for users to extract that info? I think it'll be very valuable to have this feature implemented as the same NLP rules are shown to our Data Owners/Stewards and it'll be great if I can prove that it is being generated by the AI.

As an example,

• check:
  name: AvailabilityDate cannot be blank or null
  arguments:
  column: AvailabilityDate
  function: is_not_null
  criticality: error

[hyqel-razali]

Found an existing solution!! I've managed to find a work around whereby I've mapped the NLP rules through the use of user_metadata parameter in the output of the AI-assisted DQ generation.

• check:
  arguments:
  column: AvailabilityDate
  function: is_not_null
  criticality: error
  user_metadata:
  Completeness: AvailabilityDate cannot be blank or null

[ghanse]

Yes, user_metadata is great for this. Other potential use cases:

• Mapping checks to data dimensions
• Mapping checks to assigned data steward teams
• Mapping checks to data contracts (e.g. with a contract file name)
• Mapping checks to data producers and/or consumers