joining the data lineage records

[rajendra-t]

Spline UI currently gives lineage records associated per databricks notebook, however if we have multiple inputs/outputs in one notebook, it creates multiple records arangodb, Is there a capability to join them and create end to end relationship /lineage e.g. dataset1-> dataset2->dataset3?

[wajda]

Yes, it is the main objective of Spline, and it's done automatically. If you click on an execution event record, the first you see is a end-to-end lineage overview that is comprised of jobs and data source nodes. Currently the UI can only show the history part of end-to-end lineage starting at the write event that you clicked on, but the develop version of REST API can do both backward and forward lineage (the UI has yet to come for this view).
Keep in mind however that in order to get the end-to-end view it is important for the read and write data source URI to match exactly including case of letters.

[rajendra-t]

Thanks @wajda for the feedback. Can you please share the link and details of the developed version of REST API for getting backward and forward lineage?

[rajendra-t]

Hi @wajda, do you have any feedback on the developed version of REST API for getting backward and forward lineage?

[wajda]

There are two REST API calls with identical signature but opposite meaning:

1. GET /lineage-overview?eventId=123
2. GET /impact-overview?eventId=123

Considering the following example

With respect to the portion of data produced by a given execution event (say ID 123) of the job Foo, written in the destination C, the /lineage-overview gives you the upper portion of the graph (the jobs and data sources that affects the data produced by the given event), while the /impact-overview gives you the lower part of the graph (the jobs and data sources that are affected by the given execution event)