Using Spline Agent for Data Lineage with Azure Blob Storage

[mohini-tripathi]

Spline agent seems to be a robust solution for capturing data lineage in Spark applications. However, our data is stored in Azure Blob Storage, and I am curious to know if anyone has experience or insights on integrating Spline with Azure Blob Storage.

Specifically, I would like to understand:

1. Is the Spline agent compatible with Azure Blob Storage for capturing data lineage?
2. What are the necessary configurations or steps required to set up Spline with Azure Blob Storage?

Any guidance, documentation, or examples would be greatly appreciated. Thank you in advance for your assistance!

[wajda]

As long as we are talking about Spark, Spline Agent has only limited dependency on the underlaying storage technology. In majority cases it should just work, but at the end of the day it all depends on a concrete Spark Connector that is used to connect to the underlaying storage, in this case Azure Blob Storage. I have not tried it myself, but the easiest way to figure it out is just to try it.
Configure Spark Spline agent for your Spark job normally as the doc suggests, and try to execute some writes. If you don't get lineage, look at the Spark driver logs to see if there is any error message, and then take it from there.