Azure Blob Storage integration #1189

steffen-biehs · 2025-09-18T10:07:12Z

Checklist

The PR title is short and expressive.
I have updated the CHANGELOG.md. Hints on formulation: Important information first, details second.
I have updated the Deployment Migration Notes Section in the CHANGELOG.md for any configuration / external API changes.
I have updated the Community Edition Postman-Collection if I changed existing APIs or added new APIs (e.g. for Management-API or API-Wrapper)
I have performed a self-review
I understand that my contributions will be contributed under the Contributor License Agreement (CLA) which has to be agreed to on your first contribution.
I have added / adjusted Apache 2.0 license headers to files *.ts, *.kts, *.kt, *.java and marked anything else of relevance in the NOTICE file.

CLAassistant · 2025-09-18T10:07:20Z

All committers have signed the CLA.

konradge · 2025-10-22T14:26:05Z

docs/deployment-guide/goals/local-demo-ce/docker-compose.yaml

      sovity.logging.level: 'DEBUG'
  provider-connector-db:
-    image: 'docker.io/postgres:17-alpine'
+    image: 'docker.io/postgres:16-alpine'


please use postgres:17 for both images

When I'm using v17, which was default in the version I've checked out, errors occur and the database is not working. If you want, I can reproduce the error and paste the error message as a comment

Please make sure, that you have pulled the corresponding docker container postgres:17-alpine. If it still does not work, please provide the logs.

This is the output when I'm running the docker compose file

[+] Running 4/4 ✔ Container local-demo-ce-consumer-ui-1 Recreated 0.2s ✔ Container local-demo-ce-provider-connector-db-1 Recreated 0.2s ✔ Container local-demo-ce-provider-ui-1 Recreated 0.2s ✔ Container local-demo-ce-consumer-connector-db-1 Recreated 0.2s Attaching to consumer-1, consumer-connector-1, consumer-connector-db-1, consumer-ui-1, provider-1, provider-connector-1, provider-connector-db-1, provider-ui-1 consumer-connector-db-1 | consumer-connector-db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization consumer-connector-db-1 | provider-connector-db-1 | provider-connector-db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization provider-connector-db-1 | consumer-connector-db-1 | 2025-10-30 08:19:43.449 UTC [1] FATAL: database files are incompatible with server consumer-connector-db-1 | 2025-10-30 08:19:43.449 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.5. provider-connector-db-1 | 2025-10-30 08:19:43.449 UTC [1] FATAL: database files are incompatible with server provider-connector-db-1 | 2025-10-30 08:19:43.449 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.5. dependency failed to start: container local-demo-ce-consumer-connector-db-1 is unhealthy

I've tested the docker compose at that version and it works, try resetting your volumes: docker compose down -v -t 1

konradge · 2025-10-22T14:30:51Z

frontend/src/app/(authenticated)/data-offers/create/components/data-offer-live-custom-form.tsx

+        <Button
+          dataTestId={'btn-open-azure-blob-select'}
+          type="button"
+          onClick={() => setIsAzureOpen(true)}
+        >
+          Select Blob
+        </Button>


Instead of this button, it would be better if this is another item in the "Type" Dropdown. Then decide, what looks better:

keep it in a dialog

move it directly into the form

What do you mean with "another item in the "Type" Dropdown"?

Ok nvm, I got it. I've added a new Type in the dropdown.

konradge · 2025-10-22T14:37:29Z

frontend/src/components/azure-blob-selector/azure-blob-selector.tsx

+  }, [sasToken, storageSasToken, storageAccountName, containerName, blobName, setSelectionAllowed]);
+
+  return (
+    <div className="p-6 max-w-6xl mx-auto grid gap-4">


To have this integrated in the existing UI, please take a look at other places, where we are using input forms inside dialogs, for example add-counterparty-dialog.tsx

This makes it easier to handle changes of values as well as potential validation.
Additionally, this way the form looks more similar to other forms.

Also, please think about if it would allow an easier flow for the user, if elements that should only be used later are conditionally rendered. For example, show the Blob selection only if an container was selected.

I've updated the code accordingly.

konradge · 2025-10-30T09:13:03Z

...nd/src/app/(authenticated)/data-offers/create/components/data-offer-live-azure-blob-form.tsx

+  ) as DataOfferLiveAzureBlobFormValue;
+
+  useEffect(() => {
+    console.log(`azureDataAddress: ${azureDataAddress}`);


please remove any console.log's

konradge · 2025-10-30T09:14:34Z

frontend/src/app/(authenticated)/data-offers/create/components/data-offer-live-custom-form.tsx

-  form,
-  formKeyDataOfferTypeLive,
-}: {
+                                          form,


run npm run format-all to have a consistent formatting

konradge · 2025-10-30T09:29:36Z

docs/deployment-guide/goals/local-demo-ce/docker-compose.yaml

      sovity.dataspace.kind: 'sovity-mock-iam'
      edc.participant.id: 'consumer'
      sovity.http.protocol: 'http://'
      sovity.print.config: 'true'


Make sure, that the form validation is working. Right now the following error is shown:

konradge · 2025-10-30T09:32:03Z

docs/deployment-guide/goals/local-demo-ce/docker-compose.yaml

      edc.participant.id: 'consumer'
      sovity.http.protocol: 'http://'
      sovity.print.config: 'true'
      sovity.logging.level: 'DEBUG'


I would suggest, that in the list of available containers and blobs, each entry is selectable and will automatically fill out the "Container Name" and "Blob Name" form field respectively

I've added your suggested functionality.

konradge · 2025-10-30T09:36:41Z

frontend/src/components/azure-blob-selector/azure-blob-selector.tsx

+      account: values.storageAccountName,
+      container: values.containerName,
+      blobName: values.blobName,
+      keyName: "key1",


Suggested change

keyName: "key1",

I've made changes to write down a key name (keyName) which has to be the same as used in the vault.

konradge · 2025-10-30T09:37:26Z

docs/deployment-guide/goals/local-demo-ce/docker-compose.yaml

      sovity.logging.level: 'DEBUG'
  provider-connector-db:
-    image: 'docker.io/postgres:17-alpine'
+    image: 'docker.io/postgres:16-alpine'


I've tested the docker compose at that version and it works, try resetting your volumes: docker compose down -v -t 1

richardtreier · 2025-11-10T14:30:02Z

frontend/src/components/azure-blob-selector/azure-blob-selector.tsx

+        <DialogHeader>
+          <DialogTitle>Select Azure Blob</DialogTitle>
+        </DialogHeader>


Shouldn't this be integrated into the form (same UX as surrounding), with the same sections as the other data source types and the building of the data source / data sink happening on submit?

I've based my decision on component AddCounterpartyDialog.tsx:

<DialogHeader> <DialogTitle>Add New Chat</DialogTitle> <DialogDescription> Enter the details of the counterparty you want to chat with. </DialogDescription> </DialogHeader>

richardtreier · 2025-11-10T14:30:46Z

frontend/src/components/azure-blob-selector/azure-blob-selector.tsx

+            name="sasToken"
+            render={({ field }) => (
+              <FormItem>
+                <FormLabel>SAS Token</FormLabel>


Is this the same SAS token as above?

I renamed the Labels accordingly

richardtreier · 2025-11-10T14:31:28Z

frontend/src/components/azure-blob-selector/azure-blob-selector.tsx

+            name="sasToken"
+            render={({ field }) => (
+              <FormItem>
+                <FormLabel>SAS Token</FormLabel>


As @mmaeffert mentioned, an integration of the EDC secrets, wouldn't it require backend code as well, as secrets cannot be expected to be passed into a frontend?

Unfortunately, I don't quite understand what the comment means. The token at that point in the dialog is only used for a query made here to Azure to display the available containers. The same applies to the blobs. After selecting the blob and closing the dialog, the entry expires. Only the storage account key is used to retrieve the blob via the extension. This is entered beforehand via the vault.

steffen-biehs added 3 commits September 17, 2025 14:05

feat: fix wrong db version

cfa9f66

feat: add azure blob storage connection to frontend as a blob selector

fd8039b

feat: delete usesless useEffect

0740970

konradge requested changes Oct 23, 2025

View reviewed changes

feat: add Azure Blob Selection dropdown as data source type

b832119

konradge reviewed Oct 30, 2025

View reviewed changes

richardtreier reviewed Nov 10, 2025

View reviewed changes

steffen-biehs added 2 commits November 11, 2025 14:07

feat: fix PR comments

b43a0b7

feat: remove unused import

ef52ffe

Azure Blob Storage integration #1189

Are you sure you want to change the base?

Azure Blob Storage integration #1189

Uh oh!

Conversation

steffen-biehs commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

CLAassistant commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steffen-biehs Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

steffen-biehs commented Sep 18, 2025 •

edited

Loading

CLAassistant commented Sep 18, 2025 •

edited

Loading

steffen-biehs Nov 11, 2025 •

edited

Loading