Pipelines PGFS with Azure Innovation Release
This documentation covers the current Innovation Release of
EDB Postgres AI. You may also want the docs for the current LTS version.
Overview: Azure
PGFS supports Azure Blob Storage and Azure Data Lake Storage Gen2 (ADLS Gen2).
The general syntax for using Azure is:
select pgfs.create_storage_location( 'storage_location_name', 'prefix://bucket_name', options => '{}', credentials => '{}' );
PGFS supports the following prefixes and formats to indicate an Azure Blob Storage or an ADLS Gen2 bucket:
abfs://<container>/<path> abfs://<file_system>@<account_name>.dfs.core.windows.net/<path> abfss://<container>/<path> abfss://<file_system>@<account_name>.dfs.core.windows.net/<path> az://<container>/<path> adl://<container>/<path> azure://<container>/<path> https://<account>.dfs.core.windows.net https://<account>.blob.core.windows.net https://<account>.blob.core.windows.net/<container>
Settings for the options argument in JSON format settings
| Option | Description |
|---|---|
account_name | Unique, high-level identifier for your entire storage resource within Azure. |
container_name | The container name. |
Settings for the credentials argument in JSON format
Specify either account_key for a shared access key credential, or client-id, client-secret, and tenant-id for client credentials.
| Option | Description |
|---|---|
account_key | Primary secret used when setting up a Shared Access Key credential. |
client-id | Unique identifier assigned to your application registration in Azure Active Directory. |
client-secret | The key generated for the application registration. |
tenant-id | The unique identifier for your specific Azure Active Directory instance |
Example: create a storage location in Azure Blob storage using a shared access key credential
SELECT pgfs.create_storage_location('edb_ai_example_images', 'https://edb-account.blob.core.windows.net', NULL, '{ "account_name": "edb-account", "container_name": "ai-images" }', '{ "account_key": "<your azure storage key>" }' );
Example: create a storage location in Azure Blob storage using client credentials
SELECT pgfs.create_storage_location('edb_ai_example_images', 'https://edb-account.blob.core.windows.net', NULL, '{ "account_name": "edb-account", "container_name": "ai-images" }', '{ "client_id": "<your azure clientID>", "client_secret": "<your azure client secret>", "tenant_id": "<your azure tenantID>" }' );