Create an Azure Data Factory using REST API - Azure Data Factory (2023)

  • Article
  • 12 minutes to read
  • Version 1
  • Current version

APPLIES TO: Create an Azure Data Factory using REST API - Azure Data Factory (1)Azure Data Factory Create an Azure Data Factory using REST API - Azure Data Factory (2)Azure Synapse Analytics

Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation. Using Azure Data Factory, you can create and schedule data-driven workflows (called pipelines) that can ingest data from disparate data stores, process/transform the data by using compute services such as Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics, and Azure Machine Learning, and publish output data to data stores such as Azure Synapse Analytics for business intelligence (BI) applications to consume.

This quickstart describes how to use REST API to create an Azure Data Factory. The pipeline in this data factory copies data from one location to another location in an Azure blob storage.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

Note

We recommend that you use the Azure Az PowerShell module to interact with Azure. See Install Azure PowerShell to get started. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.

  • Azure subscription. If you don't have a subscription, you can create a free trial account.
  • Azure Storage account. You use the blob storage as source and sink data store. If you don't have an Azure storage account, see the Create a storage account article for steps to create one.
  • Create a blob container in Blob Storage, create an input folder in the container, and upload some files to the folder. You can use tools such as Azure Storage Explorer to connect to Azure Blob storage, create a blob container, upload input file, and verify the output file.
  • Install Azure PowerShell. Follow the instructions in How to install and configure Azure PowerShell. This quickstart uses PowerShell to invoke REST API calls.
  • Create an application in Azure Active Directory following this instruction. Make note of the following values that you use in later steps: application ID, clientSecrets, and tenant ID. Assign application to "Contributor" role at either subscription or resource group level.

Note

For Sovereign clouds, you must use the appropriate cloud-specific endpoints for ActiveDirectoryAuthority and ResourceManagerUrl (BaseUri).You can use PowerShell to easily get the endpoint Urls for various clouds by executing “Get-AzEnvironment | Format-List”, which will return a list of endpoints for each cloud environment.

(Video) 40. Working with Token based REST API in Azure Data Factory

Set global variables

  1. Launch PowerShell. Keep Azure PowerShell open until the end of this quickstart. If you close and reopen, you need to run the commands again.

    Run the following command, and enter the user name and password that you use to sign in to the Azure portal:

    Connect-AzAccount

    Run the following command to view all the subscriptions for this account:

    Get-AzSubscription

    Run the following command to select the subscription that you want to work with. Replace SubscriptionId with the ID of your Azure subscription:

    Select-AzSubscription -SubscriptionId "<SubscriptionId>"
  2. Run the following commands after replacing the places-holders with your own values, to set global variables to be used in later steps.

    $tenantID = "<your tenant ID>"$appId = "<your application ID>"$clientSecrets = "<your clientSecrets for the application>"$subscriptionId = "<your subscription ID to create the factory>"$resourceGroupName = "<your resource group to create the factory>"$factoryName = "<specify the name of data factory to create. It must be globally unique.>"$apiVersion = "2018-06-01"

Authenticate with Azure AD

Run the following commands to authenticate with Azure Active Directory (AAD):

$credentials = Get-Credential -UserName $appIdConnect-AzAccount -ServicePrincipal -Credential $credentials -Tenant $tenantID

You will be prompt to input the password, use the value in clientSecrets variable.

If you need to get the access token

GetToken

Create a data factory

Run the following commands to create a data factory:

(Video) REST API Fetching only Required Records and Writing to Azure SQL DB in Azure Data Factory

$body = @"{ "location": "East US", "properties": {}, "identity": { "type": "SystemAssigned" }}"@$response = Invoke-AzRestMethod -SubscriptionId ${subscriptionId} -ResourceGroupName ${resourceGroupName} -ResourceProviderName Microsoft.DataFactory -ResourceType "factories" -Name ${factoryName} -ApiVersion ${apiVersion} -Method PUT -Payload ${body}$response.Content 

Note the following points:

  • The name of the Azure Data Factory must be globally unique. If you receive the following error, change the name and try again.

    Data factory name "ADFv2QuickStartDataFactory" is not available.
  • For a list of Azure regions in which Data Factory is currently available, select the regions that interest you on the following page, and then expand Analytics to locate Data Factory: Products available by region. The data stores (Azure Storage, Azure SQL Database, etc.) and computes (HDInsight, etc.) used by data factory can be in other regions.

Here is the sample response content:

{ "name":"<dataFactoryName>", "identity":{ "type":"SystemAssigned", "principalId":"<service principal ID>", "tenantId":"<tenant ID>" }, "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>", "type":"Microsoft.DataFactory/factories", "properties":{ "provisioningState":"Succeeded", "createTime":"2019-09-03T02:10:27.056273Z", "version":"2018-06-01" }, "eTag":"\"0200c876-0000-0100-0000-5d6dcb930000\"", "location":"East US", "tags":{ }}

Create linked services

You create linked services in a data factory to link your data stores and compute services to the data factory. In this quickstart, you only need create one Azure Storage linked service as both copy source and sink store, named "AzureStorageLinkedService" in the sample.

Run the following commands to create a linked service named AzureStorageLinkedService:

Replace <accountName> and <accountKey> with name and key of your Azure storage account before executing the commands.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/linkedservices/AzureStorageLinkedService?api-version=${apiVersion}"$body = @"{ "name":"AzureStorageLinkedService", "properties":{ "annotations":[ ], "type":"AzureBlobStorage", "typeProperties":{ "connectionString":"DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>" } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/linkedservices/AzureStorageLinkedService", "name":"AzureStorageLinkedService", "type":"Microsoft.DataFactory/factories/linkedservices", "properties":{ "annotations":[ ], "type":"AzureBlobStorage", "typeProperties":{ "connectionString":"DefaultEndpointsProtocol=https;AccountName=<accountName>;" } }, "etag":"07011a57-0000-0100-0000-5d6e14a20000"}

Create datasets

You define a dataset that represents the data to copy from a source to a sink. In this example, you create two datasets: InputDataset and OutputDataset. They refer to the Azure Storage linked service that you created in the previous section. The input dataset represents the source data in the input folder. In the input dataset definition, you specify the blob container (adftutorial), the folder (input), and the file (emp.txt) that contain the source data. The output dataset represents the data that's copied to the destination. In the output dataset definition, you specify the blob container (adftutorial), the folder (output), and the file to which the data is copied.

Create InputDataset

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/InputDataset?api-version=${apiVersion}"$body = @"{ "name":"InputDataset", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":{ "type":"AzureBlobStorageLocation", "fileName":"emp.txt", "folderPath":"input", "container":"adftutorial" } } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response 

Here is the sample output:

(Video) 32. Copy data from REST API which sends response in Pages using Azure data factory

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/datasets/InputDataset", "name":"InputDataset", "type":"Microsoft.DataFactory/factories/datasets", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":"@{type=AzureBlobStorageLocation; fileName=emp.txt; folderPath=input; container=adftutorial}" } }, "etag":"07011c57-0000-0100-0000-5d6e14b40000"}

Create OutputDataset

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/OutputDataset?api-version=${apiVersion}"$body = @"{ "name":"OutputDataset", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":{ "type":"AzureBlobStorageLocation", "folderPath":"output", "container":"adftutorial" } } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/datasets/OutputDataset", "name":"OutputDataset", "type":"Microsoft.DataFactory/factories/datasets", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":"@{type=AzureBlobStorageLocation; folderPath=output; container=adftutorial}" } }, "etag":"07013257-0000-0100-0000-5d6e18920000"}

Create a pipeline

In this example, this pipeline contains one Copy activity. The Copy activity refers to the "InputDataset" and the "OutputDataset" created in the previous step as input and output.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartPipeline?api-version=${apiVersion}"$body = @"{ "name": "Adfv2QuickStartPipeline", "properties": { "activities": [ { "name": "CopyFromBlobToBlob", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }, "enableStaging": false }, "inputs": [ { "referenceName": "InputDataset", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "OutputDataset", "type": "DatasetReference" } ] } ], "annotations": [] }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/pipelines/Adfv2QuickStartPipeline", "name":"Adfv2QuickStartPipeline", "type":"Microsoft.DataFactory/factories/pipelines", "properties":{ "activities":[ "@{name=CopyFromBlobToBlob; type=Copy; dependsOn=System.Object[]; policy=; userProperties=System.Object[]; typeProperties=; inputs=System.Object[]; outputs=System.Object[]}" ], "annotations":[ ] }, "etag":"07012057-0000-0100-0000-5d6e14c00000"}

Create pipeline run

In this step, you trigger a pipeline run. The pipeline run ID returned in the response body is used in later monitoring API.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartPipeline/createRun?api-version=${apiVersion}"$response = Invoke-AzRestMethod -Path ${path} -Method POST $response.content 

Here is the sample output:

{ "runId":"04a2bb9a-71ea-4c31-b46e-75276b61bafc"}

You can also get the runId by using following command

($response.content | ConvertFrom-Json).runId

Parameterize your pipeline

You can create pipeline with parameters. In the following example, we will create an input dataset and an output dataset that can take input and output filenames as parameters given to the pipeline.

Create parameterized input dataset

Define a parameter called strInputFileName , and use it as file name for dataset.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/ParamInputDataset?api-version=${apiVersion}"$body = @"{ "name": "ParamInputDataset", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strInputFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strInputFileName", "type": "Expression" }, "folderPath": "input", "container": "adftutorial" } } }, "type": "Microsoft.DataFactory/factories/datasets"}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/datasets/ParamInputDataset", "name": "ParamInputDataset", "type": "Microsoft.DataFactory/factories/datasets", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strInputFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strInputFileName", "type": "Expression" }, "folderPath": "input", "container": "adftutorial" } } }, "etag": "00000000-0000-0000-0000-000000000000"}

Create parameterized output dataset

Define a parameter called strOutputFileName , and use it as file name for dataset.

(Video) How to Read the data from Rest API and write to Blob Storage in ADF By using REST Connector in ADF

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/ParamOutputDataset?api-version=${apiVersion}"$body = @"{ "name": "ParamOutputDataset", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strOutPutFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strOutPutFileName", "type": "Expression" }, "folderPath": "output", "container": "adftutorial" } } }, "type": "Microsoft.DataFactory/factories/datasets"}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/datasets/ParamOutputDataset", "name": "ParamOutputDataset", "type": "Microsoft.DataFactory/factories/datasets", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strOutPutFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strOutPutFileName", "type": "Expression" }, "folderPath": "output", "container": "adftutorial" } } }, "etag": "00000000-0000-0000-0000-000000000000"}

Create parameterized pipeline

Define a pipeline with two pipeline level parameters: strParamInputFileName and strParamOutputFileName. Then link these two parameters to the strInputFileName and strOutputFileName parameters of the datasets.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartParamPipeline?api-version=${apiVersion}"$body = @"{ "name": "Adfv2QuickStartParamPipeline", "properties": { "activities": [ { "name": "CopyFromBlobToBlob", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }, "enableStaging": false }, "inputs": [ { "referenceName": "ParamInputDataset", "type": "DatasetReference", "parameters": { "strInputFileName": { "value": "@pipeline().parameters.strParamInputFileName", "type": "Expression" } } } ], "outputs": [ { "referenceName": "ParamOutputDataset", "type": "DatasetReference", "parameters": { "strOutPutFileName": { "value": "@pipeline().parameters.strParamOutputFileName", "type": "Expression" } } } ] } ], "parameters": { "strParamInputFileName": { "type": "String" }, "strParamOutputFileName": { "type": "String" } } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelines/Adfv2QuickStartParamPipeline", "name": "Adfv2QuickStartParamPipeline", "type": "Microsoft.DataFactory/factories/pipelines", "properties": { "activities": [ { "name": "CopyFromBlobToBlob", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }, "enableStaging": false }, "inputs": [ { "referenceName": "ParamInputDataset", "type": "DatasetReference", "parameters": { "strInputFileName": { "value": "@pipeline().parameters.strParamInputFileName", "type": "Expression" } } } ], "outputs": [ { "referenceName": "ParamOutputDataset", "type": "DatasetReference", "parameters": { "strOutPutFileName": { "value": "@pipeline().parameters.strParamOutputFileName", "type": "Expression" } } } ] } ], "parameters": { "strParamInputFileName": { "type": "String" }, "strParamOutputFileName": { "type": "String" } } }, "etag": "5e01918d-0000-0100-0000-60d569a90000"}

Create pipeline run with parameters

You can now specify values of the parameter at the time of creating the pipeline run.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartParamPipeline/createRun?api-version=${apiVersion}"$body = @"{ "strParamInputFileName": "emp2.txt", "strParamOutputFileName": "aloha.txt"}"@$response = Invoke-AzRestMethod -Path ${path} -Method POST -Payload $body$response.content$runId = ($response.content | ConvertFrom-Json).runId

Here is the sample output:

{"runId":"ffc9c2a8-d86a-46d5-9208-28b3551007d8"}

Monitor pipeline

  1. Run the following script to continuously check the pipeline run status until it finishes copying the data.

     $path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelineruns/${runId}?api-version=${apiVersion}" while ($True) { $response = Invoke-AzRestMethod -Path ${path} -Method GET $response = $response.content | ConvertFrom-Json Write-Host "Pipeline run status: " $response.Status -foregroundcolor "Yellow" if ( ($response.Status -eq "InProgress") -or ($response.Status -eq "Queued") -or ($response.Status -eq "In Progress") ) { Start-Sleep -Seconds 10 } else { $response | ConvertTo-Json break } }

    Here is the sample output:

     { "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelineruns/ffc9c2a8-d86a-46d5-9208-28b3551007d8", "runId": "ffc9c2a8-d86a-46d5-9208-28b3551007d8", "debugRunId": null, "runGroupId": "ffc9c2a8-d86a-46d5-9208-28b3551007d8", "pipelineName": "Adfv2QuickStartParamPipeline", "parameters": { "strParamInputFileName": "emp2.txt", "strParamOutputFileName": "aloha.txt" }, "invokedBy": { "id": "9c0275ed99994c18932317a325276544", "name": "Manual", "invokedByType": "Manual" }, "runStart": "2021-06-25T05:34:06.8424413Z", "runEnd": "2021-06-25T05:34:13.2936585Z", "durationInMs": 6451, "status": "Succeeded", "message": "", "lastUpdated": "2021-06-25T05:34:13.2936585Z", "annotations": [], "runDimension": {}, "isLatest": true }
  2. Run the following script to retrieve copy activity run details, for example, size of the data read/written.

     $path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelineruns/${runId}/queryActivityruns?api-version=${apiVersion}" while ($True) { $response = Invoke-AzRestMethod -Path ${path} -Method POST $responseContent = $response.content | ConvertFrom-Json $responseContentValue = $responseContent.value Write-Host "Activity run status: " $responseContentValue.Status -foregroundcolor "Yellow" if ( ($responseContentValue.Status -eq "InProgress") -or ($responseContentValue.Status -eq "Queued") -or ($responseContentValue.Status -eq "In Progress") ) { Start-Sleep -Seconds 10 } else { $responseContentValue | ConvertTo-Json break } }

    Here is the sample output:

     { "activityRunEnd": "2021-06-25T05:34:11.9536764Z", "activityName": "CopyFromBlobToBlob", "activityRunStart": "2021-06-25T05:34:07.5161151Z", "activityType": "Copy", "durationInMs": 4437, "retryAttempt": null, "error": { "errorCode": "", "message": "", "failureType": "", "target": "CopyFromBlobToBlob", "details": "" }, "activityRunId": "40bab243-9bbf-4538-9336-b797a2f98e2b", "iterationHash": "", "input": { "source": { "type": "BinarySource", "storeSettings": "@{type=AzureBlobStorageReadSettings; recursive=True}" }, "sink": { "type": "BinarySink", "storeSettings": "@{type=AzureBlobStorageWriteSettings}" }, "enableStaging": false }, "linkedServiceName": "", "output": { "dataRead": 134, "dataWritten": 134, "filesRead": 1, "filesWritten": 1, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "copyDuration": 3, "throughput": 0.044, "errors": [], "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (East US)", "usedDataIntegrationUnits": 4, "billingReference": { "activityType": "DataMovement", "billableDuration": "" }, "usedParallelCopies": 1, "executionDetails": [ "@{source=; sink=; status=Succeeded; start=06/25/2021 05:34:07; duration=3; usedDataIntegrationUnits=4; usedParallelCopies=1; profile=; detailedDurations=}" ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" }, "durationInQueue": { "integrationRuntimeQueue": 0 } }, "userProperties": {}, "pipelineName": "Adfv2QuickStartParamPipeline", "pipelineRunId": "ffc9c2a8-d86a-46d5-9208-28b3551007d8", "status": "Succeeded", "recoveryStatus": "None", "integrationRuntimeNames": [ "defaultintegrationruntime" ], "executionDetails": { "integrationRuntime": [ "@{name=DefaultIntegrationRuntime; type=Managed; location=East US; nodes=}" ] }, "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelineruns/ffc9c2a8-d86a-46d5-9208-28b3551007d8/activityruns/40bab243-9bbf-4538-9336-b797a2f98e2b" }

Verify the output

Use Azure Storage explorer to check the file is copied to "outputPath" from "inputPath" as you specified when creating a pipeline run.

Clean up resources

You can clean up the resources that you created in the Quickstart in two ways. You can delete the Azure resource group, which includes all the resources in the resource group. If you want to keep the other resources intact, delete only the data factory you created in this tutorial.

(Video) How to Use Rest API to Query Azure Data Factory From Web or By Using Postman ADF Tutorial 2022

Run the following command to delete the entire resource group:

Remove-AzResourceGroup -ResourceGroupName $resourcegroupname

Run the following command to delete only the data factory:

Remove-AzDataFactoryV2 -Name "<NameOfYourDataFactory>" -ResourceGroupName "<NameOfResourceGroup>"

Next steps

The pipeline in this sample copies data from one location to another location in an Azure blob storage. Go through the tutorials to learn about using Data Factory in more scenarios.

FAQs

Can Azure Data Factory connect to REST API? ›

Azure Data Factory allows you to add your required REST API Url and the authentication method to connect REST API to Azure products such as Azure SQL Database & Blob Storage.

How do I run Azure Data Factory pipeline using REST API? ›

Execute Azure ADF Pipeline using REST API
  1. Subscription ID.
  2. Tenant ID.
  3. Client ID.
  4. Client Secret.
  5. Authenticate using client_credentials.
  6. Request Type : POST.
  7. URL: https://login.microsoftonline.com/{{tenant_id}}/oauth2/token.
  8. Request Body:
Dec 5, 2019

How do I trigger Azure Data Factory pipeline from API? ›

To manually trigger a pipeline or configure a new scheduled, tumbling window, storage event, or custom event trigger, select Add trigger at the top of the pipeline editor. If you choose to manually trigger the pipeline, it will execute immediately.

Is Azure Data Factory an API? ›

Linked Services - REST API (Azure Data Factory)

How to connect REST API to database? ›

Connect from Postman

Using the excellent Postman app for accessing your database is a great way to explore and test your data REST API. Just install the Postman application and paste in a database REST url to get started. For authentication you also need to add a secure token as a HTTP Header value.

How do I call REST API from Azure pipeline? ›

Example
  1. From your pipeline definition, select the ellipsis button (...), and then select Add an agentless job.
  2. In your new agentless job, select the + sign to add a new task.
  3. Search for the Invoke REST API task. ...
  4. Select your Connection type and your Service connection. ...
  5. Select Save & queue when you are done.

How do I deploy REST API to Azure? ›

Publish the API app to Azure App Service
  1. In Solution Explorer, right-click the project and select Publish.
  2. In the Publish dialog, select Azure and select the Next button.
  3. Select Azure App Service (Windows) and select the Next button.
  4. Select Create a new Azure App Service. ...
  5. Select the Create button.
Nov 3, 2022

How do I create a dynamic pipeline in Azure data Factory? ›

The first step within the Azure Data Factory pipeline is to add a Lookup activity, which will retrieve the list of table names to load into the cloud from the metadata table. That set of table names will then be passed into a ForEach loop activity in order to process each table within the metadata list.

How do I run Azure rest APIs with Postman? ›

Use the Azure CLI to get a bearer token that you'll use to make API requests in Postman. Set up a Postman collection and configure the Postman REST client to use your bearer token to authenticate.

What is the best possible way to get started on creating an Azure data factory? ›

Advanced creation in the Azure portal
  1. Launch Microsoft Edge or Google Chrome web browser. ...
  2. Go to the Azure portal data factories page.
  3. After landing on the data factories page of the Azure portal, click Create.
  4. For Resource Group, take one of the following steps: ...
  5. For Region, select the location for the data factory.
Dec 13, 2022

How can I invoke an Azure Data factory pipeline from C# code? ›

  1. Specify the source Azure Blob information.
  2. Specify the Azure Cosmos DB information.
  3. Specify the Linked Service Names and Dataset Names.
  4. We can authenticate and create a data factory management client.
  5. Create data factory and wait.
  6. Create an Azure Blob Storage linked service.
  7. Create an Azure Cosmos DB linked service.
Oct 16, 2018

What are the types of triggers in Azure Data factory? ›

Azure Data Factory Triggers come in three different types: Schedule Trigger, Tumbling Window Trigger, and Event-based Trigger.

Does Azure support REST API? ›

The Azure Support REST API enables you to create and manage Azure support tickets programmatically. Using the API, you can: Open a technical, billing, subscription management, or subscription and service limits (quota) support ticket. Get a list of support tickets and detailed information about each ticket.

Does Azure have REST API? ›

Welcome to the Azure REST API Reference. Representational State Transfer (REST) APIs are service endpoints that support sets of HTTP operations (methods), which provide create/retrieve/update/delete access to the service's resources.

Is Azure data/factory ETL or ELT? ›

With Azure Data Factory, it is fast and easy to build code-free or code-centric ETL and ELT processes.

How to pull data from REST API? ›

Example 1: Pulling data from an Open source COVID API
  1. Connect to an API. At first, we need to connect to an API and make a secure connection as shown below– ...
  2. Get the data from API. ...
  3. Parse the data into JSON format. ...
  4. Extract the data and print it.

Which database is good for REST API? ›

The Snowflake SQL API is a REST API that you can use to access and update data in a Snowflake database. You can use this API to develop custom applications and integrations that perform simple queries.

How to connect REST API to SQL server? ›

You can set up the SQL Server REST API Integration by leveraging the connection manager using the following steps:
  1. Step 1: Defining a Data Flow Task.
  2. Step 2: Configuring Connection Properties.
  3. Step 3: Configuring Extraction Properties of REST Data Source.
  4. Step 4: Configuring the Microsoft SQL Server Database.
Nov 17, 2021

How do I call API from rest? ›

Step #1 – Enter the URL of the API in the textbox of the tool. Step #2 – Select the HTTP method used for this API (GET, POST, PATCH, etc). Step #3 – Enter any headers if they are required in the Headers textbox. Step #4 – Pass the request body of the API in a key-value pair.

How do I connect to Azure DevOps REST API? ›

First, provide API URL to get list of project. This URL needs to have the DevOps organization. Move to the Authorization section, sect Type as Basic Auth and provide the PAT Token to the Password field. Once done, send the request, You will have JSON Response of all the Projects.

How do I expose REST API in Azure? ›

Add a scope
  1. Sign in to the Azure portal.
  2. If you have access to multiple tenants, use the Directories + subscriptions filter in the top menu to select the tenant containing your client app's registration.
  3. Select Azure Active Directory > App registrations, and then select your API's app registration.
  4. Select Expose an API.
Oct 12, 2022

How REST API are deployed? ›

Deploy a REST API to a stage

In the APIs navigation pane, choose the API you want to deploy. In the Resources navigation pane, choose Actions. From the Actions drop-down menu, choose Deploy API. In the Deploy API dialog, choose an entry from the Deployment stage dropdown list.

How do I create a simple REST API on Azure? ›

How to Create a REST API with Azure Functions and the Serverless Framework - Part 1
  1. Step 1: Create your local Azure Function project. ...
  2. Step 2: Add your own handlers. ...
  3. Step 2.1: Test your API Locally. ...
  4. Step 2.2: Deploy. ...
  5. Step 2.3 Invoke Deployed Function.
Sep 17, 2019

How do I connect API to Azure SQL Database? ›

  1. Step 1 - Create SQL Server and Database in Azure. ...
  2. Step 2 - Set Server Firewall Rule. ...
  3. Step 3 - Create a New Table. ...
  4. Step 4 - Create a new Azure Function. ...
  5. Step 5 - Add references to NuGet Packages. ...
  6. Step 6 - Create Models. ...
  7. Step 7 - Add CRUD Operations. ...
  8. Step 8 - Add Connection String in local.settings.json file.
Jan 15, 2021

What is the difference between pipeline and data flow in Azure Data Factory? ›

Data moves from one component to the next via a series of pipes. Data flows through each pipe from left to right. A "pipeline" is a series of pipes that connect components together so they form a protocol.

How many pipelines can an Azure data/factory have? ›

Overview. A Data Factory or Synapse Workspace can have one or more pipelines.

How many pipelines can be created in Azure Data Factory? ›

Overview. A data factory can have one or more pipelines.

How do I call Azure REST API from PowerShell? ›

To use the Azure Rest API using PowerShell, we first need to connect to the Azure cloud account using the Connect-AzAccount. Once you are connected to the Azure Account, you can use the below authorization header (same has been provided on the MS website) which contains a bearer token to authenticate the rest API.

Is Postman tool a REST API? ›

Postman began as a REST client and has evolved into today's comprehensive Postman API Platform.

Can Postman BE USED FOR REST API? ›

You can use Postman REST Client to invoke a REST Service API to create and execute queries.

Which three types of activities can you run in Microsoft Azure Data Factory? ›

Data Factory supports three types of activities: data movement activities, data transformation activities, and control activities.

What are the limitations of Azure Data Factory? ›

Version 2
ResourceDefault limitMaximum limit
Total number of entities, such as pipelines, data sets, triggers, linked services, Private Endpoints, and integration runtimes, within a data factory5,000Contact support.
Total CPU cores for Azure-SSIS Integration Runtimes under one subscription64Contact support.
29 more rows

How do I deploy Azure Data Factory in production? ›

2.1 Prerequisites. 2.2 Setup Azure DevOps project. 2.3 Setup ADF DEV instance. 2.4 Create feature branch in ADF DEV.
...
2.1 Prerequisites
  1. Azure Account.
  2. Azure DevOps.
  3. Azure CLI (recommended, also for troubleshooting)

Can we write code in Azure data Factory? ›

With the changes introduced in the Data Factory V2 and Synapse pipeline Custom Activity, you can write your custom code logic in your preferred language and execute it on Windows and Linux Operation Systems supported by Azure Batch.

Can you run Python script in Azure data Factory? ›

In the Folder Path, select the name of the Azure Blob Storage container that contains the Python script and the associated inputs. This will download the selected files from the container to the pool node instances before the execution of the Python script.

How do you connect one pipeline to another Azure data Factory? ›

Invoke another Azure Data Factory Pipeline can be done using the “Execute Pipeline” Activity. Add Execute Pipeline activity available under “General” category of all the Azure Data Factory Activity List.

What is pipeline in Azure Data Factory? ›

A pipeline is a logical grouping of activities that performs a unit of work. Together, the activities in a pipeline perform a task. For example, a pipeline can contain a group of activities that ingests data from an Azure blob, and then runs a Hive query on an HDInsight cluster to partition the data.

What is difference between debug and trigger in ADF? ›

The difference between the manual trigger and debugging the pipeline, is that with a trigger you're using the pipeline configuration that is saved to the server. With debugging, you're running the pipeline as it is in the visual editor.

What is tumbling window in Azure Data Factory? ›

Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals.

Which platform is best for REST API? ›

Best API Management Tools & Solutions (2021)
  • RapidAPI.
  • Tyk.
  • Apigee API Management.
  • Azure API Management.
  • Postman.
  • IBM API Connect.
  • AWS API Gateway.
  • Mulesoft Anypoint Platform.
Jul 8, 2021

What are the 4 most common REST API operations? ›

For REST APIs built on HTTP, the uniform interface includes using standard HTTP verbs to perform operations on resources. The most common operations are GET, POST, PUT, PATCH, and DELETE.

How do I call REST API from Azure function app? ›

An Azure service that provides an event-driven serverless compute platform. You use HttpClient object to make HTTP calls. You need to create your request and use HttpClient to invoke any REST API within your Fx.

Is Azure data Factory an API? ›

Linked Services - REST API (Azure Data Factory)

Is REST API just JSON? ›

The REST architecture allows API providers to deliver data in multiple formats such as plain text, HTML, XML, YAML, and JSON, which is one of its most loved features.

What are the 3 components of a RESTful API? ›

REST Components
  • Resource Path (request target)
  • HTTP Verb.
  • Body.
  • Header.

Which language is best for ETL? ›

Python is one of the most popular programming languages for building ETL pipelines. Python frameworks can help you create more successful ETL pipelines. Top ETL Python frameworks include Bonobo, Bubbles, pygrametl, and Mara.

Which is better ETL or ELT? ›

ETL is better suited for compliance with GDPR, HIPAA, and CCPA standards given that users can omit any sensitive data prior to loading in the target system. ELT carries more risk of exposing private data and not complying with GDPR, HIPAA, and CCPA standards given that all data is loaded into the target system.

Is Azure Data factory PaaS or Saas? ›

Azure Data Factory (ADF) is a Microsoft Azure PaaS solution for data transformation and load. ADF supports data movement between many on premises and cloud data sources.

How do I connect to a SOAP API from Azure Data Factory? ›

How to Connect to a SOAP API from Azure Data Factory
  1. Step 1 - Create Linked Service. Begin by creating a linked service. ...
  2. Step 2 - Create Dataset. Create a new dataset for your API. ...
  3. Step 3 - Create Pipeline. To use your SOAP API connector, create a pipeline. ...
  4. Step 4 - Determine SOAP Headers and Body.
Jul 28, 2021

Does Azure Devops support REST API? ›

A required HTTP method (also known as an operation or verb), which tells the service what type of operation you are requesting. Azure REST APIs support GET, HEAD, PUT, POST, and PATCH methods.

How do I deploy a REST API to Azure? ›

Publish the API app to Azure App Service
  1. In Solution Explorer, right-click the project and select Publish.
  2. In the Publish dialog, select Azure and select the Next button.
  3. Select Azure App Service (Windows) and select the Next button.
  4. Select Create a new Azure App Service. ...
  5. Select the Create button.
Nov 3, 2022

How do I create an Azure REST API? ›

How to Create a REST API with Azure Functions and the Serverless Framework - Part 1
  1. Step 1: Create your local Azure Function project. ...
  2. Step 2: Add your own handlers. ...
  3. Step 2.1: Test your API Locally. ...
  4. Step 2.2: Deploy. ...
  5. Step 2.3 Invoke Deployed Function.
Sep 17, 2019

CAN REST API call SOAP? ›

Because SOAP is a protocol, and REST is an architectural style. A REST API can actually utilize the SOAP protocol, just like it can use HTTP.

Can we use SOAP in REST API? ›

A REST API response can thus contain a link to a SOAP endpoint which the client could invoke. The content negotiation may tell the client that media type application/soap+xml is required which gives the client a clue that a SOAP 1.2 service is sitting at the URI location.

Can SOAP services call REST API? ›

Yes of course. Apigee Edge can help you do this. In many cases we have the scenario in which a customer has a legacy SOAP service, which works well, but the new app developers would prefer to use something more REST-ful.

How many pipelines can be created in ADF? ›

#3 Maximum parameters per pipeline -50 and you can't increase this limit further.

What are the 3 types of data that can be stored in Azure? ›

Azure storage types include objects, managed files and managed disks. Customers should understand their often-specific uses before implementation. Each storage type has different pricing tiers -- usually based on performance and availability -- to make each one accessible to companies of every size and type.

What is Azure SQL Database REST API? ›

The Azure SQL Database REST API includes operations for managing Azure SQL Database resources.

Is Microsoft discontinuing Azure DevOps? ›

In this article

Azure DevOps Starter will be retired March 31, 2023. The corresponding REST APIs for Microsoft.

Videos

1. Read the data from Rest API JSON by using REST Connector and Write to Azure SQL DB in ADF
(TechBrothersIT)
2. Data Factory: API data to Azure Table storage
(javiBI)
3. Azure Data Factory (ADF) Quick Tip: Invoking REST APIs Using External Call Transform
(Data Platform Central)
4. Azure Data Factory - Executing a Pipeline from Azure Logic Apps using REST End Point!
(Azure Made Simple)
5. Azure Lab : Azure Data Factory V2 (ADFV2) : Copy data from Rest API to Data Lake Storage
(Cloudpandith)
6. #26. Copy data from REST API which sends the response in Pages| Azure Data Factory Tutorial |
(GeekCoders)
Top Articles
Latest Posts
Article information

Author: Francesca Jacobs Ret

Last Updated: 02/08/2023

Views: 6441

Rating: 4.8 / 5 (48 voted)

Reviews: 87% of readers found this page helpful

Author information

Name: Francesca Jacobs Ret

Birthday: 1996-12-09

Address: Apt. 141 1406 Mitch Summit, New Teganshire, UT 82655-0699

Phone: +2296092334654

Job: Technology Architect

Hobby: Snowboarding, Scouting, Foreign language learning, Dowsing, Baton twirling, Sculpting, Cabaret

Introduction: My name is Francesca Jacobs Ret, I am a innocent, super, beautiful, charming, lucky, gentle, clever person who loves writing and wants to share my knowledge and understanding with you.