Azure Data Factory - Samples - Azure Data Factory (2023)

  • Article
  • 7 minutes to read

Note

This article applies to version 1 of Data Factory. If you are using the current version of the Data Factory service, see PowerShell samples in Data Factory and code samples in the Azure Code Samples gallery.

Important

Support for Machine Learning Studio (classic) will end on 31 August 2024. We recommend you transition to Azure Machine Learning by that date.

Beginning 1 December 2021, you will not be able to create new Machine Learning Studio (classic) resources (workspace and web service plan). Through 31 August 2024, you can continue to use the existing Machine Learning Studio (classic) experiments and web services.

  • See information on moving machine learning projects from ML Studio (classic) to Azure Machine Learning.
  • Learn more about Azure Machine Learning

ML Studio (classic) documentation is being retired and may not be updated in the future.

Samples on GitHub

The GitHub Azure-DataFactory repository contains several samples that help you quickly ramp up with Azure Data Factory service (or) modify the scripts and use it in own application. The Samples\JSON folder contains JSON snippets for common scenarios.

SampleDescription
ADF WalkthroughThis sample provides an end-to-end walkthrough for processing log files using Azure Data Factory to turn data from log files in to insights.

In this walkthrough, the Data Factory pipeline collects sample logs, processes and enriches the data from logs with reference data, and transforms the data to evaluate the effectiveness of a marketing campaign that was recently launched.

JSON samplesThis sample provides JSON examples for common scenarios.
Http Data Downloader SampleThis sample showcases downloading of data from an HTTP endpoint to Azure Blob Storage using custom .NET activity.
Cross AppDomain Dot Net Activity SampleThis sample allows you to author a custom .NET activity that is not constrained to assembly versions used by the ADF launcher (For example, WindowsAzure.Storage v4.3.0, Newtonsoft.Json v6.0.x, etc.).
Run R scriptThis sample includes the Data Factory custom activity that can be used to invoke RScript.exe. This sample works only with your own (not on-demand) HDInsight cluster that already has R Installed on it.
Invoke Spark jobs on HDInsight Hadoop clusterThis sample shows how to use MapReduce activity to invoke a Spark program. The spark program just copies data from one Azure Blob container to another.
Twitter Analysis using ML Studio (classic) Batch Scoring ActivityThis sample shows how to use AzureMLBatchScoringActivity to invoke an ML model that performs twitter sentiment analysis, scoring, prediction etc.
Twitter Analysis using custom activityThis sample shows how to use a custom .NET activity to invoke an ML Studio (classic) model that performs twitter sentiment analysis, scoring, prediction etc.
Parameterized Pipelines for ML Studio (classic)The sample provides an end-to-end C# code to deploy N pipelines for scoring and retraining each with a different region parameter where the list of regions is coming from a parameters.txt file, which is included with this sample.
Reference Data Refresh for Azure Stream Analytics jobsThis sample shows how to use Azure Data Factory and Azure Stream Analytics together to run the queries with reference data and set up the refresh for reference data on a schedule.
Hybrid Pipeline with On-premises Hortonworks HadoopThe sample uses an on-premises Hadoop cluster as a compute target for running jobs in Data Factory just like you would add other compute targets like an HDInsight based Hadoop cluster in cloud.
JSON Conversion ToolThis tool allows you to convert JSONs from version prior to 2015-07-01-preview to latest or 2015-07-01-preview (default).
U-SQL sample input fileThis file is a sample file used by an U-SQL activity.
Delete blob fileThis sample showcases a C# file which can be used as part of ADF custom .NET activity to delete files from the source Azure Blob location once the files have been copied.

Azure Resource Manager templates

You can find the following Azure Resource Manager templates for Data Factory on GitHub.

TemplateDescription
Copy from Azure Blob Storage to Azure SQL DatabaseDeploying this template creates an Azure data factory with a pipeline that copies data from the specified Azure blob storage to Azure SQL Database
Copy from Salesforce to Azure Blob StorageDeploying this template creates an Azure data factory with a pipeline that copies data from the specified Salesforce account to the Azure blob storage.
Transform data by running Hive script on an Azure HDInsight clusterDeploying this template creates an Azure data factory with a pipeline that transforms data by running the sample Hive script on an Azure HDInsight Hadoop cluster.

Samples in Azure portal

You can use the Sample pipelines tile on the home page of your data factory to deploy sample pipelines and their associated entities (datasets and linked services) in to your data factory.

  1. Create a data factory or open an existing data factory. See Copy data from Blob Storage to SQL Database using Data Factory for steps to create a data factory.

  2. In the DATA FACTORY blade for the data factory, click the Sample pipelines tile.

    Azure Data Factory - Samples - Azure Data Factory (1)

  3. In the Sample pipelines blade, click the sample that you want to deploy.

    Azure Data Factory - Samples - Azure Data Factory (2)

  4. Specify configuration settings for the sample. For example, your Azure storage account name and account key, logical SQL server name, database, User ID, and password, etc.

    Azure Data Factory - Samples - Azure Data Factory (3)

  5. After you are done with specifying the configuration settings, click Create to create/deploy the sample pipelines and linked services/tables used by the pipelines.

  6. You see the status of deployment on the sample tile you clicked earlier on the Sample pipelines blade.

    Azure Data Factory - Samples - Azure Data Factory (4)

  7. When you see the Deployment succeeded message on the tile for the sample, close the Sample pipelines blade.

  8. On DATA FACTORY blade, you see that linked services, data sets, and pipelines are added to your data factory.

    Azure Data Factory - Samples - Azure Data Factory (5)

Samples in Visual Studio

Prerequisites

You must have the following installed on your computer:

  • Visual Studio 2013 or Visual Studio 2015
  • Download Azure SDK for Visual Studio 2013 or Visual Studio 2015. Navigate to Azure Download Page and click VS 2013 or VS 2015 in the .NET section.
  • Download the latest Azure Data Factory plugin for Visual Studio: VS 2013 or VS 2015. If you are using Visual Studio 2013, you can also update the plugin by doing the following steps: On the menu, click Tools -> Extensions and Updates -> Online -> Visual Studio Gallery -> Microsoft Azure Data Factory Tools for Visual Studio -> Update.

Use Data Factory Templates

  1. Click File on the menu, point to New, and click Project.

  2. In the New Project dialog box, do the following steps:

    1. Select DataFactory under Templates.

    2. Select Data Factory Templates in the right pane.

    3. Enter a name for the project.

    4. Select a location for the project.

    5. Click OK.

      Azure Data Factory - Samples - Azure Data Factory (6)

  3. In the Data Factory Templates dialog box, select the sample template from the Use-Case Templates section, and click Next. The following steps walk you through using the Customer Profiling template. Steps are similar for the other samples.

    Azure Data Factory - Samples - Azure Data Factory (7)

  4. In the Data Factory Configuration dialog, click Next on the Data Factory Basics page.

  5. On the Configure data factory page, do the following steps:

    1. Select Create New Data Factory. You can also select Use existing data factory.
    2. Enter a name for the data factory.
    3. Select the Azure subscription in which you want the data factory to be created.
    4. Select the resource group for the data factory.
    5. Select the West US, East US, or North Europe for the region.
    6. Click Next.
  6. In the Configure data stores page, specify an existing database in Azure SQL Database and Azure storage account (or) create database/storage, and click Next.

  7. In the Configure compute page, select defaults, and click Next.

  8. In the Summary page, review all settings, and click Next.

  9. In the Deployment Status page, wait until the deployment is finished, and click Finish.

  10. Right-click project in the Solution Explorer, and click Publish.

  11. If you see Sign in to your Microsoft account dialog box, enter your credentials for the account that has Azure subscription, and click sign in.

  12. You should see the following dialog box:

    Azure Data Factory - Samples - Azure Data Factory (8)

  13. In the Configure data factory page, do the following steps:

    1. Confirm that Use existing data factory option.
    2. Select the data factory you had select when using the template.
    3. Click Next to switch to the Publish Items page. (Press TAB to move out of the Name field to if the Next button is disabled.)
  14. In the Publish Items page, ensure that all the Data Factories entities are selected, and click Next to switch to the Summary page.

  15. Review the summary and click Next to start the deployment process and view the Deployment Status.

  16. In the Deployment Status page, you should see the status of the deployment process. Click Finish after the deployment is done.

See Build your first data factory (Visual Studio) for details about using Visual Studio to author Data Factory entities and publishing them to Azure.

Top Articles
Latest Posts
Article information

Author: Rueben Jacobs

Last Updated: 11/25/2022

Views: 6168

Rating: 4.7 / 5 (57 voted)

Reviews: 88% of readers found this page helpful

Author information

Name: Rueben Jacobs

Birthday: 1999-03-14

Address: 951 Caterina Walk, Schambergerside, CA 67667-0896

Phone: +6881806848632

Job: Internal Education Planner

Hobby: Candle making, Cabaret, Poi, Gambling, Rock climbing, Wood carving, Computer programming

Introduction: My name is Rueben Jacobs, I am a cooperative, beautiful, kind, comfortable, glamorous, open, magnificent person who loves writing and wants to share my knowledge and understanding with you.