Azure Data Factory Interview Questions

ETL genuinely transforms data before actually loading it into the destination. ELT genuinely loads raw data first and transforms it afterward, typically within the destination system itself. ADF genuinely supports both patterns, letting you actually build either kind of pipeline depending on your specific needs.

ADF solves the genuine problem of needing to actually move and transform data across genuinely many different systems, on-premises databases, cloud storage, SaaS applications, without manually writing and maintaining a genuinely separate, custom integration script for every single one of those connections.

A pipeline genuinely contains one or more activities, each activity genuinely performs a specific task like copying data or running a transformation, and activities genuinely reference datasets (describing the actual data) and linked services (describing the actual connection to a data source).

They genuinely share the exact same underlying pipeline engine and authoring experience. Synapse's own integrated pipelines are genuinely built directly into the broader Synapse workspace alongside SQL and Spark, while ADF is a genuinely standalone service, useful when data integration doesn't necessarily need to live within a genuinely full Synapse workspace.

ADF Studio provides a genuinely visual, drag-and-drop interface for actually building, testing, and monitoring a pipeline, letting a data engineer actually design a genuinely complex workflow without needing to hand-write the underlying JSON definition directly.

A pipeline is a genuine logical grouping of activities that together actually perform a specific task, like extracting data from a source, transforming it, and loading it into a genuine destination, all coordinated as one cohesive, orchestrated unit.

An activity represents a genuinely single processing step within a pipeline, like copying data from one location to another, running a stored procedure, or actually executing a data transformation, and a pipeline genuinely chains multiple activities together to actually accomplish its overall task.

A data movement activity, like Copy Activity, genuinely moves data from a source to a destination without transforming it. A data transformation activity, like a Mapping Data Flow, actually applies genuine logic, like filtering or aggregating, to the data as part of the pipeline's own execution.

The Delete activity genuinely removes a specified set of files or folders from a data store as part of a pipeline's execution. A genuinely practical use case is cleaning up temporary staging files left behind by an earlier Copy Activity, once that intermediate data is genuinely no longer needed later in the same pipeline.

Connecting the first activity's genuine Success output to the second activity's own input, using ADF Studio's visual designer, tells ADF the second activity should only actually execute once the first has genuinely completed successfully.

Copy Activity actually moves data from a genuine source dataset to a destination dataset, handling the underlying connection, format conversion, and data transfer automatically, making it the genuinely most commonly used activity for a straightforward data movement task.

A pipeline run genuinely represents one complete execution of an entire pipeline, from start to finish. An activity run genuinely represents the execution of just one genuinely specific activity within that pipeline, and a genuinely single pipeline run typically contains several individual activity runs.

A linked service genuinely defines the actual connection information needed to actually connect to a specific data source or destination, like a database's own connection string or a storage account's own access key, similar in concept to a genuine connection string in a traditional application.

A dataset genuinely represents a specific structure of data within a data store, like a specific table or a specific file, and it genuinely references a linked service to actually know which specific connection to use to actually access that particular data.

A linked service genuinely defines how to actually connect to a data store. A dataset genuinely defines a specific piece of data within that store, like a genuine table name or a file path, that an activity will actually read from or write to.

Within ADF Studio's Manage tab, creating a genuinely new linked service and selecting Azure SQL Database as the type, then actually supplying the server name, database name, and authentication credentials, establishes that genuine connection for pipelines to actually use.

ADF genuinely supports SQL authentication (a username and password), Azure Active Directory authentication, and a Managed Identity, letting a pipeline actually connect securely without needing a hardcoded credential stored directly within the linked service's own configuration.

Reusing one linked service avoids duplicating the exact same connection configuration, so if a genuine credential or a connection detail ever needs to actually change, you only genuinely need to update it in that one single, shared place, rather than in many, genuinely separate places.

ForEach genuinely iterates over a collection, running a set of activities once per item. If Condition genuinely branches execution based on a defined condition. Execute Pipeline genuinely calls another, genuinely separate pipeline from within the current one.

The Lookup activity genuinely retrieves a value or a small dataset from a source, like a specific configuration value from a database table, making that retrieved data available for a genuinely later activity in the pipeline to actually use, like determining which files to actually process.

The Web activity genuinely calls a REST endpoint from within a pipeline, letting you actually trigger an external system or retrieve data from a genuine API as part of the overall pipeline's own workflow.

The Wait activity genuinely pauses pipeline execution for a specified number of seconds before actually continuing to the next step. A genuinely practical use case is deliberately pausing briefly after triggering an external system, giving it time to actually begin processing before a subsequent activity checks on its own status.

A Stored Procedure activity genuinely executes a stored procedure directly against a database, useful when a genuine transformation or a business logic step is already implemented as a stored procedure and you simply need the pipeline to actually trigger it at the correct, appropriate point in the workflow.

The source configuration genuinely defines where the Copy activity actually reads data from. The sink configuration genuinely defines where it actually writes that data to, and both are genuinely configured independently, letting a Copy activity actually move data between two genuinely very different types of data store.

Using dynamic content expressions, like @activity('PreviousActivityName').output.value, lets a genuinely later activity actually reference and use a specific value produced by an earlier activity's own execution within the exact same pipeline run.

A trigger genuinely defines when a pipeline should actually run, whether on a defined recurring schedule, in response to a genuine event, or manually, on demand, letting a pipeline execute automatically without requiring someone to actually start it by hand every single time.

A Schedule trigger genuinely runs a pipeline at a defined, recurring interval, like every day at 2 AM, letting you actually automate a genuinely regular, periodic data processing task without manual intervention.

An Event-based trigger genuinely fires a pipeline run in response to a specific event, like a genuinely new file actually being uploaded to a storage container, letting a pipeline actually process new data as soon as it genuinely arrives rather than waiting for the next scheduled run.

A Schedule trigger simply genuinely runs at a fixed time interval. A Tumbling Window trigger genuinely divides time into fixed, non-overlapping windows and can actually track dependency between windows, supporting genuine backfill and retry logic for a specific, individual time window that might have actually failed.

Selecting Trigger Now from within ADF Studio's pipeline canvas genuinely starts an immediate, on-demand pipeline run, useful for actually testing a pipeline or manually re-running it once, outside its genuinely normal, automated schedule.

Yes, a genuinely single pipeline can actually be associated with multiple triggers at once, for instance a genuine Schedule trigger running it nightly and a genuinely separate Event-based trigger also running it whenever a genuinely specific file actually arrives.

An Integration Runtime is the genuine compute infrastructure ADF actually uses to perform data movement and transformation activities, providing the actual connectivity between the pipeline logic and the real, underlying data sources and destinations.

The Azure IR is a genuinely fully managed compute environment running entirely within Azure. A Self-Hosted IR genuinely runs on a machine you provide yourself, typically needed when a pipeline needs to actually access an on-premises data source not directly, publicly reachable from the actual Azure cloud.

A Self-Hosted IR is genuinely needed when the data source lives within a genuinely private, on-premises network, or a genuinely restricted virtual network, that the fully managed, cloud-based Azure IR can't directly, actually reach on its own.

The Azure-SSIS IR lets you actually run an existing SQL Server Integration Services (SSIS) package directly within Azure Data Factory, solving the genuine problem of an organization migrating to the cloud while still needing to actually run a genuinely existing, already-built SSIS package without a full, immediate rewrite.

No, ADF genuinely provides a default Azure Integration Runtime automatically, ready to actually use immediately, and you'd only genuinely need to create an additional, genuinely custom IR, like a Self-Hosted one, for a genuinely specific requirement the default one doesn't actually cover.

A Copy Activity genuinely moves data with only genuinely basic transformation, like a simple column mapping. A Mapping Data Flow genuinely supports far more sophisticated transformation logic, joins, aggregations, conditional splits, running on a genuinely dedicated, scalable Spark-based compute environment.

Join, genuinely combining two data streams based on a matching key. Aggregate, genuinely computing a summary value across a group. Filter, genuinely keeping only rows satisfying a condition. Derived Column, genuinely computing a new column's value based on an expression.

A debug session genuinely spins up a temporary, active Spark cluster, letting you actually preview data at each individual transformation step interactively while designing a Data Flow, rather than needing to run the entire pipeline genuinely end to end just to see an intermediate result.

A Wrangling Data Flow lets you actually use a Power Query-like interface to interactively explore and shape data, generally genuinely more approachable for an analyst genuinely familiar with Excel or Power BI. A Mapping Data Flow provides genuinely more powerful, structured transformation capability suited to a genuinely more complex data engineering pipeline.

A pipeline parameter lets you actually pass a genuinely dynamic value into a pipeline at the moment it's actually run, solving the genuine problem of needing genuinely separate, near-identical pipelines just to handle a different input value, like a genuinely different file name or a different date range.

A parameter's value is genuinely set once, when the pipeline is actually triggered, and can't genuinely change during that specific run. A variable's value can genuinely be set and changed multiple times during the pipeline's own execution, using a Set Variable activity.

@pipeline().parameters.parameterName genuinely references that specific parameter's value within an activity's configuration field, letting the activity actually behave dynamically based on whatever value was genuinely passed in when the pipeline was actually triggered.

Set Variable genuinely assigns a specific value to a pipeline variable during the pipeline's own execution, letting you actually update that variable's value dynamically as the pipeline progresses through genuinely several different activities.

Configuring the Execute Pipeline activity's own parameters section, mapping each genuine parameter the child pipeline expects to a specific value or expression from the parent pipeline, lets the genuine child pipeline actually receive that dynamic data from its own caller.

Configuring ForEach with the collection to actually iterate over, like a list of file names returned from a Lookup activity, and defining the genuine inner activities to actually run for each individual item, processes every item in that collection through the exact same defined logic.

A sequential ForEach genuinely processes each item one after another, in order. A parallel ForEach genuinely processes multiple items simultaneously, up to a genuinely configured batch count, which can meaningfully speed up processing a genuinely large collection of independent items.

If Condition genuinely evaluates a defined boolean expression and executes one genuine set of activities if it's true, and a genuinely different set of activities if it's false, letting a pipeline actually branch its behavior based on a genuinely dynamic condition.

Until genuinely repeats a set of activities until a defined condition actually becomes true, useful for a genuinely practical scenario like polling an external system repeatedly until a specific genuine job it triggered has actually finished processing.

Connecting an activity's genuine Failure output to a genuinely separate cleanup or notification activity, rather than only connecting its Success output, lets the pipeline actually handle a failure explicitly and continue to a genuinely defined recovery path rather than simply stopping abruptly.

The Monitor tab within ADF Studio genuinely shows every pipeline run's own status, start and end time, and, for a genuinely failed run, the specific error message from whichever activity actually caused that failure.

Debug mode genuinely runs a pipeline interactively within the authoring canvas, showing intermediate output at each individual activity, letting you actually test and iterate on a pipeline's own design quickly without needing to actually publish it or wait for a genuinely full, real pipeline run to complete.

Configuring an Azure Monitor alert rule on the pipeline's own failure metric, connected to an action group that actually sends an email or a notification, lets the genuinely relevant team be automatically informed the moment a genuinely critical pipeline actually fails, without requiring anyone to manually check the Monitor tab.

Checking the specific error message in the Monitor tab's own activity run details usually genuinely reveals the actual root cause, a genuine connectivity issue, a schema mismatch, or a throttling error from the source or destination system, letting you actually address that specific, real underlying issue.

A DIU represents a genuine unit of processing power and memory allocated to a Copy Activity's own execution. Increasing the number of DIUs genuinely allocated can meaningfully speed up a genuinely large data copy operation, at the cost of a correspondingly genuinely higher billed cost for that specific run.

Parallel copy genuinely splits a large data copy operation into multiple genuinely concurrent sub-tasks, running them simultaneously rather than sequentially, meaningfully reducing the total time needed to actually copy a genuinely large volume of data.

Within the Copy Activity's own Mapping tab, explicitly connecting a genuine source column to its corresponding destination column, rather than relying on Copy Activity's own automatic mapping by matching identical column names, handles a genuine naming mismatch between source and destination.

Increasing the DIU allocation, enabling staged copy through a genuine intermediate storage location when copying between two genuinely incompatible network environments, and choosing an appropriate genuine parallel copy degree, all meaningfully improve a large Copy Activity's own overall throughput.

Partitioning determines how data is genuinely distributed across the underlying Spark cluster's own worker nodes during a Data Flow's execution. Choosing a genuinely appropriate partitioning strategy, matching how the data will actually be processed, meaningfully improves parallel processing efficiency compared to Data Flow's own default partitioning.

TTL genuinely determines how long a debug session's own underlying cluster stays alive after your last genuine interaction before automatically shutting down. Quick re-use genuinely lets a genuinely new debug session reuse an already-running cluster instead of provisioning a genuinely brand-new one, speeding up the start of a genuinely subsequent debug session.

Enabling schema drift within the Data Flow's own source settings lets it actually accept a genuinely varying column structure across different files, and using a rule-based mapping instead of an genuinely explicit, fixed column mapping lets the Data Flow adapt to that genuine variation dynamically.

The Optimize tab lets you actually control the genuine partitioning strategy for that specific transformation step, choosing between letting ADF decide automatically or explicitly specifying a genuinely specific partition scheme, which matters meaningfully for a genuinely large or complex transformation's own actual performance.

The Data Flow's own execution details, viewable from the Monitor tab after a run completes, break down the actual time spent at each individual transformation stage, letting you actually pinpoint exactly which genuinely specific step is consuming the most time within that overall Data Flow.

ADF genuinely integrates with Git for source control, and its own ARM template export capability lets a CI/CD pipeline actually deploy a genuinely tested version of the Data Factory's pipelines, datasets, and linked services consistently to a genuinely separate environment, like production.

Git integration lets you actually work in a genuinely dedicated feature branch, review changes through a pull request, and maintain a genuine version history of every pipeline change, rather than every single change being genuinely, immediately live in the actual, live Data Factory instance without any review step.

The collaboration branch, typically main, genuinely holds the actual source JSON definitions for every pipeline, dataset, and linked service. The adf_publish branch genuinely holds the generated ARM template artifacts, automatically created when publishing, used specifically for actually deploying to a genuinely different environment.

ADF's own ARM template export automatically genuinely parameterizes environment-specific values, like a linked service's own connection string, letting the exact same template be actually deployed with a genuinely different parameter file supplying the correct, environment-specific values for each target environment.

Running the modified pipeline in Debug mode against a genuinely representative test dataset first, and, for a genuinely more rigorous approach, deploying to a dedicated test Data Factory instance and running a genuine end-to-end validation there before actually promoting the change to the real, live production environment.

Storing the genuine maximum value of a watermark column, like a last-modified timestamp, from the previous run, then genuinely filtering the source query on the next run to only actually retrieve rows with a watermark value genuinely greater than that stored value, processes only the genuinely new or changed data.

A metadata-driven pipeline genuinely reads its own configuration, like which tables to actually copy and their specific settings, from an external metadata store, like a control table, rather than genuinely hardcoding every specific pipeline's own logic separately. It solves the genuine problem of needing to build a genuinely near-identical, separate pipeline for every single table or data source.

A metadata-driven design, reading a genuinely control table listing every source table and its own specific configuration, combined with a ForEach activity genuinely iterating over that list and calling a genuinely single, shared child pipeline for each table, avoids needing hundreds of genuinely separate, individually maintained pipelines.

Configuring genuinely multiple nodes for the Self-Hosted IR provides high availability and additional throughput, so a single node failing doesn't genuinely halt the entire pipeline's ability to actually reach the required, on-premises data source.

Storing the Data Factory's own pipeline definitions in Git provides a genuine, reliable way to actually redeploy the entire factory to a genuinely different Azure region if needed, combined with regularly testing that redeployment process to actually confirm it genuinely works when it's actually needed during a real disaster.

I'd weigh Mapping Data Flow's genuinely code-free, more approachable authoring experience against Databricks' own genuinely greater flexibility and power for a truly complex, custom transformation logic that Mapping Data Flow's own visual interface might struggle to genuinely express cleanly.

Adding a Databricks Notebook activity to the pipeline, configured with a genuine linked service pointing to the Databricks workspace, lets ADF actually trigger that specific notebook's execution as one genuine step within a genuinely larger, orchestrated end-to-end pipeline.

ADF can actually orchestrate loading data into Synapse's own dedicated SQL pool, and Synapse's own integrated pipeline feature genuinely shares the exact same underlying technology as ADF, letting an organization genuinely choose whichever specific service better fits its own broader architecture.

Chaining the downstream activity, a Databricks Notebook activity or a Synapse pipeline call, to actually run only after the upstream Copy Activity or Data Flow's genuine Success output, ensures the downstream processing only actually starts once the upstream data is genuinely confirmed to be actually ready.

Key Vault lets ADF actually retrieve a genuinely sensitive credential, like a database password, at runtime through a linked service reference, rather than that genuine credential ever being stored directly, in plain text, within the Data Factory's own configuration itself.

Creating an Azure Key Vault linked service first, then referencing it within the target linked service's own connection field as an AKV Linked Service reference, rather than typing the actual secret value directly, tells ADF to actually fetch that specific secret from Key Vault at runtime whenever the connection is actually used.

ADF genuinely orchestrates ingesting raw data into Data Lake Storage, Databricks or a Synapse Data Flow genuinely transforms it through the medallion architecture's Bronze, Silver, and Gold layers, and Synapse's own SQL pool or serverless SQL genuinely serves the final, refined data for reporting and analytics.

I'd weigh the genuine benefit of a unified, single workspace experience Synapse provides against the real cost and disruption of a migration, and whether the organization's own current pain, managing genuinely several separate, disconnected tools, genuinely justifies that specific consolidation.

I'd evaluate whether the Azure-SSIS Integration Runtime can genuinely support a lift-and-shift migration for the existing packages first, migrating incrementally and validating each migrated package's output against the genuinely existing, on-premises system before actually decommissioning that legacy infrastructure.

I check whether it genuinely follows an established metadata-driven or parameterized pattern consistent with the rest of the organization's own pipelines, whether error handling and failure notification are genuinely built in, and whether incremental loading is genuinely used where the data volume actually warrants it.

I'd document the handful of standards that actually matter most, with genuine, concrete examples of a real problem each one prevents, and enforce what can genuinely be automated, like a required CI/CD deployment process, rather than relying on manual, ad hoc review.

I'd weigh the genuine benefit of consistent governance and reduced duplicated administrative effort against the real cost and genuine disruption of a consolidation, and whether the separate instances genuinely exist for a real, valid reason, like a distinct regulatory or business separation requirement.

I'd check whether the actual production data volume genuinely differs meaningfully from the development test data, since a Copy Activity or a Data Flow succeeding on a genuinely small test dataset can fail once real, much larger production data volume is genuinely involved, especially around a genuine timeout or memory limit.

Configure Azure Monitor alerts on pipeline failure, and track genuine pipeline run duration trends over time, alerting on meaningful deviation from an established baseline. A genuinely, slowly growing pipeline duration trend is often an early warning sign well before it actually causes a real, missed processing deadline.

Treat the schema and pipeline's own parameters as a genuine contract with every downstream consumer. Adding a genuinely new, optional parameter is generally safe. Changing or removing an existing one needs a documented migration plan and direct communication with every team genuinely depending on that pipeline.

I'd check the pipeline's own recent run history and error logs first, since a genuinely recent change to either the pipeline itself or the underlying source data is the most likely, genuine suspect, and consider a genuinely quick manual re-run as an immediate mitigation while properly investigating the actual root cause.

I'd load test with a genuinely realistic, larger data volume, verify Integration Runtime capacity and any relevant Azure service quotas are genuinely sufficient, and identify whether the coming bottleneck is genuinely likely to be Copy Activity throughput, Data Flow compute capacity, or a downstream system's own ingestion limit.

This is a judgment question interviewers use to see how you reason under genuine uncertainty, not to test a specific textbook fact. A strong answer names the actual constraint that forced the decision, the realistic options that were genuinely on the table, why you picked one knowing it wasn't guaranteed to be right, and what you'd do differently with what you know now.

I'd walk through the actual, real pipeline run duration and cost together as the source data volume has genuinely grown, showing concretely how a full reload's own cost keeps growing over time, rather than explaining incremental loading as an abstract best practice in isolation. Seeing the genuine, real growing cost tends to build that habit far more effectively.

I wouldn't lead with the metadata-driven pattern as an abstract best practice. I'd point to a specific, real, already-experienced maintenance burden of updating genuinely dozens of near-identical pipelines for the exact same kind of change, and show concretely how one shared, parameterized pipeline would have genuinely prevented that exact same specific burden.

I'd bring the actual, concrete complexity of the required transformation logic into the discussion, rather than a general, abstract preference for one tool over the other. Most disagreements like this genuinely resolve once both sides are looking at the exact same concrete transformation requirements together.

I'd translate the opportunity into terms leadership already tracks: a specific percentage of Data Flow or Copy Activity spend going toward a genuinely full reload that incremental loading would eliminate, and what that recovered spend could instead fund elsewhere. Framed as recovered budget with a concrete number attached, it competes far better for prioritization than framed as a general infrastructure cleanup.