Validates skills in designing and implementing data engineering solutions using Azure Databricks, including Delta Lake, Unity Catalog, and data pipelines.

Delta Lake ACID, Time Travel, MERGE, VACUUM, Unity Catalog, Auto Loader, and Structured Streaming fundamentals.

Core Concepts

Security & Secrets, Unity Catalog, Compute & Clusters, Monitoring, Data Transformation, Ingestion, Modeling, Performance, Pipeline Orchestration, Data Quality, and CI/CD.

Pipelines & Orchestration

Compute & Clusters, Unity Catalog, Pipeline Orchestration, Data Ingestion, Data Quality, Security & Secrets, Data Transformation, Monitoring, Data Modeling, and CI/CD.

Performance & Optimization

Unity Catalog access control & volumes, compute & libraries, data formats, performance, monitoring & audit, data ingestion & CDC, CI/CD & Bundles, data modeling, and pipeline orchestration.

Security & Governance

Security & Secrets, CI/CD & Git Workflows, Data Modeling, Performance & Optimization, Pipeline Orchestration, Data Transformation, Data Quality, Data Ingestion, Compute & Clusters, Monitoring, and Unity Catalog.

Security, CI/CD & Advanced Topics

Unity Catalog, Data Ingestion, Pipeline Orchestration, Performance & Optimization, Security & Governance, CI/CD & Git Workflows, Monitoring, Data Transformation, Data Quality, Data Modeling, and Compute & Clusters.

Governance, Ingestion & Orchestration

DP-750

Microsoft Certified: Azure Databricks Data Engineer Associate

Classic all-purpose compute with a large driver node

Classic all-purpose compute for the ETL pipeline; SQL warehouse for the analysts

Serverless compute for the ETL pipeline; shared compute for the analysts

Classic jobs compute for the ETL pipeline; serverless SQL warehouse for the analysts

SQL warehouse for the ETL pipeline; classic all-purpose compute for the analysts

sales, dev_sales, partner (one catalog per domain, environment as prefix, generic share name)

sales_p, sales_d, sales_s (abbreviated tokens)

Use identical catalog names across dev and prod to keep access patterns simple

Embed an explicit environment token (dev, staging, prod) in every catalog name

Encode team lead initials in every schema name as an ownership signal

Reserve a recognizable prefix or suffix for all externally shared objects (e.g., ext_share)

A second read-only connection must be created before analysts can query the foreign catalog

Granting USE CATALOG, USE SCHEMA, and SELECT on the foreign catalog objects to the analyst group

Running DESCRIBE CONNECTION on pg_conn to validate the connection before use

Converting the foreign catalog into a standard catalog so Unity Catalog can manage it

USE SCHEMA on a schema inside the external database

CREATE FOREIGN CATALOG on the connection object

Grant CREATE CONNECTION on the metastore to the analytics group

Grant USE CATALOG on the foreign catalog to the analytics group

Grant CREATE FOREIGN CATALOG on the connection to the analytics group

Grant USE SCHEMA and SELECT on the foreign catalog schemas and tables to the analytics group

A column mask SQL UDF returning a type castable to the column type

A dynamic view with IF(is_member('pii_admin'), phone, 'REDACTED') logic

A CHECK constraint that validates phone format for pii_admin members only

A column mask SQL UDF returning STRING applied to the region column

A row filter SQL UDF returning BOOLEAN applied to the sales table

A CHECK constraint referencing current_user() on the region column

A materialized view per region that pre-filters rows for each manager group

Rewrite the mask using CREATE TABLE syntax instead of ALTER TABLE

Wrap the Python UDF in a SQL UDF and attach the SQL UDF as the column mask

Convert the Python UDF to a Scala UDF to satisfy the mask type requirement

Grant EXECUTE on the Python UDF to all workspace users before attaching

The two masking functions return incompatible data types for the column

Multiple distinct masks that both resolve for the same user and column cause a runtime error

The data_steward group is missing SELECT on the table

Column masks require dedicated access mode compute and the cluster is using shared mode

Databricks-backed secret scope with a documented manual rotation runbook

Azure Key Vault-backed secret scope referenced via dbutils.secrets.get()

Cluster-level environment variables populated from the Azure Key Vault secret values

Notebook widgets that prompt engineers to paste credentials at runtime

Switch the Key Vault permission model to Vault access policy

Enable the trusted Microsoft services firewall bypass on the Key Vault

Copy all secrets into a Databricks-backed scope before creating the Key Vault-backed scope

Disable the Key Vault firewall entirely before scope creation

Delete and recreate the secret scope with the new password value

Restart the cluster so it fetches the updated secret value at startup

Use databricks secrets put-secret to push the new value into the Key Vault-backed scope

Grant the cluster service principal WRITE permission on the Key Vault-backed scope

Add per-secret READ ACLs inside the Databricks secret scope to restrict individual keys

Create a separate Azure Key Vault containing only the JDBC credentials and create a new scope backed by that vault

Change the Manage Principal on the existing scope to restrict who gets MANAGE rights

Move the non-JDBC secrets to a Databricks-backed scope and leave the vault unchanged

The workspace service principal needs MANAGE permission on the vault

The developer creating the scope needs at least Contributor or Key Vault Contributor role on the vault

The vault must contain at least 5 secrets before a scope can be created against it

The workspace must be assigned an Azure managed identity before Key Vault-backed scopes can be created

The administrator must be the table owner or have MANAGE privilege on the table

The administrator must have CREATE TABLE on the parent schema

The administrator must have EXECUTE on the masking function

All existing SELECT grants on the table must be revoked before the mask can be attached

Add a last_updated_timestamp column to the dimension table

Repartition the fact table by warehouse_id to speed up filtering

Migrate the warehouse assignment attribute to SCD Type 2 to preserve prior versions

Schedule a daily COPY INTO snapshot of the dimension table

Correcting a misspelled vendor name with no audit trail required

Tracking a customer credit tier for regulatory review of historical lending decisions

Replacing a discontinued product color code with a new standard code across all records

Preserving which sales rep owned an account at the time each contract was signed for commission attribution

Hourly COPY INTO job triggered by Azure Data Factory

Auto Loader with cloudFiles format, checkpointing, and schema evolution options

Lakeflow Connect managed connector pointing at the ADLS container

Kafka-based Structured Streaming from an Azure Event Hubs namespace

Add .option('cloudFiles.inferColumnTypes', 'true') to the stream reader

Change .option('cloudFiles.format', 'csv') to .option('cloudFiles.format', 'json')

Enable ANSI mode on the Spark session to enforce automatic type casting during ingestion

Add CAST expressions in a downstream silver transformation notebook

Auto Loader with .option('cloudFiles.format', 'json')

CREATE STREAMING TABLE backed by the Event Hubs Kafka connector

Lakeflow Connect managed connector for Salesforce

COPY INTO from a scheduled Salesforce export landed in ADLS

Full table export to CSV on a shared drive, then COPY INTO to load into Delta

Lakeflow Connect SQL Server managed connector

Auto Loader reading JSON snapshot files staged in ADLS

Incremental CDC or change-tracking extraction through the connector

SCD Type 2 to preserve full price and category history in versioned rows

A daily full-table snapshot appended to a separate history table

SCD Type 1, overwriting prior values with the current state

An append-only event log recording every price change event

GROUP BY store_id, order_status — then HAVING order_status = 'COMPLETE'

WHERE order_status = 'COMPLETE' — then GROUP BY store_id with SUM(net_revenue)

SELECT DISTINCT store_id, SUM(net_revenue) with no WHERE clause

ORDER BY store_id — then apply SUM(net_revenue) across the sorted result

Replace GROUP BY with PARTITION BY department

Move the aggregate condition SUM(salary) > 80000 into a HAVING clause after GROUP BY department

Wrap SUM(salary) inside MAX() in the GROUP BY clause to make it valid

Add ORDER BY SUM(salary) DESC to filter only departments above 80000

Use HAVING customer_id IS NOT NULL to remove null IDs before grouping

Use .filter(col('customer_id').isNotNull()) on the base DataFrame before .groupBy()

Use .agg(sum('order_value')) without .groupBy() to compute per-customer totals

Use .filter(col('total_value') > 5000) after .groupBy('customer_id').agg(sum('order_value').alias('total_value'))

The streaming table schema drifted and rejected new row insertions

The SQL Server read replica does not support CDC or change tracking, and the infrequent gateway schedule allowed source logs to expire before capture

The destination Unity Catalog table exceeded its maximum row-count quota

The pipeline compute ran out of memory during the holiday catch-up batch

.option('cloudFiles.inferColumnTypes', 'true')

.option('cloudFiles.format', 'binaryFile')

.option('cloudFiles.schemaEvolutionMode', 'addNewColumns')

Use the default RootManageSharedAccessKey namespace policy for simplicity

Create a dedicated Event Hubs SAS policy with Listen permission only

Store the SAS connection string in Databricks secrets and reference it in the pipeline configuration

Embed the SAS connection string inline in the kafka.sasl.jaas.config option in the pipeline code

The pipeline falls back to an unencrypted connection on port 9092

The 'security.protocol' key is not recognized by the Kafka connector and is silently ignored, causing the connection to proceed without SASL_SSL, which Event Hubs requires

The pipeline automatically retries using GSSAPI (Kerberos) authentication

Databricks auto-corrects unrecognized Kafka option key names during pipeline compilation

Grant the pipeline owner MANAGE on the target schema

Switch the pipeline compute back to standard access mode

Replace the Event Hubs source with Auto Loader pointing to an ADLS landing zone

Change the pipeline target to the Hive metastore instead of Unity Catalog

Schedule COPY INTO to run every 5 minutes using a Lakeflow Job trigger

Use raw Structured Streaming with Auto Loader format in an interactive notebook

Replace COPY INTO with CREATE STREAMING TABLE backed by Auto Loader

Switch the source files from JSON to Parquet to enable more efficient COPY INTO incremental loads

CREATE MATERIALIZED VIEW on the target schema

A missing JOIN condition between bronze and a lookup table caused a Cartesian product

Adding a high-cardinality column to GROUP BY raised output granularity beyond one row per customer

SUM ignores NULL values, causing row duplication in the output

Databricks cached a stale version of the DataFrame before the transformation executed

Disable ANSI mode on the Spark session so CAST returns NULL instead of raising an error

Use to_date() with an explicit format string and regexp_replace() combined with to_number() or try_cast() for amounts

Store transaction_date and amount as STRING in the silver table to avoid casting

Add a CHECK constraint on transaction_date to reject malformed values at write time

Primary key constraints in Delta tables are enforced at write time and reject rows with duplicate keys

CHECK and NOT NULL are enforced Delta table constraints — violations cause write transactions to fail

Auto Loader's _rescued_data column captures type-mismatched fields without stopping the ingestion stream

Lakeflow pipeline FAIL expectations emit per-record quality metrics in addition to WARN and DROP modes

Rows are frozen after the first successful load and cannot be updated

Prior versions of changed dimension rows are retained instead of being overwritten

All dimension rows are overwritten with the latest source values, keeping one row per supplier

All dimension updates are converted into append-only event records without effective date ranges

Disable ANSI mode for the pipeline so CAST returns NULL on failure instead of raising an error

Use to_date(raw_date, 'dd/MM/yyyy') and try_cast(raw_total AS DECIMAL(12,2)) instead of CAST

Change the silver table column types to STRING to avoid casting entirely

Add PRIMARY KEY(invoice_id) on the silver table to enforce data quality at write time

CHECK constraints are not supported on managed Delta tables in Unity Catalog

The constraint name chk_status is already reserved globally across the metastore

Some existing rows in the table contain order_status values not in the allowed set

The table must be converted to Delta format version 3 before CHECK constraints are supported

Set cloudFiles.schemaEvolutionMode to 'failOnNewColumns'

Enable the rescued data column (_rescued_data)

Add CHECK constraints on the bronze target table for all expected columns

Use Trigger.AvailableNow to process files in bounded micro-batches

Row A: Salesforce CRM → managed connector → streaming tables

Row B: ADLS Parquet → Auto Loader → streaming tables

Row C: Event Hubs → Kafka connector → continuous pipeline

Row D: Daily Salesforce → continuous pipeline as a Lakeflow Job task

Create one dynamic view per hospital with embedded CASE expressions for column masking

Partition the table by hospital_id and grant partition-level SELECT privileges to each coordinator group

Apply a row filter based on the current user's hospital assignment AND a column mask on the patient_name column

Use schema-level DENY grants to block lab technicians from accessing the patient_name column

The Log Analytics workspace is in a different Azure region than the Databricks workspace

No Databricks log categories were selected and enabled in the diagnostic settings

Azure Monitor requires a dedicated agent installed on each cluster node to collect operational logs

Photon must be disabled on the workspace before diagnostic log export is activated

Configure cluster log delivery to DBFS for all job clusters in each workspace

Create a central Log Analytics workspace and enable Databricks diagnostic log categories in each workspace Diagnostic settings blade

Install the Azure Monitor agent on each cluster driver node via an init script

Export the System.ClusterEvents Delta table to an Event Hubs topic for downstream ingestion

Increase auto-scaling maximum to 32 workers to add more parallelism

Enable Photon on the cluster to accelerate the slow shuffle operation

Diagnose and resolve the data skew causing the imbalanced task distribution

Switch to a serverless SQL warehouse for the dimension-loading transformation

notification: ALL_SUCCESS | publish: AT_LEAST_ONE_FAILED | cleanup: ALL_DONE

notification: AT_LEAST_ONE_FAILED | publish: ALL_SUCCESS | cleanup: ALL_DONE

notification: NONE_FAILED | publish: ALL_SUCCESS | cleanup: ALL_DONE

notification: ALL_DONE | publish: ALL_SUCCESS | cleanup: AT_LEAST_ONE_FAILED

Only publish_silver executes because it is listed first among downstream tasks

All downstream tasks are skipped because the upstream task failed

send_alert and archive_logs execute; publish_silver is skipped

Only archive_logs executes because ALL_DONE takes priority over all other conditions

Hard-code one task per expected partition value in the job definition YAML

Use a For each task with the partition list as input; make the summary task depend on the For each task

Trigger one child job per partition from a notebook using dbutils.notebook.run() and poll for results

Process all partitions using Python threading inside a single notebook task and write a summary at the end

The summary_report task's run_if is set to ALL_DONE instead of ALL_SUCCESS

The summary_report task does not list the For each task in its depends_on configuration