Databricks Certified Data Engineer Associate Exam Guide

1. Understand the core components of the Databricks Data Intelligence Platform
2. Understand Databricks Data Intelligence Platform’s compute services, including their characteristics, limitations, and cost models, and select the most suitable option for each workload use case

Example scenario: A nightly batch job runs for 20 minutes and nobody wants to manage clusters. Which compute option would you choose?

1. Enable and detail data ingestion patterns, including batch, streaming, and incremental loading, and import data from sources such as local files, Lakeflow Connect standard connectors, and Lakeflow Connect managed connectors
2. Use the COPY INTO command to incrementally load files from cloud object storage into Unity-Catalog-governed tables
3. Use Auto Loader with schema enforcement and schema evolution in batch modes to land data into Unity-Catalog-governed tables
4. Configure Lakeflow Connect to reliably ingest data from diverse enterprise sources into Unity-Catalog-governed tables
5. Use JDBC/ODBC or REST clients in notebooks to land data into cloud storage or directly into Unity-Catalog-governed tables, usually orchestrated and scheduled with Lakeflow Jobs
6. Prioritize between Auto Loader, Lakeflow Connect (standard and managed connectors), partner connectors, and other ingestion methods based on technical requirements such as data volume, ingestion frequency, data types, and governance needs with Unity Catalog
7. Ingest semi-structured and unstructured data via Lakeflow Connect and other managed connectors into Unity-Catalog-governed Delta tables

Example scenario: New JSON files land in cloud storage every few minutes and fields are sometimes added. How would you load them so new fields do not break the pipeline?

1. Implement data cleaning by reading bronze tables with PySpark/SQL, cleaning nulls, standardizing data types, and writing to new silver tables
2. Combine DataFrames with operations such as Inner join, left join, broadcast join, multiple keys, cross join, union, and union all
3. Manipulate columns, rows, and table structures by adding, dropping, splitting, renaming column names, applying filters, and exploding arrays
4. Perform data deduplication operations and aggregate operations on DataFrames, such as count, approximate count distinct, and mean, summary
5. Understand the basic tuning parameters and re-measure the performance
6. Understand the difference between, and how to build, Gold layer objects such as materialized views, views, streaming tables, and tables for BI and analytics teams in Unity Catalog
7. Apply data quality checks and validation rules to ensure reliable Silver and Gold datasets

Example scenario: A silver table has duplicate orders that differ only by ingestion time. How would you keep the latest record per order in PySpark?

1. Implement control flows (retries and conditional tasks such as branching and looping) using Lakeflow Jobs for pipeline orchestration
2. Configure common tasks (notebook, SQL query, dashboard, and pipeline tasks) and their dependencies using Lakeflow Jobs and its DAG-based task graph
3. Implement job schedules using Lakeflow Jobs with an understanding of trigger types (scheduled, file arrival, and table update)
4. Choose between time-based and data-driven triggers based on data availability and pipeline dependencies

Example scenario: A report job should run only after the upstream table has new data, not at a fixed time. Which trigger would you configure?

1. Manage your code development workflow within the Databricks workspace UI, including creating and switching between branches in Databricks Repos, committing and pushing changes, and creating pull requests using Databricks Git integration
2. Understand environment-specific configuration using Automation Bundle (formerly Databricks Asset Bundles) variables and overrides while promoting the same codebase across dev, test, and prod targets
3. Deploy Declarative Automation Bundles (formerly Databricks Asset Bundles) to package, configure, and promote Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, and other workspace assets across dev, test, and prod environments
4. Understand the Databricks CLI to validate, deploy, and manage Declarative Automation Bundles (formerly Databricks Asset Bundles) and other workspace assets in automated CI/CD workflows

Example scenario: The same job must deploy to dev and prod with different cluster sizes. How would you set that up in a bundle?

1. Identify trends in job performance using the Lakeflow Jobs run history view to compare current execution times against historical baselines
2. Use the Lakeflow Jobs UI to monitor pipeline health by interpreting job statuses, viewing DAG-based task graphs to spot upstream blockers, and tracking pipeline run times and failure rates
3. Identify common performance bottlenecks such as data skew, shuffling, and disk spilling by interpreting stage-level metrics in the Spark UI
4. Understand the features of Liquid Clustering and predictive optimization
5. Diagnose cluster startup failures, library conflicts, and out-of-memory issues

Example scenario: One task in a join stage takes far longer than the others in the Spark UI. What is the likely cause, and what would you try?

1. Differentiate between managed and external tables in Unity Catalog and perform basic operations (create, modify, delete, and convert between managed and external tables) on them
2. Configure access controls using the UI and SQL by applying GRANT, REVOKE, and DENY privileges to principals (users, groups, and service principals) at appropriate levels of the security hierarchy
3. Understand column-level masking and row-level security to restrict data visibility based on user groups
4. Understand Unity Catalog ABAC policies to centrally control row-level filtering and column masking for sensitive data

Example scenario: Analysts may see every column of a customer table except email addresses, which only support staff can see. What would you configure?