|
Skill
|
Required /Desired
|
Amount
|
Actual Years of Experience
|
|
Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR....
|
Required
|
4
|
|
|
Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role; OR..
|
Required
|
4
|
|
|
Relevant certifications and coursework may substitute for formal education/experience requirements
|
Required
|
4
|
|
|
Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions
|
Required
|
4
|
|
|
ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)
|
Required
|
4
|
|
|
Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT
|
Required
|
4
|
|
|
Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
|
Required
|
4
|
|
|
Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI
|
Required
|
4
|
|
|
Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control
|
Required
|
4
|
|
|
Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning
|
Required
|
4
|
|
|
Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views
|
Required
|
4
|
|
|
Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2
|
Required
|
4
|
|
|
Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)
|
Required
|
4
|
|
|
Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing
|
Required
|
4
|
|
|
ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
|
Required
|
4
|
|
|
Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
|
Required
|
4
|
|
|
Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs
|
Required
|
4
|
|
|
Strong written and verbal communication skills
|
Required
|
|
|
|
Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks
|
Required
|
4
|
|
|
Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus)
|
Highly desired
|
|
|
|
Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held
|
Highly desired
|
|
|
|
Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance
|
Desired
|
4
|
|
