Senior Data Engineer

Role: Senior Data Engineer

Location: Remote

Visa: No GC

Job Description

Experience working directly with business users and data consumers to gather and refine requirements.

Ability to translate business requirements into scalable data pipelines and platform solutions.

Experience guiding users from spreadsheet-based processes to enterprise data platforms.

Strong understanding of data ingestion patterns, including batch and streaming architectures.

Knowledge of file formats such as Parquet and Delta Lake, including ACID compliance, schema evolution, and time travel.

Experience with incremental data processing, including CDC and merge strategies.

Hands-on experience with API-based and file-based data ingestion.

Clear understanding of Spark architecture, including driver vs. executor roles, partitioning, and data distribution.

Experience making cluster sizing and autoscaling decisions based on workload requirements.

Ability to organize and manage data domains, catalogs, and governance structures.

Experience onboarding and integrating new data sources into enterprise data platforms.

Required Qualifications:

Overall, 12 years of experience required.

Hands-on experience delivering Databricks solutions and building end-to-end data pipelines.

Strong understanding of Databricks workspace strategy, Unity Catalog, and Medallion architecture.

Experience managing development, test, and production environments.

Ability to educate and support business users on Databricks capabilities and processes.

Comfortable working independently with minimal supervision.

Experience with CI/CD processes, including Git, pull requests, and promotion pipelines.

Knowledge of parameterization, secrets management, and configuration management across environments.