Hi
Professional greetings from E-IT please share with updated resume
Job Title: AWS Databricks Data Engineer
Location : Los Angeles CA (Hybrid)
Hire type : FTE / CTH
Job Description
We are seeking a highly skilled AWS Data Engineer with strong expertise in SQL, Python, PySpark, Data Warehousing, and Cloud-based ETL to join our data engineering team. The ideal candidate will design, implement, and optimize large-scale data pipelines, ensuring scalability, reliability, and high performance. This role requires close collaboration with cross-functional teams and business stakeholders to deliver modern, efficient data solutions.
Key Responsibilities
- Data Pipeline Development
- Build and maintain scalable ETL/ELT pipelines using Databricks on AWS.
- Leverage PySpark/Spark and SQL to transform and process large, complex datasets.
- Integrate data from multiple sources including S3, relational/non-relational databases, and AWS-native services.
- Collaboration & Analysis
- Partner with downstream teams to prepare data for dashboards, analytics, and BI tools.
- Work closely with business stakeholders to understand requirements and deliver tailored, highquality data solutions.
- Performance & Optimization
- Optimize Databricks workloads for cost, performance, and efficient compute utilization.
- Monitor and troubleshoot pipelines to ensure reliability, accuracy, and SLA adherence.
- Apply query optimization, Spark tuning, and shuffle minimization best practices when handling tens of millions of rows.
- Governance & Security
- Implement and manage data governance, access control, and security policies using Unity Catalog.
- Ensure compliance with organizational and regulatory datahandling standards.
- Deployment & DevOps
- Use Databricks Asset Bundles for deployment of jobs, notebooks, and configuration across environments.
- Maintain effective version control of Databricks artifacts using GitLab or similar tools.
- Use CI/CD pipelines to support automated deployments and environment setups.
Technical Skills (Required)
- Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Workflows, Delta Live Pipelines, Databricks Runtime, etc.).
- Proven ability to implement robust PySpark solutions.
- Handson experience with Databricks Workflows & orchestration.
- Solid knowledge of Medallion Architecture (Bronze/Silver/Gold).
- Significant experience designing or rebuilding batchheavy data pipelines.
- Strong background in query optimization, performance tuning, and Spark shuffle optimization.
- Ability to handle and process tens of millions of records efficiently.
- Familiarity with Genie enablement concepts (understanding required; deep experience optional).
- Experience with CI/CD, environment setup, and Git-based development workflows.
- Solid understanding of AWS cloud, including:
- IAM
- Networking fundamentals
- Storage integration (S3, Glue Catalog, etc.)
Preferred Experience
- Experience with Databricks Runtime configurations and advanced features.
- Knowledge of streaming frameworks such as Spark Structured Streaming.
- Experience developing real-time or near real-time data solutions.
- Exposure to GitLab pipelines or similar CI/CD systems.
Certifications (Optional)
- Databricks Certified Data Engineer Associate / Professional
- AWS Data Engineer or AWS Solutions Architect certification
Never miss an opportunity! Create an alert based on the job you applied for.
