Job Title :Senior Databricks Data Engineer - AI Data Platform
Employment Type: Full-Time
Location: Irving, TX (Hybrid - 2 to 4 days onsite per week)
Position Summary
We are seeking a Senior Databricks Data Engineer to build and manage the enterprise data foundation powering an Agentic AI ecosystem. This role will focus on designing and delivering scalable data pipelines, AI-ready datasets, and governed data products that support intelligent AI agents and advanced analytics.
The ideal candidate will have deep expertise in the Databricks Lakehouse Platform, data engineering best practices, and cloud-based data services. The engineer will be responsible for data ingestion, transformation, feature engineering, vector-ready data preparation, semantic retrieval enablement, and governance of enterprise data assets.
This role will support and integrate data from multiple enterprise systems, including:
- Oracle ERP
- ServiceNow
- Maintenance Systems
- Invoice Processing Systems
- Operational Data Sources
The successful candidate will ensure reliable, secure, and scalable access to enterprise data for AI and machine learning workloads.
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines using Databricks and Spark technologies.
- Build and maintain AI-ready data platforms supporting Agentic AI initiatives.
- Develop data ingestion frameworks integrating Oracle ERP, ServiceNow, and other enterprise applications.
- Create feature-engineered datasets for machine learning and AI use cases.
- Prepare and optimize vector-ready data for semantic search and retrieval applications.
- Implement Delta Lake architecture and manage data lifecycle processes.
- Develop and maintain data governance standards using Unity Catalog.
- Optimize data workflows, orchestration, and monitoring processes.
- Collaborate with data scientists, AI engineers, architects, and business teams to deliver enterprise AI solutions.
- Support CI/CD deployment processes and DevOps best practices.
Required Skills
Databricks & Data Engineering
- Databricks
- Apache Spark
- PySpark
- Delta Lake
- Delta Live Tables (DLT)
- Databricks Workflows
- Unity Catalog
Programming & Development
- SQL
- Python
- Data Modeling
- ETL/ELT Development
- REST APIs
- Integration Frameworks
Cloud & DevOps
- Azure Data Services
- AWS Data Services
- Git
- CI/CD Pipelines
- DevOps Practices
Mandatory Skills
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Databricks
- PySpark
- Delta Lake
- SQL
- Python
We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, citizenship status, age, disability, genetic information, protected veteran status, or any other characteristic protected by applicable law.
