Data Engineer (Mid)
Overview
We are seeking a mid-level Data Engineer to join a growing Enterprise AI team. This team is focused on building modern data platforms and AI-enabled solutions that drive business value, operational efficiency, and informed decision-making.
If you enjoy solving complex data challenges, building scalable data products, and working at the intersection of data engineering and AI, this is an excellent opportunity to make a meaningful impact.
About the Role
You will join a multidisciplinary team of data engineers, AI/ML engineers, data scientists, and product stakeholders building enterprise data and AI solutions.
In this role, you will build reliable data foundations that power AI-enabled products, operational intelligence, and business decision-making. You'll integrate enterprise data with AI-generated insights and contribute reusable data capabilities across a variety of AI and analytics initiatives.
This is a hands-on individual contributor role. You'll own well-defined data engineering components from development through production, partnering with senior engineers and architects on solution design and technical direction.
Key Responsibilities
- Build, maintain, and enhance scalable batch and near real-time data pipelines using Python, SQL, and Spark/PySpark.
- Integrate data from enterprise applications, operational systems, APIs, event streams, and AI inference outputs into reliable, analysis-ready datasets.
- Develop transformations that combine structured and semi-structured data across sources using consistent identifiers and timestamps, including handling time zones, duplicate records, and late-arriving events.
- Create and maintain data models and curated datasets for business analytics, AI model development and evaluation, and AI-enabled application workflows.
- Partner with AI/ML engineers to deliver data interfaces for inference workflows and capture model outputs, confidence scores, and user feedback to support continuous improvement.
- Implement data quality checks, reconciliation processes, monitoring, and alerting to ensure completeness, accuracy, freshness, and traceability.
- Troubleshoot pipeline failures and performance issues while implementing reliable retries, backfills, and incremental processing strategies.
- Collaborate with product managers, data scientists, analysts, and engineering partners to translate business requirements into practical data solutions.
- Follow data security, privacy, governance, access control, and retention requirements when handling enterprise and AI-generated data.
- Write maintainable, tested code and participate in code reviews, CI/CD practices, and documentation of schemas, data lineage, dependencies, and operational procedures.
Required Qualifications
- Bachelor's degree in Computer Science, Software Engineering, Information Systems, Engineering, Mathematics, Statistics, or a related quantitative discipline, or equivalent practical experience.
- 3-5 years of experience in data engineering or a closely related software engineering role.
- Strong hands-on proficiency with Python and SQL, including complex queries, transformations, optimization, and troubleshooting.
- Experience with Spark/PySpark and distributed data processing frameworks.
- Experience designing, building, and supporting ETL/ELT pipelines on cloud platforms such as Azure or AWS.
- Understanding of data modeling, relational databases, data lakes, data warehouses, and incremental processing patterns.
- Experience with workflow orchestration tools such as Apache Airflow or Databricks Workflows.
- Familiarity with event-driven and streaming data processing concepts including event time management, deduplication, and reliable delivery.
- Experience with Git, automated testing frameworks, monitoring tools, and production support.
- Strong communication skills and the ability to collaborate effectively with both technical and non-technical stakeholders.
Preferred Qualifications
- Hands-on experience with Databricks, Delta Lake, or lakehouse architectures.
- Experience with Databricks Asset Bundles (DABs) or similar deployment automation frameworks supporting CI/CD processes.
- Experience with streaming technologies such as Kafka, Azure Event Hubs, or Spark Structured Streaming.
- Experience working with customer, inventory, operational, financial, or transactional datasets.
- Experience preparing datasets for machine learning, anomaly detection, predictive analytics, or AI model evaluation.
- Familiarity with multimodal data, unstructured data processing, and time-based event correlation.
- Exposure to edge-to-cloud integration patterns, containerized workloads, Kubernetes, Docker, or infrastructure-as-code tools such as Terraform or Bicep.
