Data Engineer - III

Job Summary We are seeking a Data Engineer III to support the development and modernization of a cloud-based enterprise data platform. This role is responsible for designing, developing, and maintaining scalable data pipelines, integrating diverse data sources, and delivering high-quality data products that support analytics and business operations. The ideal candidate will have strong experience with Databricks, Python, PySpark, AWS, Spark-based data processing, and modern data engineering practices within Agile environments. Key Responsibilities Design, develop, and maintain scalable data pipelines to ingest, transform, catalog, and deliver trusted data from multiple enterprise data sources. Build and support end-to-end data pipelines for structured, semi-structured, and unstructured data using Apache Spark. Develop robust data engineering solutions using Python, PySpark, Databricks, and cloud-native technologies. Implement data cataloging, governance, and metadata management processes using enterprise data management tools. Build and maintain scalable analytical data stores and modern data lakehouse architectures. Monitor data pipelines and implement alerting, automation, and auto-remediation processes to improve reliability and availability. Troubleshoot and resolve issues affecting data pipelines, data quality, and analytical platforms. Apply security-first principles, automated testing, and data engineering best practices throughout the development lifecycle. Collaborate with product managers, data scientists, analysts, and business stakeholders to understand data requirements and deliver scalable solutions. Participate in Agile ceremonies and follow SAFe Agile development methodologies. Evaluate emerging technologies and recommend improvements to enhance data engineering capabilities and operational efficiency. Develop and maintain technical documentation for data pipelines, architectures, and operational processes. Required Qualifications Bachelor's degree in Computer Science, Information Systems, or a related field, or equivalent professional experience. 2+ years of experience with Databricks, Collibra, Starburst, or similar enterprise data management platforms. 3+ years of experience developing applications using Python and PySpark. Experience using Jupyter Notebooks for development, testing, and data analysis. Experience working with relational and NoSQL databases, including dimensional modeling and STAR schema design. 2+ years of experience with modern data engineering technologies including Amazon S3, Apache Spark, Apache Airflow, lakehouse architectures, real-time databases, Redshift, or Snowflake. Experience designing and supporting traditional ETL and Big Data solutions in on-premises or cloud environments. Hands-on experience with AWS data engineering services and cloud-native data platforms. Experience building end-to-end data pipelines to ingest, process, and transform structured, semi-structured, and unstructured data using Spark architecture. Strong analytical, troubleshooting, communication, and collaboration skills. Experience working within Agile or SAFe Agile development environments. Preferred Qualifications Experience implementing enterprise data governance and metadata management solutions. Experience with cloud-based data lakehouse architectures and distributed data processing frameworks. Experience deploying monitoring, alerting, and automated remediation for data platforms. Experience collaborating with cross-functional engineering, analytics, and business teams. Knowledge of data security, automation, and cloud data engineering best practices. Education: Bachelors Degree