We are looking for a strong Python/PySpark Data Engineer to build scalable data ingestion and integration pipelines across enterprise data sources and metadata platforms. The engineer will work with metadata catalogs, databases, APIs, and batch/Realtime integration patterns while contributing to enterprise data management and data warehousing initiatives.
Key Responsibilities
• Develop robust data engineering solutions using Python, PySpark and Apache Spark.
• Consume and integrate metadata/data from Ab Initio Enterprise Meta>Environment (EME/EMH), Oracle databases, and ServiceNow APIs.
• Perform source-to-target mapping, data transformation, validation, and reconciliation.
• Build and maintain batch data ingestion and integration pipelines.
• Work with data warehouse concepts, data models, metadata management, data lineage and data quality.
• Integrate technical and business metadata from multiple enterprise sources.
• Understand existing/custom data engineering frameworks and quickly develop solutions within them.
• Troubleshoot performance, data quality, and integration issues across large datasets.
Required Skills
• Strong hands-on expertise in Python-based data engineering.
• Strong experience with PySpark / Apache Spark and large-scale data processing.
• Experience working with Oracle/SQL and relational databases.
• Experience consuming REST APIs and integrating API-based data sources.
• Good understanding of ETL/ELT, source-to-target mapping, data warehousing and dimensional/relational data modeling.
• Understanding of metadata management, business/technical metadata, lineage and data governance concepts.
• Experience building production-grade batch ingestion/integration pipelines.
• Ability to understand and work with custom enterprise data frameworks.
Good to Have
• Experience with real-time/streaming integration, such as Kafka or Spark Structured Streaming.
• Exposure to Google Cloud Platform (Google Cloud Platform) and its data services.
• Experience with enterprise metadata/catalog solutions, particularly Ab Initio metadata environments.
• Experience integrating with ServiceNow.
Ideal Profile: A hands-on engineer who is particularly strong in Python + PySpark, understands enterprise data and metadata ecosystems, and can quickly learn existing frameworks rather than relying only on off-the-shelf ETL tools.
