Job Title: Senior Data Engineer (Python, SQL, Apache Flink)
Location: Columbus, OH (Hybrid)
Duration: Long Term
Job Description
We are seeking an experienced *Senior Data Engineer* with strong hands-on expertise in *Python, SQL, Apache Flink, and Apache Kafka* to design, develop, optimize, and maintain scalable data engineering solutions.
The ideal candidate should have extensive experience building real-time and batch data pipelines, developing complex SQL transformations, writing production-quality Python code, and implementing distributed stream-processing solutions using Flink and Kafka.
Key Responsibilities
· Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL for data ingestion, transformation, validation, and processing.
· Write clean, efficient, reusable, and production-ready Python code and complex SQL queries to support enterprise data engineering solutions.
· Develop Python-based data integration frameworks to extract, process, and transform data from REST APIs, relational databases, files, and streaming platforms
· Utilize Python libraries such as Pandas, NumPy, PyArrow, and SQLAlchemy for data manipulation, processing, and database integration.
· Develop and optimize advanced SQL queries using CTEs, window functions, joins, subqueries, aggregations, and analytical functions.
· Design and maintain relational database schemas, data models, views, stored procedures, and reusable data transformation frameworks.
· Implement data cleansing, deduplication, enrichment, normalization, and complex business transformations using Python and SQL.
· Develop reusable Python modules, utility functions, SQL scripts, and automation frameworks to improve data processing efficiency.
· Implement incremental data processing, Change Data Capture (CDC), and data synchronization using Python and SQL.
· Optimize Python applications and SQL queries for large-scale datasets, focusing on execution performance, memory efficiency, scalability, and resource utilization
· Analyze SQL execution plans and improve database performance through indexing, partitioning, query optimization, and efficient data access patterns.
· Develop automated data quality checks, validation rules, reconciliation processes, and monitoring solutions using Python and SQL.
· Implement robust exception handling, logging, retry mechanisms, and error recovery to ensure reliable data pipeline execution.
· Build and integrate REST APIs and external data services using Python to support enterprise data ingestion and processing.
· Develop unit tests, integration tests, and automated testing frameworks using PyTest and SQL-based validation techniques.
· Design and implement SQL transformations for historical data processing, data migrations, reporting, and analytical workloads.
· Troubleshoot and resolve Python application errors, SQL performance bottlenecks, data discrepancies, and production pipeline failures.
· Collaborate with data architects, database developers, and engineering teams to develop scalable, reliable, and high-performance data solutions.
