Job Title: Data Engineer

Experience: 3–8 Years Employment Type: Full-Time Location: [Insert Location / Remote / Hybrid]

About the Role

We are looking for a skilled Data Engineer with 3–8 years of experience to design, build, and maintain scalable data pipelines and infrastructure. The ideal candidate is comfortable working across cloud platforms (AWS, Azure, or Google Cloud Platform), has strong programming and SQL skills, and can build reliable, efficient, and secure data systems that power analytics, reporting, and machine learning initiatives across the organization.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple structured and unstructured sources.
• Build and optimize data warehouses, data lakes, and lakehouses for analytics and reporting use cases.
• Develop and manage batch and real-time/streaming data pipelines using tools such as Apache Kafka, Spark Streaming, or Kinesis/Event Hubs/Pub-Sub.
• Write efficient, reusable, and well-tested code in Python/Scala/Java for data processing tasks.
• Design and optimize relational and NoSQL database schemas for performance and scalability.
• Implement data quality checks, validation frameworks, and monitoring/alerting to ensure pipeline reliability.
• Work with cloud-native data services (AWS Glue/Redshift/S3/EMR, Azure Data Factory/Synapse/Databricks, or Google Cloud Platform Dataflow/BigQuery/Composer) to build and manage infrastructure.
• Implement data orchestration using tools like Apache Airflow, Dagster, or Prefect.
• Collaborate with Data Scientists, Analysts, and Product teams to understand data requirements and deliver clean, reliable datasets.
• Apply data modeling best practices (star schema, snowflake schema, dimensional modeling, normalization/denormalization).
• Ensure data governance, security, and compliance (data masking, encryption, access controls, GDPR/HIPAA as applicable).
• Optimize query performance and manage cost-efficiency of cloud data infrastructure.
• Implement CI/CD pipelines for data engineering workflows using Git, Jenkins, GitHub Actions, or similar tools.
• Containerize and deploy data applications using Docker and Kubernetes where required.
• Participate in code reviews, architecture discussions, and documentation of data systems.
• Troubleshoot and resolve production data pipeline issues in a timely manner.

Required Skills

Programming & Scripting

• Strong proficiency in Python (Pandas, PySpark) and/or Scala/Java
• Strong command of SQL (complex queries, window functions, query optimization)
• Shell scripting (Bash) for automation

Big Data & Processing Frameworks

• Apache Spark (PySpark/Spark SQL)
• Apache Kafka or other streaming platforms
• Hadoop ecosystem (Hive, HDFS) – good to have

Cloud Platforms (Any one or more required)

• AWS: S3, Glue, Redshift, EMR, Lambda, Athena, Kinesis
• Azure: Data Factory, Synapse Analytics, Databricks, ADLS
• Google Cloud Platform: BigQuery, Dataflow, Dataproc, Cloud Composer, Pub/Sub

Data Warehousing & Databases

• Experience with modern data warehouses: Snowflake, Redshift, BigQuery, Synapse
• Strong knowledge of relational databases: PostgreSQL, MySQL, SQL Server, Oracle
• Familiarity with NoSQL databases: MongoDB, Cassandra, DynamoDB

Orchestration & Workflow Management

• Apache Airflow / Dagster / Prefect / Azure Data Factory pipelines

Data Modeling & Architecture

• Dimensional modeling, Star/Snowflake schema
• Data Lake / Lakehouse architecture (Delta Lake, Iceberg, Hudi)

DevOps & CI/CD

• Git, GitHub/GitLab/Bitbucket
• CI/CD tools: Jenkins, GitHub Actions, Azure DevOps
• Docker and basic Kubernetes knowledge
• Infrastructure as Code: Terraform / CloudFormation (good to have)

Data Quality & Governance

• Data validation frameworks (Great Expectations, dbt tests)
• Understanding of data governance, lineage, and cataloging tools (Collibra, Alation, Purview – good to have)

Additional Tools

• dbt (Data Build Tool) for transformation
• Version control and Agile/Scrum methodology exposure
• BI tool familiarity (Power BI, Tableau, Looker) – for data validation purposes

Preferred Qualifications

• Bachelor''s/Master''s degree in Computer Science, Information Technology, Engineering, or related field
• Cloud certification (AWS Certified Data Analytics, Azure Data Engineer Associate, or Google Cloud Platform Professional Data Engineer) is a plus
• Experience working in Agile/Scrum environments
• Exposure to Machine Learning pipelines / MLOps is a plus
• Strong problem-solving skills and ability to work independently as well as in a team

Soft Skills

• Strong analytical and problem-solving mindset
• Excellent communication skills to collaborate with cross-functional teams
• Ability to manage multiple priorities and work in a fast-paced environment
• Attention to detail and commitment to data accuracy