Key Responsibilities
Design and develop scalable batch and streaming data pipelines using Apache Spark and Scala.
Build distributed data-processing solutions for high-volume structured and semi-structured datasets.
Develop reusable data frameworks, transformation logic, and ingestion components.
Optimize Spark applications for performance, scalability, partitioning, memory utilization, and execution efficiency.
Build and maintain ETL/ELT pipelines across enterprise data platforms.
Develop real-time or near-real-time data pipelines using technologies such as Kafka.
Integrate data from APIs, databases, files, event streams, and cloud storage platforms.
Partner with Data Architects, Data Scientists, Product teams, and Software Engineers to translate business requirements into technical data solutions.
Implement data-quality, reconciliation, lineage, monitoring, and error-handling controls.
Build cloud-native data solutions using AWS services.
Support CI/CD, automated testing, code quality, infrastructure automation, and production deployments.
Troubleshoot data-processing failures and performance bottlenecks across distributed systems.
Participate in Agile delivery, code reviews, design reviews, and production support.
Required Qualifications
5+ years of experience in Data Engineering, Big Data Engineering, or Software Engineering.
Strong hands-on development experience with Scala.
Strong expertise with Apache Spark / Spark SQL.
Experience building large-scale distributed data pipelines.
Strong SQL skills.
Experience with ETL/ELT and data transformation.
Strong understanding of distributed-computing concepts.
Experience with data modeling and large-scale data processing.
Experience with Git, CI/CD, and automated deployment practices.
Strong problem-solving and production troubleshooting capabilities.
Preferred Qualifications
Experience with AWS services such as:
S3
EMR
Glue
Lambda
Redshift
Athena
Kinesis
IAM
CloudWatch
Experience with Kafka or other event-streaming platforms.
Experience with Databricks.
Experience with Delta Lake, Iceberg, or similar modern data-lake technologies.
Experience with Python / PySpark in addition to Scala.
Experience with Hadoop ecosystem technologies such as Hive or HDFS.
Experience with NoSQL technologies such as DynamoDB or Cassandra.
Experience with Docker and Kubernetes.
Experience with Terraform or other Infrastructure-as-Code technologies.
Experience building secure data platforms in highly regulated environments.
Banking or financial-services domain experience is preferred.
Key Technical Skills
Programming: Scala, SQL, Python
Big Data: Apache Spark, Spark SQL, PySpark
Streaming: Kafka, Kinesis
Cloud: AWS – S3, EMR, Glue, Lambda, Redshift, Athena
Data Platforms: Databricks, Delta Lake, Hadoop, Hive
Databases: PostgreSQL, Oracle, SQL Server, DynamoDB, Cassandra
DevOps: Git, Jenkins/GitHub Actions, CI/CD, Docker, Kubernetes, Terraform
Data Engineering: ETL/ELT, Data Modeling, Data Quality, Data Lineage, Performance Optimization
Methodology: Agile / Scrum
Note- All candidate who are legally authorized to work in US without sponsership can apply
