Big Data Engineer

Job Title: BIG DATA ENGINEER

LOCATION :: Rockville, MD

Client :: Coobsystems/Fintech

Responsibilities

• Design, develop, and maintain large-scale data pipelines using modern big data technologies such as Spark, Hadoop, Hive, and Trino.
• Build scalable and reliable solutions for data ingestion, transformation, storage, and analytics.
• Architect distributed data platforms capable of processing massive (petabyte-scale) datasets.
• Optimize and enhance existing data pipelines for performance, scalability, cost efficiency, and reliability.
• Implement automated testing frameworks and continuous validation for data quality and pipeline accuracy.
• Develop unit, integration, and end-to-end test strategies for data platforms.
• Collaborate with cross-functional teams to translate business requirements into scalable data solutions.
• Support data scientists and analytics teams by delivering high-quality, production-ready datasets.
• Monitor, troubleshoot, and resolve data pipeline issues in production environments.
• Investigate and resolve challenges such as data skew, resource constraints, job failures, and large-scale system bottlenecks.
• Apply Spark tuning techniques including partitioning, caching, broadcast joins, and performance optimization.
• Ensure strong software engineering practices, including version control, code quality, and CI/CD automation.
• Stay current with emerging big data, cloud, and AI technologies to continuously improve data architecture.
• Drive AI-enabled development practices, including prompt engineering, AI-assisted coding, and workflow optimization.
• Partner with stakeholders to ensure regulatory, governance, and financial data integrity requirements are met.

Qualifications

Required:

• Bachelor’s degree in Computer Science, Information Systems, or a related discipline, or equivalent practical experience.
• 5+ years of experience designing and implementing big data and distributed systems.
• Strong expertise in Apache Spark and its architecture (executors, stages, DAG, tasks).
• Hands-on experience with big data technologies such as Hadoop, Hive, and Trino.
• Strong proficiency in Python, Scala, or Java with a focus on scalable and modular code.
• Extensive experience writing advanced SQL queries including window functions, complex joins, and aggregations.
• Experience working with large-scale datasets and troubleshooting performance or scalability challenges.
• Hands-on experience with cloud platforms such as AWS, including S3, EMR, Glue, Lambda, and Athena.
• Experience designing and maintaining production ETL and data processing systems.
• Strong understanding of distributed system performance tuning and resource optimization.
• Experience implementing CI/CD pipelines and automated testing in data engineering environments.
• Strong understanding of Agile methodologies such as Scrum and Kanban.
• Excellent communication and collaboration skills.
• Ability to work in fast-paced, dynamic environments and manage competing priorities.

Preferred:

• Master’s degree in Computer Science, Data Engineering, or related field.
• Experience in Financial Services or regulated industries.
• Exposure to petabyte-scale data processing environments.
• Experience with Kubernetes, EKS, or serverless architectures.
• Experience with data lake and modern data platform architectures.
• AWS certifications.
• Experience with AI-assisted development tools such as GitHub Copilot, ChatGPT, Claude, or similar.
• Experience in prompt engineering, AI workflow design, and AI-driven productivity.
• Strong understanding of data governance, compliance, and security best practices.
• Experience driving organizational adoption of modern data and AI engineering practices.

Candidates are required to be living in the MVD area at the time of the interview.