Big Data Engineer

Title :   Big Data Engineer

Location :: NYC, NY (Onsite)

Duration : Contract

Relocation is fine

Required Technical Skills

• As a core technical contributor, you are responsible for maintaining critical data pipelines and architectures across multiple technical areas within various business functions in support of the firm’s business objectives.

Required qualifications, capabilities, and skills

• § Formal training or certification on software engineering concepts and 5+ years applied experience
• § Extensive experience in managing the full lifecycle of data, from collection and storage to analysis and reporting.
• § Proficiency in one or more large-scale data processing distributions such as Java or Python Spark along with knowledge on Data Pipeline (DPL), Data Modeling, Data warehouse, Data Migration and so-on.
• § Hands-on practical experience in system design, application development, testing, and operational stability
• § Experience in developing, debugging, and maintaining code in a large corporate environment with one or more modern programming languages and database querying languages.
• § Solid understanding of agile methodologies such as CI/CD, Application Resiliency, and Security
• § Strong organizational, problem-solving, and critical thinking skills; Strong documentation skills
• § Proficiency in Python or Java and proficiency in Spark
• § Hands-on experience on AWS services and its components along with good understanding on Kubernetes. AWS Data Services: Proficiency in Lake formation, Glue ETL (or) EMR, S3, Glue Catalog, Athena, Kinesis (or) MSK, Airflow (or) Lambda + Step Functions + Event Bridge.
• § Data De/Serialization: Expertise in at least 2 of the formats: Parquet, Iceberg, AVRO, JSON-L
• § AWS Data Security: Good Understanding of security concepts such as: Lake formation, IAM, Service roles, Encryption, KMS, Secrets Manager

Preferred qualifications, capabilities, and skills

• § Familiarity with modern front-end technologies
• § Experience designing and building REST API services using Java
• § Exposure to cloud technologies - knowledge on Hybrid cloud architectures
• § DevOps: Linux Scripting, Jenkins, Git, CI/CD, JIRA, TDD skills
• Proficient in Big Data application development skills as well as multiple design techniques Working proficiency in Big Data development toolset to design, develop, test, deploy, maintain and improve
• Software Strong understanding of Agile methodologies with ability to work in at least one of the common frameworks Strong understanding of techniques such as Continuous Integration, Continuous Delivery, Test
• Driven Development, Cloud Development, application resiliency and security Proficiency in one or more general purpose programming languages Working proficiency in a portion of software engineering disciplines and demonstrates understanding of overall software skills including business analysis, development, testing, deployment, maintenance and improvement of software