Senior Java BackendScala Data Engineer

MUST HAVE:

• Must have a minimum of 10-12 years of hands-on development experience implementing batch and events driven applications using Java, Kafka, Spark, Scala, PySpark and Python.
• Experience with Apache Kafka and Connectors, Java, Springboot in building event driven services, Python in building ML pipelines.
• Develop data pipelines responsible for ingesting large amounts of different kinds of data from various sources.
• Help evolve data architecture and work on Next Generation real time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems.
• Write code and develop worker nodes for business logic, ETL and orchestration processes.
• Develop algorithms for better attribution rules and category classifiers.
• Work with stakeholders throughout the organization to identify opportunities for leveraging company data to drive search, discovery, and recommendations.
• Work closely with architects, engineers, data analysts, data scientists, contractors/consultants and project managers in assessing project requirements, design, develop and support data ingestions and API services.
• Work with Data Scientists in building feature engineering pipelines and integrating machine learning models during the content enrichment process.
• Able to influence on priorities working with various partners including engineers, project management office and leadership.
• Mentor junior team members, define architecture, code review, hands-on development and deliver the work in sprint cycle.
• Participate in design discussions with Architects and other team members for the design of new systems and re-engineering of components of existing systems.
• Wear an Architect hat when required to bring new ideas to the table, thought leadership and forward thinking.
• Take a holistic approach to building solutions by thinking of the big picture and overall solution.
• Work on moving away from legacy systems into next generation architecture.
• Take complete ownership from requirements, solution design, development, production launch and post launch production support. Participate in code reviews and regular on-call rotations.
• Desire to apply the best solution in the industry, apply correct design patterns during development and learn best practices and data engineering tools and technologies.
• Performs any other functions and duties assigned and necessary for the smooth and efficient operation

EDUCATION & EXPERIENCE:

• BS or MS in Computer Science (or related field) with 12+ years of hands-on software development experience working in large-scale data processing pipelines.
• Must have skills are Apache Spark, Scala and PySpark with 2-4 years of experience building production grade batch pipelines that handle large volumes of data.
• Must have at least 8+ years of experience in Java and API / Microservices.
• Must have at least 5+ years of experience in Python.
• 5+ years of experience in understanding and writing complex SQL and stored procedures for processing raw data, ETL, data validation, using databases such as SQL Server, Redis and other NoSQL DBs.
• Knowledge of Big data technologies, Hadoop, HDFS.
• Expertise with building events driven pipelines with Kafka and Java / Spark.
• Expertise with Amazon AWS stack such as EMR, EC2, S3.
• Experience working with APIs to collect and ingest data as well build the APIs for business logic.
• Experience working with setting up, maintaining, and debugging production systems and infrastructure.
• Experience in building fault-tolerant and resilient systems.
• Experience in building worker nodes, knowledge of REST principles and data engineering design patterns.
• In-depth knowledge of Java, SpringBoot, Spark, Scala, PySpark, Python, Orchestration tools, ESB, SQL, Stored procedures, Docker, RESTful web services, Kubernetes, CI/CD, Observability techniques, Kafka, Release processes, caching strategies, versioning, B&D, BitBucket / Git and AWS Cloud Ecosystem, NoSQL Databases, Hazelcast.
• Strong software development, architecture diagramming, problem-solving and debugging skills.
• Phenomenal communication and influencing skills

NICE TO HAVE:

• Exposure to Machine Learning (ML), LLM models, using AI during coding, build with AI.
• Knowledge of Elastic APM, ELK stack and search technologies such as Elasticsearch / Solr.
• Some experience in workflow orchestration tools such as Air Flow or Apache NiFi.