Lead Backend Developer

Job Title: Lead Backend Developer (Big Data, Python, DevOps)

Location: New York, NY

Department: Engineering / Technology

Job Overview:

We are seeking an experienced Lead Backend Developer with a strong background in Big Data, Python, and DevOps to join our dynamic engineering team. As the Lead Backend Developer, you will be responsible for architecting and building scalable backend systems and data pipelines that process large volumes of data. You will work closely with cross-functional teams to design, implement, and maintain infrastructure solutions that support data-driven applications. In addition, you will guide and mentor a team of developers, ensuring best practices in backend development, cloud infrastructure, and data operations.

Key Responsibilities:

• Big Data System Design & Development:
  Design and implement scalable backend systems and data pipelines using Big Data technologies (e.g., Hadoop, Spark, Kafka, Apache Flink, Hive, Presto) to process and analyze large datasets.
• Backend API & Service Development:
  Lead the development of RESTful APIs, microservices, and distributed systems that interact with big data platforms and provide seamless data access to other services.
• Cloud & DevOps Infrastructure:
  Manage and scale cloud-based infrastructure using platforms such as AWS, Azure, or Google Cloud. Implement and maintain DevOps practices, including CI/CD pipelines, automation, and containerization (e.g., Docker, Kubernetes).
• Data Pipeline Management:
  Oversee the development and optimization of ETL (Extract, Transform, Load) processes that handle real-time and batch data workflows. Leverage tools like Apache Airflow, Luigi, or Kafka Streams for orchestrating and scheduling data pipelines.
• Team Leadership & Mentorship:
  Provide technical leadership and mentorship to backend developers, ensuring adherence to coding standards, best practices, and modern data engineering techniques. Conduct code reviews, pair programming, and foster a culture of continuous improvement.
• Performance Tuning & Optimization:
  Optimize backend systems for performance, scalability, and reliability. Utilize profiling, monitoring, and logging tools to identify bottlenecks and improve the overall system architecture.
• Collaboration with Cross-Functional Teams:
  Collaborate with data scientists, analysts, and product teams to ensure backend systems align with business needs and data requirements. Work together to define data models, APIs, and infrastructure needs.
• Security & Compliance:
  Ensure backend systems and data pipelines follow security best practices, including data encryption, access control, and compliance with industry standards and regulations (e.g., GDPR, HIPAA).
• Automation & Monitoring:
  Implement monitoring and alerting systems to track backend system health and performance. Automate routine tasks and deployment processes to ensure seamless releases and minimal downtime.
• Documentation:
  Maintain up-to-date documentation for system architecture, data workflows, APIs, and infrastructure setups to ensure clear communication and knowledge sharing across the team.

Skills & Qualifications:

• Experience & Leadership:
  5+ years of backend development experience, with at least 2 years in a senior or lead position, preferably working with Big Data tools and cloud environments.
• Programming Languages:
  Proficiency in Python, with experience building scalable data systems, microservices, and APIs. Familiarity with Java or Scala is a plus, especially for Big Data tools like Apache Spark or Hadoop.
• Big Data Tools & Frameworks:
  Hands-on experience with Big Data processing frameworks such as Apache Hadoop, Apache Spark, Apache Kafka, Hive, and Presto. Experience with distributed data processing and real-time data streaming is essential.
• Cloud Platforms & DevOps Tools:
  Strong experience with cloud platforms (e.g., AWS, Google Cloud, Azure) and containerization technologies (Docker, Kubernetes). Experience with infrastructure-as-code tools like Terraform or CloudFormation is a plus.
• ETL & Data Pipelines:
  Expertise in designing and implementing ETL workflows using tools like Apache Airflow, Luigi, Kafka Streams, or AWS Glue.
• API & Microservices Development:
  Proven experience in building RESTful APIs and microservices that interact with large datasets, using modern frameworks like Flask, FastAPI, Django, or Spring Boot.
• CI/CD & Automation:
  Familiarity with CI/CD tools like Jenkins, GitLab CI, or CircleCI, and automation tools such as Ansible or Chef.
• Version Control & Collaboration:
  Expertise in using Git for version control and collaborative software development.
• Performance Tuning:
  Experience in optimizing the performance of backend systems and databases, including caching, indexing, and query optimization techniques.
• Security Best Practices:
  Understanding of security protocols for backend systems, including OAuth, JWT, data encryption, and secure API design.

Preferred Qualifications:

• Advanced Data Engineering Skills:
  Familiarity with advanced data engineering concepts like Data Lakes, Real-time Analytics, and Event-Driven Architectures.
• Data Science Collaboration:
  Experience working closely with data scientists to operationalize machine learning models and analytics solutions.
• Data Governance & Compliance:
  Knowledge of data governance practices, and familiarity with compliance regulations (e.g., GDPR, HIPAA).
• Agile Methodologies:
  Experience working in an Agile development environment, including sprint planning, daily standups, and retrospectives.

Education:

• Bachelor s or Master s degree in Computer Science, Engineering, Data Science, or a related field, or equivalent work experience.