Sr. Data Engineer

• Write new ETLs to support both real time and new features data and reporting products.
• Work with product teams to gather and understand requirements, discuss approaches, design solutions based on the requirements and create design documents for implementation keeping in mind costs, capacity, shelf-life and choosing the right tools and technologies.
• Design and develop peripheral systems and processes - monitoring and alerting services, ETL processes, automated offline workflows, automated build and deployment pipelines, systems for gathering application performance metrics, ensuring efficiency in cost.
• Ensure best practices in system development ranging from test coverage, use of enterprise standards, adherence to target architecture, high-availability and scalability, low latency, and fault-tolerance, thus ensuring delivery of high-quality systems.
• Create detailed documentation on systems as well as processes ranging from comprehensive overview of systems, runbooks for developing, monitoring and troubleshooting system issues and recommendations on future improvements.
• Managing project delivery from a day-to-day standpoint – participating in daily agile stand-ups, escalating blockers to stakeholders, ensuring timely completion of tasks & overall project & spring goals. Communicating updates to stakeholders and working with them on timelines.

• Bachelor''s degree in Computer Science or related technical field. (NOT REQUIRED TO HAVE A DEGREE)
• Experience building Data Lakes, Data Warehouse using big data and cloud stack.
• 5+ years of experience on software development using Agile, dev ops that include designing, developing, and implementing large-scale data engineering solutions.
• Expert Level Proficiency in Spark streaming and spark concepts.
• Expert level proficiency in Python Programming language and PySpark with at least 4 years of experience.
• Experience working with Kafka streams and creating ingestion ETL''s.
• Expert level proficiency in ANSI SQL /Spark SQL with at least 5 year''s experience.
• Experience working with diverse data sources and data formats ( json, parquet, csv, psv, delta)
• Experience working with cloud services such as AWS along with cloud-based data sources(S3)
• Experience building ETL with batch and real time stream processing.
• Experience using version control system such as git and CI/CD

• Experience with building workflows and ETL''s in the databricks.
• Experience ingesting data produced by real time systems like Kafka
• Guiding and mentoring skills - guiding and mentoring junior members - helping them understand technologies, the project, best practices and in general setting them up for success.
• Experience building and deploying ML models using ML Flow.

• Strong verbal communication, ability to clearly communicate ideas, work status, advocate for (or against) a particular approach or design.
• Documentation skills - ability to make the path easier for new team members / new hires / interns by organizing knowledge in Confluence, wiki, or similar.
• Ability to manage self and own time, deliver work on time, evaluate and estimate work.
• Ability to work with various teams, including tech and product, incorporate their inputs into the work being done, and manage expectations on issues & enhancements.

Our benefits package includes: (EXCLUDE on perm placements) 

• Comprehensive medical benefits 
• Competitive pay 
• 401(k) retirement plan 
• …and much more!