Job SummaryDataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing.
DataPelago Nucleus is a universal data processing engine that is designed to accelerate the processing of diverse data - structured through unstructured - with any parallel processing framework - e.g., Apache Spark - on any infrastructure - including vectorized CPU and GPU. DataPelago Accelerator for Spark (DPA-S), our product based on DataPelago Nucleus, is running large-scale production applications of many globally renowned customers.
The Opportunity:As a software engineer for DataPelago data processing customer success, you will develop workload assets and automation tools for product validation and product support which will help ensure customers deploy and operate DataPelago data processing products successfully in production. You will apply AI effectively in combination with the knowledge from both product validation and product support to proactively avoid bugs and issues and to drastically reduce time to diagnose, mitigate, and resolve bugs and issues.
What You'll Do:- Develop synthetic workload queries and datasets for testing DPA-S, which reflect the essential environment and workload characteristics represented in the field.
- Develop analysis and diagnostic tools that analyze various logs from DPA-S, Apache Spark, and their deployment environment to aid in pinpointing the root cause of functional and performance anomalies and potential mitigations and resolutions.
- Field and address or escalate customer support issues as part of the product support rotation.
- Conduct end-to-end product validation, collaborating with others to define, review, and execute test plans.
- Analyze artifacts from product validation testing and issues raised by customers to identify potential configuration tuning opportunities, product usage recommendations, and product bugs.
- Leverage automation and AI techniques effectively in the entire process of product validation and product support.
- Create bug reports, RCA documents, configuration guides, and best practice documents.
What You'll Bring:- Bachelor's degree in computer science or a related field.
- 12+ years of experience related to development, validation and support of enterprise data processing software products.
- 2+ years of experience providing direct support to external customers.
- 2+ years of experience supporting Apache Spark pipelines for data engineering, analytics. Proficient in running Spark jobs from various clients, Spark resource provisioning, Spark parameter configuration and tuning, using Spark History Server and Spark UI, analyzing Spark event logs and other Spark logs.
- Solid knowledge of SQL and Python.
- Proficient in testing data products and test automation.
- Good understanding of the deployment, and operations of data processing products and pipelines in the cloud - in SaaS, VM-based, container-based, and serverless scenarios. Experience with Google Cloud Platform, AWS, and their Apache Spark related offerings preferred.
- Good customer engagement and communication skills.
Compensation:
The target salary range for this position is 196,350 - 292,600 USD. The salary offered will be determined by the candidate's location, qualifications, experience, and education and may be outside of this range. Final compensation packages are competitive and in line with industry standards, reflecting a variety of factors, and include a comprehensive benefits package. This may cover Health Insurance, Life Insurance, Retirement or Pension Plans, Paid Time Off, various Leave options, Performance-Based Incentives, employee stock purchase plan, and/or restricted stocks (RSU's), with all offerings subject to regional variations and governed by local laws, regulations, and company policies. Benefits may vary by country and region, and further details will be provided as part of the recruitment process.
