Big Data ETL Developer (Remote)

Depends on Experience

Full Time

    Skills

    • Hadoop
    • Big Data
    • HDFS
    • Hive
    • Databricks
    • SQL
    • ETL
    • AWS
    • Java
    • Scala

    Job Description

     

    Job Title: Big Data ETL Developer

    Location: Remote/Hybrid if local to Maryland

    Job Type: Full-time with Sparksoft

     

    Position Summary: 

    This position will be supporting and working on one of Sparksoft’s technical projects. The ideal candidate will have experience in developing data ingestion and transformation ETL processes for analytical data loads, from a technical perspective.

    Responsibilities: 

    • Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.

    • Transition of legacy ETLs with Java and Hive queries to Spark ETLs.

    • Develop Spark programs on Databricks to perform tasks like data cleansing, validation, standardization, and then applied transformations as per the use cases.

    • Design, develop, test and release ETL solutions including data quality validations and metrics that follow data governance and standardization best practices.

    • Designing and Developing Databricks engineering solutions on AWS cloud.

    • Good experience working on AWS Cloud.

    • Performance tuning of end-to-end ETL integration processes.

    • Monitoring performance and advising any necessary infrastructure changes

    • Analyze and recommend optimal approach for obtaining data from diverse source systems.

    • Work closely with the data architects, who maintain the data models, including data dictionaries/metadata registry.

    • Interface with business stakeholders to understand requirements and offer solutions.

     

    Required Skills:

    • Proficient understanding of distributed computing principles and hands on experience in Big Data Analytics and development

    • Good knowledge of Hadoop and Spark ecosystems including HDFS, Hive, Spark, Yarn.

    • Experience in designing and developing applications in Spark using Scala that work with different file formats like Text, Sequence, Xml, parquet and Avro

    • Experience of using Databricks

    • Strong SQL coding; Strong experience in Scala and Java , understanding of SQL and No SQL statement optimization/tuning.

    • Experience in building ,maintaining Unix shell script

    • Ability to lead designing and implementation of ETL data pipelines.

    • Experience developing data quality checks and reporting to verify ETL rules and identify data anomalies.

    • Techniques for testing ETL data pipelines either manual or using tools.

    • Candidates must be able to obtain and maintain a Public Trust clearance

    • Candidates must have lived in the United States 3 out of the past 5 years

    Desired Skills:

    • Experience of using build tools Ant, SBT Maven (Good to have)

    • AWS development using big data technologies. (Good to have)

    • AWS cloud certified, Databricks and Snowflake experience a plus.

    Education/Experience Level:

    • Bachelor’s Degree with 5 years’ experience or 10+ years of experience in the software development field.

    • 5+ years of Bigdata ETL development experience.

    • 4+ years of AWS big data experience.

    • 3+ years of experience developing data validation checks and quality reporting.

    • 4+ years of experience tuning Spark/Java coding, SQL and No SQL.

    Sparksoft is a certified Capability Maturity Model Integration (CMMI) SVC and DEV Level 3, ISO 9001:2015, ISO 27001:2013, Small Disadvantaged Business (SDB), Women-Owned Small Business (WOSB), and Small, Women-owned, Minority-owned (SWaM), and MBE/DBE/SBE consulting firm. With our focused mission “to ignite innovation, inspire transformation, and implement digital solutions for a healthier nation”, we specialize in 6 specific digital health services: Test Automation, Cloud Services, DevOps Delivery, Cyber Security, Data Science, and Human-Centered Design. Since 2004, our exceptionally skilled people, proven leadership, and optimized processes all work together relentlessly to continuously push for more efficient solutions.

    Sparksoft is an Affirmative Action/Equal Opportunity Employer and does not discriminate against any applicant for employment or employee because of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veteran status, or any other characteristic prohibited under Federal, State, or local laws.

    In accordance with the Executive Order on Ensuring Adequate COVID Safety Protocols for Federal Contractors, Sparksoft Corporation is complying with the requirements that all employees assigned to a federal contract be vaccinated. Employees in need of an exemption from this policy due to a medical reason or because of a sincerely held religious belief must submit a physician’s note for a medical accommodation or a religious request for accommodation to the human resources department to begin the interactive accommodation process as soon as possible. Accommodations will be granted where they do not cause Sparksoft Corporation undue hardship or pose a direct threat to the health and safety of others. New hires must show proof of vaccination.

    If you need accommodation seeking employment with Sparksoft Corporation, please email Sparksoft.Accommodations@sparksoftcorp.com or call 410-424-7700. Accommodations are made on a case-by-case basis.

    At Sparksoft Corporation, we take security and protection of personal information very seriously. We will never ask you to send private personal information over email. Accordingly, we ask you to immediately contact our security team via email at abuse@sparksoftcorp.com upon receiving a suspicious request.