Senior Data Engineer - (Databricks & Hadoop Expertise)

About the Company

Blackstraw.ai is an end-to-end technology services company specializing in Artificial Intelligence (AI) and Engineering solutions across Data Science, Data Engineering, LLM/GenAI and LLMOps. Founded in 2018, we help global enterprises across North America, Europe and Asia to build and operationalize AI systems that create measurable business impact. Our mission is to make AI adoption simpler, faster and scalable through a blend of deep domain expertise, reusable accelerators and proven engineering practices.

With a 600+ strong team of engineers, data scientists and AI specialists, we partner with organizations to deliver real-world outcomes in areas such as predictive analytics, computer vision, natural language processing and Generative AI. Headquartered in Florida (USA) with operations in Canada and India, Blackstraw.ai continues to empower global enterprises to unlock the true potential of AI.

Location: USA / Canada (Full‑time only)

Experience: 7 to 13 years

Must Have: Databricks & Hadoop

Role Summary:

As a Senior Data Engineer, you will design, build, and optimize large‑scale data pipelines and analytics solutions on Microsoft Azure. You will work extensively with Azure Databricks, Hadoop ecosystem tools, and modern data engineering frameworks to support enterprise‑grade data processing, transformation, and analytics workloads.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Databricks, Spark, and Hadoop ecosystem tools (HDFS, Hive, Sqoop, Oozie, YARN).
• Build and optimize ETL/ELT workflows leveraging Azure Data Factory, Databricks notebooks, and distributed processing frameworks.
• Develop high‑performance data models and data transformations for analytics, reporting, and machine learning workloads.
• Implement and manage big‑data solutions across Azure services including Data Lake Storage (ADLS), Synapse Analytics, Event Hub, and Azure SQL.
• Integrate structured and unstructured data sources using Spark, Kafka, Hadoop connectors, and cloud-native ingestion tools.
• Optimize Databricks clusters and Spark jobs for performance, scalability, and cost efficienc.
• Manage and monitor Hadoop-based data platforms, ensuring reliability, security, and high availability.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-ready data solutions.
• Implement data governance, security, and compliance using Azure IAM, RBAC, Purview, and best practices for enterprise data management.
• Troubleshoot and resolve issues across distributed systems, cloud data pipelines, and big‑data processing environments.
• Contribute to architecture design, technical documentation, and continuous improvement of the data engineering ecosystem.

Required Skills & Experience:

• Strong expertise in Azure Databricks, Spark (PySpark/Scala), and Delta Lake.
• Hands-on experience with Hadoop ecosystem: HDFS, Hive, Sqoop, Oozie, YARN, MapReduce.
• Proficiency in Azure Data Factory, ADLS, Synapse, Event Hub, and related Azure data services.
• Solid understanding of distributed systems, big‑data processing, and cloud-native architectures.
• Experience with SQL, NoSQL, and data modeling techniques.
• Strong programming skills in Python, Scala, or Java.
• Knowledge of CI/CD, DevOps, and version control (Git).

Preferred Qualifications:

• Azure certifications (e.g., DP‑203, Azure Data Engineer Associate)
• Experience with Kafka, Airflow, or other orchestration tools
• Exposure to machine learning workflows in Databricks

Soft Skills:

• Strong analytical and problem‑solving abilities.
• Clear communication and teamwork.
• Ability to learn quickly and adapt to new technologies.
• Ownership mindset and attention to detail.

Education:

• Bachelor’s degree in Computer Science, Engineering, IT, or equivalent experience.

Blackstraw provides equal employment opportunities to applicants and employees without regard to race, color, religion, age, sex, sexual orientation, gender identity/expression, national origin, marital status, protected veteran status, disability status, or any other basis as protected by federal, state, or local law