Data Engineer

Need Consultants with 12+ years of experience.

Need consultant only Local to Plano, TX to attend Onsite interview.

Job Title: Data Engineer

Location: Plano, TX Onsite

Duration: 12+ Months - W2 Contract

Consultant needs to attend Onsite interview in Plano, TX.

Experienced Senior Data Engineer with deep expertise in Python coding and scripting, Scala, and Databricks to design, optimize, and maintain robust data architectures.

Key Responsibilities

• Architect and implement scalable data pipelines using Python for advanced scripting, data transformation, and automation, including custom libraries for ETL processes.
• Develop and optimize Spark-based applications in Scala on Databricks, leveraging Delta Lake for ACID transactions, data versioning, and performance tuning.
• Design, build, and maintain end-to-end data workflows on Databricks, including Unity Catalog for governance, MLflow for model tracking, and serverless compute for cost efficiency.
• Write production-grade Python code for data ingestion from diverse sources (e.g., APIs, Kafka, S3), processing with Pandas/PySpark, and orchestration via Airflow or Databricks Workflows.
• Profile and optimize Scala/Spark jobs for large-scale datasets, reducing latency and resource usage by 30%+ through partitioning, caching, and broadcast joins.
• Collaborate with cross-functional teams to define data models, implement CI/CD pipelines, and ensure data quality via automated testing and monitoring.
• Mentor junior engineers on best practices for Python/Scala development, Databricks clusters, and cloud-native data engineering.
• Troubleshoot complex issues in distributed systems, using tools like Ganglia, Databricks diagnostics, and custom logging scripts.

Required Qualifications

• Bachelor's/Master's in Computer Science, Engineering, or related field.
• 13+ years of experience in data engineering, with 5+ years hands-on with Python (scripting, PySpark, Pandas) and Scala/Spark.
• Proven expertise in Databricks (workspaces, clusters, Delta Live Tables, SQL warehouses); Databricks certification preferred.
• Strong proficiency in building scalable ETL/ELT pipelines with Apache Spark, Kafka, and cloud storage (S3, ADLS).
• Experience with Python scripting for automation, including Boto3 for AWS integration and dbutils for Databricks.
• Familiarity with Scala for functional programming in Spark DataFrames/Datasets and UDFs.
• Solid understanding of data modeling (Star/Snowflake schemas), SQL optimization, and big data tools (Hadoop, Hive).
• Excellent problem-solving skills with a track record of deploying production systems handling 1TB+ daily data volumes.