Senior Data Engineer

Job Title: Senior Data Engineer (W2 Position)

Experience: 12+ Years
Employment Type: W2 Only
Location: San Antonio, TX, USA
Duration: Long-Term Contract 

Job Summary

We are seeking a highly experienced Senior Data Engineer with 12+ years of IT experience in designing, building, and optimizing scalable data platforms and modern data pipelines. The ideal candidate will possess strong expertise in Python, SQL, Spark, Databricks, Snowflake, AWS, ETL/ELT, and cloud-native data engineering solutions. This role involves developing high-performance data architectures, implementing data governance standards, and enabling advanced analytics and business intelligence initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time data processing.
• Build enterprise-grade ETL/ELT workflows using modern cloud technologies.
• Develop and optimize large-scale data solutions using Python, SQL, Spark, and Databricks.
• Design and implement data lakes, data warehouses, and Lakehouse architectures.
• Build robust data models to support analytics, reporting, and machine learning workloads.
• Integrate data from multiple structured, semi-structured, and unstructured sources.
• Optimize SQL queries, Spark jobs, and overall pipeline performance.
• Develop scalable data ingestion frameworks using Kafka or other streaming platforms.
• Implement Infrastructure as Code (IaC) for data platform deployment and automation.
• Collaborate with Data Scientists, BI Developers, Analysts, Architects, and business stakeholders.
• Ensure data quality, lineage, governance, security, and compliance across enterprise data platforms.
• Implement monitoring, logging, alerting, and troubleshooting for production data pipelines.
• Support CI/CD automation for data engineering deployments.
• Perform root cause analysis and resolve production issues.
• Mentor junior data engineers and establish engineering best practices.

Required Qualifications

• Bachelor''s or Master''s degree in Computer Science, Information Systems, Engineering, or related field.
• 12+ years of overall IT experience.
• 8+ years of hands-on Data Engineering experience.
• Strong programming expertise in Python and advanced SQL.
• Extensive experience with Apache Spark and PySpark.
• Hands-on experience with Databricks.
• Strong expertise in Snowflake data warehouse.
• Experience with cloud platforms, preferably AWS.
• Strong experience building ETL/ELT pipelines.
• Experience with Airflow or similar workflow orchestration tools.
• Hands-on experience with Git and CI/CD pipelines.
• Strong understanding of data modeling, data warehousing, and dimensional modeling.
• Experience working in Agile/Scrum environments.
• Excellent analytical, troubleshooting, and communication skills.

Technical Skills

Programming Languages

• Python
• SQL
• PySpark
• Scala (Preferred)
• Shell Scripting

Big Data Technologies

• Apache Spark
• PySpark
• Hadoop Ecosystem
• Hive
• Kafka
• Delta Lake

Cloud Platforms

• AWS (S3, EMR, Glue, Lambda, Redshift, Athena, EC2, IAM, CloudWatch)
• Azure (Preferred)
• Google Cloud Platform (Preferred)

Data Platforms

• Databricks
• Snowflake
• Redshift
• Azure Synapse Analytics
• BigQuery (Preferred)

ETL/ELT & Orchestration

• Apache Airflow
• AWS Glue
• Informatica
• Talend
• dbt
• Azure Data Factory (ADF)

Databases

• SQL Server
• PostgreSQL
• Oracle
• MySQL
• MongoDB
• DynamoDB

Streaming Technologies

• Apache Kafka
• Kafka Connect
• Amazon Kinesis

Data Warehousing

• Star Schema
• Snowflake Schema
• Dimensional Modeling
• Data Vault
• Lakehouse Architecture

DevOps & CI/CD

• Git
• GitHub
• GitLab
• Jenkins
• Azure DevOps
• Docker
• Kubernetes (Preferred)
• Terraform (Preferred)

Monitoring & Logging

• CloudWatch
• Datadog
• Splunk
• Prometheus
• Grafana

Data Governance & Security

• Data Lineage
• Data Quality
• Data Catalog
• IAM
• Encryption
• RBAC
• GDPR
• HIPAA
• CCPA

Methodologies

• Agile
• Scrum
• DevOps
• DataOps

Preferred Qualifications

• AWS Certified Data Engineer – Associate.
• AWS Certified Solutions Architect.
• Databricks Certified Data Engineer Professional.
• Snowflake SnowPro Certification.
• Experience with Lakehouse architecture and Delta Lake.
• Experience supporting AI/ML data pipelines.
• Experience with real-time event-driven architectures.
• Knowledge of Terraform and Infrastructure as Code.
• Exposure to MLOps and feature engineering pipelines.

Soft Skills

• Excellent verbal and written communication skills.
• Strong analytical and problem-solving abilities.
• Ability to lead data engineering initiatives and mentor team members.
• Strong stakeholder management and cross-functional collaboration skills.
• Ability to work independently in fast-paced enterprise environments.
• Strong attention to detail and commitment to delivering high-quality data solutions.