Position: Lead - Software Engineering
Location: Bellevue/ Frisco
Duration: 6 Months
Work Type: Onsite
Job Description:
DATA PIPELINE DEVELOPMENT
· Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).
· Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).
· Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.
· Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.
· Provide technical leadership for multi-source, high-volume data integration pipelines.
PLATFORM & TOOLING
· Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.
· Define best practices for:
· o Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency)
· o Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization)
· o dbt (modular design, testing frameworks, CI/CD integration, reusable components)
· Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring.
· Evaluate and drive tooling strategy and platform standardization across teams.
CLOUD INFRASTRUCTURE
· Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents.
· Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning.
· Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation).
· Ensure platforms are scalable, secure, and production-ready.
LANGUAGES & FRAMEWORKS
· Provide deep technical leadership in:
· o Advanced SQL (query tuning, execution optimization, complex transformations)
· o Python / PySpark for distributed data processing
· Guide teams on best practices, reusable frameworks, and performance optimization.
· Oversee development standards for Spark, Scala (where applicable), and automation scripting.
STREAMING & REAL-TIME
· Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming.
· Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance.
· Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting.
DATA QUALITY & TESTING
· Establish enterprise frameworks for data quality, validation, and observability.
· Define standards for:
· o Automated testing (unit, integration, regression)
· o Data validation (completeness, accuracy, consistency)
· o Data quality tools (dbt tests, Great Expectations, custom frameworks)
· Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines.
· Drive proactive data quality and governance practices across teams.
DATA MODELING SUPPORT
· Interpret and implement architect-defined enterprise data models (star, snowflake, data vault).
· Provide guidance on:
· o SCD (Type 1/2) strategies
· o Partitioning, clustering, and performance optimization
· Collaborate with architects to evolve scalable and reusable data models.
· Support semantic layer enablement for analytics and reporting.
DEVOPS & ENGINEERING PRACTICES
· Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps).
· Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines).
· Standardize environment promotion (dev → QA → prod) and release management.
· Drive adoption of engineering excellence practices including reusable frameworks and templates.
SECURITY & GOVERNANCE
· Lead implementation of enterprise-grade security and governance controls:
· o RBAC, row/column-level security
· o PII and CPNI compliance (TISS-310)
· Define standards for secrets management and secure pipeline design.
· Ensure data lineage, auditability, and compliance readiness across platforms.
FINANCE DOMAIN KNOWLEDGE
· Deep understanding of finance and revenue data domains, including:
· o Billing and revenue systems
· o GL structures and financial reporting
· o Revenue recognition and reconciliation
· o Period-end close cycles
· Guide engineering teams on accurate implementation of finance logic.
· Ensure high data integrity standards for regulated financial data.
SOFT SKILLS & COLLABORATION
· Act as a technical leader and escalation point across engineering teams.
· Partner with architects, product managers, analysts, and business stakeholders.
· Drive cross-team alignment and solution consistency.
· Communicate complex technical topics clearly to both technical and non-technical audiences.
· Lead incident reviews and ensure continuous improvement.
PRINCIPAL-LEVEL EXPECTATIONS
· Own and drive enterprise-level data engineering strategy and execution.
· Lead delivery of large, complex, multi-domain data platforms.
· Mentor senior engineers and define technical direction for the team.
· Drive tooling, architecture, and platform decisions across programs.
· Identify and lead technical debt reduction and modernization initiatives.
· Establish best practices, reusable components, and platform standards at scale.
· Influence cross-functional teams and leadership decisions on data platform strategy.
Never miss an opportunity! Create an alert based on the job you applied for.
