Data Lake Tech. Lead
12 Months
Remote
Primary skills:
Apache Kafka
Databricks platform
DataHub or Collibra/Alation/Apache Atlas
RDBMS and Modern Data lake
Kubernetes - Functional knowledge
Required Qualifications
Technical Expertise around 12-15 years of experience.
Data Streaming & Processing
5+ years with Apache Kafka (streaming architecture, Kafka Connect, Schema Registry, stream processing)
3+ years with Databricks (Delta Lake, Apache Spark optimization, Unity Catalog, cluster management)
Deep understanding of data lake architecture patterns (Bronze/Silver/Gold, medallion architecture)
Data Governance & Cataloging
3+ years with DataHub or similar metadata management platforms (Alation, Collibra, Apache Atlas)
Deep experience building and operating enterprise data catalog systems
Expertise in automated metadata extraction, lineage tracking, and impact analysis
Experience with data quality frameworks and metadata-driven data operations
Knowledge of data governance policies, data classification, and compliance automation
Understanding of data discovery, access control, and self-service analytics enablement
Relational Database Systems
5+ years with enterprise RDBMS platforms including:
SQL Server(T-SQL, SSIS, SSRS, replication, Always On Availability Groups)
PostgreSQL(advanced query optimization, partitioning, extensions, streaming replication)
MySQL(replication, clustering, performance tuning)
Strong SQL skills (complex queries, stored procedures, window functions, query optimization)
Database design principles (normalization, indexing strategies, schema design, partitioning)
Change Data Capture (CDC) patterns and implementation (Debezium, Azure Data Factory, AWS DMS, custom solutions)
Database migration experience (schema migration, data migration, zero-downtime migrations)
Cloud & Infrastructure
5+ years with Azure or AWS Cloud including:
Azure: Data Lake Storage (Gen2), Event Hubs, AKS, Azure SQL, Key Vault, Azure AD, Monitor
AWS: S3, MSK/Kinesis, EKS, RDS/Aurora, Secrets Manager, IAM, CloudWatch
Cloud-native data services, networking, security, and IAM
3+ years with Kubernetes (deployment strategies, scaling, monitoring, service mesh, Helm)
3+ years with Terraform (modules, state management, multi-environment deployments, multi-cloud)
Programming & Development
Strong proficiency in any programming languages like C#, Go
Expert-levelSQLacross multiple database platforms
Experience with Python for data engineering tasks (preferred)
Familiarity with Shell scripting (Bash)
Leadership & Experience
7+ yearsin data engineering, platform engineering, or database engineering roles
3+ yearsin technical leadership capacity (Tech Lead, Principal Engineer)
Proven track record of delivering large-scale data infrastructure projects
Experience leading teams of 5-10+ engineers
Strong architectural design and system thinking capabilities
Experience migrating legacy RDBMS workloads to modern data lake architectures
Demonstrated ability to balance technical excellence with business needs
Regards,
Ram
Celersoft LLC
