Position: Lead Data Engineer With Strong Thoughtspot Experience
Location: San Francisco, CA - Hybrid
Duration: 9+Months with possible extension
Interviews: (3 Rounds): All the interviews within one week.
Main skills: SQL, Databricks, ADF, DataStage (or other ETL tool), SSAS cubes, Cognos/other, Tableau/other, ThoughtSpot
Job Description:
We are seeking a Lead Data Engineer to design, implement, and maintain our data infrastructure and business intelligence solutions. This role requires a technical expert who can architect scalable data solutions while collaborating across teams to drive business insights.
Required Experience:
- 8-10 years of software development and deployment experience
- At least 7 years of hands-on experience with:
- SQL
- Databricks
- Azure Data Factory (ADF)
- DataStage (or other ETL tools)
- SSAS cubes
- BI tools (Cognos, Tableau, ThoughtSpot, etc.)
- Strong Database knowledge (SQL & No-SQL preferred)
- Advanced SQL and Python skills
- Experience working with APIs for data collection/ingestion
Key Responsibilities:
Technical Development:
- Write SQL for processing raw data, Kafka ingestions, ADF pipelines, data validation, and QA
- Lead, design and implement innovative analytical solutions using Tableau, SQL, and Big Data technologies
- Experience with storing, joining, filtering, and analyzing data using SQL, Spark, Hive, etc.
Data Architecture & Design:
- Design comprehensive data architecture and data modeling strategies for BI systems
- Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
- Collaborate with stakeholders to translate business requirements into robust, scalable data models
- Perform detailed analysis of business problems and technical environments to design quality solutions
Integration & Implementation:
- Integrate data from various sources (databases, warehouses, external systems).
- Develop strategies for ETL processes ensuring data accuracy, consistency, and availability.
- Evaluate and select appropriate BI tools based on organizational needs.
- Run POCs and implement BI software, including visualization tools and analytics solutions.
Optimization & Governance:
- Monitor and optimize BI system performance for efficient data retrieval, processing, and reporting
- Identify and resolve performance bottlenecks through query tuning and data structure optimization
- Establish data governance policies ensuring data quality, consistency, and security
- Define access controls, privacy measures, and retention policies to meet regulatory requirements
Collaboration & Leadership:
- Build partnerships with architects, technical leads, and stakeholders across functional groups.
- Actively participate in code reviews and testing to ensure adherence to best practices.
- Work with product and engineering teams to evaluate requirements and new features.
- Build and foster a high-performance engineering culture.
- Mentor team members and provide tools and motivation for success.
Additional Skills & Requirements:
- Experience building data ingestion pipelines and data warehouse architecture
- Ability to write design documentation, source-to-target mapping documentation, and manage Confluence pages
- Experience converting business functionalities into technical Jira stories
- Strong experience with data modeling, design patterns, and building scalable BI solutions
- Knowledge of cloud platforms (Azure, AWS or equivalent)
- Experience analyzing large datasets to identify trends, patterns, and outliers
Never miss an opportunity! Create an alert based on the job you applied for.