Job Title: Data Architect Duration: 12 months to start ( Will go long term) Position Type: Long Term contract. * Understanding of Snowflake's cloud data warehouse architecture, including components like warehouses, databases, tables, stages etc * Familiarity with Snowflake-specific features like Time Travel, Zero-Copy Cloning, and Snowflake Streams for CDC * Familiarity with the SnowSQL command-line interface for interacting with Snowflake. * Proficiency in SQL, including Snowflake-specific syntax and functions. Ability to write SQL queries for complex data transformations. * Ability to write snowflake UDFs (Python, SQL, JavaScript) and stored procedures * Ability to read, review and implement data models optimized for Snowflake's architecture * Experience with loading data into Snowflake from various sources (files, databases, APIs) and unloading data from Snowflake to other destinations * Understanding of snowflake data shares * Knowledge of Snowflake's security features, including role-based access control (RBAC), data masking, and encryption * Understanding of query performance tuning in Snowflake, including the use of clustering, micro-partitions, and result caching * Understanding and ability to implement code based on recommended best practices * Building and orchestrating ETL/ELT pipelines using MWAA AWS and Snowflake's native functionality * Experience using Snowflake features like Snowpipe for continuous data ingestion from S3 * Best practices for loading bulk data into Snowflake using COPY INTO from S3. * Managing semi-structured data formats (e.g., JSON, XML, Parquet) and working with VARIANT data type in Snowflake * Knowledge of Snowflake's integration with external identity provider solution. * Understand Service accounts in Snowflake * Integrating Snowflake with AWS S3 for storage and AWS services for orchestration. * Working with tools like Informatica IICS, Python for data extraction and loading into Snowflake * Setting up DAGs (Directed Acyclic Graphs) in Airflow to manage workflow dependencies, schedules, and task automation * Understanding of how to structure workflows to efficiently orchestrate jobs, including task dependencies, retries, and task parallelization * Writing Python scripts to create custom Airflow operators and hooks * Understand commonly used Python libraries in data engineering (example: Pandas) * Ability to write unit tests in Python * Experience with AWS MWAA, including the setup of Airflow environment variables, connections, and Airflow plugins * Troubleshooting and monitoring Airflow tasks and DAGs using the Airflow UI and logs * Experience setting up continuous integration and deployment pipelines for DAGs and code artifacts * Automating the deployment of new workflows and updates to existing DAGs using GitHub Actions * Knowledge of Git for managing and deploying code, both for Snowflake scripts and Airflow DAGs * Write automation scripts in Python, Shell, or similar languages to orchestrate jobs * Diagnose and resolve issues related to data pipelines, performance optimization * Familiarity with AWS services, including S3, Lambda (nice to have), CloudWatch, for data engineering workflows * Experience with different schema designs (Star, Data Vault 2.0) * Designing and implementing raw, stage, core, and presentation layers * Experience with reference data, historical data, and slowly changing dimensions (SCDs) * Ability to work independently, quickly evaluate opensource tools in collaboration with team members
Never miss an opportunity! Create an alert based on the job you applied for.
