Data Engineer

Data Engineer

Join our dynamic team as a Data Engineer, where you will play a pivotal role in designing, constructing, and managing our data architecture. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data systems are robust, efficient, and scalable. Your primary focus will be on developing data pipelines, optimizing data storage, and ensuring data quality and integrity.

Responsibilities

• Design, build, and maintain efficient data pipelines and ETL processes to support data-driven decision making.
• Collaborate with data scientists, analysts, and business stakeholders to understand their data needs and provide solutions that meet those requirements.
• Develop and manage data storage solutions, ensuring data is stored in a way that is both secure and easily accessible.
• Implement data quality checks and validation processes to ensure the accuracy and reliability of our data.
• Optimize data retrieval and query performance to support real-time analytics and reporting.
• Monitor and maintain data infrastructure, ensuring high availability and reliability of data systems.
• Stay up-to-date with the latest trends and technologies in data engineering and recommend improvements to our data architecture.
• Write and maintain comprehensive documentation for data processes and systems.

Requirements

• Strong understanding of data engineering principles and practices.
• Proficiency in programming languages such as Python, SQL, and Java.
• Experience with data pipeline tools and frameworks such as Apache Kafka, Apache Spark, and Airflow.
• Knowledge of data storage solutions, including relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• Familiarity with cloud platforms such as AWS, Google Cloud, or Azure.
• Strong problem-solving skills and the ability to work independently and as part of a team.
• Excellent communication skills and the ability to explain complex technical concepts to non-technical stakeholders.

• Experience with data visualization tools such as Tableau or Power BI.
• Knowledge of containerization technologies such as Docker and Kubernetes.
• Experience with big data technologies such as Hadoop and Spark.
• Familiarity with machine learning frameworks and tools.
• Bachelor’s degree in Computer Science, Engineering, or a related field.