Data Transformation Engineer

Centralizing data pipeline management with Matillion Hub involves

leveraging the Data Productivity Cloud to unify ETL/ELT tasks, visibility, and security in a single, browser-based interface

The Hub acts as a central control plane, connecting to various data sources and cloud data platforms (Snowflake, Databricks, Redshift) while using agents for execution.

• Deployment Options: Choose between full SaaS (Matillion hosts and manages infrastructure) or hybrid-SaaS (you deploy and manage agents in your private cloud).
• Agent Management: Use the Hub to manage, monitor, and configure data plane agents that run your pipelines.
• Git Integration: Enable Git features within the Hub to manage version control, allowing teams to contribute, push transformations, and manage data products centrally.

• Connectivity: Configure pre-built or custom connectors in the Hub to move data from databases (PostgreSQL), SaaS applications (, Google Analytics), or files (Amazon S3).
• Target Configuration: Define the default cloud data warehouse (Snowflake, etc.) for ELT processes.

• Shared Pipelines: Create reusable, shared pipelines to avoid duplicating logic across projects, ensuring consistency in auditing, data masking, and transformations.
• Advanced Scheduling: Use the Scheduler in the Designer interface to manage, monitor, and fine-tune pipeline execution times using either standard or cron expressions.
• Visibility & Monitoring: Use the centralized Dashboard to monitor all pipeline activities, view logs, and troubleshoot with a single click.
• User Roles & Security: Assign user permissions and roles within the Hub to ensure governance.

• Low-Code/High-Code Interface: Utilize the Designer interface within the Hub for visual, low-code transformation, or incorporate Python/SQL scripting.
• DataOps Enabled: Use the Matillion Pipeline Language (DPL) to automate the creation of folders, projects, and pipelines, integrating with tools like Azure DevOps.