Role Summary
We are looking for a Software Engineer to build an end-to-end Digital Brain CI/CD asset deployment automation framework. The engineer will design and develop a multi-file asset deployment workflow where multiple passport files are ingested, validated, sorted into a dependency DAG, and executed in the correct deployment order using topological sorting.
The solution should ensure that each asset runs through the full deployment loop independently, while downstream assets and downstream systems are triggered only after required upstream deployments are successful.
Strong experience with Python, PySpark, Spark SQL, SQL, JSON, and YAML.
Hands-on experience with Databricks, Unity Catalog, Delta Lake, Databricks Workflows, and Databricks Asset Bundles.
Strong experience with Jenkins CI/CD pipelines, Jenkinsfiles, build stages, deployment stages, credentials, pipeline parameters, PR validation, and environment-based deployments.
Experience with Docker, Dockerfiles, developer containers, image versioning, dependency packaging, and containerized CI/CD execution.
Experience with Bitbucket, Git branching, pull requests, merge validation, release branches, and controlled deployment workflows.
Strong understanding of CI/CD automation for data platforms, including build validation, bundle validation, deployment gating, rollback/stop conditions, and auditability.
Experience building metadata-driven or configuration-driven deployment frameworks.
Strong knowledge of dependency management concepts including DAG creation, topological sorting, dependency validation, safe execution ordering, and failure propagation.
Experience generating and validating structured deployment artifacts such as JSON, YAML, SQL scripts, DDL files, metadata payloads, API payloads, and audit logs.
Experience integrating with Alation, SharePoint, Knowledge Graph platforms, Vector Databases, Ontology/Semantic Model platforms, REST APIs, and metadata services.
Never miss an opportunity! Create an alert based on the job you applied for.
