Staff Data Engineer

Our Company

Changing the world through digital experiences is what Adobe's all about. We give everyone-from emerging artists to global brands-everything they need to design and deliver exceptional digital experiences! We're passionate about empowering people to create beautiful and powerful images, videos, and apps, and transform how companies interact with customers across every screen.

We're on a mission to hire the very best and are committed to creating exceptional employee experiences where everyone is respected and has access to equal opportunity. We realize that new ideas can come from everywhere in the organization, and we know the next big idea could be yours!

About the Role

We are looking for a highly skilled staff level Data Engineer to join our team as a technical lead. This role involves designing and handling scalable data platforms and machine learning solutions using Databricks and cloud technologies. Your tasks will include coordinating Databricks environments, optimizing cloud infrastructure, and developing production-grade data pipelines and ML models. A key aspect of the role is to ensure strong data governance, security, and compliance, while seamlessly integrating cloud-native services.

You will collaborate with data scientists, product teams, and external engineering partners to advance standard methodologies in MLOps, infrastructure automation, and data processing. This position is ideal for an individual skilled in managing data platforms and deploying ML models, while effectively working with multi-functional teams and senior collaborators. It presents chances for creativity, leadership, and significant impact through data-driven product decisions and large-scale data solutions.

Key Responsibilities

Databricks Platform Administration

• Coordinate and manage Databricks workspaces, clusters, and workflows.
• Configure role-based access control (RBAC), manage user groups, entitlements, and integrate with identity providers like Okta.
• Optimize cluster configurations, auto-scaling, and cost management.
• Monitor performance, debug Spark job failures, and fix performance bottlenecks in notebooks and SQL queries.

Cloud Platform Expertise

• Proficient in managing cloud infrastructure on AWS and Azure, including IAM, networking (VPCs, VNets), and storage integrations (S3, ADLS, Blob Storage).
• Experienced in securing cloud environments with identity management, role-based access control, and policy configurations.
• Integrate Databricks with cloud-native services and manage storage credentials/mount points.

Security & Networking

• Manage secure VPCs, networking, and cloud storage (S3, ADLS).
• Implement access controls, encryption, and ensure compliance with data governance policies (GDPR, HIPAA).

Automation & DevOps

• Automate Databricks tasks using REST APIs, Python or Bash.
• Implement job orchestration with Databricks Workflows, Apache Airflow.
• Develop CI/CD pipelines using Jenkins, Azure DevOps, or GitHub Actions.

Machine Learning & Data Science

• Apply Databricks to build, deploy, and manage scalable machine learning models in production environments.
• Monitor and maintain ML models in production, ensuring accuracy, performance, and business relevance.
• Implement anomaly detection systems to monitor key business metrics and ensure data quality and integrity.

Collaboration & Leadership

• Collaborate with data scientists, product, and external data engineering teams.
• Develop documentation, runbooks, and training sessions to upskill data scientists.
• Establish and advocate for standard methodologies in areas like notebook development, database design, scalable data processing, and MLOps.

Must-Have Skills

• 10+ years of professional experience in software engineering, data engineering, and machine learning.
• Experience with Databricks administration and database management, including building, deploying, and managing ML models and data pipelines.
• Proficiency in PySpark and Python (including Pandas, Scikit-learn) for scalable data processing.
• Ability to write and maintain scalable, distributed data processing code optimized for performance and reliability.
• Strong expertise in cloud platforms: AWS or Azure.
• Experience with CI/CD pipelines and infrastructure automation.
• Proficient in data governance, compliance, and security standard methodologies.
• Experience with container orchestration and deployment systems like Docker.
• Familiarity with networking concepts, including VPC setup, cloud storage integrations, and security configurations.
• Strong communication, collaboration, and guidance skills, demonstrated through successful teamwork with engineering, product, and business colleagues

Preferred Qualifications

• Experience with designing, building, and supporting machine learning pipelines and collaborating with ML teams.
• Understanding of generative AI models (e.g., GPT, Claude) and experience working with custom models tailored to specific product needs.
• Expertise in anomaly detection systems and related monitoring solutions.
• Knowledge of retrieval systems, including vector search and embedding-based search, to extract relevant information from large unstructured datasets (e.g., documents, knowledge bases).
• Advanced proficiency in SQL and Apache Spark tuning.

Application Window Notice
If this role is open to hiring in Colorado (as listed on the job posting), the application window will remain open until at least 04/02/2025 12:01 AM Pacific Time, in compliance with Colorado pay transparency regulations. If this role does not have Colorado listed as a hiring location, no specific application window applies, and the posting may close at any time based on hiring needs.

Our compensation reflects the cost of labor across several U.S. geographic markets, and we pay differently based on those defined markets. The U.S. pay range for this position is $145,600 -- $276,400 annually. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. Your recruiter can share more about the specific salary range for the job location during the hiring process.

At Adobe, for sales roles starting salaries are expressed as total target compensation (TTC = base + commission), and short-term incentives are in the form of sales commission plans. Non-sales roles starting salaries are expressed as base salary and short-term incentives are in the form of the Annual Incentive Plan (AIP).

In addition, certain roles may be eligible for long-term incentives in the form of a new hire equity award.

Adobe will consider qualified applicants with arrest or conviction records for employment in accordance with state and local laws and "fair chance" ordinances.

Adobe is proud to be an Equal Employment Opportunity and affirmative action employer. We do not discriminate based on gender, race or color, ethnicity or national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, or any other applicable characteristics protected by law. Learn more.

Adobe aims to make Adobe.com accessible to any and all users. If you have a disability or special need that requires accommodation to navigate our website or complete the application process, email or call .

Adobe values a free and open marketplace for all employees and has policies in place to ensure that we do not enter into illegal agreements with other companies to not recruit or hire each other's employees.