Job Title: Senior Data Platform Architect - Databricks (PySpark)
Location: Dallas, TX - Onsite
Job Summary
We are looking for an experienced Senior Data Platform Architect - Databricks (PySpark) to design and drive enterprise-scale data platform architecture and modernization initiatives. The role requires strong expertise in Data Engineering, Databricks Lakehouse, Python, PySpark, Spark SQL, distributed data processing, and cloud-native data architectures.
The selected candidate will collaborate closely with the Enterprise Architecture team to define target-state architectures, establish technology roadmaps, and develop scalable, secure, and reusable data architecture solutions. The role will also provide technical leadership to engineering teams and support AI/ML and Generative AI initiatives through the development of AI-ready data platforms.
Key Responsibilities
- Design and define scalable enterprise data platform architectures using Databricks Lakehouse and modern cloud-native technologies.
- Collaborate with Enterprise Architecture teams to establish target-state architectures, technology roadmaps, and data modernization strategies.
- Create, maintain, and govern reference architectures, solution blueprints, design patterns, and architecture standards.
- Design and implement scalable batch, streaming, and real-time data processing solutions.
- Provide technical direction for distributed data processing using Python, PySpark, Spark SQL, and the Hadoop ecosystem.
- Drive the architecture and evolution of enterprise data platforms across Azure, AWS, or Google Cloud Platform cloud environments.
- Define architecture standards and best practices for data ingestion, transformation, storage, processing, and integration.
- Support AI/ML and Generative AI initiatives by designing AI-ready data platforms and enabling scalable, high-quality data pipelines.
- Conduct architecture reviews and ensure solutions meet enterprise requirements for security, scalability, performance, governance, and reliability.
- Evaluate emerging data technologies and recommend suitable solutions aligned with enterprise architecture standards.
- Mentor data engineering teams and provide technical leadership throughout solution design, development, and implementation.
- Collaborate with business stakeholders, technical teams, and architecture leaders to influence technology decisions and ensure alignment with business objectives.
Required Technical Skills & Experience
- 7-12 years of experience in Data Engineering, Data Platforms, or Solution Architecture.
- Strong hands-on experience with Python, PySpark, and Spark SQL.
- Strong understanding of Apache Spark and distributed data processing.
- Experience working with the Hadoop ecosystem, including technologies such as HDFS, Hive, and related big data platforms.
- Hands-on expertise in Databricks Lakehouse architecture and implementation.
- Experience designing and implementing modern cloud-native data architectures.
- Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud Platform (Google Cloud Platform).
- Strong knowledge of batch, streaming, and real-time data processing architectures.
- Experience defining enterprise data architecture standards, reference architectures, solution blueprints, and reusable design patterns.
- Understanding of data security, governance, scalability, performance optimization, and architecture best practices.
- Strong communication, stakeholder management, and technical leadership skills.
Preferred Qualifications
- Databricks certification.
- Microsoft Azure, AWS, or Google Cloud Platform cloud certification.
- Previous experience working with Citibank consumer banking or wealth management platforms.
- Exposure to Enterprise Architecture frameworks, preferably TOGAF.
- Experience in Banking, Financial Services, or large-scale enterprise data modernization programs.
- Exposure to AI/ML and Generative AI data platform architecture.
Key Skills
- Data Platform Architecture
- Databricks Lakehouse
- Python
- PySpark
- Apache Spark / Spark SQL
- Hadoop Ecosystem
- Distributed Data Processing
- Cloud Data Architecture - Azure / AWS / Google Cloud Platform
- Batch, Streaming & Real-Time Data Processing
- Enterprise Architecture
- Reference Architecture & Solution Blueprints
- Data Governance & Security
- AI/ML & Generative AI-Ready Data Platforms
- Technical Leadership & Stakeholder Management
