Job Title: AWS Data & Lakehouse Architect
Location: Plano, TX (Hybrid)
Type: Contract Position
Job description :
Role Overview
Client is looking for an experienced AWS Data & Lakehouse Architect to support enterprise data modernization initiatives involving migration of legacy/on-premises data platforms to a modern AWS-based Data Lakehouse architecture using Databricks and Apache Spark.
The ideal candidate should combine strong architecture capabilities with hands-on engineering experience. This is not purely an advisory or high-level solution architect role. The candidate should be comfortable designing the target architecture as well as working directly with AWS services, Databricks, Spark, data pipelines, security configurations, performance optimization, and migration patterns.
Key Responsibilities
1. AWS Data & Cloud Architecture
- Design scalable, resilient, secure, and cost-effective enterprise data platforms on AWS.
- Define reference architectures for migration from on-premises data platforms to AWS Data Lake/Lakehouse.
- Architect AWS solutions leveraging services such as:
- Amazon S3
- IAM
- AWS Glue
- AWS Lake Formation
- Amazon Redshift
- Amazon RDS/Aurora
- AWS Lambda
- Amazon EC2
- Amazon VPC
- AWS KMS
- CloudWatch
- CloudTrail
- Secrets Manager
- AWS Step Functions
- Amazon EventBridge
- Define appropriate patterns for networking, security, data ingestion, storage, processing, consumption, and monitoring.
2. Databricks & Lakehouse Architecture
- Design and implement enterprise Databricks Lakehouse solutions on AWS.
- Strong hands-on experience with Databricks, Apache Spark, PySpark, SQL and Delta Lake.
- Design Bronze, Silver and Gold data-layer patterns based on workload requirements.
- Define scalable ingestion, transformation, orchestration, and data-serving patterns.
- Implement and optimize Delta tables, partitioning strategies, Spark workloads, cluster configurations, and large-scale ETL/ELT pipelines.
- Experience with Databricks Unity Catalog, governance, access controls, lineage, and workspace architecture is highly desirable.
3. On-Premises to AWS Modernization
- Assess existing on-premises data applications, ETL workloads, databases, file systems, and batch processing environments.
- Define migration and modernization approaches based on rehost, replatform, refactor, or retire decisions.
- Modernize legacy ETL/data-processing workloads into cloud-native and Lakehouse-based architectures.
- Identify application and data dependencies, migration risks, sequencing, and integration requirements.
- Support coexistence and transition patterns between legacy/on-premises and AWS platforms during migration.
4. Data Engineering
The architect must be capable of going hands-on when required, including:
- Developing and reviewing PySpark/Spark and SQL workloads.
- Designing complex ETL/ELT pipelines.
- Debugging Spark jobs and data-processing failures.
- Reviewing code and engineering standards.
- Optimizing Spark transformations, joins, partitioning, caching, and processing patterns.
- Troubleshooting performance bottlenecks across data pipelines and platform components.
- Designing reusable data-engineering frameworks and patterns.
5. Security, Governance & Compliance
- Design security using IAM roles/policies, KMS encryption, network isolation, secrets management, and least-privilege principles.
- Establish data access patterns across Databricks, S3, databases, and downstream applications.
- Work with enterprise security and infrastructure teams on secure connectivity between on-premises platforms and AWS.
- Incorporate data governance, metadata management, lineage, auditing, and access controls into the architecture.
- Strong understanding of enterprise data protection and security standards.
6. Performance, Scalability & Cost Optimization
- Architect for large-scale enterprise data workloads.
- Optimize Spark/Databricks workloads for performance and scalability.
- Evaluate compute, storage, cluster sizing, and workload patterns to improve cloud cost efficiency.
- Establish monitoring, logging, alerting, and operational support patterns.
- Identify opportunities to reduce unnecessary cloud consumption while maintaining required performance and SLAs.
7. DevOps / Infrastructure Automation
- Experience implementing CI/CD for data and Databricks workloads.
- Familiarity with Infrastructure as Code, preferably Terraform and AWS-native automation.
- Experience integrating Git-based development, automated deployments, environment promotion, and release-management processes.
- Establish standards for DEV, QA/UAT and Production environments.
Mandatory Technical Skills
Vendors should prioritize candidates with strong hands-on depth across the following:
Area | Required Expertise |
AWS | Strong hands-on AWS architecture and implementation |
Databricks | Strong architecture + hands-on Databricks experience |
Apache Spark | Advanced Spark/PySpark development and optimization |
Data Lakehouse | Strong Data Lake/Lakehouse architecture patterns |
AWS Storage | Amazon S3 and related data-lake architecture |
Data Engineering | ETL/ELT, batch and scalable data pipelines |
SQL | Advanced SQL and query optimization |
Delta Lake | Strong experience with Delta-based architectures |
Migration | On-premises to AWS data migration/modernization |
Security | IAM, KMS, secrets, encryption, network/security patterns |
Governance | Unity Catalog/Lake Formation or equivalent concepts |
DevOps | CI/CD, Git and deployment automation |
IaC | Terraform preferred |
Monitoring | CloudWatch and Databricks/platform monitoring |
Programming | Python/PySpark |
Preferred Qualifications
- AWS Certified Solutions Architect – Professional or Associate.
- Databricks Data Engineer/Databricks Architect certifications are desirable.
- Experience modernizing large enterprise data platforms onto AWS.
- Experience with Redshift, Glue, Lake Formation and AWS data services.
- Experience integrating Databricks with enterprise AWS environments.
- Exposure to streaming/event-driven data processing such as Kafka or Amazon MSK/Kinesis.
- Experience with enterprise data governance and metadata solutions.
- Automotive industry experience, particularly large-scale manufacturing, sales, vehicle, supply-chain, or connected-vehicle datasets, is an advantage but not mandatory.
