Fullstack Data Engineer -(Telecom domain)

Job Summary:

- Location & Work Mode: Hybrid role; applicants must be local to Texas (TX) or New Jersey (NJ).
- Industry Experience: Must have recent experience in the telecom domain.

- Role Overview:
- Responsible for designing and implementing a real-time data platform using Medallion Architecture.
- Manage end-to-end development of data pipelines from ingestion to consumption-ready output.
- Act as a “Full Stack” data professional, handling infrastructure automation (DataOps) and complex data modeling.

Key Responsibilities:
- Build scalable real-time ingestion pipelines using Spark Structured Streaming and Auto Loader for sources like Kafka/Event Hubs/CDC.
- Develop ELT logic to cleanse, deduplicate, and normalize data into relational formats with ACID compliance.
- Design optimized JSON data structures in Delta tables for low-latency API consumption.
- Implement and manage workflows using Delta Live Tables (DLT), streaming tables, and Databricks Workflows.
- Enforce data governance and security using Unity Catalog for access control and maintain a searchable data catalog.
- Own and automate deployment lifecycle with Databricks Asset Bundles (DABs) and CI/CD tools (GitHub Actions/Azure DevOps).
- Optimize streaming processes for performance and cost efficiency.

Required Skills & Qualifications:
- Technical Core:
- Expert in PySpark/Scala, Spark internals, and Delta Lake features (Z-Ordering, Liquid Clustering, CDF, Time Travel).
- Hands-on experience with advanced streaming patterns (watermarking, checkpoints, handling late data).
- Data Modeling & Languages:
- Advanced SQL skills for complex transformations.
- Mastery in handling and generating complex nested JSON objects in Spark.
- Experience with Medallion Architecture (Bronze, Silver, Gold layers) and data contracts.
- Full Stack & DevOps:
- Proficient in automating deployments (CI/CD, Git-based workflows).
- Capable of setting up observability/monitoring (Databricks SQL Alerts, Grafana).
- Soft Skills:
- Strong architectural thinking to balance cost and latency in streaming solutions.
- Client-focused mindset, understanding API consumption needs for JSON Gold layer.

Other Requirements:
- Must be able to work in a hybrid setup and be located in TX or NJ.
- Recent hands-on experience in the telecom industry is required.