Senior Data Engineer- Databricks

12+ years of proven experience in Data Engineering, worked on designing and developing software with Big Data, Data Lake/ Lake House ecosystem, Data Analytics, backend microservices architecture, and heterogeneous data types at scale
Proven in-depth experience in creating ELT/integrations pipelines using Databricks, Spark, Python, SQL, Scala, Kafka, Presto, Parquet, Streaming, events, bots, AWS/cloud ecosystem.
Proficient in developing Micro Services and using AWS frameworks such as SQS, Stream, Kubernetes, EC2, S3, Lambda etc.
Experience with data pipelines/analysis/visualization tooling such as Elastic stack, Logstash, Kibana, Kafka, Grafana, Splunk, Pandas, Message brokers, Data modeling.
Expertise in Data Lakehouse architecture and end-to-end Databricks techniques.
Have worked on connecting Data Lake to SAP Ecosystem, aware on technological ecosystem on how to onboard data from SAP ECC application layer and ingest back to SAP via application layer
Have designed and built PB sized scalable data lake and structured/unstructured data query interfaces and microservices to ingest, index, mine, transform, and compose large datasets.
Worked on end-to-end data lifecycle from Data Ingestion, Data Transformation, and Data Consumption layer.
Strong experience building Data Lake-led APIs and its usability for consuming large-scale data and ingest in real-time ecosystem.
Expert in Spark, Parquet, steaming, events, Kafka, telemetry, MapReduce, Hadoop, Hive, Presto, Spark, data query approaches, and dashboarding.
The one who have implemented Enterprise use cases like CMDB, Governance, time series classification, telemetry anomaly detection, logs, and real-time data ingestion through APIs.
Experience with structured data such as Avro, Parquet, Protobuf, Thrift, and concepts like schema evolution.