Data Engineer

Columbus, IN, US • Posted 2 days ago • Updated 2 days ago
Contract W2
On-site
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • python
  • Pyspark
  • Databricks

Summary

Job Title: Data Engineer
Location: Columbus, IN
This role requires strong expertise in Python, PySpark, Databricks, and Machine Learning/AI, along with hands-on experience in React, Node.js, Azure Web Applications, and OpenAI API integration. The engineer will design, build, and deploy scalable data quality and observability solutions - including AI/ML-powered validation
Deliverables:
  • Analyzing complex business systems, industry requirements, and/or data regulations
  • Processing and managing large data sets
  • Design and development for a Big Data platform using open source and third-party tools 4. Data Quality Engineering & Automation
  • Design and develop data quality profiling pipelines using PySpark, Python, and Databricks to ensure data accuracy and reliability.
  • Build automated validation, cleansing, and anomaly detection frameworks leveraging ML/AI models.
  • Integrate OpenAI APIs for intelligent rule generation, data summarization, and NLP-based insights.
  • Full-Stack Development (exp in Web & API)
  • Develop responsive React-based user interfaces for self-service data quality monitoring and visualization.
  • Build and maintain RESTful APIs using Node.js and Python (Flask/FastAPI) to serve ML models and DQ metrics.
  • Host and manage applications using Azure Web Apps, ensuring secure API connectivity and scalable architecture.
  • Cloud and Data Integration
  • Work with Azure SQL Database, Azure Storage, and Databricks for seamless data access and validation.
  • Integrate data pipelines with APIs, Dataverse, Snowflake, or other cloud data sources. 15. Support Azure Application Insights and monitoring integration for performance tracking. 16. ML/AI Enablement
  • Contribute to ML projects focused on data quality improvement and predictive monitoring.
  • Apply techniques like anomaly detection, clustering, or deep learning to identify data issues proactively.
  • Use OpenAI / LLM APIs to automate root cause analysis, data profiling, and rule suggestions. 20. Collaboration and Documentation
  • Partner with Data engineers, Architect team, Cloud support team and business stakeholders to translate requirements into technical solutions.
  • Document DQ frameworks, ML workflows, APIs, and integration processes for reusability and transparency.
  • SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent college coursework
  • SQL query language
  • Clustered compute cloud-based implementation experience
  • Developing applications requiring large file movement for a Cloud-based environment and other data extraction tools and methods from a variety of sources
  • Build analytical solutions 28. Experience with IoT technology, Experience in Agile software development
  • Manage Offshore delivery team from India, work assignment review, weekly report review with client
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10106248
  • Position Id: 2026-4611/175640
  • Posted 2 days ago
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.