Observability Engineer

Job Title: Observability Engineer

Location: San Antonio, TX (Hybrid)

Duration: Long-Term Contract

Position Summary

We are seeking a highly skilled Senior Observability Engineer to design, implement, and optimize modern observability solutions across cloud-native environments. The ideal candidate will have strong expertise in infrastructure automation, backend development, and monitoring platforms, with hands-on experience in Terraform, Python, FastAPI, OpenTelemetry, and Datadog.

This role will focus on building scalable observability frameworks, improving system reliability, and enabling proactive monitoring for critical applications and infrastructure.

Key Responsibilities

Infrastructure as Code (IaC)

·        Design, build, and manage scalable cloud infrastructure using Terraform.

·        Develop reusable infrastructure modules and maintain infrastructure provisioning standards.

·        Implement and maintain CI/CD pipelines to automate infrastructure and application deployments.

·        Improve deployment reliability through automation and infrastructure best practices.

Backend Development

·        Design, develop, and maintain RESTful APIs using Python and FastAPI.

·        Build backend services that support observability workflows, telemetry processing, and integrations.

·        Optimize services for performance, scalability, reliability, and maintainability.

·        Troubleshoot application and API performance issues.

Observability & Monitoring

·        Design and implement observability solutions using OpenTelemetry for distributed tracing, metrics, and logging.

·        Configure and maintain Datadog dashboards, monitors, alerts, and Service Level Objectives (SLOs).

·        Develop monitoring strategies to improve system visibility and reduce incident response times.

·        Analyze telemetry data to identify performance bottlenecks and reliability issues.

·        Establish alerting thresholds and monitoring standards to support production operations.

Required Skills & Qualifications

·        Degree in Computer Science, Engineering, or related field (or equivalent practical experience).

·        10+ years of experience in cloud infrastructure, software engineering, or observability engineering.

·        Strong hands-on experience with Terraform and Infrastructure as Code practices.

·        Proficiency in Python and backend API development using FastAPI.

·        Experience implementing observability frameworks using OpenTelemetry.

·        Strong experience with Datadog, including dashboards, monitors, alerting, and SLO management.

·        Solid understanding of distributed systems, cloud-native architectures, and microservices.

·        Experience with CI/CD tools and deployment automation.

·        Strong troubleshooting, analytical, and problem-solving skills.

Preferred Qualifications

·        Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud.

·        Knowledge of container orchestration platforms such as Kubernetes.

·        Familiarity with logging and monitoring tools such as Prometheus, Grafana, or Splunk.

·        Experience supporting high-availability production environments.

• Must Have & Desired Skills:  • 10+ years of experience in cloud infrastructure, software engineering, or observability engineering.
  • Strong hands-on experience with Terraform and Infrastructure as Code practices.
  • Proficiency in Python and backend API development using FastAPI.
  • Experience implementing observability frameworks using OpenTelemetry.
  • Strong experience with Datadog, including dashboards, monitors, alerting, and SLO management.