Datadog Principal Engineer

Our client is growing their enterprise observability practice and they are seeking experienced Datadog engineers to help lead and scale monitoring, analytics, and reliability capabilities. Depending on your experience and scope of ownership, you will join as either a Principal Datadog Engineer or a Senior Datadog Engineer / Architect.

This role is 100% REMOTE.

Duration: 6 months plus with potential to convert

Qualified candidates must be authorized to work in the US without sponsorship.

Responsibilities:

• Design, architect, and scale enterprise observability solutions using Datadog across applications, infrastructure, cloud, and security platforms


• Own and drive enterprise-level observability strategy and architecture


• Serve as a final decision-maker for observability standards, tooling approaches, and long-term platform direction


• Architect dashboards, monitors, and alerting frameworks aligned to business, operational, and reliability requirements


• Define and implement best practices for metrics, logs, traces, and anomaly detection


• Lead deployment and configuration of Datadog agents, APIs, integrations, and automation across complex, multi-cloud environments


• Integrate Datadog with CI/CD pipelines, logging platforms, and collaboration tools (e.g., GitLab, ServiceNow, Jira, Slack)


• Identify observability gaps and drive improvements in signal quality, reliability, and incident response


• Optimize Datadog usage and licensing costs while maintaining strong coverage and actionable insights


• Partner closely with DevOps, SRE, Cloud, Application, and Security teams to embed observability into daily operations


• Produce clear documentation and contribute to knowledge sharing across teams

Required Qualifications

• Recent enterprise-level ownership is required


• Highly preferred: ownership in your current role


• Minimum: ownership within the last 12 months


• Senior: 8+ years of engineering experience


• Principal: 10+ years of engineering experience


• Hands-on experience designing and owning Datadog observability solutions (not execution-only roles)


• Proven experience as a technical decision-maker in a modern software development environment


• Strong experience with:


• Datadog Monitoring, APM, Distributed Tracing, and alerting


• Cloud platforms and DevOps / SRE practices


• CI/CD integrations and automation


• Scripting and configuration skills (Python, Bash, PowerShell, YAML, etc.)


• Strong communication skills with the ability to collaborate across engineering, product, and business stakeholders


• Demonstrated ability to articulate measurable impact and outcomes (e.g., MTTR reduction, reliability improvements, cost optimization)

Preferred Qualifications

• Datadog certifications (Fundamentals, APM, Distributed Tracing, or equivalent)


• Experience optimizing observability cost and usage at enterprise scale


• Experience mentoring teams and influencing technical standards


• Background working in large, complex, or regulated enterprise environments