AI Systems Performance & Benchmarking Engineer
Trebecon LLC
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Python
- Artificial Intelligence
- Benchmarking
- Docker
- SQLite
- Linux
Summary
Role: AI Systems Performance & Benchmarking Engineer
Location: Onsite – All over the USA - Near to client location
Experience: 8+ years in systems or performance engineering (suggested)
Role summary:
Owns the evolution of AgentSysPerf into a pluggable benchmarking framework that can be reused.
Key responsibilities:
Review AgentSysPerf's architecture end to end: benchmark adapters, measurements, analyzers, storage, replay, reporting and dashboards.
Review state persistence, container lifecycle, concurrency and the AgentInvoker protocol implementation.
Integrate up to 4 new agentic benchmarks (e.g., τ-bench, BrowseComp, GAIA, BFCL). This includes configuration, CLI execution, model/runtime integration, metrics capture and results persistence.
Measure latency, throughput and resource use on Intel hardware.
Improve run variance, repeatability, fault handling and SQLite write concurrency during large sweeps.
Add new schema fields (policy hashes, topology IDs, route decisions, resource placement) to the storage and telemetry pipelines.
Must-have skills:
Python, Linux internals, Docker, cgroups and Linux perf tools.
Hardware profiling.
SQLite and OTLP/OpenTelemetry exporters.
Benchmarking LLMs or agents, including inference engines and runtime configurations.
Nice to have:
Xeon platform tuning.
Terminal-Bench.
Experience with AgentSysPerf itself.
- Dice Id: 91171215
- Position Id: 9113045
- Posted 7 hours ago
Company Info
About Trebecon LLC
Trebecon is a global leader in innovative IT solutions, product development, and professional staffing services. Our parent company is founded in 2015, we have grown into a trusted partner for Fortune 50 companies, U.S. state governments, and international organizations. We specialize in delivering scalable, sustainable, and future-ready technology solutions designed to drive operational efficiency, empower growth, and create lasting business value.
Our vision is to set a benchmark for excellence by aligning technology, talent, and trust. We believe in harnessing cutting-edge technologies while fostering meaningful client partnerships that go beyond project delivery. By combining innovation with integrity, Trebecon has become synonymous with delivering business outcomes that exceed expectations.
Our Mission
At Trebecon, our mission is rooted in integrity, collaboration, and performance. We are committed to:
- Delivering exceptional consultancy and IT services tailored to evolving client needs.
- Creating cost-effective, agile solutions that drive measurable impact.
- Nurturing long-term relationships built on trust, transparency, and mutual success.
- Fostering a thriving workplace culture that inspires innovation and teamwork.
Our Journey
Trebecon’s parent company started its journey in 2015 with a vision to transform industries through technology. By 2020, we had become a trusted technology partner for government agencies, insurance companies, and global enterprises. Strategic collaborations with Fortune 50 clients and international organizations enabled us to rapidly expand our footprint.
Despite market challenges in 2022, our resilience and innovation led to a strong comeback, resulting in tripled office space and workforce by 2024. Today, we are a forward-thinking company that combines global expertise with a passion for delivering world-class IT solutions.
What Sets Trebecon Apart?
- Innovative Solutions: We create tailored technologies that solve unique challenges and ensure scalability and future readiness.
- Expert Teams: Our professionals bring deep domain knowledge and technical excellence to every project.
- Client-Centric Approach: We prioritize understanding business needs, maintaining transparent communication, and ensuring long-term partnerships.
- Seamless Execution: From strategy to deployment, we minimize risks and maximize business value through agile processes.
Our Services & Expertise
We offer end-to-end services spanning AI & ML integration, product engineering, IT staffing, and cloud transformation. Whether building customized digital platforms or assembling dedicated remote teams, Trebecon ensures every solution is aligned with the client’s business objectives.
The Trebecon Edge lies in our ability to combine technology leadership with workforce empowerment. We build high-performing remote and hybrid teams to deliver next-gen solutions for enterprises worldwide.