Sr. LLM Application Engineer Punch List Digital is hiring a Senior LLM Application Engineer to own the application layer of its AI platform: the chat and reporting engine our teams and clients rely on for performance insight. We are looking for an experienced Laravel engineer who has shipped production features on top of large language model APIs in customer-facing settings, and who understands how engineering decisions translate into client outcomes.
The work is applying LLM capabilities inside a production application, not model research. The core disciplines are prompt design, structured outputs, tool-use, and retrieval - and depth in Laravel matters more than theoretical AI expertise.
About this role PLD builds the AI platform that turns live client performance data and our marketing methodology into answers and reports our teams and clients act on. Performance data enters the platform through API integrations with the marketing systems our clients use, including Google Analytics, Google Ads, SEMrush, Monday.com, and WhatConverts.
You own the Laravel application layer behind both surfaces, and with it the questions that define it:
- What information the model receives, and whether it is best sourced from the database, from retrieval, or from a live API call.
- When prompting, tool-use, or retrieval is the right mechanism for a given task.
- Where human review belongs in the workflow.
- What "ready for clients" means in practice, and how that standard is verified before a change ships.
This is a senior, hands-on position on a small team that is still taking shape. The system's output directly shapes client relationships, and engineering decisions are expected to reflect that responsibility.
The opportunity- Production data at scale. Live performance data across five marketing product lines and several hundred clients, delivered through per-source API integrations.
- Users who depend on the output daily. Strategists, client success, sales, and external clients build the platform's output into their regular work.
- Meaningful consequences. An inaccurate answer can reach a client conversation. The engineering standard follows from that.
- Genuine ownership. The product direction is set and in production. The architecture of the application layer - how chat, report generation, and the Knowledge Vault fit together within Laravel - is yours to shape and defend.
The perspective we expect The role takes more than strong implementation. The engineer we hire will:
- Understand how each piece of work affects clients and internal teams - retention, expansion, time returned - and let that understanding inform engineering decisions.
- See the system end to end: how a retrieval choice surfaces in a report, how a prompt change lands in a client conversation, and how an upstream API change becomes an incorrect figure if nothing detects it.
- Treat an incorrect number in front of a client as the most expensive defect the system can produce, and design grounding, validation, and review gates accordingly.
- Explain technical trade-offs to strategists and executives in their terms, and defend deferring a release when shipping would put client trust at risk.
- Measure their own success by the accuracy, adoption, and time savings the system delivers.
Key responsibilities- The answer pipeline. The flow from a user question or scheduled trigger, through data retrieval and reasoning, to a grounded, client-safe output - structured outputs, per-product context assembly, and multi-tenant scoping that cannot be bypassed.
- The chat interface. Answering performance questions from live, multi-tenant client data, scoped so that no client's data can ever surface to another.
- Report generation. The monthly report generator: it draws KPI outputs from PostgreSQL, assembles report sections through the LLM with methodology grounded in the Knowledge Vault, and produces a client-ready document or PDF. The workflow runs as durable, observable, retryable queue jobs rather than request handlers with hidden state.
- Knowledge Vault retrieval. Ingestion and search across the curated knowledge base that grounds the system's answers, including relevance evaluation and choosing the right retrieval method per use case. The default is pgvector within Postgres; a separate vector store would be a documented decision, not an assumption.
- API integrations. Three integration surfaces: the LLM provider (Anthropic tool-use, structured outputs, streaming, and prompt caching); the KPI feed published by our data platform; and direct partner and third-party API calls where chat needs live data. The craft here is authentication (OAuth, service accounts, API keys, token refresh), pagination and cursor handling, rate limits with retry and backoff, schema-drift detection so an upstream change never silently corrupts an answer, contract tests against recorded fixtures, and observability on every outbound call. Ingestion pipelines are owned elsewhere; this role owns how the application consumes their output and the written contract between the two.
- The prompt library. Prompts maintained as versioned, reviewed code artifacts with a change history - never inline strings.
- Quality gates. A lightweight validation step before any prompt or report change ships - for example, sampled reports and chat answers compared against known-correct figures - grown over time into structured evaluation banks that releases must pass.
- Production reliability. Hallucinations, retrieval misses, tool-use failures, upstream API outages and quota errors, output drift, cost variance, model deprecations, and the rollbacks that follow. Instrumentation precedes conjecture, and decisions are recorded in writing.
Qualifications- 4-6+ years of backend engineering experience with strong PHP and Laravel. Eloquent, queues and jobs, the service container, events, and an established testing practice with PHPUnit or Pest - plus a record of well-structured production systems.
- Production experience integrating LLM APIs , Anthropic preferred: structured outputs, tool-use, and prompt design in customer-facing systems, with the discipline to keep prompts, schemas, and their tests under version control.
- Hands-on third-party and partner API integration in production. OAuth and service-account authentication, pagination, rate limiting with backoff, weathering schema and contract changes without silent data corruption, contract or fixture-based testing, and instrumentation that flags a failing call before users notice.
- Solid PostgreSQL. Schema design, performant reporting queries, indexing, query-plan analysis, and sound judgment on when aggregation belongs in the database rather than in application code. Source data arrives through APIs, but the reporting layer lives in PostgreSQL and strength here is essential.
- Practical retrieval-augmented generation. Calling embeddings and similarity search in production, assembling context, and judging whether retrieval improved the result. Building a retrieval system from first principles is not expected.
- Clear written communication. Concise design notes for a small team, and one-page proposals that a non-AI engineer, a strategist, and a CTO can each read and respond to.
- Sound judgment on data accuracy. An incorrect figure in front of a client is the most expensive defect this system can produce, and the design of grounding and review gates should show that you believe it.
Fine-tuning, deep evaluation-infrastructure research, and vector-database internals are not required.
Preferred qualifications- pgvector or a comparable embeddings-in-Postgres deployment run in production.
- Production experience with Anthropic Claude, including tool-use, prompt caching, and streaming.
- Familiarity with any of the APIs we integrate: Google Analytics 4, Google Ads, SEMrush, Monday.com, or WhatConverts.
- Multi-tenant client data and the access-control patterns that come with it.
- PDF and document generation from templated content.
- Familiarity with the marketing or agency domain: SEO/SEM, paid media, local search, or answer-engine visibility.
How we work- We write things down. Architectural choices get a short write-up of what we chose and why, so the next person (or you, six months later) doesn't have to reverse-engineer the reasoning. A page is usually enough.
- We validate before we ship. Prompt and report changes get checked against known-correct figures first - it's far cheaper to catch a drift in a sample run than in a client meeting. You'll help decide what that validation looks like.
- We learn from human review. Where our team reviews or overrides the AI, that's by design - and every correction is signal we feed back into prompts, retrieval, and checks so the system keeps improving.
- We agree on interfaces early. The boundary with the data platform is a written contract we change together, not a table someone happens to read - which frees you to work independently everywhere behind it.
- Pace follows risk. Experiments and internal tooling move fast and loose; anything on the path to a client gets more care. You'll have room to try things without ceremony where mistakes are cheap.
How success is measured Success shows up in the accuracy of chat answers and generated reports against known-correct figures, in adoption by the teams and clients the platform serves, in the time it returns to our strategists, and in the absence of incorrect or misattributed figures reaching a client.
Hiring process- Culture Index assessment (5 minutes) - short questionnaire to help determine culture fit
- Introductory call (30 minutes). Mutual fit, background, and questions.
- Technical deep dive (60-90 minutes). A walkthrough of one production Laravel system you owned that integrated an LLM API and at least one external API, covering:
- how prompts, structured outputs, and tool-use were versioned and tested, and the failure modes designed against;
- an external API integration that failed in production - a rate limit, an authentication change, a schema change - how it was detected, and what changed as a result;
- a quality check that caught a genuine defect before release, or an incorrect figure that nearly reached a user and the safeguard added afterwards;
- an architectural decision between prompting, retrieval, and tool-use, and the business reasoning behind it.
- Working session with the team. A representative problem from our domain, worked through together.
- Offer.
Candidates who can speak to those four experiences in concrete terms can expect the process to move quickly.
Never miss an opportunity! Create an alert based on the job you applied for.
