Overview

Senior AI Agent Engineer - Voice AI at Zendesk. This role focuses on building and evolving a high-performance voice-first AI agent with real-time, low-latency spoken dialogue capabilities.

What You Will Do

• Design and develop robust, stateful, and scalable voice-first AI agents using Python, optimized for real-time voice interactions, managing turn-taking, interruptions, and low-latency responses.
• Integrate real-time Speech-to-Text (STT), Text-to-Speech (TTS), and Voice Activity Detection (VAD) services to create seamless conversational flow.
• Connect voice agents with existing enterprise systems, databases, and third-party APIs to enable end-to-end automated workflows initiated and managed through voice.
• Establish and own the evaluations for voice agent performance and behavior; iterate to improve performance, reliability, and user experience.
• Build end-to-end conversational flows with reasoning, planning, and dynamic tool use beyond pre-scripted voice experiences.
• Collaborate cross-functionally with product managers, ML scientists, and engineers to understand user needs and voice interaction goals.
• Implement fallback, recovery, and error-handling strategies for noisy audio input or speech recognition inaccuracies.
• Define and track voice-specific evaluation metrics (e.g., word error rate, latency, conversational naturalness).
• Develop observability tools and guardrails to monitor performance, ensure safety, and handle edge cases in spoken interactions.
• Document development, architecture decisions, and research findings to share knowledge across the team.

Requirements

• LLM-Oriented System Design: Experience building multi-step, tool-using agents (LangChain, Autogen). Familiar with prompt engineering, context management, and reasoning strategies like Chain-of-Thought and ReAct.
• Voice AI Expertise:
• Experience building low-latency, streaming voice applications. Expertise in integrating and managing real-time STT/TTS models and APIs. Proficient with Voice Activity Detection (VAD), noise suppression, and interruption logic.
• Experience with integrating third-party voice AI APIs, including STT and TTS services from providers like OpenAI, Deepgram, ElevenLabs, etc.
• Understanding of latency, timing, and streaming audio constraints.
• Tool Integration & APIs: Comfortable connecting agents to external APIs, tools, and databases in secure environments.
• RAG (Retrieval-Augmented Generation): Building pipelines with vector stores, chunking strategies, and hybrid retrieval.
• Evaluation & Observability: Implementing and using monitoring tools and evaluation frameworks to score AI agents.
• Safety & Reliability: Familiarity with prompt injection defenses, guardrails, and failover logic.
• Performance Optimization: Managing token budgets and latency using caching, model routing, etc.
• Programming & Deployment: Proficient in Python, FastAPI, and LLM SDKs. Experience deploying AI apps to cloud platforms (AWS, GCP, Azure) with CI/CD practices.

Nice-to-have

• MS/PhD in Computer Science, NLP, Machine Learning, or related field.
• Background in spoken dialogue systems or conversational UX design.
• Familiarity with real-time streaming architectures (e.g., WebRTC, gRPC, socket.io).
• Multilingual ASR/TTS pipeline experience.

About Zendesk

Zendesk builds software for better customer relationships. It empowers organizations to improve customer engagement and better understand their customers. Zendesk products are easy to use and implement, providing flexibility to move quickly, focus on innovation, and scale with growth.

Zendesk is an equal opportunity employer and fosters diversity, equity, and inclusion. We are committed to accessibility and reasonable accommodations in the recruitment process.

If you are based in the United States you can access EEO rights and related information through Zendesk’s resources. Zendesk’s Candidate Privacy Notice explains how personal information may be processed in recruitment.