Own end-to-end LLMOps for Ask Sovi products: gate agents/models, design evaluation frameworks, build golden datasets and review pipelines, monitor hallucinations/drift/latency/cost/safety, convert feedback into measurable quality improvements, and drive AI adoption across product portfolio.
Build your future with Sovos.
Sovos is transforming tax compliance from a business requirement into a force for growth. Trusted by more than 100,000 customers worldwide, including many of the world's leading organizations, we are leveraging AI to build smarter, more impactful customer experiences.
We're looking for an AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You'll drive AI quality, safety, evaluation, observability, and continuous improvement, ensuring our AI-powered experiences are reliable, scalable, and production-ready.
What You'll Do- Gate new AI agents, prompts, and models before production release
- Design and run evaluation frameworks using deterministic testing and LLM-as-a-Judge methodologies
- Build and maintain golden datasets, annotation pipelines, and human-review workflows
- Monitor AI quality, hallucinations, drift, latency, cost, and safety metrics
- Convert customer feedback and production learnings into measurable quality improvements
- Drive AI adoption and quality standards across your assigned product portfolio
- 1-3 years of experience in AI Engineering, AI Operations, AI Quality Engineering, LLMOps, MLOps, Test Automation, or related disciplines
- Experience with Langfuse, LangSmith, Phoenix, or similar LLM observability/evaluation platforms
- Strong understanding of LLMOps, AI quality engineering, and evaluation frameworks
- Experience with hallucination detection, model drift monitoring, prompt regression management, and AI safety evaluations
- Hands-on experience building evaluation datasets, quality benchmarks, and feedback loops
- Familiarity with NLP concepts such as embeddings, semantic similarity, classification, and retrieval
- Strong product mindset and stakeholder management skills
- Applied AI Engineers from AI-native or AI-first organizations
- Product-minded Engineers with strong customer and business awareness
- QA/Test Automation Engineers who have transitioned into AI evaluation
- AI Quality Analysts with expertise in hallucination detection, drift monitoring, and LLM-as-a-Judge frameworks
UK Shift
Why Join Sovos?- Work on cutting-edge Generative AI products
- Global, remote-first collaboration with AI, Product, and Engineering leaders
- Bi-Weekly Meeting Free Days
- Mentoring and career development programs
- Globally recognized training opportunities
- Comprehensive health and wellness benefits
- Opportunity to shape AI experiences used by customers around the world
- CCAT & EPP Assessment
- Automated Video Interview
- Hiring Manager / Team Interview
- Final Interview
Interested in building the future of Enterprise AI with Sovos? We'd love to hear from you.
#Hiring #AIOps #LLMOps #AIEngineering #GenerativeAI #LangSmith #Langfuse #SovosCareers
Similar Jobs
HR Tech • Information Technology • Professional Services • Sales • Software
Manage a portfolio of SaaS HR accounts in the EU, build account relationship frameworks, drive renewal and expansion, improve adoption and customer outcomes, conduct QBRs and success planning, mitigate churn, meet retention and CSAT/NPS KPIs, collaborate with Product/Dev/Marketing/Sales and provide ongoing product enablement.
Top Skills:
Ai ToolsAsanaChurnzeroSalesforceSFDCWeb-Based Technologies
Cloud • Information Technology • Security • Software • Cybersecurity
Drive pipeline generation by qualifying inbound and marketing leads, creating excellent customer experiences, and mapping customer pain points to Cloudflare solutions. Use Salesforce (SFDC) to track activities, hit quota metrics, and collaborate with account executives and cross-functional teams. Develop sales skills, product knowledge, and contribute to process iteration.
Top Skills:
OutreachSalesforceSFDC
Cloud • Information Technology • Security • Software • Cybersecurity
Generate pipeline and qualify inbound and marketing leads for Sales by creating excellent customer experiences, mapping customer pain to Cloudflare solutions, meeting quota, tracking activity in Salesforce (SFDC), and collaborating with account executives and stakeholders. Role requires Danish, Norwegian, or Swedish language skills and is based in Lisbon on a hybrid schedule.
Top Skills:
OutreachSalesforceSFDC
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
