Classification: Contract
Contract Length: 12-months

Position Summary

The Conversational AI Business Quality Analyst – The Conversational AI Business Quality Analyst plays a critical role in ensuring the quality, performance, and customer experience of Virtual Agent (VA) and IVR solutions across the organization. This role blends business analysis, quality assurance, and contact center technology expertise to validate that conversational solutions are designed, tested, implemented, and operating as intended—while continuously identifying opportunities for improvement.

This role partners closely with Contact Center Operations, IT, Product, Conversation Design, and external platform teams to support the full lifecycle of Virtual Agent and IVR solutions, from requirements review and sprint validation through UAT, go-live readiness, and post-implementation optimization. The analyst ensures all Virtual Agent experiences adhere to approved designs and quality standards, with a 100% audit model across all service lines.

The ideal candidate thrives in a fast-moving environment, operates independently, and brings strong analytical rigor, attention to detail, and a passion for delivering exceptional customer experiences. Experience with contact center technologies in the healthcare industry, cloud-based CCaaS platforms, and conversational AI solutions is highly valued.

Responsibilities

• Review and evaluate all Virtual Agent (VA) interactions using standardized scoring criteria and audit rubrics
• Ensure 100% interaction coverage, validating that each experience meets approved quality, design, and compliance standards.
• Identify and document intent misclassifications, Natural Language Understanding (NLU) gaps, Automatic Speech Recognition (ASR) failures
• Assess conversational user experience (UX), including, turn efficiency and clarity, prompt effectiveness, customer effort and friction points
• Validate accurate task completion and proper backend logic execution across all supported journeys.
• Confirm correct handling of fallback, recovery, and escalation scenarios, including transfers to live agents where applicable.
• Monitor and validate authentication workflows to ensure compliance with business and regulatory requirements.
• Identify, document, and escalate potential compliance risks or incidents discovered during interaction reviews.
• Partner with IT, operations, and compliance stakeholders to support timely investigation and mitigation of identified risks.
• Document functional and experience defects with clear expected vs. actual behavior, supporting screenshots, transcripts, or call references as needed.
• Categorize defects using established classifications (e.g., NLU gap, conversation design flaw, logic error, compliance issue).
• Create and manage Azure DevOps (ADO) stories for each identified issue, assigning priority using the approved evaluation and prioritization matrix.
• Collaborate with product, IT, and design teams to clarify defects and recommended remediation approaches.
• Perform regression testing to validate defect resolution and ensure no unintended impacts to existing functionality.
• Track and analyze Virtual Agent performance trends
• Leverage audit findings and interaction data to surface systemic issues and improvement opportunities.
• Partner closely with Conversation Designers and Computational Linguists to refine prompts, intents, decision logic, and conversational flows.
• Produce weekly and monthly operational summary reports highlighting quality trends, high-risk issues, experience breakdowns, recommended actions and prioritization

Other Core Duties

• Support special projects related to Virtual Agent and IVR initiatives as assigned
• Assist with training, documentation, and knowledge-sharing efforts as needed
• Participate in ad-hoc analysis, initiatives, or process reviews outside primary quality responsibilities
• Perform other duties as assigned

Requirements

• Bachelor’s degree in computer science, Analytics, Information Systems, Business Administration, Engineering, or related field.
• 3+ years in business analysis or product analysis roles required.
• Contact Center experience is required.
• 2+ years in QA roles within contact center technologies preferred.
• Experience working in Agile work environments preferred.
• Experience with data visualization tools like Tableau, Power BI, etc. preferred.
• Experience in related industries a plus (e.g., healthcare, collections, financial, data analytics and processing).
• Experience with virtual agent and IVR systems such as Google CCAI, Dialogflow, Avaya, Noble, or CCaaS platforms preferred.
• Familiarity with natural language processing (NLP) concepts and conversational design principles
• Ability to collaborate effectively with conversation designers, linguists, and product owners
• Experience applying design thinking to improve customer journeys
• Basic scripting or text manipulation skills (e.g., Python) for data review or QA support
• Knowledge of relational database concepts and data processing workflows
• Strong stakeholder management and influencing skills, including comfort presenting insights to senior leadership 
• Initiative to proactively identify issues, own investigations, and drive improvements through resolution

Required skills

• Business Analysis & QA
• Contact Center & Virtual Agent Expertise
  • Solid understanding of contact center environments, IVR flows, call routing, and customer experience metrics
  • Hands on experience supporting or QA testing Virtual Agents, Voicebots, IVR, or CCaaS platforms
  • Ability to conduct call audits and interaction reviews across multiple platforms to validate performance and experience outcomes
  • Experience identifying trends in call volume, FSS, containment, transfer rates, and other KPIs
• Data & Analytical Skills
  • Strong SQL proficiency for validation, querying, and issue investigation
  • Advanced Microsoft Excel skills for analysis, reporting, and audit tracking
  • Ability to synthesize quantitative and qualitative data into executive level summaries and insights
• Professional & Behavioral Skills
  • Excellent written and verbal communication skills, including the ability to produce clear QA findings and executive summaries
  • Strong organizational skills with the ability to manage multiple priorities simultaneously
  • Demonstrated ability to work independently with minimal oversight
  • High attention to detail and commitment to quality
  • Customer focused mindset with a drive to exceed expectations and deliver “wow” experiences
• Platforms & Tools
  • Experience with Google Conversational AI / Dialogflow, Genesys, or similar CCaaS platforms
  • Familiarity with cloud-based contact center technologies (CCaaS, Voicebots, Virtual Assistants)
  • Experience using BigQuery, SSMS, or similar data environments
  • Familiarity with data visualization tools (Power BI, Tableau)
• Automation & Testing
  • Exposure to test automation tools (e.g., Selenium, QTP, or equivalent)
  • Experience performing system or performance testing in contact center or digital channels
• Domain Experience
  • Prior experience supporting healthcare contact center use cases
  • Understanding of regulatory or compliance considerations impacting healthcare customer interactions