Key Responsibilities
- Lead QA strategy and define testing approaches for agentic AI systems and ML applications
- Design and implement comprehensive test automation frameworks using Python
- Develop specialized testing methodologies for AI models including accuracy, fairness, robustness, and drift detection
- Build and maintain CI/CD pipelines with integrated automated testing
- Test complex Azure architectures including microservices, serverless functions, and distributed systems
- Validate data pipelines, ETL processes, and analytics workflows in Azure Databricks
- Perform advanced performance testing, load testing, and scalability validation
- Establish quality metrics, KPIs, and reporting dashboards for AI applications
- Mentor junior QA engineers and promote best practices across the team
- Collaborate with architects and developers on testability and quality requirements
- Implement monitoring and validation strategies for production AI systems
- Lead root cause analysis for critical production issues
- Evaluate and integrate new testing tools and frameworks
- Ensure compliance with security, privacy, and regulatory requirements
Required Skills & Qualifications
QA Leadership:
- 8+ years of experience in software QA with at least 2+ years testing AI/ML systems
- Proven track record of establishing QA processes and testing frameworks
- Experience leading testing efforts for production-grade applications
- Strong understanding of QA best practices, methodologies, and industry standards
Use LLM or AI for testing or Python working experience in testing.
Technical Expertise:
- Expert-level proficiency in Python for test automation and scripting
- Advanced SQL skills for complex data validation and quality checks
- Deep knowledge of Azure services (Functions, ML, OpenAI, Databricks, Storage, API Management)
- Extensive hands-on experience with Azure Databricks including testing data workflows
- Strong experience building and maintaining CI/CD pipelines (Azure DevOps, GitHub Actions)
- Proficiency with test automation frameworks (pytest etc...)
- Experience with API testing and microservices validation
AI/ML Testing Expertise:
- Strong understanding of Machine Learning, Natural Language Processing, and Deep Learning
- Specialized knowledge of AI/ML testing strategies (model evaluation, data quality, bias detection, adversarial testing)
- Experience testing LLM-based applications including prompt validation and response quality
- Knowledge of MLOps practices and testing ML pipelines
- Understanding of A/B testing and experimentation frameworks
- Familiarity with model monitoring and drift detection
Azure & Databricks:
- Deep knowledge of Azure architecture and deployment patterns
- Experience testing serverless applications and event-driven systems
- Proficiency with Azure monitoring tools (Application Insights, Log Analytics)
- Strong understanding of Databricks notebooks, jobs, and cluster configurations
- Experience with infrastructure testing and validation
Problem-Solving & Quality:
- Exceptional analytical and debugging skills
- Experience with performance testing tools.
- Knowledge of security testing principles and tools
- Strong understanding of observability and monitoring strategies
Preferred Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, or related field
- Azure certifications (Azure Administrator, Azure Solutions Architect)
- Databricks certification (Spark Developer, ML Associate)
- Experience with resilience testing
- Experience testing multi-agent systems or autonomous AI applications
- Familiarity with MLflow, Azure ML, or other MLOps platforms
- Experience with test data management and synthetic data generation
