The Automation Engineer will design and implement scalable automation testing frameworks for AI applications, develop automated test scripts, validate AI models, manage test datasets, integrate tests into CI/CD pipelines, conduct performance testing, and collaborate with cross-functional teams while ensuring documentation and continuous improvement of testing methodologies.
Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.
Key Responsibilities for Automation QA (AI Testing Suite):
- Framework Development:
- Design and implement a scalable automation testing framework for AI applications using .NET technologies.
- Build reusable libraries and components that enhance the efficiency of the testing process.
- Test Script Development:
- Write, maintain, and optimize automated test scripts for functional, regression, and integration tests using frameworks like Selenium, SpecFlow, or NUnit.
- Ensure test scripts are designed to handle the unique requirements of AI applications, including validating model outputs and performance.
- AI Model Validation:
- Collaborate with data scientists and engineers to understand the architecture and outputs of AI models.
- Develop specific test cases to evaluate the accuracy, performance, and reliability of AI models, including edge case scenarios.
- Data Management:
- Create and manage diverse test datasets that accurately reflect real-world scenarios, ensuring comprehensive coverage for testing AI models.
- Implement data validation processes to ensure input data integrity and correctness.
- Integration into CI/CD:
- Integrate automated testing processes into CI/CD pipelines using tools like Azure DevOps or Jenkins to facilitate continuous integration and delivery.
- Collaborate with DevOps teams to ensure smooth deployment and execution of automated tests.
- Performance Testing:
- Conduct performance and load testing of AI applications using tools like JMeter or LoadRunner to assess scalability and responsiveness under various conditions.
- Analyze performance metrics and work with development teams to identify and resolve performance bottlenecks.
- Collaboration and Agile Practices:
- Work closely with cross-functional teams, including developers, product managers, and data scientists, to gather requirements and clarify testing objectives.
- Participate in agile ceremonies, contributing to sprint planning, daily stand-ups, and retrospectives.
- Documentation and Reporting:
- Maintain clear and comprehensive documentation of the automation framework, test cases, and testing outcomes.
- Utilize bug tracking tools like Azure DevOps, JIRA to report, track, and prioritize defects, ensuring clear communication with development teams.
- Continuous Improvement:
- Stay updated with the latest trends in automation and AI testing, evaluating new tools and techniques for potential integration into the testing process.
- Proactively suggest improvements to existing testing methodologies and frameworks to enhance efficiency and effectiveness.
Expected Technology Stack:
- AI/ML Tools:
- Familiarity with frameworks like TensorFlow or PyTorch (to understand AI model validation)
- Tools for model evaluation (e.g., Scikit-learn)
- End-to-end development cycle of LLM-based AI applications – (e.g. Promptflow)
- Understanding on Prompt Evaluation Metrics.
- Programming Languages (preferred):
- C# (primary language for .NET development)
- Python (for AI-related testing and data manipulation)
- JAVA (at least beginner level understanding)
- Protractor (Good to Have)
- Automation Frameworks:
- Selenium (for web application testing)
- SpecFlow (for behavior-driven development)
- NUnit (for unit testing)
- Continuous Integration/Continuous Deployment (CI/CD):
- Azure DevOps
- Jenkins
- GitHub Actions
- Performance Testing Tools:
- Apache JMeter
- LoadRunner
- Bug Tracking and Documentation:
- Azure DevOps / JIRA (for documentation, defect logging and tracking)
- Version Control: Git (for source control)
Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.
Candidate Privacy Policy
Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:
- What information we collect during our application and recruitment process and why we collect it;
- How we handle that information; and
- How to access and update that information.
Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.
Top Skills
C#
Java
Python
Similar Jobs
Be an Early Applicant
The Automation Software Engineer III is responsible for designing, developing, and troubleshooting software solutions within an agile team. Duties include producing architecture artifacts, maintaining high-quality code, improving systems through data analysis, and contributing to team culture. A focus on automation testing and modern programming languages is essential.
The Automation Engineer at Atlassian will design, develop, and deploy UiPath workflows to automate finance processes. Responsibilities include process analysis, maintenance of existing automations, documentation, testing, and collaboration with IT and finance teams to enhance efficiency.
As an Automation Engineer, you will design, develop, and test automation solutions primarily using PowerShell and bash. You will analyze system requirements, create workflows, maintain documentation, and integrate with APIs. Collaborating closely with delivery consultants is key to addressing customer infrastructure issues.
What you need to know about the Mumbai Tech Scene
From haggling for the best price at Chor Bazaar to the bustle of Crawford Market, the energy of Mumbai's traditional markets is a key part of the city's charm. And while these markets will always have their place, the city also boasts a thriving e-commerce scene, ranking among the largest in the region. Driven by online sales in everything from snacks to licensed sports merchandise to children's apparel, the local industry is worth billions, with companies actively recruiting to meet the demands of continued growth.
