Why AI Models Pass Benchmarks but Fail in Production
Headline

Why AI Models Pass Benchmarks but Fail in Production

by Software Testing Magazine in Headline 0

There are many cases where AI models that worked fine in development start to behave badly in production. In this article, Kanika Vatsyayan discusses the causes that lead to this situation. He recommends moving from static assessments to continuous quality engineering for AI models. share share share share 

Featured

Software Testing Videos

Rethinking Software Quality: a New Approach to Testing API

October 9, 2026 0

This session, “Rethinking Software Quality: Why API Testing Needs to Change,” unpacks the challenges of testing API-driven ecosystems, the limitations of conventional QA, and the new pressures created by AI and evolving digital platforms.

Featured

Why Pull Requests Should Review Tests, Not Just Code
Headline

Why Pull Requests Should Review Tests, Not Just Code

October 12, 2026 0

When you add new code to your applications, it is associated to the test scripts that should validate it. Do you give enough importance to these tests? In this article, Eli Lopian explains that when pull requests are opened, you should review the tests scripts as much as the code.

Knowledge

Tools

  • Turning Postman Collections into Committable Pytest Suites

    June 23, 2026 2
    In this article, Mikhail Golikov, the sole QA on a seven-team backend platform, explains how he turned a drawer full of unrun Postman collections into committable pytest suites, walks through the conversion request by request, and shows the folder-scoped command-line tool he built so one team’s pattern could scale to [...]
  • Open Source Test Data Generators

    June 15, 2026 10
    In most of your software testing activities, you need data. Sometimes you can rely on a small sample, but if you want to perform some load testing or if you want to test a feature that needs to produce a multipage invoice, then you start to need more than just [...]
  • Open Source API Testing Tools

    March 3, 2026 6
    The rising trend of using SOA and then microservices as a software architecture has led to the creation of multiple tools for automated testing of the services API. This article presents a list of open source API testing tools. share share share share 
  • Open Source Test Management Tools

    December 2, 2025 4
    Test management is defined by Wikipedia a part of the software testing process that includes the planning of tests and test cases, their execution and the storage and analysis of the tests results. This is achieved also by the integration with requirements management tools, functional software testing tools like Selenium [...]

Videos