Interest in continuous testing puts pressure on performance-testing vendors

Traditionally, performance testing is the last step before production and deployment, making it a dreaded gate with the potential to derail launch timelines or, worse, a step that gets skipped in an effort to launch quickly. In a survey earlier this year commissioned by Microsoft and conducted by 451 Research, we found that speeding time to market is an increasingly important goal for most companies, even at the expense of lowering costs and growing revenue. As most development shops feel increasing pressure to get products and services to market fast, the question of how to efficiently do performance testing has been in the spotlight.

The 451 Take

Performance testing was a relatively stable market until recent demand for easier testing processes emerged. Now, businesses want to do performance testing early and often in order to find problems and fix them quicker. With new products and services aimed at enabling this kind of continuous testing, startups are challenging the entrenched vendors that have dominated the market (a decade ago, LoadRunner held nearly 70% of the performance-testing space). We think that the sector will continue to be disrupted, with opportunities for new entrants to develop the latest continuous-testing products and adjacent players, including continuous integration and delivery (CI/CD) and automation vendors, to enter the fray.

Continuous testing

Advanced development shops are adopting CI/CD practices in order to speed time to production. These organizations want to begin doing performance testing more frequently, including earlier in the development process, and integrate performance testing into their CI/CD processes. This so-called 'shift left' of performance testing to long before deployment can surface problems sooner so that developers can fix them as they iterate, ideally without significantly holding up deployment.

To enable load testing earlier in the development process, vendors are offering key integrations and automation. Integrations with popular tools like Jenkins, AWS CodePipeline, Bamboo, CircleCI, GitHub and Chef support the automation of load testing with each code submit, ensuring that testing happens as apps and websites are continually being updated.

Companies are also working hard to make their products easier to use since the process of shifting left involves a new kind of user. While popular entrenched load-testing products like HPE's LoadRunner are typically deployed by dedicated trained performance test engineers, the logical way to shift left is for developers to run performance tests. Rather than require developers to learn a challenging new product, some providers are trying to remove complexity from their offerings, making them easier to use and at the same time decreasing the amount of time it takes to set up and run load tests. Some firms argue that making their products easier to use spares developer time so that they can focus on their primary development work. Ease of use can also reduce ownership cost since an additional performance-testing team may no longer be required.

Take-up lags

While this scenario – where developers handle performance testing throughout the continuous integration and development process – is the ideal, it's not yet the norm. The number of organizations that are even doing CI/CD is likely relatively low. Our Q4 2015 Voice of the Enterprise Survey found that 65% of decision-makers say they've adopted agile development. That's often a step before implementing CI so we expect that the segment of the market doing CI is significantly smaller. Performance-testing vendors we spoke with said that their customers want to adopt CI/CD and do continuous testing, but few actually are.

To start, some businesses instead are doing weekly testing, a first step before automating a test that is triggered by code submissions within a CI/CD tool. Depending on the complexity of the testing tool, some organizations are retaining performance-testing team members and positioning them like consultants or subject-matter experts who work with various agile teams on their testing needs. If a business sticks with dedicated test engineers, it's becoming more common for QA experts to help with performance testing, rather than have dedicated performance engineers who once were responsible for performance testing.

The vendor opportunity

The demand to shift testing left represents an opportunity for existing vendors as well as those in adjacent markets. Existing performance-testing specialists with offerings that appeal to developers may attract many more end users since they are targeting essentially all developers and potentially additional types of end users as well, rather than one discrete performance-testing team of users. BlazeMeter reports that one of its customers has 350 users of the product.

We think that vendors in the CI/CD and automation markets make logical suitors for performance-testing companies, particularly those with stand-alone products aimed at developer users. As more developers assume performance-testing responsibilities, it makes sense for the required tools to come from the same firms that offer the CI/CD and automation platforms that they already use.

Key players

Shifting load testing left is a popular topic that incumbents and relatively young vendors alike talk about. The degree to which they enable it varies. These are some companies that we've seen position themselves to respond to interest in doing performance testing earlier in the development process:

  • Apica: Apica offers load-testing products and services to very large users and sets itself apart by delivering high-volume loads in a short amount of time. It reports that one client achieves two million test users in about eight milliseconds, the kind of load test that can serve large customers preparing for a popular campaign. To encourage users to test frequently, Apica's pricing allows for 50 or more tests per month – it notes that it offers an unlimited number of tests but has set a 'soft limit' of 50 tests per month.

    The company aims to reduce the pain around creating scenarios by allowing developers to record a user path to set up tests rather than requiring them to write scripts. Apica has integrations with CodePipeline, TeamCity, Jenkins, Bamboo and Microsoft Visual Studio Test Professional to enable testing as part of a CI/CD process. JPMorgan Chase, Apple, Virgin America and the NBA are among Apica's 500 customers.

  • BlazeMeter BlazeMeter was founded as a service for delivering load testing from the cloud based on JMeter, the open source testing tool for stress, load and functional testing, and now also supports other open source tools including The Grinder and Locust. The company's mission is to democratize performance engineering, enabling a range of roles such as QA pros, developers, product managers and operations staff to build, run, analyze and automate a range of tests. It recently released an open source tool for defining tests employing YAML and JSON syntax, a process designed to be easier than writing scripts that might appeal to developers who may appreciate the familiar approach.

    It has integrations with CI services including Jenkins, Bamboo, Travis, Teamcity and CircleCI. To automate tests, users can integrate BlazeMeter with Chef, Puppet, AWS CodePipeline and IBM Bluemix. BlazeMeter can run tests from AWS, Azure, Google Compute and other cloud services and can test one million concurrent users. Customers include Disney, the NFL, Walmart, KPMG, Sony, Target, BT, Tesla Motors, Mattel and others.

  • Dynatrace: Dynatrace's Load offering has roots in the performance-testing product from Keynote, which Dynatrace merged with last year. The company has added capabilities to Load to support continuous testing, including the option to build tests using JSON. It also offers plug-ins for Jenkins and TeamCity for test automation.

  • Neotys: Like its competitors, Neotys' load-testing offering NeoLoad aims for ease of use, in its case by offering a GUI for designing tests rather than requiring users to write scripts. One feature that stands out is the company's Team Server, which allows developers and QA team members to share test scenarios and test results. With this collaboration feature, developers and QA team members can reuse materials and QA teams can enforce processes and standards. NeoLoad has integrations with Jenkins, Hudson and Bamboo. Its pricing includes an unlimited number of tests and customers include Best Buy, Toyota, Boston University, Wells Fargo, Siemens and Deloitte.

  • Nouvola: Nouvola, a startup with 100 paying customers and 1,000 users of its free tier, offers performance testing exclusively from the cloud, primarily deploying AWS. Like some of its rivals, the company hopes to achieve ease of use by avoiding requiring clients to write scripts – developers can record their activity and import the recording to Nouvola to create the test. It reports that one of its customers does tests that generate 1.2 million virtual users per minute and another does three million virtual users in five minutes. Nouvola has Jenkins and GitHub plug-ins.

  • SmartBear: While SmartBear positions its LoadComplete product as easy to use by developers, it isn't integrated with CI/CD or automation tools – the vendor reports that few of its customers are employing CI/CD tools. It serves medium-sized businesses and enterprises, although enterprise customers tend to deploy SmartBear in a department rather than company-wide.

  • SOASTA: SOASTA is an entrenched testing specialist with a reputation for offering a comprehensive suite of tools that are complex to use. However, the company is tracking demand for shifting testing left and providing features to serve this demand. For instance, it recently began supporting JMeter in its CloudTest platform, a move aimed at inviting developers to do early performance testing. SOASTA also supports integrations with Jenkins, Bamboo, Travis CI and Microsoft Visual Studio Online to enable automated testing. Still, half of CloudTest revenue comes from customers that hire the vendor to run the testing for them, indicating that it's a complex product to deploy.


The newer breed of performance-testing vendors, including Nouvola, Apica and Neotys, are challenging the traditional market leaders, namely HP LoadRunner, IBM Rational Performance Tester, Micro Focus' Borland Silk Test and Microsoft Visual Studio. Some of those giants are, however, beginning to enable continuous testing and test automation and so will respond to existing customer needs for this capability. For instance, both Microsoft and IBM outline steps that customers can take so that developers can run performance tests more frequently as part of a continuous-testing initiative.

In addition to the legacy vendors, employing open source JMeter to manage performance testing is popular. Both SmartBear and SOASTA report that they commonly win deals among customers that were previously exclusively using JMeter.

New Alert Set

"My Alert"

Failed to Set Alert

"My Alert"