As a vendor providing the Assertthat BDD, Cucumber & Test Management plugin for Jira Data Center, it is crucial for us to ensure that our application meets the highest performance standards. To achieve this, we undergo an annual/periodic performance testing process that complies with Atlassian's Marketplace approval requirements. In this blog post, we'll look into the specifics of this performance testing process and the necessary steps we take to ensure optimal functionality in Data Center environments.
The Data Center Performance Testing Requirement
Atlassian Marketplace's approval process necessitates rigorous performance testing, ensuring that our plugin functions seamlessly in environments equivalent to our Data Center customers. The testing environment, akin to a 4 node Jira instance type, plays a significant role in accurately gauging performance impact. Key requirements for our performance testing include:
Test Environment Setup:
A clustered test host with a 4 node configuration (32 CPU cores minimum).
Preloaded Large Dataset:
Using a base dataset that mirrors the one included in the Data Center App Performance Toolkit to emulate real-world usage.
Test Scenarios:
Performance testing on a 4 node Data Center instance, both with and without our app installed.
Testing various actions on the base Jira product, including login, issue creation, viewing and editing issues, dashboard and board views, commenting, JQL search, and logout.
Concurrency Details:
Simulating a substantial load with 200 concurrent users performing 54,500 actions per hour, reflecting real-world usage patterns.
Test Duration:
Each performance run lasts for 45 minutes, capturing a substantial amount of data to analyze performance effectively.
Conducting Performance Testing
To meet the performance testing requirements, we follow a structured process:
Environment Setup: Configure a test host that mirrors the Data Center environment, meeting the outlined specifications.
Dataset Preparation: Load a preconfigured large dataset similar to the Data Center App Performance Toolkit to simulate realistic usage scenarios.
Test Execution: Conduct performance tests on the Data Center instance, both with and without our app installed, using the specified test scenarios and concurrency levels.
Data Collection and Analysis:
Capture performance metrics for each transaction, listing the response times with and without our app installed.
Analyze the data to identify any performance regressions and areas for optimization.
Performance Regression Report
Following each performance test, we generate a comprehensive performance regression report. This report includes a detailed analysis of the tested transactions, highlighting response times with and without our app.
Scalability Testing
Scalability testing is vital for assessing how well our plugin performs when operating across multiple nodes in a Jira Data Center (DC) deployment. The main goal is to understand the impact as the environment is scaled to different node configurations.
Tests are repeated on Jira DC environments with the following configurations:
1 node Jira DC environment
2 node Jira DC environment
4 node Jira DC environment
By conducting these tests on different node setups, we gain valuable insights into how the app scales. This ensures that our plugin can effectively meet the demands of our users as their requirements grow and evolve.
Conclusion
At AssertThat, ensuring top-notch performance of our BDD, Cucumber & Test Management plugin for Jira Data Center is fundamental. Our commitment to compliance with Atlassian's performance testing requirements guarantees a reliable and high-performing experience for our users. If you have any questions or need further information about our performance testing processes, feel free to reach out. Your satisfaction with our high-performance plugin is our priority.