Intro

The Fredhopper Product Discovery platform facilitates a range of testing options conducted by Fredhopper customers or their authorized third-party partners, provided that all prerequisites are satisfied.

Terminology clarification

Testing serves a purpose: it validates assumptions, explores limits, and assesses functionality, all aimed at minimizing the risks associated with changes before they are implemented in a production environment. Typically, during the testing process, the goal is to gain insights into the behaviour of a system by subjecting it to a defined set of conditions and interacting with it in a specific way. The results of these tests provide information that leads to informed conclusions, ultimately fulfilling the objectives of the testing process.

Types of testing

In our practice, we categorize testing into distinct types to ensure comprehensive evaluation.

1. Functional Testing: Does a feature function as we expect it to according to specification?
2. Non-Functional Testing: This encompasses several key areas:
   • Performance Testing: We evaluate the system's response time when we trigger features.
   • Load Testing: This examines the system's performance and stability under a predefined volume and rate of traffic.
   • Stress Testing: Here, we determine the threshold at which the system ceases to function or perform effectively under increased traffic.
3. Business Continuity Testing: This type focuses on the system's resilience and recovery capabilities:
   • Endurance Testing: We assess how long the system can maintain a predefined load without any intervention.
   • Penetration Testing: This evaluates the system's vulnerability to known attack vectors.
   • Disaster Recovery Testing: We measure the time required to restore the system to full functionality after an outage.

Service offering

	Functional	Performance	Load	Stress	Live
Provisioning condition	Default	On-demand via TC	On-demand via TC	On-demand via TC	On-demand via CSM
Number of items	<= live	<= live	<= live	<= live	up to 4 million
Number of configuration rules	<= live	<= live	<= live	<= live	unlimited
Maximum traffic	< 1 QPS	< 10 QPS	agreed upfront constant	increasing until failure	dynamic
Maximum data update rate	best effort	best effort	best effort	best effort	best effort
Uptime % SLO for QUERY API	N/A	N/A	N/A	N/A	99.9% or higher
Uptime % SLO for CONFIG API	None	None	None	None	None
Data processing performance speed SLO	None	None	None	None	None
Mean Time To Repair	24-48 hours	< 24 hours	< 24 hours	< 24 hours	ASAP

 

Standard: Functional Testing Environment

By default, unless otherwise agreed upon, each test environment will be provisioned as a Functional Testing Environment. This serves as a dynamic experimentation space; it can be used in most unexpected ways. As a non-live environment, it does not include redundancy measures, and its available capacity is designed to support the traffic generated by a single user engaging in typical manual actions. For any other types of testing, a separate environment will be provisioned, which incurs additional costs.

Live Testing Recommendations

For customers choosing to conduct live testing, we strongly recommend purchasing a load testing environment. This option provides the same level of validation while minimizing the risk of impacting live traffic.

 

If this recommendation does not suit your needs, we kindly ask that you direct your API calls to a designated URL, which we can provide upon request. This measure will enable us to swiftly respond and mitigate any potential impact on other customers who utilize our service during your testing efforts.

Prerequisites

1. The customer has explicitly acquired a non-functional environment, such as performance, load, or test.
2. The test environment is provisioned by Technical Consultant, with assistance from the Cloud team where necessary.
3. The customer has submitted all required test details.
4. The FCS team has confirmed their support for the requested testing.

Required test details

To ensure an effective testing process, please provide the following information:

1. Test Purpose: Select one of the following options:
   • Performance
   • Load
   • Stress
   • Live
   • Penetration
2. Duration: Specify the testing period by providing both the start and end dates/times in UTC ISO format:
   • Start Date/Time:
   • End Date/Time:
3. Traffic Type: Choose one of the following options for the type of traffic:
   • Live
   • Realistic
   • Synthetic
   • Human
4. Projected Peak Queries Per Second (QPS): Indicate the expected volume of queries hitting the Fredhopper Query API by selecting one of the following options:
   • Precise: Provide the exact maximum number of queries per second anticipated at peak. For example, "150 QPS."
   • Estimation: Offer a traffic rate in relation to the production (live1) traffic. For example, if the normal traffic is approximately 50 QPS and the expected load is 150 QPS, indicate that this represents a threefold increase (300% of normal).
5. Caching of Fredhopper Responses: Specify the caching settings:
   • On/Off
   • Hit Rates

Supported tests for Non-Functional testing

Non-functional testing will be supported provided that all prerequisites are met, including the following criteria:

• The traffic directed at the FAS must mirror the production traffic source.
• The test traffic should target the FAS's published test endpoints.
• The queries utilized in the tests should closely resemble real-world scenarios; they should be synthesized rather than synthetic.

Unsupported activities

Please note that the Service Level Agreement (SLA) does not apply to unsupported activities, and as such, Crownpeak cannot guarantee business continuity in these scenarios. Sudden traffic peaks are dealt with assuming it's organic traffic.

 

The following activities are categorized as unsupported:

• Tests for which at least one of the prerequisites has not been met are not supported
• Load testing that impacts production (live environments) service instances is not permitted.
• Testing conducted with synthetic traffic is not supported.
• Exhausting testing (i.e. increasing load until it breaks) is not supported

Synthetic Traffic

Automated security systems out of Crownpeak’s control operated (e.g. ISPs, AWS etc.) may misclassify synthetic traffic as a malicious anomaly, such as a DDoS attack, leading to unintended disruptions. Once this classification occurs, these systems may block the activity, making it challenging and time-consuming to identify and rectify the resulting blockage.

Capacity

By default, a load test service instance is scaled to handle the production (live environment) capacity. During the purchasing process, customers have the option to specify their capacity requirements. It is recommended that this specification be expressed in terms of peak Queries Per Second (QPS) directed at the Fredhopper Query API endpoints for optimal clarity.