Fredhopper search performs a number of operations upon input search terms, the Query Pipeline, before performing the actual comparison of the search term during the Search Passes and outputting the results to the Results Pipeline. You can read a more thorough explanation of the entire Fredhopper Search function in the following document, but an overview of the process is illustrated below:

Configure Fredhopper search

Fredhopper's Business Manager rules are applied at various points throughout the search process, although this knowledge base article focuses on Stop Words, which are applied by the Linguistic Analyzers in the Query Pipeline and are therefore filtered from the input search query before they arrive at the Search Passes.

Linguistic Analyzers and Stop Words

To provide the most relevant results possible in the most efficient manner, Fredhopper search applies various "Linguistic Analyzers" to input search queries. Each Linguistic Analyzer applies various operations, known as filters, in order to break the search query down into a compound query that can be compared against the Search Index, (created during the Indexing process). The various filters from the English Default Linguistic Analyzer can be seen below:

Applying a series of filters to search queries is common functionality in search engines, and in fact virtually all search engines apply some form of Linguistic Analyzer(s) for the same reasons that Fredhopper does, but the performed analysis can sometimes lead to confusing and/or misleading search results. Most commonly the erroneous results are caused by the implementation of the "Stop Words" filter, so this Knowledge Base article aims to explain the functionality of Stop Words and their potential impact on your search results so that you can manage them effectively.

What are Stop Words and why do they matter?

A Stop Word is a word that generally has no significant meaning in a search query and does not help in identifying a user's search intent. Words that fall into this category typically include articles, conjunctions, pronouns, etc. Fredhopper can be configured to ignore these words when indexing items and when executing search queries in order to improve the performance of the search function. Fredhopper is shipped with various Linguistic Analyzers, for a variety of languages, and each of these has a predefined list of Stop Words that are stripped from a search query.

A list of the pre-defined Stop Words from the Default English Linguistic Analyzer are included below as an example:

Examples of how Stop Words can affect the quality of search results

In most search configurations, frequently used non-descriptive words, (such as the, and, from, with) need to be ignored by the search to produce more meaningful search results and improve the overall performance of the search function. However, sometimes Stop Words can play a significant role in describing a product or brand, and as a result this can impact the quality of the returned search results.

Provided below are some examples of search queries that contain some of the predefined Stop Words included in the Default English Linguistic Analyzer:

The below examples will only use part of the input search query, where one or more of the words will be stripped by the Stop Word filter.

Example 1: By Night
Example 2: Only You
Example 3: After Dark

The below examples will generate an empty input search query because all of the included words will be stripped by the Stop Word filter.

Example 1: To be or not to be
Example 2: The Who
Example 3: Why Why Why

Given the examples above, it is worth considering whether any of your products/brands are likely to be impacted by the Stop Word filter, and where necessary test the impact on your search results to see whether it is necessary to remove the specific Stop Word(s).

Stop Words affecting redirects, synonyms and key word triggers

Stop Words will be stripped by the Linguistic Analyzers before being processed for synonyms and "keyword triggers", (used in campaigns, redirects and result modifications).

As of Fredhopper 7.5 a warning will be shown when trying to configure a redirect or result modification that contains a stop word, and in this case Fredhopper will not allow you to save the redirect/result modification, as illustrated below:

Stop Words are applied in the Search Query Pipeline, before other Business Manager rules, so you should always test any changes to your Business Manager rules thoroughly before deploying to production to ensure you receive the expected results!

Testing Fredhopper search for the impact of the Stop Word filter

If the search query contains none of the pre-defined Stop Words, and you have not added any yourself, then the Stop Word filter will not impact your search query. But if you suspect that your search queries are not returning the expected results because Stop Words, (or other Linguistic Anlayzer filters), are impacting your search then you can use the Fredhopper Search Engine Tester to see whether filters are indeed affecting the results. The Search Engine Tester can be be found on the following link, and an screenshot of the page can be seen below: <FREDHOPPER_BASE_URL>/fredhopper/sysadmin/searchtest.jsp

The results from a search for the Stop Word "without" is illustrated below, and as can be seen the Stop Word filter has stripped the search term leading to empty results being returned for that specific query:

Removing Stop Words from the Linguistic Analyzer(s)

There is no such thing as a perfect default search configuration, and it is quite possible that some of the products or content of your catalogue use words that are blocked by the default Stop Words shipped with Fredhopper. For this reason you might need to delete some of the Stop Words, although it is advisable to only delete the words you really need to so as not to negatively impact the quality and performance of your search results!

When removing Stop Words from a Linguistic Analyzer you should ensure to remove the word from all Analyzers, (for example English Default, English Natural and English Stemmed), to ensure that the removed word is filtered by all the appropriate analyzers

To remove Stop Words:

1) Navigate to Business Manager > System > Search > Linguistic analysers:

2) Select the appropriate Linguistic Analyzer for which you wish to remove a Stop Word

3) Click on the Stop Word filter to see the list of Stop Words in use by that Analyzer's filter

4) Find the required word in the list and then click on the red "X" to delete it from the Analyzer's filter

Once the configuration has been changed in the Business Manager Fredhopper will know not to apply the Stop Word filter on the search query, but the catalogue still needs be re-indexed before this will have a complete effect on the search results, (because the search indexes need be reconstructed according to the newly defined rules). Therefore please ensure you perform a full re-index before evaluating the search results so that the catalogue is also indexed according to the new rules.

Adding new Stop Words to the Linguistic Analyzer(s)

As you may have realised, it is also possible to use Stop Words in your search configuration to prevent certain terms being entered as search queries. This can be advantageous to prevent searches using profanities or undesired terms, (such as competitor products etc).

When adding Stop Words to a Linguistic Analyzer you should ensure to add the word to all Analyzers, (for example English Default, English Natural and English Stemmed), to ensure that the newly added word is filtered by all the appropriate analyzers

To add Stop Words:

1) Navigate to Business Manager > System > Search > Linguistic analysers:

2) Select the appropriate Linguistic Analyzer for which you wish to add a new Stop Word

3) Click on the "Add Stop Word" button on the required Linguistic Analyzers page

4) Add the Stop Word in the box, as illustrated below, and press Enter to add the new Stop Word

Once the configuration has been changed in the Business Manager Fredhopper knows not to apply the Stop Word filter on the search query, but the catalogue still needs be re-indexed before this will have a complete effect on the search results (as the search indexes need be reconstructed according to the newly defined rule). Therefore please ensure you perform a full re-index before evaluating the search results so that the catalogue is also indexed according to the new rules.