Incremental Updates

Incremental Updates (otherwise known as "partial updates" or "deltas") permit customers to update or add catalog data ´on the fly without the need to complete a full re-index. This method is applicable for a number of day to day use cases (see below) and aside from being much faster than the full update process (users need only present a sub-set of catalog data rather than the complete collection), users can also continue to access the Business Manager tool throughout their execution - so business productivity is unaffected.

Typical Use Cases

Experience has shown that many product attributes change in value throughout the course of trading. For example:

• an item's stock level may decrease
• an item's price value may increase
• a new variant has been added to the product catalog

In each case above incremental updates allow the customer to inform Fredhopper of these changes in a lightweight manner.

Standard Input Formats

Overview

The table below lists all of the files that can be expected as part of the incremental update. Note that if your data model is such that there is no concept of variants then variants.csv and custom_variant_attributes.csv can always be omitted.

Required Files

For most use cases, all the files listed above are required. There are a few exceptions for specific cases that allow you to send only the files that have the changed data. This can be helpful if changes are managed by a system that does not have an overview of the catalog and keeps the update small.

*Note that for standard transformations with a version before version 4, all files need to be present. However, the ones not specified can contain static dummy values, as in the attached example feed minimal-data-incremental.zip, which can be found at the bottom of the article.

 

Operation Type

The full re-index process always expects that the entire product catalog is provided within the data files provided; this differs in comparison to the incremental update process as here we may wish to update an attribute value, add a new product or even delete a variant from the catalog entirely. The means by which we inform Fredhopper which action to perform is defined by the "operation_type" special attribute. The locale for this attribute does not matter and if specified it will be disregarded. For example, if a variant with two locales has to be removed it is sufficient to include only one delete operation type in custom_variant_attributes.csv (see in the sample data below how this is done).

The "operation_type" attribute is defined within custom_attributes_values.csv and (if required) custom_variant_attributes.csv for each product/variant respectively and can be one of four values below. If no "operation_type" attribute is configured for an item then add is used as a default.

*An item can be either product or variant.

 

Sample Files

The data model depicted below represents a single-universe, multi-locale deployment that has two categories (women & shoes) that contain two products and three variants in total.

The scenario below represents the following:

• p1 is updated with a new colour attribute value (blue).
• p2 is added to the product catalog.
• v2 is updated with a new price value (30.00).
• v1 is removed from the product catalog.

categories.csv

products.csv

custom_attributes_meta.csv

custom_attributes_values.csv

variants.csv

custom_variant_attributes.csv

Providing the Files to Fredhopper

Dependent on the deployment model of your choice, the incremental update csv files need be provided differently.

 

On-Premise

Managed Services

FAQ (Frequently Asked Questions)

How long should it typically take before I see the incremental update changes reflected in the Fredhopper response?

This is dependent on both the amount of data provided within the input files as well as the complexity of the transformation logic, but generally, the changes should be reflected within five minutes or less.

Do incremental updates work if I don't use the standard CSV format?

Yes, however, you will need to work with your Fredhopper consultant to develop custom data transformations to support your file format.

Can I add other files (e.g. product rating information) in addition to the set of standard CSV files?

Yes, though you will need to work with your Fredhopper consultant to develop the data transformations to incorporate the new files.

If I delete a product id, do I also need to explicitly delete all its associated variants?

No. Fredhopper will automatically delete any child dependencies from the product id in question.

What happens if I try to add a product with an existing secondid value?

Fredhopper will reject the product id from being added to the catalog. An error statement reflecting this can be found in the xmlloader.log.

What happens if I try to replace a product with a non-existing secondid value?

Fredhopper will reject the product id from being added to the catalog. An error statement reflecting this can be found in the xmlloader.log.

Can I add a new category to the category tree as part of the incremental update?

Yes. Be sure to append this within categories.csv, detail the product to category relationship within products.csv, and then finally provide an update operation type statement for the secondid within custom_attributes_values.csv.

Can I remove a category from the category tree as part of the incremental update?

Yes though in this case it is best to persist the category id within categories.csv and replace the products which are assigned to it (stating the new product to category relationships) within products.csv and custom_attributes_values.csv.

Can I add new attributes as part of the incremental update?

Yes. In addition to providing these within custom_attributes_values.csv / custom_variant_attributes.csv please also remember to define the base type within custom_attributes_meta.csv.