Services API. Full employment of fail-safe features

This document is an extension of the information provided in the API reference documentation and describes the fail-safe setup of the API end-points of Attraqt Fredhopper Services and provides a guideline for the integration of these APIs.
We believe that these insights will help you better understand the capabilities of the service and therefore consume it more optimally.

Scope & prerequisites

The document covers the essential technical details of the setup of the Services API end-points.
One requires to have a good understanding of the service, its purpose and functionality.

Technical information

Services API

The Services API accepts requests to supply or retrieve data and trigger the data processing.

 Although we support both FTP and REST protocols for Services API, we strongly recommend using REST.

The diagram below depicts the overview of the infrastructure setup and indicates the fail-safe components.

Infrastructural overview

Components

Front-end

By front-end we mean the complete set of components on the customer's side involved with the integration of the Services API.
One could think of the application back-end servers and the underlying infrastructure, proxies and other network appliances, etc.

Internet

The front-end connects to the API end-point via the Internet.

DNS authority

We register the DNS names for the API end-points at a 3rd party DNS service provider.
To ensure high availability, each DNS record for the Services API end-point contains a set of public IP addresses of our Data API servers and adheres to the Round-robin DNS principle. In short, this means that every time the DNS name gets resolved into an IP address different from the previous time. In order to minimize the chance of issues related to DNS caching, we set the TTL to 30 seconds.

 Note that the host name alone does not identify any specific service instance. Instead the service instance is referenced in the path. E.g.

https://my.<region>.fredhopperservices.com/<service>:<instance>

Data API node

Each of the Data API servers (or nodes) fulfill the following functions:

• Authentication: each request gets authenticated based on the supplied account credentials
• Processing: once authorized, the request gets processed. Depending on the type of the request it might be either handled directly by the Data API node, or by other internal components.
• Storage: the data supplied via the Services API gets stored for a short period of time alongside the information regarding the trigger processing etc.

All Data API nodes have identical configuration and functionality and share the underlying storage and processing clusters.

Mechanisms

Data flow (Full load, automatic id)

1. The front-end prepares the data export, compresses it, calculates the md5 checksum and sends it to the Services API end-point
2. The Services API end-point DNS name gets resolved. The DNS authority answers with the external IP of one of the Data API nodes.
3. The front-end sends the data to the IP address, indicating the service instance in the path and supplying the account credentials.
4. The Data API node processes the request, verifies and stores the data and assigns it a data ID, which it sends back as the response.

Integration guideline

DNS resolving and caching

• Make sure the DNS record is resolved each time before a request is sent and only send 1 request to 1 IP at a time.
• Make sure the TTL advertised with the DNS record (30 seconds) is respected.

Implementation optimization

• For a given service instance, send the requests sequentially, awaiting the response each time. For example, if the data uploading process is still running, sending a trigger would result in a failure.