Automate Full Reindex and Incremental Updates – Rezolve Ai Support Portal

The full reindexes and incremental update jobs consists of a series of commands that are tedious to repeat frequently. Further we want these commands to run on specific moments in time and a warn us in case of system, logic or business failures. This can be achieved by combining commands in scripts, add additional safety checks and add extensive logging.

This article assumes you have an on-premise setup that makes use of the Fredhopper Deployment Package

The full and incremental index process

In the full reindex process the entire existing index is wiped and a new index is created. During this process the live side should continue to be available at any time, even when the new index is replacing the previous one. While the reindex is running incremental updates should be stalled until the reindex is ready. Further we need proper logging of each run, we should ensure only one reindex at the time is running, test of a step succeeds and bail out and alert staff when not.

The two provided scripts below do exactly this and more.

The standard full reindex process

#!/bin/bash
BASE=`pwd`
. config/fasrc      # part of deployment package
. bin/functions.sh  # part of deployment package

IDX=${1-indexer}
TODAY=`date "+%Y%m%d"`
mkdir -p $IDX/log
LOG=$IDX/log/$TODAY.full-reindex.log
exec 1>>$LOG
exec 2>>$LOG

# This lock should also lock the incremental updates
LCK=$IDX/tmp/full-reindex.lck
if [ -s $LCK ]; then
    echo `date` "WARN Full Reindex already running, exiting"
    exit
else
    date > $LCK
fi

echo `date` "INFO Full reindex ETL started"
if ! bin/run-etl-job run.kjb -DINSTANCE=$IDX -DTRIGGER=load-data; then
   echo `date` "ERROR Step csv to FAS xml failed, check kitchen.log"
   `alert $MAILTO "Indexing failed $TODAY" $LOG`
   rm -f $LCK
   exit 1
fi

echo `date` "INFO Fredhopper reindex started"
if bin/reindex $IDX ; then
   if bin/chk-valid-index $IDX ; then
      echo `date` "INFO Indexing succeeded, bring index to live qservers"
      bin/fresh-index-to-live $IDX
   else
      echo `date` "ERROR Reindex succeeded but the index is not proper"
      `alert $MAILTO "Indexing failed $TODAY" $LOG`
      rm -f $LCK
      exit 1
   fi
else
  echo `date` "ERROR Fredhopper reindex failed. Check xmlloader.log, treebuilder logs and searchindexer logs"
fi
rm -f $LCK
echo `date` "INFO Done"

A standard incremental process

#!/bin/bash
#
IDX=${1-indexer}
LCK=$IDX/tmp/full-reindex.lck

TODAY=`date "+%Y%m%d"`
mkdir -p $IDX/log
LOG=$IDX/log/$TODAY.incrementals.log
exec 1>>$LOG
exec 2>>$LOG

if ! -s $LCK ; then
	echo `date` "INFO start incrementals"
	bin/run-etl-job run-incremental.kjb "-DINSTANCE=$IDX" "-DTRIGGER=load-data"
        # Make sure  bin/incremental-update is executable
        bin/incremental-update $IDX   # new step to copy file to xmlloader input.
        echo `date` "INFO finished incrementals"
else
	echo `date` "INFO incremental postponed during full reindex"
fi