Skip to content

Importing Data with Incremental Processing#

Find here all information about incremental processing when ingesting data into Spectrum.

INFO

Ingesting data with incremental processing into Spectrum reduces the resources that are required to process data sets by computing against a subset of the data, usually the latest changes in the data set.

Requirements#

  • a Data Warehouse Workbook must have a column of the integer data type
  • the tracking column must have ascending ID data, e.g. user ID 1, user ID 2, etc.
  • duplicates are not allowed

Incremental Processing in a Data Warehouse Workbook#

To use the incremental data ingest:

  1. Right-click the Data Warehouse in the File Browser and select "Add to New Workbook". A new Workbook opens and the dialog 'Add Data Warehouse Sheet' is displayed.

  2. Select the "Schema" from the drop-down. The corresponding datasets are loaded.

  3. Select the "Table" from the drop-down and confirm with "Submit". The new Workbook is loaded.

  4. If needed, prepare the Workbook so that the tracking column with the IDs is sorted by ascending integer values.

  5. Click on the Workbook Inspector tab "Sheet" and scroll to the section ""Incremental Processing Column"".

  6. Select from the two options how to use incremental processing: "Inactive" does not activate incremental processing. "Only processing new data" provides the option to process only the new data. For that, select the integer-only column from the drop down and confirm with "Save".

  7. Safe and run the workbook by clicking "Save" and then "Run". Configuring the incremental processing for Data Warehouse Workbooks is finished.

Incremental Processing in Depending Workbooks#

INFO

You can use sheets from a Data Warehouse Workbook source that are executing incremental processing in other Workbooks. Incremental processing is possible when the sheet from the source Data Warehouse Workbook is kept and run initially.

To use incremental processing sheets from a source Data Warehouse Workbook:

  1. Make sure the sheet is kept in the source Data Warehouse Workbook and the Workbook has run. To check that, open the Workbook configuration page from the source Workbook and, if needed, mark the checkbox for the sheet to be kept, the checkbox for running the Workbook after saving and confirm with "Save".

  2. Click the "+" and then select "Workbook" to create a new Workbook. Or open the already existing Workbook in which you want to use the source sheet. The dialog 'Add Data' opens.

  3. Select the Workbook that contains the needed sheet and confirm with "Add Data". The dialog 'Import Worksheets' opens.

  4. Select the sheet from the source Workbook and confirm with "Import Sheets". The new Workbook opens

  5. Move to the Workbook Inspector and open the "Config" tab -> section "Incremental Processing". Note that the incremental processing mode is set as 'Inactive' per default.

  6. Select the incremental processing mode and confirm with "Update". Incremental processing configuration for the new Workbook is finished.

    "Inactive" does not use incremental processing for the sheet.

    "Only process new data" processes the data that is additional to the last data processing, e.g. when the integer-only tracking column has the value 'ID57' as the highest value, only values higher than 'ID57' are being processed.

    "Advanced" precesses the data that is additional to the last data processing but depends also on the entered expression, e.g. '§jobStartTime <=NOW()-5h'. For that, enter the corresponding expression in "Expression (returning a boolean value)".