FileFeeds Transforms

Transforms are recorded when a user corrects un-normalized data in the process of creating their initial FileFeed. All saved transforms will run on uploaded documents in order, ensuring that the output data is cleaned according to the template requirements.

If you click on the “Add transform” button in the upper right hand corner of the FileFeeds tab of your OneSchema admin panel, you will be presented with a list of prebuilt transforms.

Data Parsing

These transforms happen on upload, prior to Data Setup and Mapping.

Data Setup

These transforms happen just after upload, similar to post-upload code hooks in Importer.

Standardize Data Format

Similar to Autofixers in Importer, these transforms perform data reformatting and JSON manipulation.

Data Editing

These transforms perform various operations to normalize column data.

Column Editing

Custom Transforms

You can also write your own custom transforms in JavaScript or TypeScript, similar to post-mapping code hooks in Importer - more on them below.

FileFeeds Custom Code Actions

Custom FileFeeds transforms, as mentioned above, are custom code actions on your data, and equivalent to code hooks in Importer (but parametrized). There are two categories of custom code actions: Data Setup (similar to post-upload hooks in Importer), and Transforms (similar to post-mapping hooks in Importer), both reachable by clicking on the “Custom Code Actions” section of the FileFeeds tab in your OneSchema admin panel.

You can then click the “Create code action” button toward the upper right corner of the admin panel to get to a selection dialog, where you can pick whether you want to work on Data Setup or Transforms.

Data Setup

As shown above, Data Setup custom code actions allow you to process uploaded data prior to template mapping (equivalent to post-upload hooks in Importer). Clicking on the “Data Setup” button will bring you to the following set of screens, where you can create your action through several steps, as shown below.

Set an identifier for this data setup action

Add your label, key, and description to the code action.

Define variables

Here you will define variables that can be set in the Transforms section of the FileFeed creation, and then used inside the code action:

• Variable name: the name you'll be referring to in the code
• Data type: Boolean, String or Array of Strings
• Label: what is shown to the user in the transform dialog when adding to FileFeed
• Placeholder: shown to the user as an example of the expected value

Defined variables will be displayed in the form builder preview pane on the right half of the screen.

Code Editor

The last step is to write your custom code action logic in JavaScript or TypeScript, using the programming editor interface below the variable definition element.

Example: Add New Header

Let's create a function that creates a new header as row 1 in the file (useful for customer files that comes without a header). Each string entered as a parameter will be a new column name.

export default function(rows: string[][], parameters: { [name: string]: any }): string[][] {
    const newHeader: string[] = parameters["headerArray"]; // this is how you use variables you defined

    return [newHeader, ...rows];
}

Save the transform. Your custom Data Setup can now be chosen in the UI of the FileFeed builder by clicking on “Add transform”.

Transforms

Transformation custom code actions allow you to transform data post-mapping. The process for adding a transformation code action is identical to that for Data setup–just select Transformation after hitting the “Create code action” button and go through the above steps as described for Data setup.

Example: Delete row if column contains "forbidden" values

Imagine a situation: an accounting spreadsheet contains rows that have Total sums per column - we don't need to import such rows and would like to skip them after mapping. Let's write a function that accepts a list of "forbidden values" (e.g. "Total:", "Grand Total") and a list of columns (e.g. "Customer Name"), and we'll delete the rows that contain any of such values in the columns specified.

We can define variables like so:

Now let's write a function that iterates through each specified column, and looks for values the user entered in FileFeeds UI, and if any of those values are present in that respective column, we'll delete the whole row. We'll make this case-insensitive, because customer may have e.g. "Total:" and "total:".

import {
  createColumn, createColumns, hasColumn, findColumn, forEachRow,
  transformColumn, transformColumns, deleteColumn, deleteRows,
} from "oneschema/postmapping"

export default function (parameters: { [name: string]: any }) {
	const valuesToCheck = parameters["forbiddenValues"];
	const valuesToCheckSet = new Set(valuesToCheck.map(v => v.toLowerCase()));
  parameters["columns"].forEach((col: string) => {
    deleteRows((row) => {
      if (hasColumn(col)) {
        if (valuesToCheckSet.has(row[col].toLowerCase())) {
          return true;
        }
      }
      return false;
    })
  })
}

API Reference & NPM packages

To see a full list of OneSchema methods from oneschema and oneschema/postmapping packages, as well as supported NPM packages, refer to Using Code Hooks.