You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 60 Next »

Import engine allows administrators to collect data from various data sources, and then map this data into relevant tables.
The SimpleOne import engine features the following:
- different data sources types are supported;
- different file formats and retrieval methods are supported (if you choose "File" as the data source).
You can choose the data source, convert the data from it using different import scripts, and after that, with the transform maps and the field maps, you can map the data from the source with the target tables.

You can find graphical procedure view of the import process below. For a more detailed description, please check out this article

Key terms


ConceptDescription
Import sourceA record specifying what data an import set should import. 
Import set

A record containing references to raw data uploaded to the staging location. Also, this record returns information about this import attempt state.

Import set tableA table that is used as a staging location for records importing from a data source before transforming them. The fields in these tables are generated automatically based on imported data.
TransformationThe conversion of data from an import set table to another table according to the rules defined in a transform map.
Transform MapA set of field map specifying the relationships between a field in an import set table and a field in a target table.
Field MapA relationship between a field in an import set table and a field in a target table.


General import procedure description


  1. Prepare raw data;
  2. Create an import source;
  3. Create a transform map;
  4. Create field sets;
  5. Run transformation.
  6. Verify the results.


Role required: export_admin.

Import Sources


Creating an import source is necessary to load raw data into system for further processing and transformation. You can set up an import source able to work with the different data source types, file format, and able to retrieve files in a different ways if this option was chosen.

  1. Navigate to Import -> Import Sources;
  2. Click New, fill in the form, and click Save.
FieldDescription
NameDisplayed import source name.
Import set table nameImport set table name.

Type

Type of the import source. Available choice options:

  • File;
  • Text;
  • LDAP.
File retrieval method

Choose the method of the file adding to the import source (if relevant Type was chosen earlier). Available choice options:

  • Attachment.
Format

Choose the file format (if the relevant Type was chosen earlier). Available choice options:

  • JSON;
  • XML;
  • Excel.

After you are done with an Import Source, click Load All Records in the Related Links area, and an import set will be created. Perform some checks of it:

  1. Import set state must be equal to Loaded;
  2. Import set rows state must be equal to Pending (you can know this by entering the import set, it's located in the Import Sets tab in the Related Lists area).

For more information about LDAP import source, please refer to the Importing using LDAP article.


Import Sets


Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources.

Import Set field description

FieldDescription
NumberImport set unique number.
State

Import set loading state. This field is populated automatically and always read-only. Available options:

  • Loading;
  • Loaded;
  • Processed;
  • Cancelled.
Import SourceSelect the import source to use in this import set.
Import Set TableSelect the import set table to use in this import set.
Short DescriptionDescribe your import set.


Running a transformation

  1. Navigate to Import → Import Sets;
  2. Open an import set you want to transform;
  3. Click Transform.

Transform Maps


Every import operation requires at least one transform map which specifies the relationships between the import set table and the target table.

To create a transform map, please complete the steps below:

  1. Navigate to Import → Transform Maps.
  2. Click New, fill in the form, and click Save.

Transform Maps field description

FieldDescription
NameDisplayed transform map name.
ActiveSelect this checkbox to make the transform map available to use.
Source tableSelect the table containing the import set data.
Target tableSelect the table where you want imported data to be placed.
Silent loadSelect this checkbox to ignore business rules, client-side scripts, and other additives while the transformation inserts or updates data in the target table.

Ignore mandatory fields

Select this checkbox to ignore mandatory fields on the target table.
Use scriptSelect this checkbox to display the Script field.
Script

Insert a transform map script you want to use to transform field values.

Do not unselect the Use script checkbox after inserting a script, otherwise it will not be executed.

Transform map script

Transform map script allows you to customize import operations using a JS-scripts written using a SimpleOne Server-Side API

Transform Script field description

FieldDescription
Transform mapSpecify a transform map to which this script is related to.
ActiveSelect this checkbox to make the script active or inactive.
When

Specify the script type to occur:

  • onComplete - this script runs at the end of the transformation;
  • onStart - this script runs at the start of the transformation;
  • onAfter - this script runs after the source row is transformed into the target row;
  • onBefore - this script runs before the source row is transformed into the target row.
Order

This field specifies the order of transform scripts; fill in this field with the integer number. Scripts will be sorted in the descending order.

ScriptEnter the mail script using SimpleOne Server-Side API

Field Sets

Field set specifies relationships between fields in an import set table and fields in the target table determining what values from the source table will be added to the target table.

To create a field set, please complete the steps below:

  1. Navigate to Import → Transform Maps;
  2. Open a transform map you want to configure;
  3. In the Field Maps related list, click New, fill in the fields and click Save.
  4. Repeat the previous step for each field mapping.

Field set fields description

FieldDescription
Transform mapDisplays the transform map using this field set. This field is populated automatically.
Source fieldSelect the field of the source table to be transformed. This field is mandatory.
Target fieldSelect the field of the target table where the values from the source table should be stored.
Coalesce

When selected, the import set application attempts to match source values to records with values from an existing record. If a match is found, the transform map updates the record instead of creating a new record. When false, the import set application is always creating new records for each transformation. 

This can be set up in the widget. By right-clicking a field in the heap-select tool, a popup menu with either Add Coalesce or Drop Coalesced choice will open.

Use source script

Select this checkbox to display the Script field.

Script

Insert a script to determine the source instead of the Source field.

Do not unselect the Use source script checkbox after inserting a script, otherwise it will not be executed.

When creating field sets, create the first binding for the sys_id fields of the source and the target tables using the Coalesce option. It will tie the tables with each other for further importing. After that, create other pairs for all the fields you want to transform.

  • No labels