Data Import

Import engine allows administrators to collect data from various data sources, and then map this data into relevant tables.
The SimpleOne import engine features the following:
- different data sources types are supported;
- different file formats and retrieval methods are supported (if you choose "File" as the data source).
You can choose the data source, convert the data from it using different import scripts, and after that, with the transform maps and the field maps, you can map the data from the source with the target tables.

You can find graphical procedure view of the import process below. For a more detailed description, please check out this article

Key terms

Concept	Description
Import source	A record specifying what data an import set should import.
Import set	A record containing references to raw data uploaded to the staging location. Also, this record returns information about this import attempt state.
Import set table	A table that is used as a staging location for records importing from a data source before transforming them. The fields in these tables are generated automatically based on imported data.
Transformation	The conversion of data from an import set table to another table according to the rules defined in a transform map.
Transform Map	A set of field map specifying the relationships between a field in an import set table and a field in a target table.
Field Map	A relationship between a field in an import set table and a field in a target table.

General import procedure description

Prepare raw data;
Create an import source;
Create a transform map;
Create field sets;
Run transformation.
Verify the results.

Role required: export_admin.

Import Sources

Creating an import source is necessary to load raw data into system for further processing and transformation. You can set up an import source able to work with the different data source types, file format, and able to retrieve files in a different ways if this option was chosen.

Navigate to Import -> Import Sources;
Click New, fill in the form, and click Save.

Field	Description
Name	Displayed import source name.
Import set table name	Import set table name.
Type	Type of the import source. Available choice options: File; Text; LDAP.
File retrieval method	Choose the method of the file adding to the import source (if relevant Type was chosen earlier). Available choice options: Attachment.
Format	Choose the file format (if the relevant Type was chosen earlier). Available choice options: JSON; XML; Excel.

After you are done with an Import Source, click Load All Records in the Related Links area, and an import set will be created. Perform some checks of it:

Import set state must be equal to Loaded;
Import set rows state must be equal to Pending (you can know this by entering the import set, it's located in the Import Sets tab in the Related Lists area).

For more information about LDAP import source, please refer to the Importing using LDAP article.

Import Sets

Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources.

Import Set field description

Field	Description
Number	Import set unique number.
State	Import set loading state. This field is populated automatically and always read-only. Available options: Loading; Loaded; Processed; Cancelled.
Import Source	Select the import source to use in this import set.
Import Set Table	Select the import set table to use in this import set.
Short Description	Describe your import set.

Running a transformation

Navigate to Import → Import Sets;
Open an import set you want to transform;
Click Transform.

Transform Maps

Every import operation requires at least one transform map which specifies the relationships between the import set table and the target table.

To create a transform map, please complete the steps below:

Navigate to Import → Transform Maps.
Click New, fill in the form, and click Save.

Transform Maps field description

Field	Description
Name	Displayed transform map name.
Active	Select this checkbox to make the transform map available to use.
Source table	Select the table containing the import set data.
Target table	Select the table where you want imported data to be placed.
Silent load	Select this checkbox to ignore business rules, client-side scripts, and other additives while the transformation inserts or updates data in the target table.
Ignore mandatory fields	Select this checkbox to ignore mandatory fields on the target table.
Use script	Select this checkbox to display the Script field.
Script	Insert a transform map script you want to use to transform field values. Do not unselect the Use script checkbox after inserting a script, otherwise it will not be executed.

Transform map script

Transform map script allows you to customize import operations using a JS-scripts written using a SimpleOne Server-Side API.

Transform Script field description

Field	Description
Transform map	Specify a transform map to which this script is related to.
Active	Select this checkbox to make the script active or inactive.
When	Specify the script type to occur: onComplete - this script runs at the end of the transformation; onStart - this script runs at the start of the transformation; onAfter - this script runs after the source row is transformed into the target row; onBefore - this script runs before the source row is transformed into the target row.
Order	This field specifies the order of transform scripts; fill in this field with the integer number. Scripts will be sorted in the descending order.
Script	Enter the mail script using SimpleOne Server-Side API.

Field Sets

Field set specifies relationships between fields in an import set table and fields in the target table determining what values from the source table will be added to the target table.

To create a field set, please complete the steps below:

Navigate to Import → Transform Maps;
Open a transform map you want to configure;
In the Field Maps related list, click New, fill in the fields and click Save.
Repeat the previous step for each field mapping.

Field set fields description

Field	Description
Transform map	Displays the transform map using this field set. This field is populated automatically.
Source field	Select the field of the source table to be transformed. This field is mandatory.
Target field	Select the field of the target table where the values from the source table should be stored.
Coalesce	When selected, the import set application attempts to match source values to records with values from an existing record. If a match is found, the transform map updates the record instead of creating a new record. When false, the import set application is always creating new records for each transformation. This can be set up in the widget. By right-clicking a field in the heap-select tool, a popup menu with either Add Coalesce or Drop Coalesced choice will open.
Use source script	Select this checkbox to display the Script field.
Script	Insert a script to determine the source instead of the Source field. Do not unselect the Use source script checkbox after inserting a script, otherwise it will not be executed.

When creating field sets, create the first binding for the sys_id fields of the source and the target tables using the Coalesce option. It will tie the tables with each other for further importing. After that, create other pairs for all the fields you want to transform.

Page tree