Versions Compared

Old Version 51

changes.mady.by.user Igor Shakhbazyan

Saved on

compared with

New Version 52

changes.mady.by.user Igor Shakhbazyan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Import engine allows administrators to collect data from various data sources, and then map this data into relevant tables.
The SimpleOne import engine features the following:
- different data sources types are supported;
- different file formats and retrieval methods are supported (if you choose "File" as the data source).
You can choose the data source, convert the data from it using different import scripts, and after that, with the transform maps and the field maps, you can map the data from the source with the target tables.

Key terms

ConceptDescription
Import sourceA record specifying what data an import set should import. 
Import set

A record containing references to raw data uploaded to the staging location. Also, this record returns information about this import attempt state.

Import set tableA table that is used as a staging location for records importing from a data source before transforming them. The fields in these tables are generated automatically based on imported data.
TransformationThe conversion of data from an import set table to another table according to the rules defined in a transform map.
Transform MapA set of field map specifying the relationships between a field in an import set table and a field in a target table.
Field MapA relationship between a field in an import set table and a field in a target table.


General import procedure description

  1. Prepare raw data;
  2. Create an import source;
  3. Create a transform map;
  4. Create field sets;
  5. Run transformation.
  6. Verify the results.

Image Modified

Note

Role required: export_admin.

Import Sources

Creating an import source is necessary to load raw data into system for further processing and transformation. You can set up an import source able to work with the different data source types, file format, and able to retrieve files in a different ways if this option was chosen.

  1. Navigate to Import -> Import Sources;
  2. Click New, fill in the form, and click Save.
FieldDescription
NameDisplayed import source name.
Import set table nameDisplayed import set table name.

Type

Type of external data source. Available choice options:

  • File;
  • JDBC;
  • LDAP;
  • Text.
File retrieval method

Choose the method of the file adding to the import source (if relevant Type was chosen earlier). Available choice options:

  • Attachment;
  • HTTPS;
  • FTP(S):
  • SFTP;
  • SCP.
Format

Choose the file format (if the relevant Type was chosen earlier). Available choice options:

  • CSV;
  • Excel;
  • JSON;
  • XML.

After you are done with an Import Source, click Load All Records in the Related Links area, and an import set will be created. Perform some checks of it:

  1. Import set state must be equal to Loaded;
  2. Import set rows state must be equal to Pending (you can know this by entering the import set, it's located in the Import Sets tab in the Related Lists area).

Import Sets

Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources.

Import Set field description

FieldDescription
NumberImport set unique number.
State

Import set loading state. This field is populated automatically and always read-only. Available options:

  • Loading;
  • Loaded;
  • Processed;
  • Cancelled.
Import SourceSelect the import source to use in this import set.
Import Set TableSelect the import set table to use in this import set.
Short DescriptionDescribe your import set.


Running a transformation

  1. Navigate to Import → Import Sets;
  2. Open an import set you want to transform;
  3. Click Transform.

Transform Maps

Every import operation requires at least one transform map which specifies the relationships between the import set table and the target table.

To create a transform map, please complete the steps below:

  1. Navigate to Import → Transform Maps;
  2. Click New, fill in the form, and click Save.

Transform Maps field description

FieldDescription
NameDisplayed transform map name.
ActiveSelect this checkbox to make the transform map available to use.
Source tableSelect the table containing the import set data.
Target tableSelect the table where you want imported data to be placed.
Silent loadSelect this checkbox to ignore business rules, client-side scripts, and other additives while the transformation inserts or updates data in the target table.

Ignore mandatory fields

Select this checkbox to ignore mandatory fields on the target table.
Use scriptSelect this checkbox to display the Script field.
Script

Insert a transform map script you want to use to transform field values.

Note

Do not unselect the Use script checkbox after inserting a script, otherwise it will not be executed.


Field Sets

Field set specifies relationships between fields in an import set table and fields in the target table determining what values from the source table will be added to the target table.

To create a field set, please complete the steps below:

  1. Navigate to Import → Transform Maps;
  2. Open a transform map you want to configure;
  3. In the Field Maps related list, click New, fill in the fields and click Save.
  4. Repeat the previous step for each field mapping.

Field set fields description

FieldDescription
Transform mapDisplays the transform map using this field set. This field is populated automatically.
Source fieldSelect the field of the source table to be transformed. This field is mandatory.
Target fieldSelect the field of the target table where the values from the source table should be stored.
CoalesceWhen selected, the import set application attempts to match source values to records with values from an existing record. If a match is found, the transform map updates the record instead of creating a new record. When false, the import set application is always creating new records for each transformation. 
Use source script

Select this checkbox to display the Script field.


Script

Insert a script to determine the source instead of the Source field.

Note

Do not unselect the Use source script checkbox after inserting a script, otherwise it will not be executed.



Info

When creating field sets, create the first binding for the sys_id fields of the source and the target tables using the Coalesce option. It will tie the tables with each other for further importing. After that, create other pairs for all the fields you want to transform.


Table of Contents
absoluteUrltrue
classfixedPosition