Page History

Import engine allows administrators to collect data from various data sources, and then map this data into relevant tables.

The SimpleOne import engine features the followingsupports:

• different data sources source types are supported.
• different file formats are supported (if you choose "File" as the data source).

You can choose the data source, place the data from it to the temporary table (also can be called "an import set"), and after that, with the transform maps and the field maps, you can map the data from the source with to the target tablestable.

You can find a graphical procedure view of the import process in the diagram below. A more detailed description is provided further in this article.Image Removed

concepts

General import procedure description

Anchor
Import Sources Import Sources

Import Sources

Creating an import source is necessary to load raw data into the system for further processing and transformation. You can set up an import source able to work with the different data source types and file and file formats.

Image Removed

1. Navigate to Import → Import Sources.
2. Click New and fill in the fields.

3. If the File type is chosen, click on the attachment icon   and attach the file you need.

   Tooltip
   onlyIcon true appendIcon info-filled iconColor blue
   You can attach JSON, XML or Excel files.

   
   

4. Click Save or Save and Exit to apply changes.

Type of the import source. Available choice options:

• File
• Text
• LDAP.

Choose the method of the file adding to the import source. Available choice options:

• Attachment.

This field appears if the File type is chosen.

Select the file format and attach it to the form. Available file format options:

• JSON
• XML
• Excel.

This field appears if the File type is chosen.

1. .

After loading data into the import source, saving the record and calling the one of the data loading actions, a new record appears in the Import Sets related list.

Also, several UI actions responsible for the further import processing appear on the import source form (see description below).

After loading data into the import source, saving the record and calling one of the data loading actions, a new record appears in the Import Sets related list. 

Using JSON format

You can use JSON files containing data formatted in JSON as a an import source. In this case, make sure that these the files follow the criteria below:

1. JSON files should be valid. Use Use the RFC 8259  document as a guideline.
2. Data types available to use in JSON structure are listed below. Dot-walking is not supported.

  {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  }

Result: an import set with related table created (consisting of these fields: top500,name,sector) and containing one elementone element. The table has top500, name, sector fields and contains one record.

[ 
 {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  },
  {
    "top500": 2,
    "name": "Nevermore",
    "sector": "Oil and gas"
  }
]

Result: an import set with the related table created (consisting of these fields: is created. The table has top500, name, sector) containing two elements fields and contains two records.

// Path for Each Row == exportField
{
   "exportField":[
      {
         "top500":1,
         "name":"More",
         "sector":"Oil and gas"
      },
      {
         "top500":2,
         "name":"Nevermore",
         "sector":"Oil and gas"
      },
	  {
		"top500": 4,
		"name": "Givememore",
		"sector": "Oil and gas",
		"more_info": "https://instance.example.com/company/34"
	  }
   ]
}

Result: an import set with the related table created (consisting of these fields: is created. The table has top500, name, sector, more_info) containing three elements fields and contains three records.

Anchor
Import Sets Import Sets

Import Sets

Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources.

Import Set field description

Import set unique number. This field is populated automatically.

This field contains related import set record number. This field is populated automatically.

Import set loading state. This field is populated automatically and always read-only.

Possible values are:

• Loading – data loading into the import set is in progress.
• Loaded – data loading into the import set has been finished.
• Processed – the import set transform has been processed.
• Cancelled – data loading has been cancelled.

Field value example:

Table structure: (top500 | name | sector | more_info | created_at | created_by | record_id)

Anchor
Transform Maps Transform Maps

Transform Maps

Transform map is a set of field maps.  In In this set of field maps, the correlations between fields in of an import set table and fields in of a target table are specified.

Every Each import operation requires at least one transform map which specifies the relationships relationship between the import set table and the target table.

To After creating the first import set, create a related transform map. To do this, please complete the steps below:

1. Click on the View Transform Map UI action located on the import source form you are configuring.
2. In the info appeared toast message appeared, follow the link Create Transform Map.
3. Fill in the fields in the appeared form appeared.
4. Click Save or Save and Exit to apply changes.

Transform Maps field description

Select this checkbox to display the Script field, where you can define the transform script.

Transform Scripts

Transform

Transform map script

Transform map script allows you to customize import operations using a JS-scripts written using a SimpleOne Server-Side API. 

Transform Script field description

Specify the script type to occur:

• onComplete – this script runs at the end of the transformation.
• onStart – this script runs at the start of the transformation.
• onAfter – this script runs after the source row is transformed into the target row.
• onBefore – this script runs before the source row is transformed into the target row.

scripts

with

SimpleOne Server-Side API. 

Anchor
Field Maps Field Maps

Field Maps

Field maps are used to define correlation between fields in an the correlation between fields of the import set table and fields in of the target table by determining . For this, you need to determine the values from the source table to that will be added to the target table.

To create a field map, please complete the steps below:

1. Navigate to Import → Transform Maps.
2. Open a the transform map you want to configure.
3. Click Create Field Maps UI actionfield map.
   • The Field
   Maps
   • Mapping widget appears
   (see
   • . See the screenshot in the table below
   )
   • .
4. Configure field mapping within the current transform transformation process.
5. Click Set to save changes or Cancel to discard changes.

Field map fields description

If this option is active within the transform process, then the engine checks for values in the Target table field whether they are equal to the Import set table field value. 

If values are equal, then the record in the Target table is updated; otherwise, a new record is created.

Map form fields 

Running a transform

1. Navigate to Import → Import Sets.
2. Open an import set you want to transform. Make sure that the state is Loaded.
3. Click Transform.

As a result, the toast message Import is completed appears. Imported data will be transferred into the target table.

Transform sequence

The scheme below shows the sequence of events triggered by the transformation: 

Image Added

1. The sequence begins with the onStarttransform script. It will be executed at the start of the transformation before any source row is read.
2. The field map script takes the record object of the Import Set Row table as an argument and returns the value to put

1. into the Target Field.

This field appears when the Use Script checkbox is selected.

1. The transform map script transforms field values from the source row to the target row.
2. The onBefore transform script is executed before the source row is transformed into the target row. 
3. Between the execution of onBefore and onAfter scripts, the target record is inserted or updated.
4. The onAfter transform script runs when the source row has been transformed into the target row.
5. The onComplete transform script is executed when all source rows are transformed.
   

Running a transform