Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The Import engine allows administrators to collect data from various data sources and then map this data into relevant tables.

The SimpleOne import engine supports:

  • different data source types
  • different file formats (if you choose File as the data source)

You can choose the data source, place the data from it to the temporary table (also can be called "an import set"), and after that, with the transform maps and the field maps, you can map the data from the source to the target table.

Tip

Role required: admin.

You can find a graphical procedure view of the import process in the diagram below. A more detailed description is provided further in this article.

Section


Column
width33%

    


Column
width33%


Column
width33%

    



Anchor
Key Terms
Key Terms
Key concepts

ConceptDescription
Import sourceA record that specifies what data an import set should import. 
Import set

A record that contains references to raw data uploaded to the staging location. The record also returns information about this import attempt state.

Import set table

An automatically generated table that is used as the temporary storage for imported records before transformation. The structure of the table is automatically generated depending on the imported data.

Transform

The process of converting data according to the transform map and transform scripts that define the source table (the import set) and the target table.

Transform map

In this set of field maps, a record that specifies correlations between fields in an import set table and fields in a target table.

Transform scriptA script that allows for customizing import operations using native JavaScript extended with SimpleOne Server-Side API.
Field mapA record that specifies the relationship between a field in an import set table and a field in a target table.
Coalesce

This option allows you to update existing table records within the transformation process.

For more information, navigate to the relevant article part: Coalesce.

Anchor
Import Sources
Import Sources
Import sources


Creating an import source is necessary to load raw data into the system for further processing and transformation. You can set up an import source with different data source types and file formats.

  1. Navigate to Import → Import Sources.
  2. Click New and fill in the fields.
  3. If the File type is chosen, click the attachment icon   and attach the file you need.

    Tooltip
    onlyIcontrue
    appendIconinfo-filled
    iconColorblue

    You can attach JSON, XML, or Excel files. 


  4. Click Save or Save and Exit to apply the changes.

Import Source form fields

FieldMandatoryDescription
NameYSpecify the import source name.
Import Set Table NameYSpecify the import set table name.

Type

N

Choose the type of import source. Available choice options:

File Retrieval MethodN

Choose the method to add the file to the import source. Available choice options:

  • Attachment

This field will appear if the File type is chosen.

FormatY

Select the file format and attach it to the form. Available file format options:

  • JSON
  • XML
  • Excel (only when the File type is chosen)

This field will appear if the File or Text type is chosen.

Import TextN

Specify the text you need to import. This field appears if the Text type is chosen. 

Code Block
languagejs
themeEclipse
titleText format
collapsetrue
[
  {
    "property": "value"
  },
  {
    "property": "value"
  }
]


LDAP DefinitionN

Click the magnifier icon Image Addedto choose the protocol you need. This field appears if the LDAP type is chosen. 

Tip

For more information about the LDAP import source, refer to the LDAP Data Source article.



After saving, the following UI actions appear on the import source form. They are responsible for further import processing.

ActionDescription
Test load (20 records)

After clicking, a pilot import set is created with a 20-records capacity, intended for importing data structure analysis.

Note

Do not use the test set for further transformation.


Load all records

This action creates a new import set record by loading full data from the import source there.

This import set can be used for further transformation.

After loading data into the import source, saving the record and calling one of the data loading actions, a new record appears in the Import Sets related list. 

Using JSON format


You can use files containing data formatted in JSON as an import source. In this case, make sure that the files follow the criteria below:

  1. JSON files should be valid. Use the RFC 8259 document as the guideline.
  2. Data types available to use in JSON structure are listed below. Dot-walking is not supported.
Code Block
languagejs
themeEclipse
titleJSON Object
linenumberstrue
  {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  }

Result: one element. The table has top500, name, sector fields and contains one record.

Code Block
languagejs
themeEclipse
titleArray of objects
linenumberstrue
[ 
 {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  },
  {
    "top500": 2,
    "name": "Nevermore",
    "sector": "Oil and gas"
  }
]

Result: an import set with the related table is created. The table has top500, name, sector fields and contains two records.

Code Block
languagejs
themeEclipse
titleObject with array of objects
linenumberstrue
// Path for Each Row == exportField
{
   "exportField":[
      {
         "top500":1,
         "name":"More",
         "sector":"Oil and gas"
      },
      {
         "top500":2,
         "name":"Nevermore",
         "sector":"Oil and gas"
      },
	  {
		"top500": 4,
		"name": "Givememore",
		"sector": "Oil and gas",
		"more_info": "https://instance.example.com/company/34"
	  }
   ]
}

Result: an import set with the related table is created. The table has top500, name, sector fields and contains three records.

Anchor
Import Sets
Import Sets
Import sets


Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources. 


Anchor
Transform Maps
Transform Maps
Transform maps


A transform map is a set of field maps. In this set of field maps, correlations between fields of the import set table and fields of the target table are specified.

Each import operation requires at least one transform map that specifies the relationship between the import set table and the target table.

After creating the first import set, create a related transform map. To do this, complete the steps below:

  1. Click View Transform Map on the import source form you are configuring.
  2. In the message that appears, follow the link Create Transform Map.
  3. Fill in the fields in the form that appears.
  4. Click Save or Save and Exit to apply the changes.
Tip

A transform map can also be created from scratch. To do this, complete the steps below:

  1. Navigate to Import → Transform Maps to open the transform maps list.
  2. Click New and fill in the fields.
  3. Click Save or Save and Exit to apply the changes.



Anchor
Field Maps
Field Maps
Field Maps


Field maps are used to define the correlation between fields of the import set table and fields of the target table. For this, you need to determine the values from the source table that will be added to the target table.

Info

During the import process, you may need to update the existing records in the target table. To set up the key relationships between the Source Table and the Target Table, create at least one field map record, with the Coalesce attribute enabled. This attribute should be enabled for a field map that contains relationships of columns with unique values. It can be an email for a user or a serial number for a CI.

To achieve stricter mapping, you can enable the Coalesce option for more than one Field Map record.

To create a field map, complete the steps below:

  1. Navigate to Import → Transform Maps.
  2. Open the transform map you need to configure.
  3. Click Create field map.
    • The Field Mapping widget appears. See the screenshot in the table below.
  4. Configure field mapping within the current transformation process.
  5. Click Set to save the changes or Cancel to discard the changes.
Tip

Here is another way to create field maps:

  1. Navigate to Import → Transform Maps.
  2. Open the transform map you need to configure.
  3. In the Field Maps related list, click New and fill in the fields.
  4. Click Save or Save and Exit to apply the changes.
  5. Repeat the previous steps for each field map that needs to be set.


Run a transform


  1. Navigate to Import → Import Sets.
  2. Open an import set you need to transform. Make sure that the state is Loaded.
  3. Click Transform.

As a result, the message Import is completed appears. Imported data will be transferred into the target table.

Table of Contents
absoluteUrltrue
classfixedPosition