Data Import

Import engine allows administrators to collect data from various data sources, and then map this data into relevant tables.

The SimpleOne import engine features the following:

• different data sources types are supported
• different file formats are supported (if you choose "File" as the data source).

You can choose the data source, place the data from it to the temporary table (also can be called "an import set"), and after that, with the transform maps and the field maps, you can map the data from the source with the target tables.

You can find graphical procedure view of the import process below. A more detailed description is provided further in this article.

Key terms

General import procedure description

1. Prepare raw data.
2. Create an import source.
3. Load raw data into the import set table.
4. Create a transform map.
5. Create field maps.
6. Run a transform.
7. Verify results.

Import Sources

Creating an import source is necessary to load raw data into system for further processing and transformation. You can set up an import source able to work with the different data source types and file formats.

1. Navigate to Import → Import Sources.
2. Click New and fill in the fields.
3. Click Save or Save and Exit to apply changes.

After loading data into the import source, saving the record and calling the one of the data loading actions, a new record appears in the Import Sets related list.

Also, several UI actions responsible for the further import processing appear on the import source form (see description below).

Using JSON format

You can use JSON files containing data formatted in JSON as a import source. In this case, make sure that these files follow the criteria below:

1. JSON files should be valid. Use RFC 8259 as a guideline.
2. Data types available to use in JSON structure are listed below. Dot-walking is not supported.

  {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  }

Result: an import set with related table created (consisting of these fields: top500,name,sector) and containing one element.

[ 
 {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  },
  {
    "top500": 2,
    "name": "Nevermore",
    "sector": "Oil and gas"
  }
]

Result: an import set with related table created (consisting of these fields: top500,name,sector) containing two elements.

// Path for Each Row == exportField
{
   "exportField":[
      {
         "top500":1,
         "name":"More",
         "sector":"Oil and gas"
      },
      {
         "top500":2,
         "name":"Nevermore",
         "sector":"Oil and gas"
      },
	  {
		"top500": 4,
		"name": "Givememore",
		"sector": "Oil and gas",
		"more_info": "https://instance.example.com/company/34"
	  }
   ]
}

Result: an import set with related table created (consisting of these fields: top500,name,sector, more_info) containing three elements.

Import Sets

Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources.

Import Set field description

Table structure: (top500 | name | sector | more_info | created_at | created_by | record_id)

Transform Maps

Transform map is a set of field maps. In this set of field maps, the correlations between fields in an import set table and fields in a target table are specified.

Every import operation requires at least one transform map which specifies the relationships between the import set table and the target table.

To create a transform map, please complete the steps below:

1. Click on the View Transform Map UI action located on the import source form you are configuring.
2. In the info message appeared, follow the link Create Transform Map.
3. Fill in the fields in the form appeared.
4. Click Save or Save and Exit to apply changes.

Transform Maps field description

Transform map script

Transform map script allows you to customize import operations using a JS-scripts written using a SimpleOne Server-Side API. 

Transform Script field description

Field Maps

Field maps are used to define correlation between fields in an import set table and fields in the target table by determining the values from the source table to be added to the target table.

To create a field map, please complete the steps below:

1. Navigate to Import → Transform Maps.
2. Open a transform map you want to configure.
3. Click Create Field Maps UI action.
4. The Field Maps widget appears (see screenshot in the table below).
5. Configure field mapping within current transform process.
6. Click Set to save changes or Cancel to discard changes.

Field map fields description

Running a transform

1. Navigate to Import → Import Sets.
2. Open an import set you want to transform. Make sure that its state is Loaded.
3. Click Transform.