Data Import

Import engine allows administrators to collect data from various data sources, and then map this data into relevant tables.

The SimpleOne import engine supports:

• different data source types
• different file formats (if you choose File as the data source).

You can choose the data source, place the data from it to the temporary table (also can be called "an import set"), and after that, with the transform maps and the field maps, you can map the data from the source to the target table.

You can find a graphical procedure view of the import process in the diagram below. A more detailed description is provided further in this article.

Key concepts

Import sources

Creating an import source is necessary to load raw data into the system for further processing and transformation. You can set up an import source with different data source types and file formats.

1. Navigate to Import → Import Sources.
2. Click New and fill in the fields.

3. If the File type is chosen, click the attachment icon   and attach the file you need.

4. Click Save or Save and Exit to apply changes.

After saving, the following UI actions appear on the import source form. They are responsible for further import processing.

After loading data into the import source, saving the record and calling one of the data loading actions, a new record appears in the Import Sets related list. 

Using JSON format

You can use files containing data formatted in JSON as an import source. In this case, make sure that the files follow the criteria below:

1. JSON files should be valid. Use the RFC 8259 document as a guideline.
2. Data types available to use in JSON structure are listed below. Dot-walking is not supported.

  {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  }

Result: one element. The table has top500, name, sector fields and contains one record.

[ 
 {
    "top500": 1,
    "name": "More",
    "sector": "Oil and gas"
  },
  {
    "top500": 2,
    "name": "Nevermore",
    "sector": "Oil and gas"
  }
]

Result: an import set with the related table is created. The table has top500, name, sector fields and contains two records.

// Path for Each Row == exportField
{
   "exportField":[
      {
         "top500":1,
         "name":"More",
         "sector":"Oil and gas"
      },
      {
         "top500":2,
         "name":"Nevermore",
         "sector":"Oil and gas"
      },
	  {
		"top500": 4,
		"name": "Givememore",
		"sector": "Oil and gas",
		"more_info": "https://instance.example.com/company/34"
	  }
   ]
}

Result: an import set with the related table is created. The table has top500, name, sector fields and contains three records.

Import sets

Generally, an import set should be created automatically based on the raw data uploaded into the import source. Import sets are intended to be a staging area for records imported from the data sources. 

Transform maps

Transform map is a set of field maps. In this set of field maps, correlations between fields of an import set table and fields of a target table are specified.

Each import operation requires at least one transform map which specifies the relationship between the import set table and the target table.

After creating the first import set, create a related transform map. To do this, complete the steps below:

1. Click View Transform Map on the import source form you are configuring.
2. In the appeared toast message, follow the link Create Transform Map.
3. Fill in the fields in the appeared form.
4. Click Save or Save and Exit to apply changes.

Field Maps

Field maps are used to define the correlation between fields of the import set table and fields of the target table. For this, you need to determine the values from the source table that will be added to the target table.

To create a field map, please complete the steps below:

1. Navigate to Import → Transform Maps.
2. Open the transform map you need to configure.
3. Click Create field map.
   • The Field Mapping widget appears. See the screenshot in the table below.
4. Configure field mapping within the current transformation process.
5. Click Set to save changes or Cancel to discard changes.

Run a transform

1. Navigate to Import → Import Sets.
2. Open an import set you need to transform. Make sure that the state is Loaded.
3. Click Transform.

As a result, the toast message Import is completed appears. Imported data will be transferred into the target table.