Configuration Packs

The SimpleOne system allows exporting and importing configuration packs to keep unrelated instances synchronized.

Concept

In brief, the VCS configuration pack conception can be defined as an ability to transfer data from one instance to another, in an automatic or semi-automatic way. It is intended for application version control. Configuration packs help to automate transfer data and configurations from one instance to another, lessen the manual effort. The key point is configurations transfer, for not to reiterate it more and more on every instance, you can just import a configuration pack. So, this technology really is to make it easy and automate a migration update from one instance to another.

Application configurations are stored in configuration packs which are represented as a .SOP file containing a set of the application version records. In an OOB supply, the SimpleOne platform contains only the Simple application, but administrators (users with the admin role) can create their own Applications if needed.

Initially, the configuration pack should be assembled on the source instance (the instance containing source data) within the local pack related to the specific application. After that, the local pack can be exported to the configuration pack as a .SOP file.

Using configuration packs allows deploying this configuration on other instances, or, if needed, rollback versions of the specified record or some local pack as a whole to the previous state.

Technically, the configuration pack is the set of the saved records of the VCS Record (sys_vcs_record) table. Every record of this table contains JSON formatted changes and other attributes described below.

{"value": "Report Item", "policy": "Open", "sys_id": 159653803414986194, "column_id": 156941403909472422, "record_id": 159653803414985080, "language_id": 156628684306541141, "application_id": 155931135900000002, "sys_created_at": "2020-08-04 10:47:14", "sys_created_by": 155931135900000001, "sys_updated_at": "2020-08-04 10:47:14", "sys_updated_by": 155931135900000001}

The configuration pack transfer between instances can be also called "import". On the picture below, you can find the cross-instance configuration versions import flow:

Record versions

After any transaction (create/update/delete) for the versioned table object, the record is created in the VCS Record (sys_vcs_record) table corresponding to the object state after the transaction. The record version contains the information described below.

VCS Record form fields

Local packs

Technically, a local pack is a record in the VCS Local Pack (sys_vcs_local_pack) table related to versions of the related records in a VCS Record (sys_vcs_record) table. Every single version is an atomic state of versioned tables (those ones which have the is_VCS_enabled attribute turned ON). All record versions in the local pack are displayed on a related list on a local pack form.

To create a local pack, please complete the steps below:

1. Navigate to Configuration Pack → Local Packs.
2. Click New and fill in the fields.
3. Click Save or Save and Exit to apply changes.

VCS Local Pack form fields

Version restoring

A non-actual version (the is_current attribute value is equal to FALSE) can be restored if needed. To perform this, please complete the steps below:

1. Navigate to Configuration Pack → Records.
2. Find the record to be recovered. You can use list search for search criteria narrowing, or a Condition Builder.
3. Click on a desirable record to navigate into it.
4. Click Restore Version.

After that, a new VCS record is created in the current local pack; the Restored by field value of this record is equal to the ID of the version used for restoring.

Default local pack

Default local pack is a local pack named "Default 1" created automatically after creating a new application. It is not recommended to use this local pack to store versions for the purpose of further import to other instances; create separate packs for these tasks.

If a version has been created in a default local pack erroneously, then you can move it to the required local pack. For this, please complete the steps below:

1. Navigate to Configuration Pack → Records.
2. Find the record to recover. You can use list search for search criteria narrowing, or a Condition Builder.
3. Click on a desirable record to navigate into it.
4. Click the magnifier icon on the left of the Local Pack field and choose the required local pack.
5. Click Save or Save and Exit to apply changes.

Protection Policy

The Protection Policy (policy) attribute is responsible for the possibility of the record changing (for example, overwriting). It it used, in particular, for data protection purposes.

How it works

When the record in the versioned table is created, its Record and Record Policy attribute values are equal to Open. After first updating, its Record and Record Policy attribute values changes to Changed.

Records with these two states for the Record and Record Policy attributes are freely modifiable.

Also, one more protection policy state for record is Protected. Records with this state cannot be changed or deleted.

During the importing process,. the records with the policy state allowing overwriting can be updated. Both record policy values set in the system and in configuration pack are taken into account. All possible policies combinations are described in the table below 

В процессе обновления платформы или приложения (которое заключается в установке конфиг-пака, содержащего это обновление) обновляются все записи, Policy которых допускает возможность обновления. Учитываются Policy, установленные для записей в системе и в локалпаке. Возможные варианты установленных Policy и результаты попыток установки представлены ниже в таблице:

Local Pack Exporting

After you are done with record versions collection within the task implementing, you need to export the changes for further import to other instances. For this, please complete the steps below:

1. Navigate to Configuration Pack → Local Packs.
2. Find the local pack to export. You can use list search for search criteria narrowing, or a Condition Builder.
3. Click on a desirable local pack to navigate into it.
4. Change the state value to Completed.
5. Click Export Local Pack link.

As a result, 

Merging Local Packs

Two local packs can be merged in one if necessary. For this, please complete the steps below:

1. Navigate to Configuration Packs → Local Packs.
2. Choose local packs you need to merge selecting the appropriate checkboxes and click the Merge button.
3. Fill in the name and description of the consequent local pack.
4. Wait until local packs are merged.
5. As a result, a new local pack is created referenced with the last versions of the VCS Record (sys_vcs_record) for every record.
6. Resulting local pack transits into the In Progress state.
7. The original local packs are removed.
8. If any versions were referenced to original local packs and didn't get into the resulting one, then this reference is reset.

Local Pack Importing

To deliver an exported local pack on a target instance, the import procedure should proceed. What is implemented within this procedure:

• The transactions stored in configuration pack (a local pack that was exported) are applied on a target instance.
• A local pack copy created on a target instance.

To import a local pack, please complete the steps below:

1. Navigate to Configuration Pack → Retrieved Packs.
2. Click New to create a new configuration pack and fill in the fields.
3. Attach previously exported .SOP file to this configuration pack.
4. Click Save or Save and Exit to apply changes.

VCS retrieved pack form fields

The import process includes three phases:

1. Preview
2. Import
3. Apply.

Preview

During this phase, the import engine checks for the possible conflicts before the pack is imported. To preview the configuration pack, please complete the steps below:

1. Navigate to Configuration Pack → Retrieved Packs.
2. Find the configuration pack to preview (it should not be previewed before, so the state should be equal to New).
3. Click on it to navigate into it.
4. Click the Preview Local Pack link.

As a result:

1. The VCS Preview Log records are created and displayed at the related list appropriate for this configuration pack.
2. The configuration pack state changes to Previewed.

VCS Preview Log fields description

Collisions processing

Generally, collisions may appear during the configuration pack import process. Their leading cause is that the dependent table or record does not exist in the database.

Consider a common use case:

1. We have exported the Incident table, child for the Task table;
2. We are importing it to the instance without the Task table;
3. In this case, the Incident table cannot be created on the target instance, because there is no parent table from which to inherit.

After that, you can find the erroneous record at the VCS preview log related list (in the VCS retrieved pack section). This record has the state=collision and the appropriate Error Text containing the error cause. Erroneous records cannot be included in the saved records pack (VCS records) bound to the retrieved pack that is used for data recovery.

Collisions disposal methods

The idea in the collision fixing process is to try to dispose of collisions in the highest level tables; this method lets to remove collisions in the lower-level tables automatically. It can be done about like this:

1. Filter the records in the VCS preview log by the state (state=collision);
2. Analyze errors for the records at the top of the list;
   1. Alternative way: you can filter the list by the Table field. The highest-level tables to analyze are, for example, sys_db_table, sys_number, sys_db_column, and some others.
3. After the error cause was found, it must be fixed on the source instance, and the configuration pack is to be exported and previewed again. clicking Preview Again Local Pack.
4. If collisions are gone, then this configuration pack now can be imported. Herewith, all the records from the last configuration pack attached, are imported and retrieved pack status changes to loaded. The records are added to the VCS record (sys_vcs_record) table. The last preview log is available in the Related Lists area.
5. After importing, the retrieved pack applying possibility appears. For this, click Apply Local Pack. The system restores records from this configuration pack (in other words, from the VCS record (sys_vcs_record) table records. attached to this retrieved pack). The retrieved pack changes its state to Committed.
6. If the collisions are not gone, then return to the previous steps.

Collision case 1

When installing a configuration pack in the system, some record “X” is added. At the same time, it is referenced to another records “Y” in this system and in this configuration pack. Both records that the record ‘X’ is referenced to do not exist. When the export admin initiates a configuration pack preview (using a Preview Local Pack button), the collision is thrown on this record, which is an expected result.

Collision solving:

The solution is the configuration pack rebuilt in order to get the record “Y” into it.

Collision case 2

The record “X” is deleted from the system. At the same time, there is a reference to this record in the configuration pack. When the export admin initiates a configuration pack preview (using a Preview Local Pack button), the collision is thrown on this record, which is an expected result.

Collision solving:

The solution is the configuration pack rebuilt in order to remove this record from it.

Collision case 3

Create a table on a source instance, with the field number equal to N.

Create the same table with the same name on a target instance, with the field number equal to N-1.

On the source instance, perform the export of the table fields into a configuration pack, after that download a .SOP file and unpack it on the target instance.

The result will be a collision thrown, and a message that this object is not valid, any actions on it cannot be taken.

Collision solving:

To solve this collision, create or update an object, excluding attributes missing on the target instance. In the above example, the N-fielded table will be successfully imported to the target instance, where the (N-1)-fielded table is created. But, the fields that exist in the source instance and are not in the target instance will not be created in the target instance. See the schema below.

Collision case 4

The records on a target instance are newer than in a configuration pack to install. It can be determined by the date and time of record creation.

Collision solving:

Export admin needs to initiate a local pack preview. In the process:

1. Collision-causing records will be marked with the Skipped state.
2. When applying the local pack, the records with this state will be skipped. The records that were present on the target instance initially will stay.
3. In the case of record in the configuration pack is newer than on the instance, then it will be marked with the Good state, and it will replace the record on the instance.

Collision case 5

A table deleting from instance contains records (is not empty). If the table is deleted being empty, then there is no collision.

Collision solving:

1. Collision-causing records will be marked with the Skipped state.
2. When applying the local pack, the records with this state will be skipped.
3. If the record was marked with the Good state, then the table will be deleted.

Collision case 6

Use case description: configuration pack contains objects related to the application X. At some moment, export admin tries to apply this configuration pack on the instance that do not have this application installed.

Collision solving:

The solution is the configuration pack rebuilt in order to add records related to the missing application.

Collision case 7

Partial attribute mismatch between instances.

Use case description.

Create a table on a source instance, with the field number equal to N.

Create the same table with the same name on a target instance, with the field number equal to N-1.

On the source instance, perform the export of the table fields into a configuration pack, after that, download a .SOP file and unpack it on the target instance.

The field names on source and target instances are slightly different. See the schema below.

Import

After the configuration pack has been previewed, preview log records have been analyzed and possible collisions have been solved, you can proceed with further configuration pack importing. For this, please complete the steps below:

1. Navigate to Configuration Pack → Retrieved Packs.
2. Find the previously previewed configuration pack (the state should be equal to Previewed).
3. Click the Import Local Pack button.

As a result:

• VCS records are created on the target instance with the attributes as listed below:
  • Is Current = FALSE
  • Not related to any local pack in the system.
• The retrieved pack state changes to Loaded.
• The date and time of this operation are displayed at the Loaded at field.

Apply

The final import phase is an applying. For this, please complete the steps below:

1. Navigate to Configuration Pack → Retrieved Packs.
2. Find the previously imported configuration pack (the state should be equal to Loaded).
3. Click Apply Local Pack.

As a result:

• A local pack record on a target instance is created.
• All versions created on the previous stage are copied onto this local pack.
• The is_current attribute value for these records changes to TRUE.
• The retrieved pack state changes to Committed.
• The date and time of this operation are displayed at the Committed at field.

Snapshots

In version control systems (VCS), snapshot is the fixed system or database status description. To create a table snapshot, a table managing form can be used, and also, server-side API class SimpleVcs was implemented.

Role required: admin.

Option 1

1. Navigate to System Definition → All Tables.
2. Using the search bar and/or a Condition Builder, find the table you want to create a snapshot and navigate into it.
3. Click the Create VCS Snapshot button at the right top. This button is visible only when the versioning for this table is on (the is_vcs_enabled attribute is set to TRUE).

Option 2

1. Navigate to System Definition → Scripts.
2. Write down a script and call SimpleVcs#createTableSnapshot(tableName) method in it.

In both cases, the result is the snapshot, that is, the set of the VCS records for all the documents in this table that do not have such records at the moment of the snapshot creating.

Rollback

Rollbacks in brief

1. Any completed local pack can be rolled back;
2. Rollback will restore the versions of all the records in this local pack as they were before update collection within this local pack started, or before committing this local pack.
3. Deleted records are restored, and created records are deleted.

Example

• Before local pack committing, the record had a version X.
• After the local pack committing, it has version Y.
• And after rollbacking, it will have version X again, even if after local pack completing it has had any other version.

How to rollback a local pack

To rollback a local pack, please complete the steps below:

1. Navigate to Configuration Pack → Local Pack.
2. Click on the local pack you need to rollback and navigate into it.
3. Click Rollback Preview to preview the list of the records that are going to be backed out.
4. Review the contents of the local pack, fix the possible collisions if required. After fixing the collision, please complete the steps below:
   1. Navigate to Configuration Pack → Preview Log.
   2. Click on the record that was causing the collision.
   3. Click on the State choice list and select the allowed value.
   4. Click Save or Save and Exit to apply changes.
5. Click Rollback to perform the rollback.

6. Resolve the issues encountered.

   1. The currently selected application affects what options are available for the local pack. So please make sure that you select the application, such as Platform, that matches the content of the local pack.
   2. Do not rollback the Default 1 local pack without an absolute necessity; otherwise, your SimpleOne instance configuration may be damaged.
      1. Rollback of any local pack is an operation that shouldn't be invoked without an emergency need; so use it cautiously.

   
   

This process both record updates and dictionary changes. Some changes caused by the rollback may lead to data loss.

Here is the expected result of the process: