Verify the Migration Status

Note: If any of the commands fail to copy the data to the target, run those commands again.

Check the status of the index in the following files in the same directory where the jar is located:

  • migration_output*success.csv
  • migration_output*failure.csv

Example filenames:

  • metadata_migration_output__ALL_SUCCESS_1691428484168.csv
  • migration_output_Final__ALL_SUCCESS_1691021040147.csv
  • metadata_migration_output_HAS_FAILURE_1691428484168.csv
  • migration_output__HAS_FAILURE_1691428484168.csv

After the migration is successful, enable job compaction for the INDEX_COMPACTION_PARENT_JOB index using the following curl command :

JSON 
curl -H 'Content-type: application/json' -XPOST 'http://<ELB_hostname_of_Events_Service_23.x>:<ElasticSearch_Port>/job_framework_job_details_v1/_update_by_query' -d '{
 "query": {
  "term": {
   "name": "INDEX_COMPACTION_PARENT_JOB"
  }
 },
 "script": {
  "inline": "ctx._source.jobDataMap.enabled = true",
  "lang": "painless"
 }
}'

Reset Migration

To undo any changes during migration and start afresh, perform the following steps:

  1. Delete dataMigration.db or rename/move existing db file.

  2. Delete metadataIndices_history.json or rename/move metadataIndices_history.json.

  3. Repeat the migration steps.

To redo the migration for any particular index, perform the following steps for the respective index:

  1. Connect to SQLite from the command line:
    CODE 
    sqlite3 <Your_Database_name>
  2. Run following command to delete status of index.
    CODE 
    DELETE FROM tasks WHERE candidate_index = <index_name> and task_status = "SUCCESS";
  3. Run the data migration again for that index. See Migrate Events Service Data.

Useful commands

To get information regarding the migration utility, run the following command:

CODE 
java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar --help

SQL Commands

You can optionally determine the status of the migration using the dataMigration.db instead of the logs. The migration database (dataMigration.db) stores the migration details for customer indices in the following tables:

  • WORKFLOW — Contains each iteration of the data migration job. Every iteration has a unique workflow ID.
  • CANDIDATES — Contains the list of indices that the utility has detected for migration.
  • TASKS — Tracks each task. It contains common and index specific tasks, the common task has the candidate_index value = "ALL" .
  • REINDEX_TASK_STATUS — Contains the re-indexed state of each candidate's index.
Note: Note that the dataMigration.db does not store any status for metadata indices. To know the details of metadata indices migration, see metadataIndices_history.json.
  • To monitor the migration status using SQLite, run the following command:
    PYTHON 
    sqlite3 dataMigration.db (dataMigration.db will be generated in the same folder where the jar file resides, hence you must run this either on the same folder or provide absolute path).
select * from tasks;

  • To obtain the indices that have not registered any failures during migration:

    • Total number of documents to be re-indexed or successfully re-indexed.

    • Created documents in the Events Service 23.x.

    • Updated documents in the Events Service 23.x.

      PYTHON 
      SELECT candidate_index, total, created, updated  from REINDEX_TASK_STATUS where completed == 1 and is_failed = 0
  • To obtain the indices that have registered any failures during migration:

    • Total number of documents to be re-indexed or successfully re-indexed.

    • Created documents in the Events Service 23.x.

    • Updated documents in the Events Service 23.x.

      PYTHON 
      SELECT candidate_index, total, created, updated  from REINDEX_TASK_STATUS where completed == 1 and is_failed = 1

  • To view the status of the documents where re-indexing is in progress:

    PYTHON 
    SELECT candidate_index, total, created, updated  from REINDEX_TASK_STATUS where completed == 0

You may come across some errors during migration. See Troubleshoot Migration Issuesand resolve them.