MobileMap: Data Backup, Recovery and Export

Overview

MobileMap provides several methods for automated and manual data backup and recovery.  These tools are intended to prevent data loss in the event of a mobile device failure.  They are not intended to provide offline workflows for data management, but can be used to help protect data prior to data upload when data upload will be delayed due to lack of internet access.

Automated Backup

Settings

Backup Interval

• Automated data backup is currently only enabled for MobileMap Cruise, as it is triggered by save of specific cruise features such as stands, plots or trees.  If not using cruise functionality, use manual backup to save data.  If automated backup of cruise data is desired, enable this in MobileMap settings by setting the backup frequency (Settings > MobileMap Cruise > General > Database Backup Interval) to the appropriate interval (e.g., every plot save operation, every stand save, every 10 trees, etc.)

Backup Format

• The default backup format is a single text file, containing JSON data which includes only the new and modified features (see description below).  An alternative SQLite database format is also available (see description below) and can be enabled in settings (Settings > MobileMap > General > Data Backup in SQLite Format).)

Backup File Location

All backups will be stored in an 'Export' folder in the selected Data Directory.  WSG recommends the use of MicroSD cards for MobileMap base data and thus for the Data Directory.  By using MicroSD card, automated backups will be stored on external media that can be removed is a device is damaged in the field.  When automated backups (MobileMap Cruise only) are used with a MicroSD card, it is possible to recover all data collected during the day even if the mobile device is damaged at the end of a day (e.g., dropped in a stream) by removing the SD card, drying the SD card off and inserting it into a PC, accessing the most recent backup file (e.g. JSON text file) and uploading the new/modified data to the feature service by cutting and pasting into a browser.

Manual Backup / Export

In addition to the automated backup described above, data can be backed up manually (Actions > Export Database).  This may be necessary in situations where a device is damaged and no longer able to access the internet (and thus cannot upload) or if upload errors cannot be resolved.  One situation where this can occur is when features have been deleted on the server (e.g., by InventoryManager) then a MobileMap users modifies those feature and tries to upload their changes.  In this case, the change cannot be applied, because the feature does not exist in the service.  It is possible to export these data and manually load them into the service. 

Export to JSON

This approach is the preferred (default) format for automated and manual backup of data from MobileMap.  This approach exports only the new/modified feature data and stores it in a single text file that contains a JSON representation of the Esri feature data.  It includes the links to the feature service endpoints for adding/updating data for each layer that contains new/modified features.  While it is intended for use by data managers, standard users can be trained to use this approach.

Because this approach exports only new/modified data, there are scenarios where the JSON files produced will not initially include the desired data.  Consider the case where a user successfully uploads data from MobileMap, then deletes those data using AGOL, InventoryManager, etc., but decides at a later time that they want to recover those data from MobileMap.  In this case, MobileMap will not see those data as new/modified, since they have not changed since they were uploaded.  One way to address this issue is to re-open and save each feature that needs to be recovered.  Simply opening and closing the tree list for each plot, for example should re-save the plot and all trees, and thus include these plots and trees in the JSON export file.

When data are exported to JSON format, they will be saved in the data directory's Export sub-folder, in a file named MobileMap_XXXX_todays_date_incremental_backup_JSON.txt.  To load these data into the feature service, copy this file to a PC, open it using a text editor, and then load the data by copying in the JSON representation of the data into the matching feature service layer endpoint. 

For example, considering the following information in an export JSON file:

This indicates that the layer Plots has one feature that needs to be updated in the feature service. This feature is not a new feature created in MobileMap an only on this device, but rather one that already exists on the server. We know this because it is an update and not an add. To apply this update to the service, we will access the updateFeatures REST endpoint for this service using a web browser and then paste in the updated data and submit.

For each Layer in the JSON export file, and for each layer action (adds, updates, deletes) complete the following steps:

• Paste the full URL into into the address bar of the browser then hit enter. This will open up the relevant (Add, Update or Delete) REST endpoint for this layer in your browser.  You might be prompted to log in if you are not already logged into ArcGIS Online site.

• Next, paste in the entry data text string, which will start with [{ and end with }] into the Features text box. 

• Set Rollback on Failure to false

• Click the submit button (Add Features, Update Features or Delete Features) and note the success or error message(s) that are displayed

Open

If an update failed because you are attempting to update features that have been deleted from the server, they will fail using this process as well.  If you believe that the deletion from the service was unintended, and you wish to re-create these features in the service, change the final characters of the URL from .../updateFeatures to .../addFeatures and then repeat the process to save these as new features.  

If the update (or add) fails due to a validation error, you may need to use a text editor to modify your data.  Consider the case where a feature service hosted in ArcGIS Portal was updated (overwritten) by a data manager and a data type was changed from Text to Integer.  The JSON data will still contain text values like "small", "medium", "large" while the new service will expect values like 1, 2 or 3.  Before the data can be saved in the service, they will need to be manually fixed to address this disconnect. If this is needed, consider using an advanced text editor that can display and validate JSON data (e.g., Notepad++, Atom, Sublime), and online JSON pretty-print tools like https://jsonformatter.org/json-pretty-print to help you edit these data.

Export to SQLite Database

This approach produces a full SQLite database (all data, even data that has been downloaded from the feature service and is unchanged).  This approach can be useful if a large amount of data must be recovered and the data have not been modified since they were uploaded.  In that case, they would not initially be included in the default JSON export format, and using the SQLite format would avoid having to re-save all features just to get them to export.  The SQLite format, however, is not a standard Esri offline geodatabase, it is a MobileMap-specific SQLite format.  This approach is for expert users as it requires knowledge of SQL/SQLite, and a SQLite viewer/editor (e.g., DB Browser for SQLite https://sqlitebrowser.org/).  WSG can provide instructions for recovering data from this SQLite file to data managers if needed.  WSG can also perform this data recovery as a service upon request. 

Manual Backup of Export Cruise Data to CSV files

Similar to the Cruise Export functionality in InventoryManager, this approach exports cruise data as spreadsheet files for use in Excel or other software. This approach differs from the InventoryManager export in that it exports to multiple CSV files instead of a tabbed Excel file. When using this option MobileMap will create separate CSV files for for Stands, Plots, Trees and Logs and will include attribute data only (no geometries) .  This approach is typically used by users who want to transfer to a PC to run Excel functions for QA prior to data upload to the feature service, or to share recently collected data while out in the field and without an internet connection. 

Similar to the export as SQLite option described above, the CSV export includes all data in the tables that are included in the export configuration. This means that the export files may be large, and may take several minutes to generate, if there a large number of features in the database. This is different than the JSON export approach described above, in which only the new and edited data are exported.

In order to use this functionality, you must first create an export configuration file. This export configuration file is the same format as the file used by InventoryManager for cruise export, but should not include the dataType field which is required for export to FVS and TCruise.  The export configuration file should contain all of the fields that need to be exported.  For more details on export configuration file creation see: https://masonbruce.atlassian.net/wiki/spaces/MD/pages/307790025/InventoryManager+Export+Inventory+Data#Export-Configuration-File

The export configuration file must be named ‘cruiseExportMapping.csv' and must be stored in a folder named 'Export' that is within a MobileMap project folder on your device (e.g., SD card > MobileMap > MyProject > Export > cruiseExportMapping.csv). When the Export Cruise Data option is selected, MobileMap will use the currently selected map cache as the location to find the configuration file (in the Export subfolder).  This means that you should use the ‘Select Map Cache’ and select the project name which you are using prior to running Export Cruise Data.

When the Export Cruise Data operation is run, the CSV files will be stored in the MobileMap Export folder in a subfolder that is named with the current date. The filenames will include the Android ID of the device, today’s date, and the layer name (e.g., SD card > MobileMap > Export > 1_1_2021 > fddadd45533cd6da_9_13_2021_20_25_40_Stands.csv).