Introduction

This document describes how to upgrade database from one version of AllegroGraph to a later version of AllegroGraph. Using the same technique, you can also make a copy of a database. The tools and methods described in this document work for all versions of AllegroGraph, 4.4 and later.

If you have AllegroGraph version 3.x and want to upgrade to version 5.x please contact Franz Inc. Customer Support ([email protected]).

If you want to upgrade from a version v4.x where x < 4 (e.g., from 4.3) to a later version (such as 7.0.0), see pre-v4.4 Upgrade Guide.

The issue with upgrading

You want two things to be moved from the earlier version to the later: the data and the server settings. The data is the set of triples. The server settings are (more or less) everything else: user information, security settings, etc.

While you can simply upgrade databases (when necessary) to a new AllegroGraph release or (if upgrading is not required) simply shut down the old server and open the databases in the new server, we recommend that you (1) back up your databases before using them in the new release and (2) test the new release with your data before committing to switching over. That means having more than one copy of the data, at least for a while. For large databases, that can mean using a great deal of disk space. We make suggestions about how to proceed to minimize disk usage in various situations but we cannot anticipate all cases. All AllegroGraph users have to devise upgrade plans which suit their specific needs using the available backup and upgrading tools.

Quick and simple upgrading

When you install a new release of AllegroGraph, running the following programs will make a copy of all databases in a new location suitable to be used by the new release and do any database conversions that are necessary. Note that this quick and simple method trades disk space for simplicity. So if your databases use X bytes, about 3X bytes are needed by this method. If you have large databases and disk space might be an issue, consider the more efficient methods described below.

Suppose you have version X and a new version Y, with version X <= version Y. The steps are these:

  1. Using agtool archive of version X, archive all the repositories and settings into the directory specified as an argument (see Repository Backup and Restore for agtool archive usage). 
    2. agtool archive --port X-port backup-all <directory>
  2. Using agtool archive of version Y, restore all the repositories and settings. If the database format has changed, it is this step that reformats the data. 
    3. agtool archive --port Y-port restore-all <directory>

After restoring the settings, you must restart the new (version Y) AG server to reflect the changes.

If the original installation uses catalogs other than root, the same catalogs need to exist in the destination configuration.

Is upgrading necessary?

AllegroGraph has a release number (such as the current release 7.0.0). AllegroGraph databases have an internal format and this format is changed from time to time. If the internal database format changes between releases, databases must be upgraded in order to be used. (The upgrading is done either by the agtool archive program with the restore or restore-all option, or by the program agtool upgrade, described below.) But if the internal database format is the same in two release, databases need not be upgraded. The Release Notes say whether the internal database format has changed. If you try to open a database from an earlier database format that has not been upgraded, you will get an error and the database will not open. See the chart below for information on when the format has changed.

If the upgrade program agtool upgrade notices that a repo is already running in the version that it is upgrading to, it will exit with a message that upgrading is not required.

If upgrading is not necessary, can a database from the later release be opened in an earlier release?

No. If both later release Y and earlier release X use the same internal database format, you can open X databases in Y. However, you cannot open Y databases in X. Even though the database structure is the same, other aspects may be different.

What is the cost of upgrading?

Upgrading takes time and uses disk space. Following the quick upgrade path described above -- backup in old version, restore in new version -- there are for a time at least three copies of the data on disk. That is significant if you have a large amount of data. Upgrading can take appreciable time, but it only needs to be done once or maybe twice per new release. (If upgrading is required, you must upgrade at least once. When you need to have the server running without interruption and you want to test the new release, you may find you need to upgrade twice: once to create databases for testing and once again when testing is complete and you want the server data -- which now has data added since the first upgrade -- moved to the new release.)

Suggested procedures when an upgrade is not required

In this discussion, your current version is release-current and the new version is release-new.

Even though an upgrade is not required, you still have to test the new release before switching over and it is desirable to test it with your actual databases.

The issue is how much downtime you can tolerate and how much new data you are willing to risk if things go wrong. In a development or test system, a reasonable downtime is often OK, as is some data loss. In a production system, downtime should be as short as possible and data lossage is unacceptable. (Data might be lost if the release-current server is stopped and testing of the release-new server fails: data added during the release-new test may be difficult or impossible to recover.)

Upgrade is not required, case 1 (minimal testing)

This is a minimal test method. You backup and stop the current version and start and restore to the new version. If the new version is unsatisfactory, you can restart and restore the backup to the old version.

  1. Create a backup in the current release of the running databases using the agtool archive from release-current with the backup or backup-all option, like the following: 
    2. agtool archive --port release-current-port backup-all <directory>

  2. Stop the current AllegroGraph version (release-current).

  3. Start the new version (release-new) and restore the backed up data with: 

    4. agtool archive --port release-new-port restore-all <directory>

Upgrade is not required, case 2 (more testing)

Here are suggested steps for a testing track upgrade:

  1. Create a backup in the current release of the running databases using the agtool archive from release-current with the backup or backup-all option, like the following: 
    agtool archive --port release-current-port backup-all <directory>
  2. While leaving the current release running, start the new version (release-new) and restore the backup up data:

    3. agtool archive --port release-new-port restore-all

  3. Test while both versions are running.

If testing fails, continue with release-current while you solve the problem, whatever it is. If testing is ok,

  1. Stop release-current and continue with release-new.

Suggested procedures when an upgrade is required

When the internal database version is changed, then to use a database created by release-current in release-new, it must be upgraded/converted. The agtool archive backup/restore procedure described above does the upgrade, but at a significant space cost when databases are large (since for a while at least, three copies of the data exist simultaneously). agtool upgrade (see below) does an in-place conversion, so in theory only one copy of a database needs to exist but having a backup is recommended so two copies will in fact exist.

Again, consider two cases: (1) you need the server running while you test the later release (roughly, do you have a continuously running production system), or (2) you can tolerate noticeable downtimes and possible data loss (roughly, you are developing and can recover without too much work).

Upgrade required, case 1 (minimal server downtime)

If you are in the case where the server downtime must be minimal, then possible steps are:

  1. Create a backup in release-current of the running databases using the release-current agtool archive with the backup or backup-all option: 
    2. agtool archive --port release-current-port backup-all <directory>
  2. Restore that backup to release-new using the release-new agtool archive with the restore or restore-all option. 
    3. agtool archive --port release-new-port restore-all <directory>

  3. Keep release-current server running while you test the new release.

  4. When testing is complete, delete the new release databases (they do not include changes made to the old release versions while testing).

  5. Make a backup in release-current of the running databases using the release-current agtool archive with the backup or backup-all option (this backup protects you in case there is a problem with the upgrade; the backup can be deleted when the databases are successfully upgraded): 

    6. agtool archive --port release-current-port backup-all <directory>
  6. Either user the restore or restore-all option to release-new agtool archive (the easiest but least space efficient way) or use agtool upgrade to upgrade the old release databases to the new release (the harder but more space efficient way).

Upgrade required, case 2 (server can be down for some time)

If you are in case (2) where server downtime is allowed, then the steps are:

  1. Create a backup of the release-current databases using agtool archive (use the agraph-backup program if release-current is prior to 6.2.0, which was when *agtool was introduced). Use the backup or backup-all option: 
    2. # For releases prior to 6.2.0  
 
agraph-backup --port release-current-port backup-all <directory>  
 
# For release 6.2.0 and later  
 
agtool archive --port release-current-port backup-all <directory>

  2. Restore to the new release using the restore or restore-all options to release-new agtool archive. Test the new release.

  3. If testing goes well, continue using the new release server. If you need to revert to the old release server, restore from the backup made in step 1. You will have to make any changes made with the new release server again. (Recall, case (2) assumes that downtime and some lossages are acceptable. Recall too that once a database is opened by the new release server, it cannot be opened by the old release server.)

Upgrading when you have distributed repositories

Distributed repositories (see the Distributed Repositories Setup document) allow data to be distributed over several repositories called shards, typically stored on different AllegroGraph servers.

The upgrade process when you have distributed repositories is similar to upgrading when you have only regular repositories. But your new version must have a agcluster.cfg file which defines the same servers and server characteristics as that of your older version (essentially have the same or at least sufficiently similar with respect to existing distributed repos' agcluster.cfg file).

Pre 7.0.0 warning: upgrading pre-7.0.0 distributed repos into version 7.0.0 or later is not supported. Export the data and reload from the resulting data files. (While not officially released or supported, test versions of distributed repositories were available in both earlier official releases and in test releases.)

The agtool upgrade program

The agtool upgrade program will do an in-place upgrade of a single database from an earlier release to the current release (the release containing the agtool program being run). The AllegroGraph server must be running.

The agtool program is the general program for AllegroGraph command-line operations. (In earlier releases, there was a separate convertdb program. A change from earlier releases is the -p short form of the --port option and the -c short form of the --catalog option.)

Backup databases before converting with agtool upgrade

Before using agtool upgrade, be sure to backup your databases. You can do this with agtool archive (agraph-backup for releases prior to 6.2.0). Using the agtool archive or agraph-backup from the earlier release, do the following (this is the same as the first step of the quick and simple upgrading: 

# For releases prior to 6.2.0  
agraph-backup --port X-port backup-all <directory>  
 
# For release 6.2.0 and later  
agtool archive --port X-port backup-all <directory>

Now you can upgrade the individual databases with agtool upgrade. If something goes wrong, you are in a position to restore the databases, using agtool archive/agraph-backup with the restore or restore-all. (You can restore to the old version or the new version, depending on which agtool archive/agraph-backup you use.)

agtool upgrade usage

The usage is: 

agtool upgrade GROUND-REPO-SPEC

A GROUND-REPO-SPEC specifies a simple repository. Catalog, port, host, and username information can be specified as part of the GROUND-REPO-SPEC. See the Repository Specification document.

Because the upgrade is in-place, this is the most space efficient way to upgrade a database.

Only one database at a time in a catalog can be upgraded. If a catalog has five separate databases, agtool upgrade must be run five times.

The --noconvert option

When an archive created by older version of agtool archive is restored, either by restore or restore-all, the database is converted to the current version by default.

To suppress this operation, give --noconvert option to agtool archive.

Recent releases where the internal format changed

As we said, upgrades are required when the internal database format changes. The table below shows the release where the format changed. Contact support ([email protected]) if you have questions about the internal format of earlier releases. 

Recent releases where the internal format changed  
---------------------------------------------------------------  
      4.14.1  
      5.0  
      6.0  
      6.1  
      6.4.0  
      6.5.0

So a database from a release prior to 4.14.1 must be upgraded in order to be used in any release starting with 4.14.1. Databases from releases 4.14.1 and after but before 5.0 must be upgraded to be used with release 5.0 and after. Databases from releases 5.0 to 6.0.2 need to be upgraded to be used with release 6.1. Once a database has been opened in a newer release, it should not be used with an earlier release, even if the format has not changed.