AllegroCache Administrator's Guide

  The Files in an AllegroCache Database
  Standalone vrs Client/Server
  Starting and Stopping an AC database
  Log Files
  Upgrading AllegroCache
  Upgrading a Database
  Log Compression
  Backups

The Files in an AllegroCache Database

All AllegroCache files for a database are stored in a directory. The name of the directory is not stored in the database so you can rename or move or copy the directory at will. Those operations should not, however, be done while the database is open.

Here is a sample AC database:

  5136 -rw-r--r-- 1 jkf jkf   5242890 Nov 13 09:42 ac000000.dat
  5136 -rw-r--r-- 1 jkf jkf   5242889 Nov 13 09:42 ac000001.dat
  5136 -rw-r--r-- 1 jkf jkf   5242898 Nov 13 09:43 ac000002.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000003.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000004.dat
  5136 -rw-r--r-- 1 jkf jkf   5242887 Nov 13 09:45 ac000005.dat
  5136 -rw-r--r-- 1 jkf jkf   5242881 Nov 13 09:46 ac000006.dat
  5136 -rw-r--r-- 1 jkf jkf   5242884 Nov 13 09:46 ac000007.dat
  4924 -rw-r--r-- 1 jkf jkf   5026128 Nov 13 09:48 ac000008.dat
     4 -rw-r--r-- 1 jkf jkf         5 Nov 13 09:41 acache-params.cl
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:47 admin.bt
116972 -rw-r--r-- 1 jkf jkf 119668736 Nov 13 09:47 class12.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 class1.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 index10.bt
 46468 -rw-r--r-- 1 jkf jkf  47529984 Nov 13 09:47 index11.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 itab.bt

The presence of the file acache-params.cl indicates that this directory contains an AC database.

The acNNNNNN.dat files are log files. A detailed description of log files is below.

The *.bt files are b-trees, which are used to store key/value pairs on disk. The API for the b-tree package is in a separate document. The AllegroCache programmer or administrator need never work directly with the b-trees.

admin.bt and itab.bt contain information about the database. classN.bt maps object id and transaction number to a location in a log file. indexN.bt maps a value and a transaction number to an oid.

Standalone vrs Client/Server

The file format for a standalone database is identical to that of a client/server database. In fact you can open a database in standalone mode, close it, and then open it it client/server mode.

A database can only be opened by one process at a time. Opening a database by two processes will result in the log files being scrambled. The Windows operating system enforces the rules that a database is only opened once. Unix does not enforce this on its own but we intend to use locks in a future version of AC to prevent this problem from occurring.

If you need to have multiple clients access the database then use the client/server version. In this version the server opens the database and the clients contact the server to access the database.

Starting and Stopping an AC database

In standalone mode the database is opened with open-file-database or create-file-database and closed with close-database. The database files are not modified except when a commit is done (except that the admin.bt btree is modified occasionally in the middle of a transaction). When a commit is not in progress the database on disk is in a consistent state. The close-database function merely closes file descriptors. It does not modify the database.

In client/server mode the start-server function opens the database. The server then accepts connections from clients when they call open-network-database. When a client is finished accessing the database it calls close-database. This only closes the client connection to the database and not the database itself. There are two ways to close the server's connection to the database

on the server run stop-server.
have a client connect to the server using open-network-connection and then have the client call (close-database :stop-server t).

When the server shuts down all clients currently connected to it will get an error message the next time they try to contact the server.

When an existing database is opened in standalone mode (open-file-database) or in client/server mode (start-server) the program can specify a :verify argument. This argument allows the program to tell AC how hard it should check to see if the the files on disk that comprise the database are error free. There are four choices

nil - don't check at all
:quick - check the end of the last log file and if it looks good then assume that AC was shut down cleanly. If it doesn't look good then scan the last log file from the beginning to find where it starts to go bad and truncate the log file at that point.
t - omit the quick scan of the last log file. Start with a scan of the last log file.
:full - do a scan of the last log file and then scan all of the b-trees for errors. If errors are found signal an error and suggest that the database be rebuilt from the log files.

Here are some factors to determine what kind of verification you should use:

If you're not sure if the database was shut down cleanly then do a :full verification if time permits.
Never specify no verification at all. Due to buffered output to the log file it's very easy for the end of the log file to be a partial record which must be removed before the log file can be used. A quick verify can detect this and fix it.
The cost to scan last log file is a function of its length. This argues against specifying a very very large log file size when the database is opened.

In order to konw what kind of verification should be done we suggest that you keep a marker in the filesystem that allows you tell if the database was shut down cleanly. This would then allow you to choose the appropriate argument to :verify.

Log Files

The log files contain everything needed to recreate the database again using the recover-from-log function. The log files are the sole repository for the objects in the database and therefore cannot be deleted.

The log files are numbered beginning with ac000000.dat. Only the newest log file is ever modified and information is only added at the end of this file. This means that for database backup purposes you need only save each log file once after that log file is no longer the newest log file.

You can specify the maximum size of each log file when you open the database. It's best to choose a very large size as that means that fewer log files need to be open. In the example above we chose a very small size (5Mb) for log files just so we could demonstrate a multi-log file database.

Because log files contain the whole history of the database they will take up more and more space on the disk. At some point you will may want to reduce the size of the database. There are three ways to do this. Two methods require that you stop the database and then rebuild it and then open it:

save-database/restore-database - the database is written out in an xml format and then read back in. This is time consuming but you are left with a text version of your database you can archive. Be sure to specify :verbose nil to save-database to make this operation faster.
recover-from-log - build a new database using only the current objects in the database. The resulting database has smaller log files and smaller b-trees as well. This will usually be much faster than save-database/restore-database.

The third possibility is compressing the log files. This is done while the database is still running. There is more on this below.

Upgrading AllegroCache

New versions of AllegroCache are released periodically. You can receive notification of these new versions by viewing the AllegroCache RSS feed or you can periodically check the AllegroCache Download Page

The AC Download page describes how to get the latest version of AC downloaded to your machine. Unlike other patch downloads, when you download AC it will intentionally not replace the version of AC you're now using. This is because the new AC may be incompatible with your current AC and simply replacing your current AC could make your data inaccessible.

Before upgrading you'll first want to check out the compatibility chart at the bottom of the AC download page. If there are any no's in the column labeled read one version older db between the latest version and the version you're currently running then you must upgrade your database. If there are any yes's in the third column labeled must rebuild then even upgrading is impossible and you must build your database again from scratch. We'll discuss upgrading the database below.

When you upgrade your AllegroCache you'll want to recompile all your source code that uses AC. This will ensure that any macro definition changes in AC are used by your code. If you're using the client/server version then all clients and and the server should be recompiled and rebuilt. Generally older clients will work with newer servers as very rarely does the existing AC network protocol change incompatibly. However to be safe it's best to keep the client and server versions in sync.

Upgrading a Database

You can use either of two techniques to upgrade a database from one version to another (assuming the the compatibility matrix at the end of the AC download page says that such an upgrade is legal):

run save-database in the application running the old version of AC and then run restore-database in the application running the new version of AC
close the old database and then start a lisp with the new AC loaded and run recover-from-log. It will create a new database based on those logs.

Using recover-from-log is usually the fastest and easiest way to upgrade. That's what we show in the example below.

These are the steps for upgrading a database

In standalone mode just close the database. In client/server mode shut down all the clients and then shut down the server. See "Starting and Stopping an AC database" above.
Move the current database to a new location so the new database can be built where our application still expects to find it.
```
tiger :: mv /paw/cl8/src/acache/bigac.db /paw/cl8/src/acache/save-bigac.db
tiger ::
```

Start lisp and load in the new version of AllegroCache. Suppose the newest version is 1.2.1.

cl-user(1): (require :acache "acache-1.2.1.fasl")
; Fast loading /paw/cust-acl80/code/acache-1.2.1.fasl
AllegroCache version 1.2.1
;   Fast loading /paw/cust-acl80/code/sax.001
;;; Installing sax patch, version 1
t
cl-user(2):

Run the recover-from-log function. The log files will come from the directory /paw/cl8/src/acache/save-bigac.db and the other files in that directory will be ignored. None of the files in that directory will be modified. The database will be rebuilt in the directory /paw/cl8/sc/acache/bigac.db.

cl-user(4): (db.ac:recover-from-log "/paw/cl8/src/acache/save-bigac.db" "/paw/cl8/src/acache/bigac.db")
; Fast loading /paw/cust-acl80/code/regexp2.fasl
;   Fast loading /paw/cust-acl80/code/yacc.fasl
Log Files will used for recovery:
  /paw/cl8/src/acache/save-bigac.db/ac000008.dat
  /paw/cl8/src/acache/save-bigac.db/ac000007.dat
  /paw/cl8/src/acache/save-bigac.db/ac000006.dat
  /paw/cl8/src/acache/save-bigac.db/ac000005.dat
  /paw/cl8/src/acache/save-bigac.db/ac000004.dat
  /paw/cl8/src/acache/save-bigac.db/ac000003.dat
  /paw/cl8/src/acache/save-bigac.db/ac000002.dat
  /paw/cl8/src/acache/save-bigac.db/ac000001.dat
  /paw/cl8/src/acache/save-bigac.db/ac000000.dat
; Fast loading /paw/cust-acl80/code/fileutil.001
;;; Installing fileutil patch, version 1
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000008.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000007.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000006.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000005.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000004.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000003.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000002.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000001.dat"
Recover from log "/paw/cl8/src/acache/save-bigac.db/ac000000.dat"
Rebuilding indexes
17086992 bytes have been tenured, next gc will be global.
See the documentation for variable *global-gc-behavior* for more information.
18221168 bytes have been tenured, next gc will be global.
See the documentation for variable *global-gc-behavior* for more information.
17113616 bytes have been tenured, next gc will be global.
See the documentation for variable *global-gc-behavior* for more information.
17508240 bytes have been tenured, next gc will be global.
See the documentation for variable *global-gc-behavior* for more information.
database recovered
#<AllegroCache db "/paw/cl8/src/acache/bigac.db" -closed- @ #x1000f90cc2>
cl-user(5):

Optionally open the newly upgraded database and verify that there are no errors:

cl-user(7): (db.ac:open-file-database "/paw/cl8/src/acache/bigac.db" :verify :full)
Check tree: /paw/cl8/src/acache/bigac.db/admin.bt
Check tree: /paw/cl8/src/acache/bigac.db/itab.bt
Check tree: /paw/cl8/src/acache/bigac.db/class12.bt
Check tree: /paw/cl8/src/acache/bigac.db/index11.bt
Check tree: /paw/cl8/src/acache/bigac.db/index10.bt
Check tree: /paw/cl8/src/acache/bigac.db/class1.bt
#<AllegroCache db "/paw/cl8/src/acache/bigac.db" @ #x1006605442>
cl-user(8): (db.ac:close-database)
#<AllegroCache db "/paw/cl8/src/acache/bigac.db" -closed- @ #x1006605442>
cl-user(9):

Rebuild your application: remove all fasl files and, using the latest allegrocache fasl file, rebuild all applications that call any AllegroCache functions or use any macros from AllegroCache.
Restart you application. In client/server mode start the server before starting the clients.

Log Compression

Over time log files contain more and more information that will never be accessed. One way to remove this unused information is to compress the log files. Compression is done much like a copying garbage collector: the parts of the log file still in use are copied to new files and then the new files replace the old files.

Log compression is done while the database is open (unlike save-database/restore-database and recover-from-log). In the client/server version you can even do log compression while other clients are accessing the database.

Let's compress this database:

  5136 -rw-r--r-- 1 jkf jkf   5242890 Nov 13 09:42 ac000000.dat
  5136 -rw-r--r-- 1 jkf jkf   5242889 Nov 13 09:42 ac000001.dat
  5136 -rw-r--r-- 1 jkf jkf   5242898 Nov 13 09:43 ac000002.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000003.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000004.dat
  5136 -rw-r--r-- 1 jkf jkf   5242887 Nov 13 09:45 ac000005.dat
  5136 -rw-r--r-- 1 jkf jkf   5242881 Nov 13 09:46 ac000006.dat
  5136 -rw-r--r-- 1 jkf jkf   5242884 Nov 13 09:46 ac000007.dat
  4924 -rw-r--r-- 1 jkf jkf   5026128 Nov 13 09:48 ac000008.dat
     4 -rw-r--r-- 1 jkf jkf         5 Nov 13 09:41 acache-params.cl
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:47 admin.bt
116972 -rw-r--r-- 1 jkf jkf 119668736 Nov 13 09:47 class12.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 class1.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 index10.bt
 46468 -rw-r--r-- 1 jkf jkf  47529984 Nov 13 09:47 index11.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 itab.bt

We open the database and call compress-log-files:

cl-user(4): (open-file-database "test")
#<AllegroCache db "/paw/cl8/src/acache/test" @ #x1001c3ebf2>
cl-user(5): (compress-log-files)
Number of full Commits: 180
Commit in progress at the end: 185
Processing log file /paw/cl8/src/acache/test/ac000000.dat
111025 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000001.dat
113975 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000002.dat
110000 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000003.dat
110022 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000004.dat
114018 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000005.dat
110960 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000006.dat
110000 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000007.dat
113029 changes in log file
nil
cl-user(6):

Now let's examine the database directory:

  3188 -rw-r--r-- 1 jkf jkf   3257198 Nov 13 14:20 ac000000.dat
  5136 -rw-r--r-- 1 jkf jkf   5242890 Nov 13 09:42 ac000000.dat.b00001
  3124 -rw-r--r-- 1 jkf jkf   3191339 Nov 13 14:20 ac000001.dat
  5136 -rw-r--r-- 1 jkf jkf   5242889 Nov 13 09:42 ac000001.dat.b00001
  3192 -rw-r--r-- 1 jkf jkf   3262898 Nov 13 14:21 ac000002.dat
  5136 -rw-r--r-- 1 jkf jkf   5242898 Nov 13 09:43 ac000002.dat.b00001
  3192 -rw-r--r-- 1 jkf jkf   3262498 Nov 13 14:21 ac000003.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000003.dat.b00001
  3120 -rw-r--r-- 1 jkf jkf   3190570 Nov 13 14:21 ac000004.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000004.dat.b00001
  3176 -rw-r--r-- 1 jkf jkf   3245607 Nov 13 14:21 ac000005.dat
  5136 -rw-r--r-- 1 jkf jkf   5242887 Nov 13 09:45 ac000005.dat.b00001
  3192 -rw-r--r-- 1 jkf jkf   3262881 Nov 13 14:21 ac000006.dat
  5136 -rw-r--r-- 1 jkf jkf   5242881 Nov 13 09:46 ac000006.dat.b00001
  3140 -rw-r--r-- 1 jkf jkf   3208362 Nov 13 14:21 ac000007.dat
  5136 -rw-r--r-- 1 jkf jkf   5242884 Nov 13 09:46 ac000007.dat.b00001
  4924 -rw-r--r-- 1 jkf jkf   5026155 Nov 13 14:20 ac000008.dat
     4 -rw-r--r-- 1 jkf jkf         5 Nov 13 09:41 acache-params.cl
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 14:20 admin.bt
116984 -rw-r--r-- 1 jkf jkf 119668736 Nov 13 14:20 class12.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 14:20 class1.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 index10.bt
 46468 -rw-r--r-- 1 jkf jkf  47529984 Nov 13 09:47 index11.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 14:20 itab.bt

What we see is that the log files (the files acNNNNNN.dat) are now smaller. We see additional files ending in ''.b00001'. These are the original log files. Our policy is to never remove or modify the older log files. However you can delete these files if wish to recover the space. AC will never reference them.

Now suppose we run for a while longer and do the compression operation again.

cl-user(8): (compress-log-files)
Number of full Commits: 180
Commit in progress at the end: 185
Processing log file /paw/cl8/src/acache/test/ac000000.dat
15001 changes in log file
Processing log file /paw/cl8/src/acache/test/ac000001.dat
No changes to log file
Processing log file /paw/cl8/src/acache/test/ac000002.dat
No changes to log file
Processing log file /paw/cl8/src/acache/test/ac000003.dat
No changes to log file
Processing log file /paw/cl8/src/acache/test/ac000004.dat
No changes to log file
Processing log file /paw/cl8/src/acache/test/ac000005.dat
No changes to log file
Processing log file /paw/cl8/src/acache/test/ac000006.dat
No changes to log file
Processing log file /paw/cl8/src/acache/test/ac000007.dat
No changes to log file
nil
cl-user(9):

We can see that the changes this time happened to be all in the first log file.

The contents of the database directory is now:

  2944 -rw-r--r-- 1 jkf jkf   3007422 Nov 13 14:28 ac000000.dat
  5136 -rw-r--r-- 1 jkf jkf   5242890 Nov 13 09:42 ac000000.dat.b00001
  3188 -rw-r--r-- 1 jkf jkf   3257198 Nov 13 14:20 ac000000.dat.b00002
  3124 -rw-r--r-- 1 jkf jkf   3191339 Nov 13 14:20 ac000001.dat
  5136 -rw-r--r-- 1 jkf jkf   5242889 Nov 13 09:42 ac000001.dat.b00001
  3192 -rw-r--r-- 1 jkf jkf   3262898 Nov 13 14:21 ac000002.dat
  5136 -rw-r--r-- 1 jkf jkf   5242898 Nov 13 09:43 ac000002.dat.b00001
  3192 -rw-r--r-- 1 jkf jkf   3262498 Nov 13 14:21 ac000003.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000003.dat.b00001
  3120 -rw-r--r-- 1 jkf jkf   3190570 Nov 13 14:21 ac000004.dat
  5136 -rw-r--r-- 1 jkf jkf   5242894 Nov 13 09:44 ac000004.dat.b00001
  3176 -rw-r--r-- 1 jkf jkf   3245607 Nov 13 14:21 ac000005.dat
  5136 -rw-r--r-- 1 jkf jkf   5242887 Nov 13 09:45 ac000005.dat.b00001
  3192 -rw-r--r-- 1 jkf jkf   3262881 Nov 13 14:21 ac000006.dat
  5136 -rw-r--r-- 1 jkf jkf   5242881 Nov 13 09:46 ac000006.dat.b00001
  3140 -rw-r--r-- 1 jkf jkf   3208362 Nov 13 14:21 ac000007.dat
  5136 -rw-r--r-- 1 jkf jkf   5242884 Nov 13 09:46 ac000007.dat.b00001
  5072 -rw-r--r-- 1 jkf jkf   5181078 Nov 13 14:28 ac000008.dat
     4 -rw-r--r-- 1 jkf jkf         5 Nov 13 09:41 acache-params.cl
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 14:28 admin.bt
117892 -rw-r--r-- 1 jkf jkf 120602624 Nov 13 14:28 class12.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 14:28 class1.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 09:41 index10.bt
 46468 -rw-r--r-- 1 jkf jkf  47529984 Nov 13 14:28 index11.bt
     8 -rw-r--r-- 1 jkf jkf      8192 Nov 13 14:20 itab.bt

Now we can see that we've just added one file ac000000.dat.b00002 which is the log file ac000000.dat just before the second call to compress-log-files started. Again this file can be deleted.

Backups

To backup an AC database you must, at minimum, save the log files (the files named acNNNNNN.dat). If you wish to save the whole directory then you should ensure that no database operations are being done while you're saving the b-tree files or the resulting file on the backup tape could be garbage.

Doing an incremental dump of the log files can save a lot of backup tape as the old log files are never changed.