Introduction

agtool is a program for performing a variety of operations on an AllegroGraph server or repository.

agtool is included with the AllegroGraph distribution, in the bin/ subdirectory of the installed AllegroGraph directory. It is also available for separate download from https://franz.com/agraph/downloads/ (the same directory containing the latest AllegroGraph downloads). It is packaged in a compressed tar* file. This command unpacks it: 

% tar zxf linuxamd64.64/agtool-8.0.0-linuxamd64.64.tar.gz

That creates a directory named agtool with the agtool executable and all necessary additional files. You can use that version of agtool interchangeably with the one included in the AllegroGraph distribution. Note you can install the downloaded agtool on any suitable x86-64 Linux machine whether or not AllegroGraph is installed. This version of agtool should only be used with AllegroGraph 8.0.0 images and associated data.

The general calling sequence for agtool and the tools it supports which operate on repositories is 

% agtool <tool> [options] [REPO-SPEC] [arguments]

Some tools operate just on the running AllegroGraph server. The general for for such tools is 

% agtool <tool> [--server SERVER-SPEC] [options] [arguments]

Some (like version) do not require a spec argument at all.

Most tools are documented in separate documents linked to from this document.

REPO-SPECs

A repository specification or REPO-SPEC argument identifies a repository and also contains the necessary SERVER-SPEC information. The Repository Specification document describes the REPO-SPEC format. In its concise form, a REPO-SPEC encodes all the necessary information to identify a repository. Its general form is: 

[USER:PASSWORD@][[HOST][:[PORT][s]]/][CATALOG:]REPO

The various elements are:

The scheme
Either http or https. The default is http. The scheme is encoded by the presence or absence of the s following the PORT: https when the s is persent, http when it is not.
USER and PASSWORD
A valid AllegroGraph user and that user's password.
HOST
The host on which the AllegroGraph server is running. The default is 127.0.0.1, which is the same as localhost.
PORT
The port which the HOST is listening on. The default is 10035.
CATALOG
The name of the catalog which contains the specified REPOSITORY. Catalogs are defined in the configuration file and can only be created at server startup time so the specified CATALOG must already exist. If not specified, defaults to the root catalog.
REPOSITORY
The repository of interest. Depending on the tool, this repository may or may not exist (some tools create repositories).

In its simplest form, a REPO-SPEC is just a repository name, which then expands to 

localhost:10035/repository-name

If anything must have a value other than the default, more elements must be used. See Repository Specification for numerous examples.

So for example, to load an Ntriples file mydata.nt into the my-repository repository, the command might be 

 % agtool load my-repository mydata.nt

or equivalently 

 % agtool load localhost:10035/my-repository mydata.nt

Tools that operate on repositories accept REPO-SPECs for identifying repositories.

Older repository specification arguments

In earlier releases, most tools which acted on repositories had options --scheme, --user, --host, --port, and --catalog (some of which had abbreviations). --catalog is no longer supported at all. For the others, some tools still accept the remaining arguments. For those tools, use of these older arguments is deprecated and will signal a warning when used. If these arguments are accepted and used and a REPO-SPEC is also specified, values must match the values specified in the REPO-SPEC argument.

Because the required matching includes default values, some commands may have unmatched values even though they are not apparent. Note the username and password are considered part of the host but do not have default values. The --scheme argument is handled specially. Here are some examples using agtool archive backup: 

# Allowed, as no host and port values are present in the REPO-SPEC:  
$ agtool archive backup repo1 \  
         --host user:pass@host --port 30035 repo1.bak  
 
# Allowed, as host and port values are the same.  
$ agtool archive backup user:pass@host:30035/repo1 \  
         --host user:pass@host --port 30035 repo1.bak  
 
# Not allowed, since --host conflicts with the REPO-SPEC.  
# --host user:pass@host2 without a --port arguments (as above)  
# is interpreted as --host user:pass@host2:10035 which conflicts  
# with user:pass@host2:30035 in the REPO-SPEC.  
$ agtool archive backup user:pass@host2:30035/repo1 \  
         --host user:pass@host2 repo1.bak  
 
 
# Not allowed, the host in the REPO-SPEC defaults to 127.0.0.1  
# (i.e. localhost) which does not match the value of --host (host2).  
$ agtool archive backup user:pass@30035/repo1 \  
         --host user:pass@host2 --port 30035 repo1.bak

The following two examples show how --scheme is handled specially. 

# Allowed to override the default scheme (http).  
$ agtool archive backup http://host:10035/repositories/repo1 \  
         --scheme https repo1.bak  
 
# Not allowed to override non-default scheme.  
   $ agtool archive backup https://host:10035/repositories/repo1 \  
             --scheme http repo1.bak

SERVER-SPECs

A SERVER-SPEC specifies the running AllegroGraph server which the tool will operate on. If the tool operates on a repository, the SERVER-SPEC information is included in the REPO-SPEC which identifies the repository but some tools, like agtool user, do not operate on repositories.

A SERVER-SPEC is typically the value of the --server option (some tools use other option names to identify servers; replicate, for example, uses --primary and --secondary). The option defaults to 

--server http://127.0.0.1:10035

You must specify --server SERVER-SPEC if any of the defaults do not apply.

A SERVER-SPEC can be 

HOSTNAME  
HOSTNAME:PORT  
SCHEME://[user:password@]HOSTNAME  
SCHEME://[user:password@]HOSTNAME:PORT

Missing elements except for user and password are taken from the default above, so --server myhost becomes --server http://myhost:10035.

SCHEME can be http or https.

Note that the SERVER-SPEC does not use the same encoding as a concise REPO-SPEC. In particular, the scheme cannot be specified by an s after the port number as it can be in a concise REPO-SPEC.

If the user who started the AllegroGraph server is executing agtool on the machine on which AllegroGraph server is running, agtool will then run under the same uid that the AllegroGraph server is running under. In that case for most tools, agtool acts as if it is being run by a superuser if no user and password are specified. This behavior is called OS authentication. It is common for production versions to be run this way, and it is also typical for users who are testing personal copies.

A superuser can generally run any agtool operation. Other users can run some, depending on their actual permissions. See Managing Users for information on user permissions.

agtool run on localhost can bypass user limitations

The user with the same user id as that running the AllegroGraph server runs an agtool command on the machine where the server is also running, that user effectively gets AllegroGraph superuser permissions.

When the server spec is localhost or 127.1 or foo.com where that name dns resolves to the server machine, and the user running agtool is the same as started the AllegroGraph server, the request bypassses user/password permission aurhentication and performs the requested task. That means is machine USER1, with AllegroGraph username non-super-1 can execute, assuming that user started the AllegroGraph server, 

% agtool users permissions --server localhost:10035 non-super-1 super+

non-super-1 will then have AllegroGraph superuser permission.

This special access, however, is unlikely to actually cause problems as typically the AllegroGraph server is run either by the special agraph user or by someone with AllegroGraph superuser rights anyway.

Help

% agtool --help

displays a help string giving a brief description of each tool. 

% agtool TOOL --help

displays a help string for the specific TOOL.

Tools

agtool can run the following tools. Use agtool TOOL --help to get more information on any tool.

There are additional agtool tools that are not documented here. Some provide information useful for dealing with problem reports and user may be asked to run those as part of dealing with a problem report. Others are associated with other AllegroGraph features and should only be run as part of using those features. Those tools are documented with the feature rather than here.

Most agtool tools take the same options as the program they are replacing, but some have either new options or no longer accept previous options (usually the abbreviation is not accepted but the long form is). These changes are noted for each relevant tool.

Accessing and operating on files on Amazon S3

Some agtool commands may access files located on Amazon S3. You can specify such files by providing an S3 URI, which has a format s3://bucketname/path/to/the/file.

Tools that operate on files include load, export, and archive.

In order to access files on S3 you must provide AWS credentials with necessary permissions to agtool.

Automated S3 access with EC2 instance profile

If agtool runs on an EC2 instance with an IAM role attached and AWS credentials are not set in any other way, then agtool will call the EC2 metadata service to request temporary credentials assuming attached IAM role.

If the IAM role attached to the EC2 instance has access to S3 objects passed to agtool commands, then no further configuration is necessary.

By default agtool will be looking for S3 objects in the same AWS region where EC2 instance is located. If you use EC2 instance profile, but need to access S3 objects in a different region, use AWS_REGION environment variable to set it.

For more information about EC2 instance profiles please see AWS documentation.

Manual configuration of S3 access with AWS credentials

Generally, AWS services require "AWS Access Key ID" and "AWS Secret Access Key" to be presented for authentication. "AWS Session Token" also must be presented for temporary credentials.

It does not matter how these values are provided to agtool as long as they are. They may be (in order of precedence):

Example: if AWS_ACCESS_KEY_ID environment variable is set, but --aws-access-key-id command line argument is passed too, then the command line argument value will be used and environment variable - ignored.

Configure S3 access with agtool command line arguments

You can set AWS credentials with additonal command line arguments to agtool commands:

--aws-access-key-id ACCESS-KEY-ID
AWS Access Key ID.
--aws-secret-access-key SECRET-ACCESS-KEY
AWS Secret Access Key.
--aws-session-token SESSION-TOKEN
AWS Session Token for temporary credentials
--aws-configuration-file CONFIGURATION
The location of the AWS configuration file. Default: ~/.aws/config
--aws-credentials-file CREDENTIALS
The location of the AWS credentials file. Default: ~/.aws/credentials
--aws-profile PROFILE
The profile to select in the Amazon configuration file. Default: default

Configure S3 access with environment variables

You can set AWS credentials and related options using standard AWS CLI environment variables. Here is a list of those that agtool reads:

AWS_ACCESS_KEY_ID
AWS Access Key ID.
AWS_SECRET_ACCESS_KEY
AWS Secret Access Key.
AWS_SESSION_TOKEN
AWS Session Token for temporary credentials
AWS_SHARED_CREDENTIALS_FILE
The location of the Amazon credentials file. Default: ~/.aws/credentials
AWS_CONFIG_FILE
The location of the Amazon configuration file. Default: ~/.aws/config
AWS_PROFILE
The profile to select in the Amazon configuration file. Default: default
AWS_REGION
The AWS region where S3 files are located. Default: us-east-1

Configure S3 access with AWS settings files

agtool can read standard AWS CLI configuration files, usually located in ~/.aws directory.

agtool takes into account the following fields from AWS credentials and config files:

aws_access_key_id = AKIAIOSFODNN7EXAMPLE
AWS Access Key ID.
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS Secret Access Key.
aws_session_token = AVeRyVeRyLONGStRiNGMuCHLoNGERTHANTHISEXAMPLEblah/blahTOKEN
AWS Session Token for temporary credentials
region = us-east-1
The AWS region where S3 files are located. Default: us-east-1

Temporary credentials

AWS allows users to generate temporary credentials that include a session token besides the usual access key ID and the secret access key itself. When temporary credentials are used, failure to specify the session token results in a 403 HTTP response with the error message like 

<Code>  
  InvalidAccessKeyId  
</Code>  
<Message>  
  The AWS Access Key Id you provided does not exist in our  
  records.  
</Message>  
...

For more information see AWS documentation.

Attribute and filter support

AllegroGraph supports triple attributes (key/value pairs associated with individual triples) and static filters (statements that can restrict access to triples based on their attribute values. These features are described in the Triple Attributes document.

Attributes can only be associated with a triple when the attribute has been defined. agtool can be used to define attributes (agtool define-attribute), delete attribute definitions (agtool delete-attribute-definition), lookup attribute definitions (agtool lookup-attribute-definitions), set and delete a static attribute filter (agtool set-static-attribute-filter and agtool delete-static-attribute-filter) and set the attributes for a particular user (agtool users set-attributes) or role (agtool roles set-attributes).

To see the calling sequence for these commands, execute 

% agtool COMMAND --help

For example 

% agtool define-attribute --help

The archive tool

Command calling template: 

% agtool archive [options] command [command-args]

options are prefixed with double dashes (single dash in some cases) and may also take arguments. To get usage information, enter 

% agtool archive --help

The commands which write to or read from files (backup, backup-all, backup-settings, restore, restore-all, and restore-settings) are passed a directory and perhaps additional information like a database name. The specific files are located and named within that directory following standard rules (described here. Unless the --supersede option is specified to backup commands, the archive directory must either not exist or be empty.

agtool archive can be used to upgrade a database from the format on one AllegroGraph release to the format of a later release. This is the recommended way to upgrade since it provides a backup copy of the database in the earlier format which allows easy recovery if there are problems with the later version. The agtool upgrade tool does not (by itself) backup a database before upgrading.

See the Repository Backup and Restore document for a complete list of commands and options.

The auto tool

The auto tool sets up background tools that run to perform certain tasks, either forever (until the server is restarted) or for a specified period of time. In this release, there is only one task: optimize which we describe now. 

agtool auto optimize [options] repo-spec

The options for optimize are:

--idle SECONDS
An index must be idle for this number of seconds (default 100) before being considered for optimization, thus (most times) preventing optimization from interfering with queries and other tasks.
--interval SECONDS
Seconds between checking for possible optimization. Default: 120.
--operate SECONDS
Operate for this many seconds and then exit (optimizations which have started will complete). Zero means run forever (until the agtool process is interrupted with Control-C or killed). Default: 0.
--quiet YES-OR-NO
If yes messages will only be printed when there is an index to optimize. When no, message are printed every interval number of seconds about what the optimizer is doing. Default is no
--verbose
If specified, print more information. Can be specified multiple times.
--workers POSITIVE-INTEGER

Maximum number of optimization operations to perform at once. Default: 1.

Once auto optimize is started, it optimizes indices whose oscores are greater than 1. The automatic optimizer can also be started and stopped in Traditional WebView (see here) and in New WebView using the Repository Control menu. Triple indices are described in the Triple Indices document. See particularly the Optimizing Indices section.

The catalogs tool

The catalogs tool can be used to create and delete dynamic catalogs, which are described in the Catalog definitions section of the Server Configuration and Control document. You must have a DynamicCatalog definition in your configuration file to be able to create dynamic catalogs. If you do have such a definition, then this command will create one or more new dynamic catalogs: 

agtool catalogs create SERVER_SPEC CATALOG_NAME*

Dynamic catalogs can be deleted with 

agtool catalogs delete SERVER_SPEC CATALOG_NAME*

Deleting a dynamic catalog also deletes all the repos it contains.

Finally, all catalogs (dynamic and static) are listed by 

agtool catalogs list [SERVER_SPEC]

The SERVER_SPEC can only be left out if the server is http://localhost:10035.

The repos tool

The repos tool can be used to:

We describe each action in turn.

Using 'agtool repos create' to create new or supersede existing repos

agtool repos create will create new ordinary repos, new multi-master replica sets, each of which has a controlling instance and one or more replicas, or new distributed repositories.

The older tool agtool create-db is still supported but simply as an alias for the agtool repos create command.

Here we create a new, ordinary repository: 

agtool repos create http://user:password@host:port/repositories/newrepo

We have provided a complete REPO_SPEC. Bits can be left out, as described in the REPO_SPEC section above.

If newrepo already exists, the command will fail unless the --supersede option is specified, in which case the existing repo will be deleted (along with all its data) and the new empty repo will be created.

Here we create a new multi-master replica set with a controlling instance and a single additional instance: 

% agtool repos create '<http://USERNAME:PASSWORD@HOST:POST/repositories/mmrep1>|<http://USERNAME:PASSWORD@HOST:POST/repositories/mmrep2>'

Some notes:

  1. Neither repo (mmrep1 and mmrep2) should already exist.
  2. At least two repos must be specified, with specs enclosed in angle brackets and separated by a |.
  3. The repo specs must be enclosed in quotes (as shown) so the | is not interpreted as a shell character.
  4. The --supersede option is not supported.

Here we create a new FedShard distributed repository cluster. 

% agtool repos create http://USERNAME:PASSWORD@HOST:POST/repositories/ClusterRepoName

The --supersede option is not supported.

Using 'agtool repos create' to create vector stores

You can use agtool repos create to create vector stores. 

agtool repos create --vector-store --embedder EMBEDDER [--model MODEL] [--api-key KEY] [--supersede] REPO_SPEC ...

where EMBEDDER is currently either openai or demo. The KEY need be specified if openai is the EMBEDDER. The MODEL need only be specified if the EMBEDDER supports multiple models and you wish to specify the model.

A vector store can also be created with agtool repos ensure-vector-store.

See LLM Embed Specification for information on how vector databases can be populated.

Using 'agtool repos ensure-vector-store' to create or supersede vector stores in regular triple stores.

You can use agtool repos ensure-vector-store to make sure that a given repo has a vector store in it. This command is usually used when you have an existing normal repo without a vector store associated with it and you want to add an associated vector store. You can also use this command, specifying --supersede if you want to replace an associated vector store and attach a new one. 

agtool repos ensure-vector-store --embedder EMBEDDER [--model MODEL] [--api-key KEY] [--supersede] REPO_SPEC ...

The behavior of this operation depends on the current state of the REPO_SPEC repo.

If the REPO_SPEC doesn't exist then this will create a new vector store and it will only contain triples that describe the parameters of the vector store.

If the REPO_SPEC exists and is not a vector store then a vector store will be added to the repo and existing triples in the store will not be deleted. Adding a vector store just means adding triples to the given repo that describe the parameters of the vector store..

If the REPO_SPEC exists and already has a vector store in it then the behavior depends on whether the --supersede option is given.

If --supersede is not given then the vector store will not be modified and the other arguments such as --embedder will be ignored.

If --supersede is given the the existing vector store will be removed and a fresh new vector store added to the repo. However none of the triples in the store that aren't used for the vector store will be deleted.

Using 'agtool repos delete' to delete repos

agtool repos delete REPO_SPEC

Using 'agtool repos list' to list repos

agtool repos list [SERVER_SPEC]

lists all existing repos.

The SERVER_SPEC can only be left out if the server is http://localhost:10035.

The materializer tool

Command calling template: 

% agtool materialize [options] command [command-args]

See the Command Line Interface section in the Materializer document for details on the options.

The fedshard tool

The fedshard tool is used to define and monitor the dynamic sharded database system introduced in version 7.4.0. The use of agtool is described in the Dynamic Cluster Setup document and the Fedshard Definition document.

The gruff tool

Gruff is a graph visualization and graphical query builder designed to work with AllegroGraph. It runs in a browser initiated by links from either New WebView or Traditional WebView. The gruff tools can be used to manage Gruff in AllegroGraph. See Gruff in AllegroGraph.

The llm tool

The llm tool is used to prepare Large Language Model vector databases for querying. See the Large Language Models document and the LLM Embed Specification for details.

The namespaces tool

With agtool namespaces you can list, add and remove namespaces that will be used in queries. See the Namespaces and query options document. In brief, this command 

% agtool namespaces add user:pass@aghost:10035/repo \  
         ex 'http://example.org#' \  
         franz 'http://franz.com#'

This adds to the namespace prefixes ex and franz to the user namespace list for the repo.

The optimize tool

See also agtool auto optimize below. The optimize command calling template is 

% agtool optimize [ OPTIONS ] REPO-SPEC

The options are:

--indices INDICES
A comma separated list of indices to optimize (such as spogi,psogi). If unspecified or all, all indices are optimized.
--level 2-or-3
Optimization effort level: 2 (the default) writes out in-memory triple data and then performs an index optimization of all overlapping chunks; 3 - same as 2, but will process all chunks regardless of overlap (useful for ensuring that all purgeable deleted triples are purged).
--wait BOOLEAN
If true, do not return until optimization completes. Default is false.

The query tool

Command calling template: 

% agtool query [options] REPO-SPEC [QUERY-FILE]*

See the Querying using agtool document for details on the arguments and options.

The query-options tool

With agtool query-options you can list, add and remove query options that will be used in queries. See the Namespaces and query options document. In brief, this command 

% agtool query-options add user:pass@aghost:10035/repo \  
         engine mjqe \  
         memoryLimit 2G

This sets the query engine to mjqe and the memoryLimit to 2G. agtool query-options list REPO prints the options in effect. agtool query-options remove REPO QUERY-OPTIONS removes the listed option.

RDF-star mode

AllegroGraph supports RDF-star (see the RDF-star document), but it must be enabled in a repo for that repo to accept and use quoted triples, as supported by RDF-star. The command 

% agtool rdf-star-mode show REPO-SPEC

reports the surrent setting. enable and disable in place of show enables or disables the mode in the repo.

read-only mode

Command calling template 

% agtool read-only-mode COMMAND [ OPTIONS ] REPO-SPEC

COMMAND can be:

When a repository is in read-only mode, no commits may be made. Although triples can be added and deleted in a session as usual, those changes will not be seen by other users and cannot be committed. Only a superuser can enable or disable read-only-mode. (See the Managing Users document for information on superusers.)

The export tool

Command calling template: 

% agtool export [options] REPO-SPEC FILE

This will export the data in the repository identified by REPO-SPEC to FILE. For example, 

% agtool export --output-format rdfxml user1:my-pw@agmachine/lesmis lesmis.rdf

exports the data in the lesmis repo (in the root catalog, since no catalog is specified) to the file lesmis.rdf. The format is rdfxml. user1 must have read permission in lesmis.

See the Repository Export document for details on the options.

The load tool

The agtool load tool can be used to load data into a store from a file or a collection of files. See the Data Import document for details.

The tool calling template is: 

% agtool load [options] REPO-SPEC SOURCE*

where REPO-SPEC identifies the repository into which the data should be loaded (the catalog can be specified as part of the REPO-SPEC, as can user and connection information). The SOURCEs are the file(s) to be loaded. Names of files on Amazon S3 must be preceded by s3://. The SOURCE can be standard input. Numerous options control the loading process. All are described in the Data Import document for details on the options.

agtool load can be run on the same machine as the AllegroGraph server by the same user who started the server, in which case no username or password is required, or on a different machine and/or by a different user, in which case the username and password of a user with AllegroGraph superuser privileges must be specified as part of the REPO-SPEC.

Here are a couple of examples of typical uses. In this first example, the user executing the command must be the user who started the AllegroGraph server and the command must be run on the same machine as the server: 

% agtool load my-repository mydata.nt

Load the Ntriples file mydata.nt into the store my-repository in the root catalog. Since no port is specified, the port is the default, 10035.

In this example, the command is again run on the same machine as the Allegrograph server and by the user who started AllegroGraph. A non-default port is used. 

% agtool load 10077/mycatalog:my-repo2 mydata-2.ttl mydata-3.ttl

Load the turtle format files mydata-2.ttl and mydata-3.ttl (the format determined by the file extensions) into the store my-repo2 in the catalog mycatalog. The server listening on port 10077.

Finally, a call from a different machine. Here the host running the server must be specified by name and the user and password must be provided:

% agtool load test:xyzzy@agmachine:10077/repo-7 mydata.ttl mydata.nq

Load the turtle format file mydata.ttl and an NQUAD format file mydata.nq (the format determined by the file extensions) into the store repo7 in the (default) root catalog. The server is running on the host agmachine and is listening on port 10077.

The recover tool

Command calling template: 

% agtool recover [options] archive database

See the Point-in-Time Recovery document for details on the options and general usage.

The repl tool

The repl tool allows you to manage a multi-master replication cluster. See Using the agtool repl command in the Multi-master Replication document for more information.

A multi-master cluster should not be confused with single-master replication described below. Single-master replication allows for one master repository, where modifications (adding and deleting triples) can be made and replicas which cannot make modifications. Multi-master replication allows each replica to make modifications.

The replicate tool

The replication tool allows you to set up single-master replication where you have several identical instances of AllegroGraph running simultaneously, so if one fails one of the others can take over. The replicate command calling template is: 

% agtool replicate [options]

See Replication Details document and also the Replication document for more information on agtool replicate and its options.

Single-master replication should not be confused with multi-master replication, described above.

The storage-report tool

This tool (command line agtool storage-report repo-spec) shows storage used by the specified repository. Each index is listed with its storage use. The percentage of space taken by deleted triples is also displayed. See the Purging Deleted Triples document for more information in deleted triples.

Here is a portion of a storage report for a repo (information on some indices is deleted, as shown by the syspension points ([...]): 

% agtool storage-report http://USERNAME:PASSWORD@HOST:POST/repositories/sapnew  
## DB storage report: sapnew  
-- Mem chunks --  
Chunk #xffffd44df8c (offset #x1e7698c): Type: mem, Refcount: 7, Used:   68896  
-- Flavors --  
[ 0] spogi: style: 1 last-id: 0      #chunks: 1     Oscore:     1.000, size: 1.5 MiB (1,560,576), deleted 27%, 15.61 bytes/triple, idle  
[ 8] posgi: style: 1 last-id: 0      #chunks: 1     Oscore:     1.000, size: 2.0 MiB (2,146,304), deleted 27%, 21.46 bytes/triple, idle  
[... lines deleted for space]  
[22] gospi: style: 1 last-id: 0      #chunks: 1     Oscore:     1.000, size: 1.9 MiB (2,035,712), deleted 27%, 20.36 bytes/triple, idle  
[24] i    : style: 1 last-id: 0      #chunks: 1     Oscore:     1.000, size: 1.3 MiB (1,388,544), deleted 27%, 13.89 bytes/triple, idle  
Total live disk chunks size          : 12.2 MiB (12,836,864)  
Triple count                         :  142,009  
Total live disk chunk triples        :  700,000  
Index triples deleted                :  16%  
 Bytes/triple/index (disk chunks only):       18.338  
Active index count                   :        7  
Bytes/triple (disk chunks only)      :      128.369 (assumes each triple exists in all active indices)  
Obsolete chunks                      :        0 (0.0% of all available)  
Total obsolete disk chunks size      :  0 bytes (0)  
nil  
%

The Oscore value of each index shows the number of index chunks which must be examined to find triples matching a pattern. The optimal Oscore value is 1.0. When you see values higher than 1.0, consider optimizing an index, as queries will run faster when Oscores are lower (sometimes very significantly faster). See agtool optimize.

The information on each index includes the approximate percentage of deleted triples and the summary at the end also shows the approximate percentage of deleted triples. A graphical display of much of this information is found in the Triple indices graph, in New WebView storage overview (on the Repository menu) or in traditional AGWebView storage reports.

The tokens tool

Tokens are long character strings used as password equivalents for logging in and managing sessions. They can be be time limited and can expire and can be deleted at any time. They are useful, for example, in giving other users temporary access to an account without revealing the actual password.

Tokens are created with the following agtool command (the last, server, argument can be expressed in various ways, see % agtool tokens create --help for details). 

% agtool tokens [command] user:password@server:port

where [command] is create, delete, or list (which lists all tokens for a user.

Thus 

% agtool tokens create user1:foo@:10035  
VfQ-4Ge28eARREqAizAWi-tBveC9njdFjGNRF6DQPpU=  
%

But note: 

% agtool tokens list user1:foo@:10035  
|          TOKEN | DESC  | EXPIRES |  TOUCHED AT         | IDLE TIMEOUT |  
|----------------|-------|---------|---------------------|--------|  
| VfQ-4Ge28eARRE | login |  null   | 2023-09-19T13:18:37 |   null |  
%

Only the beginning of the token is shown. There is no way to obtain the full token from AllegroGraph or any of its associated tools after the reveal at creation time.

The upgrade tool

agtool ugrade will do an in-place upgrade of a single database from an earlier release to the current release (the release containing the agtool being run). The upgrade is done without backing the database up. Backing up databases prior to upgrading is strongly recommended. See Repository Backup and Restore.

Command calling template: 

% agtool upgrade GROUND-REPO-SPEC

A GROUND-REPO-SPEC specifies a single repository (not one that is federated or includes reasoning). See the agtool upgrade program in the Repository Upgrading document for further details. Be sure to backup any database before upgrading.

Managing users and roles

agtool users can be used to add and remove users and to set user permissions and access rights. agtool roles can be used to create and manage roles. See Managing Users for information on managing users with agtool and with New WebView or traditional AGWebView.

The agtool users/roles subcommands set-attributes, delete-attributes and list-attributes are complex and the following is a detailed description of them.

Filtering triples using attributes uses three pieces of data:

  1. The attributes given to a triple when it's added to the repository
  2. The attributes assigned to a user when retrieving triples from the repository
  3. The static attribute filter function which, for each triple, is a function of (1) and (2) and returns true if this triple is visible to this user.

The agtool users set-attributes and agtool roles set-attributes commands are used to specify item 2 above. Only an account with the super capability can use these commands (as there would be no security if each user could specify their own attributes).

For the commands agtool users and agtool roles the subcommand set-attributes specify the attributes for a given user or role and for one or more repositories. The subcommand delete-attributes removes that specification. The subcommand list-attributes show the attributes currently set for a user or role.

The forms of the set-attributes subcommand is: 

agtool users set-attributes --server user:pwd@host:port user-name cat-repo attributes  
agtool roles set-attributes --server user:pwd@host:port role-name cat-repo attributes

The argument to --server must specify an account with super capability.

cat-repo is either a the name of a respository in the root catalog, or catname:reponame for the repo named reponame in the catname catalog. Either or both of the catalog and repo names can be "*" meaning all catalogs and/or repos. So "*:*" would match all repos in all catalogs.

The attributes argument is a json object like 

{"security":"low","dept":["hr","intern"]}

As attribute specifiers, the value of each key/value pair is either a string or a comma separated list of strings in square brackets.

In general a json object is a comma separated set of key/value pairs surrounded by braces. There is a colon between the key and the value. Whitespace is ignored outside of strings.

Here we specify the attributes for user joe accessing the hr:people repo 

agtool users set-attributes --server super:secret@machine:20035 joe hr:people '{"security": "low"}'

Here we specify the attributes for user jane accessing any repo in any catalog 

agtool users set-attributes --server super:secret@machine:20035 jane "*:*" '{"security": "high"}'

Here we set the attributes for the role exec on any repo in the hr catalog: 

agtool roles set-attributes --server super:secret@machine:20035 exec "hr:*" '{"security": "high"}'

Each user and role can have multiple attribute specifications for different catalog and repo values. For a user who is a member of any number of roles the specifications from the user and all his roles are collected and sorted from most to least specific. The most specific matching specification is used by AllegroGraph when retrieving data from the repository.

If there are no specifications matching the repo being queried then no user attributes are given for the query.

You can see all the specifications applicable for a user in sorted order with 

agtool users list-attributse --server user:pwd@host:port username

Each specification in the list will show where it came from (either the user or one of his roles).

As noted a user's attribute specifications are the merger of the specifications for the user and all of specifications for all of the roles to which that user belongs.
If a user has multiple roles and there is no single best matching specification for a given catalog and repo, then the specification chosen can't be predicated so such a situation should be avoided.

For example: 

agtool roles set-attributes --server super:secret@machine:20035  exec "hr:*" '{"security": "high"}'  
 
agtool roles set-attributes --server super:secret@machine:20035  admin "hr:*" '{"security": "low"}'

Suppose user fred has both roles exec and admin. What is the value of the "security" attribute for fred when doing a SPARQL query in hr catalog? The answer is that it can't be predicated since AllegroGraph makes no guarantees about how the role data is merged.

There are two other ways of specifying the user attributes for a query

  1. SPARQL PREFIX franzOption_userAttributes (only applies to SPARQL queries)
  2. HTTP header x-user-attributes

These will take precedence over anything set by agtool attributes

In order for an account to use one of the two above mentioned methods a super user must authorize the account to specify attributes that way. However once an account can specify its own attributes you've lost the ability to control what that user can see using attributes. The advantage of agtool attributes is that the super user and only the super user has control of what attributes are given for a user account.

Exporting user and role information to a file: agtool users and agtool roles both support export and import commands to write user and role information to a file and to read it back. The files are written in Lineparse Format.

Note that exporting user information will display the password in an encoded form. While you can't easily guess the password from the encoded form one should be careful to protect a file containing the encoded passwords just to be safe.

SHACL

SHACL, the SHApe Constraint Language supported by AllegroGraph, can be used to verify that a set of triples meets various constraints. The validation is done by a call to agtool shacl-validate. See the SHACL document for more information.

The event scheduler tool

Users can schedule events to run scripts at specified times in the future. The scripts can be run once or repeated on a regular schedule. See the Event Scheduler document for more information. The utility for using agtool to schedule events is scheduler. The various commands are: 

bin/agtool scheduler --help  
Usage: agtool scheduler COMMAND [ OPTIONS ] ...  
 
where COMMAND is one of:  
* help - Display scheduler usage.  
* delete-events - Delete events.  
* log - Display the log of events  
* events - List pending events.  
* create-event - Create a new event.  
 
To describe a particular command run:  
agtool scheduler COMMAND --help

The triple-count tool

The triple-count tool returns the number of triples in a repository. The command format is 

agtool triple-count REPO-SPEC

For example, running on the same computer as the AllegroGraph server and as the same user who started the server (so username:password is not needed) we find the kennedy database example has 1214 triples: 

% agtool triple-count localhost:10035/kennedy  
1214

The vload tool and virtualized graphs

The vload tool communicates with an instance of the Ontop program which materializes a relational database as RDF triples. agtool vload then loads the materialized triples. See the Creating Virtualized Graphs document for more information.