Introduction

The AllegroGraph Loader is one of the agtool utilities. It imports data into a repository quickly and easily. When possible, it makes use of multiple CPU cores to load the data in parallel.

The agtool program is the general program for AllegroGraph command-line operations. (In earlier releases, there was a separate agload program. A change from earlier releases is the -u option, previously a short version of the --base-uri option, is no longer supported.)

Bulk data can be loaded with tools besides agtool load. AGWebView has a loading tool. See also Adding Triples in the Lisp Reference.

Usage

Note that agtool load must be run on the same machine as the AllegroGraph server, and as the same user as the user who started the AllegroGraph server. Further, the version of agtool must be the same as the AllegroGraph server (that is, agtool must be the one distributed with the server, not from an earlier or later version of AllegroGraph).

The agtool load command line is: 

agtool load [OPTIONS] DBNAME SOURCE*

where

Here is an agtool load command line with the port and input format specified: 

agtool load --port 10035 --input ntriples my-database mydata-1

Here we also specify the catalog (which otherwise defaults to root) and multiple input files. The ttl file extensions indicate the file data formats so --input need not be specified: 

agtool load --port 10035 --catalog mycatalog  my-database mydata-2.ttl mydata-3.ttl

The specified catalog must already exist on the server.

If a database named by the DBNAME argument does not exist, it will be created. If the database exists, it will be opened and data from the input files will be added to it. If the --supersede option is suppled and the database exists, it will be deleted and a new one created. Use that option with caution as superseded data is gone and cannot be recovered except from backups. Also note that DBNAME is affected by the presence of the --catalog option: different catalogs can contain databases with the same names so if you specify a catalog, you are calling for the use or creation of the database DBNAME in that catalog.

More on the FILE argument

If the character - is given as a SOURCE, it will be interpreted as standard input. This allows agtool load to accept data directly from other programs. For example: 

cat foo.nt bar.nt baz.nt | agtool load --input ntriples test_db -

Loading files from HDFS filesystems

agtool load supports loading files from HDFS (Hadoop Distributed File System). In order for the load to succeed, the following conditions must be met:

Here is a sample command line for loading the file lubm-50.nt.gz: 

agtool load repo hdfs:///user/bruce/lubm-50.nt.gz

When those conditions are met, agtool load works with HDFS files just as with other files. The command line can mix HDFS files with other files.

Loading from HDFS file systems has been tested with the Cloudera Hadoop distribution.

File encoding

N-Triples, N-Quad, and NQX files must have character encoding UTF-8, as must Turtle files. RDF/XML, TriG, and TriX files use character encodings as defined by standard XML parsing rules. Conversion programs outside of AllegroGraph (such as iconv) can be used to convert files to UTF-8 format if necessary. All such conversions must be done prior to processing by agtool load.

Loading triple attributes

AllegroGraph supports attributes for triples, as described in the Triple Attributes document. Attributes are name/value pairs associated with triples. Attributes can only be associated with triples when the triple is added to the repository. Using agtool load to load an NQX file is one way to associate attributes with triples. Only Extended N-Quad files (NQX), with type .nqx, can specify different attributes for each triple. Here are two lines from an NQX file specifying attributes: 

<http://dbpedia.org/resource/Arif_Babayev> <http://dbpedia.org/property/placeOfDeath> <http://dbpedia.org/resource/Baku> <http://ex#trans@@1142684573200001> {"color": "red"} .  
<http://dbpedia.org/resource/Arif_Naqvi> <http://purl.org/dc/terms/subject> <http://dbpedia.org/resource/Category:Businesspeople_from_Karachi> {"color": "blue"} .

The first includes a graph, the second does not. The loader distinguishes them because it can recognize the JSON input {"color": "red"} and {"color": "blue"}. The "color" attribute name must be defined in the repository prior to loading an NQX file containing these lines.

The --attributes option (see below) allows specification of default attributes that will be applied to every triple that does not have attributes specified. Note that the default attributes will be applied to every triple loaded from any type of file other than an NQX file, since attributes cannot be specified in any such file.

Attribute names must be defined before use, so all attribute names appearing in an NQX file or as the value of the --attributes option must be already defined. See Defining attributes for information on defining attributes.

Optimal file types

agtool load is optimized for loading N-Triple and N-Quad files. Loading of files of those types will be spread over multiple cores even if only a single file is specified. agtool load will spread loading of multiple files of other types (Turtle, TriX, TriG, RDF/XML) over multiple cores but each single file will be loaded by a single core. Therefore loading a single, large turtle file takes much longer that loading a single, equivalently large N-Triple or N-Quad file on machines with multiple cores.

The Rapper program, from http://librdf.org/raptor/rapper.html, will convert files from one format to another.

Loading lists of files

Loading entire directories (by specifying the directory as the FILE argument) is not supported, but wildcards may be used. Wildcards will be expanded by the shell, as expected. Also, you can specify a single file which contains a list of files to load, as we describe next.

If FILE begins with the character @, agtool load interprets everything after the @ as a file name which contains a list of actual source files. By default the file name items are separated by a CR, LF or CRLF sequence. That is the usual case and covers most actual situations.

Because UNIX allows newlines to appear in any file name, agtool load will permit the files to be separated by NULL characters. If you want this, you must specify the -0 option (the numeral zero, not an uppercase letter o). Here is an example: 

agtool load -0 test_db @null-separated-list-of-sources.txt

Use the @file-name syntax when there are more source files than the shell will pass to agtool load as arguments.

agtool load failure

agtool load might fail (meaning fail to successfully load all data from all the specified files) for various reasons, including hardware failure and network problems. The most common cause of failure is corrupted or invalid input files.

If agtool load encounters invalid data, its behavior depends on the --error-strategy option desribed below, but by default it will stop loading data.

Note that agtool load adds triples to the database as it works. Therefore, if agtool load fails (for whatever reason) after it has begun, you may find some but not all triples added, and you should be prepared for this possibility. Here are some ways to be prepared:

But sometimes you have to load with agtool load while triples are being added from other sources, and you must be prepared in that case to deal with an agtool load failure. (We do not have specific recommendations as there are many possible cases and equally many ways to deal with them but note that if the graph field is not otherwise used, you can often arrange it so triples loaded with agtool load have a different graph than triples from other sources and that makes it easy to tell which were loaded by agtool load and which were added by other means.)

See the --error option below for more information on handling errors.

A more complex example

The following example loads a single source file into AllegroGraph. 

agtool load --supersede -v --with-indices "ospgi,posgi,spogi" -e ignore -p 10035 lesmisload lesmis.rdf

In this example, there are several OPTIONS, the DBNAME is lemisload and the SOURCE is lemis.rdf. agtool load will load lesmis.rdf into the lesmisload repository on a running AllegroGraph server that is using port 10035. If there is an existing repository named lesmisload, then it will be deleted (as specified by --supersede). The program will generate verbose messages (-v) and will ignore any errors (-e ignore). The repository will generate three triple indices: ospgi, posgi and spogi.

The complete list of OPTIONS follows.

Options

The following options may be used on the agtool load command line.

Repository options

These options control the creation and settings for the repository.

--port PORT, -p PORT
Set this to the port number of the server you would like to use. As said above, agtool load must be run on the same machine as the server and by the same user, and the agtool load version must be the same version as the server (that is, distributed with server). The default port value is 10035.
--catalog CATALOG,-c CATALOG
Use this option to assign the database to a catalog. If left off, then the root catalog will be used 1 .

--supersede, -s
If the database DBNAME exists, it will be deleted before loading data. Supersede should be used with care as there is no way to recover the deleted repository other than restoring it from backup. If the database DBNAME exists and is open, the agtool load command will fail.
--fti NAME, -f NAME

Create a free-text index named name.

This index will include both any newly added triples and all existing triples. Managing the free-text index will slow down loading speed somewhat.

--with-indices INDICES
Specify the repsoitory's indices. When supplied, the parameter should be a list of index names separated by commas (for example: spogi,posgi,ospgi,i). If not specified, newly created repositories will use the standard set of indices and the indices of existing repositories will remain unchanged. 2

--bulk

Enable bulk mode while processing the loading job. Bulk mode turns off transaction log processing and can provide considerable performance gain for large jobs. Because an unexpected failure during a bulk load can result in an unrecoverably corrupted database, we recommend you make a backup before using this option.

Specifying this option will cause the load to fail when the repository DBNAME is a replication instance (see the Multi-master Replication document) as replication depends on complete transaction logs.

--attributes

Specifies attributes for any triples which do not have attributes specified in the file being loaded. See the Loading triple attributes section above for more information. Since attributes can only be specified in NQX (extended N-Quad) files, attributes specified by this argument will be applied to all triples loaded from files of all other types. Attributes must be already defined within the repository in order to be values of this argument.

Attributes are specified as a JSON object. Here is an example:

--attributes '{ "pet": "dog", "key2": [ "red", "blue" ] }'
We have surrounded the value with single quotes to avoid interpretation of the braces ({ and }) or the double quotes as shell metacharacters. Three attributes are added, one with name "pet" and and two with name "color". The "pet" attribute has the value "dog", one "color" attribute has the value "red", and the other has the value "blue".

--parameter NAME=VALUE

When the action of agtool load will create a new repository, either because the DBNAME argument names a repository which does not already exist in the specified catalog (the root catalog if no catalog is specified), or because the --supersede argument is specified, then the repository will be created using the various catalog directives listed in the Catalog directives section of the Server Configuration and Control document. Such directives include StringTableSize, StringTableCompression, ExpectedStoreSize, and so on. Many of these directives are inheritable, meaning they can be specified at the top-level and apply to any catalog that does not also specify a value in its definition, so if you are looking in a configuration file for the value which will be used, look at the top-level directives as well as the specific catalog definition.

Only one directive (NAME=VALUE) can be specified for each occurrence of --parameter but the option can be specified as often as desired.

Loading options

These options control how agtool load processes data sources.

--input FORMAT, -i FORMAT

Specify the input format to use. The recognized values are: rdfxml, ntriples, nquads, nqx, trix, trig, turtle and guess.

The default is guess. Use guess if (1) All sources have a recognizable extension, and (2) Every file is actually of the format indicated by its extension.

  • Recognized N-Triple extensions: 
    .nt, .ntriple, .ntriples, .nt.gz, .ntriple.gz, .ntriples.gz,  
.nt.bz2, .ntriple.bz2, and .ntriples.bz2.
  • Recognized N-Quads extensions: 
    .nq, .nquad, .nquads, .nq.gz, .nquad.gz, .nquads.gz, .nq.bz2,  
.nquad.bz2, and .nquads.bz2.
  • Recognized Extended N-Quad extension (see the Loading triple attributes section above for information on NQX files): 
    .nqx
  • Recognized RDF/XML extensions: 
    .rdf, .rdfs, .owl, .rdf.gz, .rdfs.gz, .owl.gz, .rdf.bz2,  
.rdfs.bz2, and .owl.bz2.
  • Recognized Turtle extensions: 
    .ttl, .turtle, .ttl.gz, .turtle.gz, .ttl.bz2, and .turtle.bz2.
  • Recognized TriX extensions: 
    .trix, .trix.gz, and .trix.bz2.
  • Recognized TriG extensions: 
    .trig, .trig.gz, and .trig.bz2.

If a format other than guess is specified, it will take precedence over a file's extension. For example, if you have an N-Triples file named triples.rdf and specify --input ntriples then triples.rdf will be parsed as an N-Triples file.

If you use multiple source formats in one agtool load command, then you need to ensure that the source file's extensions match their contents. Otherwise, you will need to use multiple command invocations.

The only two compression formats handled by agtool load are gzip and bzip2. Any files which are compressed must be named with .gz or .bz2 extensions in order to be decompressed. All supported formats permit .[format].gz and .[format].bz2 extensions, allowing agtool load to determine the data format from the [format] portion. If a file has extension .gz or .bz2 without also specifying the format, you must use the --input option. For example, if you are loading a gzip'ed N-Quads file and the file is named btc-2011-chunk-045.gz, then you must specify --input nquads.

The use of standard input with agtool load (by specifying FILE to be -) always requires a non-default value for the input format, since standard input has no file type.

--error-strategy STRATEGY, -e STRATEGY

The available options for error strategy are cancel, ignore, print and save. Cancel is the default, and will stop the loading process as soon as an error is detected.

Error strategy ignore will attempt to silently continue upon an error, although it will print a warning (and then continue) when an entire file is skipped because its input format could not be determined (usually because the file type is not recognized or is missing).

Error strategy print is like ignore but will print the error to standard output. Error strategy save is like print but will also log the error to agload.log in the current working directory.

Error strategy applies to all recoverable errors, not just parsing errors.

Remember that if agtool load fails for whatever reason, some triples will have been added to the database and some will not have been added (except for very unusual edge cases).

--error-strategy ignore will in some cases print warnings even though it will not stop processing files or loading triples. For example, if a file does not have a recognizable type and its format has not been specified with --input, the file will be skipped when the --error-strategy is ignore but a warning will be printed: 

agtool load -e ignore repo wrong-file  
Cannot guess the format of wrong-file. Use --input to specify it.  
Load finished 0 sources in 00:00:01 (1.00 seconds).  No triples added.  
Terminating agtool load processes, please wait...

--loaders COUNT, -l COUNT

The loaders option specifies the number of processes which will be connecting to AllegroGraph and committing triples. It is for optimization of agtool load performance. The system will try to use as many cores as you specify.

The default depends on the number of cores on your server. If you have 1 or 2 cores, loaders will default to 1. If you have 4 cores, loaders will default to 3. For more cores the default is the number of cores minus one, up to a maximum of 32. agtool load also has a task dispatcher process and AllegroGraph has its own processes.

If you are not getting satisfactory performance for your load, try increasing or decreasing the number of loaders. If your data has no blank nodes, you may want to set the number of loaders to the number of logical cores on the machine and use --blank none. If you have files dense with blank nodes try decreasing the number of loaders to free up machine resources. For example on an 8 core, 48GB hyperthreaded server, we use --loaders 5 for good performance while loading Billion Triple Challenge. For Lubm-8000 we use --loaders 16.

--base-uri URI
Specify a base URI for formats that support it, such as RDF/XML or Turtle. Note that if standard input is a source and rdfxml is the input format, --base-uri must be specified. (In earlier releases, -u was accepted as a short version of this option. -u is no longer accepted. You must use --base-uri.)
--graph GRAPH, -g GRAPH
Specify a default graph for formats that support it, such as N-Triples. Special values are:
  • :default, use the default graph node for the repository (this is the default value for graph).
  • :source, use the source of the triple (i.e. the filename) as the default graph. This cannot be used if standard-input is used a source.
  • :blank, generate a blank node before loading triples and use it as the default graph.

Any other value is interpreted as a resource (URI) or literal and used as the default graph. Note that strict RDF does not allow literals to be used in the graph slot.

The three special values start with a colon (:) to allow for the usage of default, source, or blank as graph names. See the Examples using the graph option section for more information on the use of this option.

Formats that include the fourth element (like N-Quads) will use the default-graph only for the data that does not explicitly specify it.

--external-references, -x
If specified, then external references in RDF/XML source files will be followed during load.
--metadata FILENAME
Load attribute definitions and the static filter definition from FILENAME. See Triple Attributes for information on attributes.

Less common options

These options are useful in specific circumstances but do not generally need to be used.

--host
agtool load must be run on the same machine as the AllegroGraph server but it still makes an initial HTTP request to the server to get the information on how to make a direct (non-HTTP) connection to the server. In the default, the AllegroGraph server listens on all IP interfaces, but it can be configured to only listen at a specific IP address by specifying a value for the HostName configuration directive described with other top-level directives in the Server Configuration and Control document. When this directive is used (and its value is not something that resolves to 127.0.0.1), the --host option to agtool load must be suppplied with a matching value. Again, the value must specify the machine running both the AllegroGraph server and agtool load. This argument cannot be used to specify a server host which is different from the machine on which agtool load is being run.
--help, -h
Print the command line summary.
--verbose, -v

The presence of the verbose option will cause additional information to be printed to standard output. This argument can be specified multiple times to increase the verbosity. We recommend using --verbose --verbose if you encounter a problem with loading, but note --verbose --verbose generates a lot of output when loading many files, so it may fill a terminal's scrollback. See also the --debug option which also may be useful when an error occurs during loading. --debug -v -v provides maximum information about loading.

Specifically, the verbosity levels are:

  • 0 (-v not supplied): Only report periodic load rate information.

  • 1 (-v): As above, plus print the job option summary before starting the operation.

  • 2 (-v -v): As above, plus print the name of every file that has been processed.

Specifying more than two -vs is equivalent to specifying two.
--blank STRATEGY, -b STRATEGY

Determine how to handle blank node identifiers in N-Triple and N-Quad files. STRATEGY must be one of file, job or none.

By default, blank node identifiers are scoped to the source in which they appear. I.e., the blank node _:b1 in file file1.nt is considered to be different than the blank node _:b1 in file file2.nt. AllegroGraph calls this the file strategy and uses it as the default.

Blank node strategy job will consider all blank nodes found in N-Triple and N-Quad files to be in the same "scope". This means that the _:b1 in file1.nt will be considered to be the same as the one found in file2.nt.

Blank node strategy none will cause agtool load to error if any sources contains blank nodes. Loading is faster when the blank node strategy is none.

Note that the blank node strategies of job and file only apply to N-Triple and N-Quad sources. Other formats such as RDF/XML or Turtle are defined to have a blank node scope of the file and this option is ignored. Using a blank node strategy of none will, however, still signal an error if any source files contains a blank node.

--debug

If specified, additional information will be printed when an error occurs.

Typically, --debug is useful only when agtool load returns an exit code of 1, indicating that there was an unhandled error. If this occurs, re-run agtool load using the debug option and send the output to AllegroGraph support for more assistance. The option also causes information to be written to agload.log. See also the --verbose option, which causes other information about loading to be output. --debug -v -v provides maximum information about loading.

--relax-syntax
For N-Triples and N-Quad files, this flag tells AllegroGraph to ignore certain syntax errors. In particular:
  • Blank node names may use underscore (_) and dash (-) characters.
  • Literals may be used in the Graph position (for N-Quads).
  • URIs are not required to include a colon (:).

This is a non-standard parser extension and should only be used when necessary.

--duplicates SETTING, -d SETTING
Changes the handling of duplicate triples in the repository. The valid values for SETTING are keep (the default); delete, meaning delete all but one of all groups of triples that match subject, predicate, object, and graph; and delete-spo, meaning delete groups that match subject, predicate, and object, regardless of graph. In AllegroGraph, triples are deleted only by the function delete-duplicate-triples. If duplicate deletion is specified, that function is (in effect) called at the end of data loading with arguments suitable for the specified argument, and the load completes when duplicates are deleted. This argument does not affect things after the load is complete as future duplicate deletions are only done when delete-duplicate-triples is called.
--dry-run

If specified, then print the loading strategy and stop. I.e., no triples will be loaded.

agtool load loads N-Triples and N-Quads files most efficiently so it can be faster to convert source files before loading them. More information on using rapper and AllegroGraph is described in our documentation.

--null, -0
Use to specify that the file specified after an @ sign in the SOURCE inputs is a null separated list rather than a newline separated one. This is useful for loading files with newlines or other strange characters in their names.

Deprecated and Removed Options

The following option is no longer supported:

--encoding ENCODING, -C ENCODING
In earlier releases, this option could be used to specify the character encoding of the file(s) being loaded. This is no longer supported. N-Triple, N-Quad, NQX, and Turtle files must use UTF-8 encoding. RDF/XML, TriG, and TriX files use character encodings as defined by standard XML parsing rules. There are conversion programs, such as iconv, which will convert files to UTF-8 character encoding.

The use of the following option has been deprecated as it is no longer needed:

--dispatch-strategy STRATEGY

STRATEGY must be auto or file. The dispatch strategy tells agtool load how it to parallelize loading.

The default is auto which is combination of dispatch strategies based on file format. N-Triple and N-Quad files will be broken up and loaded in pieces. --dispatch-strategy file means that no files are broken into pieces for loading. There is no reason to specify file (which was useful in much earlier releases and is kept for backward compatibility). Note that RDF/XML, Turtle, TriG, and TriX formats are always dispatched on a file basis regardless of the value of this option.

Examples using the graph option

We have an N-Triples file /tmp/test.nt containing the following single line specifying a subject, predicate, and object but no graph: 

<http://franz.com/Node60469> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://franz.com/OFFICER> .

Here are some examples using various values for the --graph argument. If the triple were displayed in N-Quads format, it would look like: 

<http://franz.com/Node60469> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://franz.com/OFFICER> [graph] .

with various values of [graph]. In all cases except the first, we only show the value of [graph].

RDF compliance: To be RDF-compliant, graphs must be resources. Other values, such as literals, are not permitted. However AllegroGraph will accept and store non-resource values such as literals. This makes AllegroGraph more flexible with regard to what can be stored, but SPARQL queries which involve graphs will not work with non-compliant graph values. Compliant values are recommended unless there is some good reason to use non-compliant values.

First, we specify :default as the value of the --graph argument, or we equivalently (since the default is :default) leave the argument out: 

agtool load --graph :default --verbose --supersede foo /tmp/test.nt

or 

agtool load --verbose --supersede foo /tmp/test.nt

Either call uses the default graph node for the particular AllegroGraph database, which is not displayed in N-Quads format, so the N-Quads output for the triple is: 

<http://franz.com/Node60469> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://franz.com/OFFICER> .

Contrast that with this call: 

agtool load --graph :source --verbose --supersede foo /tmp/test.nt

The graph is the source file and [graph] is then: 

<file:///tmp/test.nt>

In this next call, --graph is :blank: 

agtool load --graph :blank --verbose --supersede foo /tmp/test.nt

agtool load generates a new blank node and uses that as the default graph for the whole job. [graph] is then similar to: 

_:bC87E16D1x1

Here we use the resource <http://foo.com/abc#123> for the default graph: 

agtool load --graph http://foo.com/abc#123 --verbose --supersede foo /tmp/test.nt

[graph] is then: 

<http://foo.com/abc#123>

Non-compliant --graph values such as literals

As said above, AllegroGraph will accept values which are not resources as graph values. Such graph values cannot be used with SPARQL queries (the queries can involve compliant subjects, predicates, and objects even if the graph value is non-compliant). Here are some examples which show how AllegroGraph deals with non-complaint values of --graph:

The value of --graph is a string with escaped quotation marks: 

agtool load --graph \"abc123\" --verbose --supersede foo /tmp/test.nt

[graph] is then (this is not RDF-compliant): 

"abc123"

The value of --graph is \"adios\"@es-mx: 

agtool load --graph \"adios\"@es-mx --verbose --supersede foo /tmp/test.nt

[graph] is then (this is not RDF-compliant): 

"adios"@es-mx

Some --graph examples which likely do not do what is wanted

Here are a couple of examples where the result is likely not what is intended. These values are accepted by AllegroGraph despite not being RDF-compliant but may not be what was intended.

In this example, the value of --graph is a string with unescaped quotation marks. The shell will remove unescaped quotation marks so the resulting value is not a literal: 

agtool load --graph "abc123" --verbose --supersede foo /tmp/test.nt

[graph] is then: 

<abc123>

This is not legal RDF. Just above we show how to specifiy a string value by escaping the quotation marks.

In this example, the graph is the word default with no colon: 

agtool load --graph default --verbose --supersede foo /tmp/test.nt

[graph] is then: 

<default>

This is not legal RDF. The value :default, described above, is likely what was intended.

Notes

Footnotes

  1. Note that the root catalog has no name (and so can be specified, if necessary, with the empty string ""). It is not named "root". ↩
  2. Spaces can also be used as separators but this is deprecated. If spaces are used they must be escaped from the shell in some fashion, such as wrapping the index names in quotation marks (for example: "spogi posgi ospgi i"). ↩