Change History | AllegroGraph 8.1.1

Version 8.1.1 user visible changes

AllegroGraph Server: General Changes

No significant changes.

HTTP Client

No significant changes.

SPARQL

AG-643 - incorrect results when using complex `FILTER [NOT] EXIST` clauses

Due to a regression introduced in AllegroGraph v8.0.0, queries with complex FILTER [NOT] EXISTS clauses (e.g. with inlined VALUES and inner FILTER) could potentially produce fewer results because of a join operation misconfiguration. This has been fixed.

AG-737 - SPARQL parser fails on space-separated `+` property path modifiers

AllegroGraph's SPARQL parser used to try to interpret + character after the predicate part of a triple pattern as an explicit numeric sign of the object part if it was separated from the predicate by a space:

?s :p + ?o  ->  ?s :p (+ ?o)  
?s :p +?o   ->  ?s :p (+ ?o)  
?s :p+ ?o   ->  ?s (:p +) ?o

This has been fixed and now the correct grouping is used:

?s :p + ?o  ->  ?s (:p +) ?o  
?s :p +?o   ->  ?s :p (+ ?o)  
?s :p+ ?o   ->  ?s (:p +) ?o

The + character is grouped with the object part of the pattern as its explicit numeric sign only if there is no space between it and the object, and as a path modifier in all other cases.

AG-770 - inconsistent results of joins with `UNION` on SBQE

SPARQL queries with UNION operations could lose some of the results on SBQE if the operands had differing sets of bound variables, because the result of the UNION would incorrectly inherit bound variables of its left-hand side operand. For example the result of the following operation

{ VALUES ?s { :person1 :person2 } }  
UNION  
{ VALUES ?o { :person1 :person2 :person3 } }

would have ?s marked as bound, while it is only bound in 2 out of 5 results, causing a subsequent join operation to lose results. This has been fixed by correctly computing the set of bound variables for the result of the UNION.

AGWebView

AG-354 - Fix losing content of query tabs in New WebView after reload

Before 8.1.1, in New WebView unsaved changes in queries were lost after page reload, page closing, and user logout (explicit or implicit due to inactivity). Now, in New WebView all query changes persist on the server side per user for a repository. In other words, even unsaved queries are saved automatically. The difference between saved and unsaved queries is the following:

Unsaved queries don't have persistent names.
Unsaved queries aren't listed in the right sidebar in the "Saved queries" group, only in the query tab.
Unsaved queries have user scope. Saved queries can be saved per repo, e.g. any user with read access could see them.
Natural language queries were saved in browser's local storage which is limited to 5MB only. Now, New WebView stores results on the server side which is limited only by server's storage.

AG-367 - Make the query language option local to a query tab instead of being global for a user

Before 8.1.1, the query language was part of "In-Browser query option" group and global for a user. Now, New WebView has language attribute for each query (saved or unsaved). The language is displayed as an icon in the query tab. Additionally, rename the "Free text" to "Text Search", and "Query documents" to "ChatStream - Natural Language" queries.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

No significant changes.

Version 8.1.0 user visible changes

AllegroGraph Server: General Changes

No significant changes.

HTTP Client

No significant changes.

SPARQL

No significant changes.

AGWebView

Option to stay logged in in New WebView

On the New WebView login dialog (shown at the beginning of the New WebView document) there is a Stay logged in checkbox which keeps you logged in for 30 days. Without checking that box, you are logged out after 30 minutes of idle time.

New WebView page now lists regular repos and vector repos

See the example images in the New WebView document.

Links in New WebView rearranged

To reduce clutter, some links on the left side of the New WebView page, such as Documentation and Examples have been combined under the heading Learning Resources.

Large Language Model changes.

Vector database are now backed up when their associated repo is backed up

You can create a vector database by embedding text contained in a repository. See the LLM Embed Specification document. When a repository is backed up, any associated vector databases are also back up automatically.

Lineparse changes

Lineparse is a file format used in various places in AllegroGraph. See the Lineparse document.

Carriage returns are ignored in Lineparse files

Linefeeds (or EOFs) indicate line endings.

Changes to the Lisp API

No significant changes

Prolog

No significant changes.

Documentation

New LLM Split document

See LLM Split Specification

Distributed AllegroGraph

No significant changes.

Version 8.0.0/8.0.1 user visible changes

This section lists changes in the current release. See the Change History for a list of changes in all earlier releases.

AllegroGraph Server: General Changes

Some changes to install-agraph

You run the install-agraph program to install the tar file distribution of AllegroGraph. Relative pathnames are now supported and there are new arguments including --help (describe how the program works), --license-file (specify a file containing the AllegroGraph license saving you from having to edit the config file after creation) and --super-password (actually not new but now works more intuitively) and --super-password-file (the latter in particular allowing you to avoid entering the password in the transcript which might be visible to others).

Users will be logged out after 30 minutes of idle time

Logging back in will return to the state when the logout occurred. If you have multiple browser tabs open to AllegroGraph, all will be logged out and logged back in simultaneously.

Two new AuthPolicy values external-token and token-external

See AuthPolicy in the Server Configuration and Control document.

Further allowed abbreviations of server specs

Now allowed are:

user:password@ meaning user:[email protected]:10035 user:password@:1234 meaning user:[email protected]:1234

127.1 is the same as localhost. 10035 is the default port number which is used unless another port is specified. See the Repository Specification document where server specs and associated repository specs are discussed.

Some changes when configuring AllegroGraph

User password is by default handled within AllegroGraph, but users may wish to use some external authentication via LDAP in order to authorize users. In that case supplying passwords within AllegroGraph is neither necessary or desired. During installation if using the install-agraph script, supplying a blank superuser password can be accepted but only after confirming you wish to disable password authentication:

SuperUser account password: [user simply enters Return]  
 
Password can't be blank!  
Would you like to disable authentication via password?:  
[n]: y

Also the default instance timeout is now 1 hour (3600 seconds).

Exiting shared backend processes

Shared Backend processes (not session backend processes) will now exit if they are haven't been used for the last ten minutes. This will free up system resources and allow repos to close if the only reason they are open is that a backend process has them open. See the agraph.cfg parameter BackendMaxIdle to control how long a backend must be idle before it is eliminated.

SHACL now allows specifying all graphs

Previously you could specify default or a list of graphs. Now all also works for specifying the union of all graphs in the triple-store. This works with:

HTTP [repo]/validate/shacl - dataGraph and shapesGraph args
Lisp shacl-validate - :data-graphs and :shapes-graphs keyword args
agtool shacl-validate - --data-graph and --shapes-graph options
SPARQL Magic Properties - dataGraph and shapesGraph args

Text starting with a `#` is a comment in agraph.cfg

Previously, # had to be the first character of a line to make the following text a comment. Now everything after # on a line is a comment regardless of where the # appears. So the line

Auditing no   # may enable if needed

suppresses auditing and ignores # may enable later.

Names of Fedshard shards now end in .fshard

The Fedshard utility (see the Fedshard-def document) requires that names or shard repos end in .shard, in part to distinguish them from normal repos. Some users may have used earlier experimental versions of Fedshard available in earlier releases where that was not a requirement. They must change names to conform with the new specification.

Backing up settings now backs up attributes data

See the description of backing up settings in Repository Backup and Restore.

Rfe17027: Add delete-duplicate-triples and get-dupliate-triples support to agtool

Duplicate triples can now be obtained with agtool get-duplicate-triples ... and deleted using agtool delete-duplicate-triples .... See agtool.

Bug26968: Occasional segmentation faults when using user attributes

User attributes could cause segmentation faults on repositories that used advanced features like reasoning or security filters. This has been fixed.

Bug26926 - Bogus results produces by joins on optional variables

When joining on a variable that is bound in some results and unbound in others, like the variable ?x3 in the following query

SELECT ?x0 ?x1 ?x2 ?x3 ?x4 {  
  ?x0 :pred1 ?x1 .  
  ?x0 :pred2 ?x2 .  
  OPTIONAL { ?x0 :pred3 ?x3 }  
  OPTIONAL { ?x3 :pred4 ?x4 }  
}

the correct behavior for the rows where ?x3 is unbound is to compute a cross-product of the triples with predicate :pred4. Due to a bug in the internal join machinery, AllegroGraph computed cross-products for incorrect rows instead, producing bogus results. This has been fixed.

Bug26397 - login failures caused by AGWebView tabs of different versions

WebView login used to fail when there were multiple browser tabs with different WebView versions. This has been fixed.

HTTP Client

Rfe17140 - support prefixed names in HTTP API endpoints accepting parts

Many HTTP API endpoints taking triple parts as arguments used to only allow complete IRIs to denote resources. This is no longer the case and now prefixed names like rdf:type are also allowed, but only if the given namespace is defined in that context (e.g. only server-level namespaces are used for security filters, while user namespaces are used for API calls <repository>/statements; see Namespaces hierarchy for more details).

Rfe17040 Add RedirectHTTPOnHTTPS AG directive

When set, HTTP requests sent to the AG SSLPORT will result in a 301 Moved Permanently response with the location set to the HTTPS version of the URL.

SPARQL and GRAPHQL

Single quotes work like double quotes to designate strings in GraphQL

Strings in GraphQL can be written 'foo' as well as "foo". Also '''string''' is supported just like """string"""

This is an AllegroGraph extension to the GraphQL spec. See the GraphQL document.

New configuration and query options for query caching

The query option allowCachingResults can cause query results (unaffected by LIMIT or OFFSET) to be cached so running the same query repeatedly (perhaps with different OFFSETs and LIMITs) can simply access the chached results. New configuration options QueryResultsCacheSize and QueryResultsCacheStorageSize allow specifying the maximum number of query caches and space used by query caching or to disable cahcing altogether.

Rfe17214 - Opt-in unification of `owl:sameAs` duplicates in SPARQL queries

If multiple items are declared owl:sameAs, you may wish only one returned in response to a query. This can be done with the new query option unifyOwlSameAsDuplicates. See its documentation in the SPARQL Reference for further details and an example.

Rfe17183 - support for `owl:SymmetricProperty` in materializer

The materializer now supports OWL 2 prp-symp (owl:SymmetricProperty) axiom as called for by the dynamic RDFS++ reasoner. (It was not supported in earlier versions of the materializer.)

Rfe16975 - Use the same default base URI for all backends/sessions

A base URI for the SPARQL query can be supplied via one of the two query parameters (uri or baseURI), but if neither is present, AllegroGraph falls back to using the URL of the query request itself. This could result in different base URIs for different SPARQL query invocations, because the request URL would contain a port of the actual AllegroGraph backend or session that was handling the request, for example the following query

SELECT ?v { VALUES ?v { <> <v0> </v1> <#v2> } }

would return

| v                                                  |  
|----------------------------------------------------|  
| http://127.0.0.1:40123/repositories/test           |  
| http://127.0.0.1:40123/repositories/v0             |  
| http://127.0.0.1:40123/v1                          |  
| http://127.0.0.1:40123/repositories/test#v2        |

when run through the main port on the repository test and

| v                                                  |  
|----------------------------------------------------|  
| https://127.0.0.1:50456/sessions/UUID              |  
| https://127.0.0.1:50456/sessions/v0                |  
| https://127.0.0.1:50456/v1                         |  
| https://127.0.0.1:50456/sessions/UUID#v2           |

when run through a session (actual UUID replaced with a placeholder).

Staring with AG v8.0.0, the default base URI is defined as

http://<host>:<port>/[catalogs/<cat>/]repositories/<repo>/sparql

for queries handled by backends and sessions that wrap a single concrete triple store and

http://<host>:<port>/sparql

for all other sessions. For example the query above will now return

| v                                                  |  
|----------------------------------------------------|  
| http://127.0.0.1:10035/repositories/test/sparql    |  
| http://127.0.0.1:10035/repositories/test/v0        |  
| http://127.0.0.1:10035/v1                          |  
| http://127.0.0.1:10035/repositories/test/sparql#v2 |

for backends and sessions wrapping a single concrete triple stores, and

| v                                                  |  
|----------------------------------------------------|  
| http://127.0.0.1:10035/sparql                      |  
| http://127.0.0.1:10035/v0                          |  
| http://127.0.0.1:10035/v1                          |  
| http://127.0.0.1:10035/sparql#v2                   |

for all other sessions.

Rfe16328: Warn about unknown constants in SPARQL `DESCRIBE` clauses

AllegroGraph used to be silent about non-existent IRIs in the DESCRIBE clauses of simple SPARQL DESCRIBE queries like the following

DESCRIBE <http://non-existent/resource>

This has been changed and now, just as for SELECT queries, an unknown-constants query warning is included in the response.

Bug27048 - query parsing failures with aggregates and outer `VALUES`

SPARQL queries containing both aggregates and a VALUES clause after the WHERE clause, like

SELECT (COUNT(?o) AS ?count)  
WHERE { ?s ?p ?o }  
VALUES ?s { <http://example.org/s0> }

used to fail during parsing. This has been fixed.

Bug27090 - SPARQL `SERVICE` calls do not follow HTTP redirects

SPARQL SERVICE calls used to only perform a single HTTP request and fail if a redirect response was received. This has been fixed and HTTP calls in SERVICE clauses now follow redirects up to 5 times.

Bug26880 - Saved queries HTTP API ignores `uri`/`baseURI` arguments

When using session's saved queries API to save a named query that contains a relative IRI, it was impossible to supply a base URI via the uri/baseURI query arguments, as they would be ignored and the operation would fail with something like

MALFORMED QUERY: No base URI for URI ...

The only way to supply a base URI was through the BASE declaration in the query itself. This has been fixed and the uri/baseURI parameters now work as expected.

Bug26772: Obscure errors in patterns with repeated variables on MJQE

When using MJQE, if a query contains a pattern with repeating variables or blank nodes, and either a VALUES clause that injects bindings into that pattern, for example

SELECT * {  
  VALUES ?o { :o1 :o2 }  
  ?a ?a ?o.  
}

or a numerical range FILTER clause on one of the variables in that pattern, for example

SELECT * {  
  ?a ?a ?o.  
  FILTER (?o > 10)  
}

it might fail with an incomprehensible error. This has been fixed.

Bug26675: Possible incorrect bindings order in SPARQL solutions

Queries with OPTIONAL clauses in rare cases can return solutions with wrong binding order. For example, the following rather synthetic query

PREFIX : <http://example.org/>  
 
SELECT ?p ?o {  
  VALUES ?g { :g0 }  
  VALUES ?b { :b }  
  GRAPH ?g {  
    :a :next ?b.  
    ?b :next ?c.  
    OPTIONAL { ?c ?p ?o }  
  }  
}

could return a solution with values for variables ?p and ?o switched. This has been fixed.

Bug25697 - Wrong results with `OPTIONAL` clauses sharing a variable

SPARQL queries that perform a join on variable that might be unbound, like, for example, variable ?x2 in the following query

SELECT ?x0 ?x1 ?x2 {  
  ?x0 <http://example.org/p1> ?x1 .            # 1  
  OPTIONAL { ?x1 <http://example.org/p2> ?x2 } # 2  
  OPTIONAL { ?x2 <http://example.org/p3> ?x3 } # 3  
}

might return incorrect results, because patterns (2) and (3) get inlined into BGP (1) during optimization, but the proper join is not introduced, which results in a cross product between patterns (2) and (3). This optimization can turned off by setting

PREFIX franzOption_inlineSingleBGPOptionals: <franz:no>

but the query might still return incorrect results due to bug26926. Both issues have been fixed.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes other than new files for new features.

Distributed AllegroGraph

No significant changes.

Version 7.3.1 user visible changes

AllegroGraph Server: General Changes

spr45740 - SHACL: errors in franzOption_profileQuery and resultMessage

These valid SHACL requests might signal the following errors:

"Attempt to take the car of :time which is not listp." error when the SHACL SPARQL query has PREFIX franzOption_profileQuery:
An error when trying to format a resultMessage template with one of the variables being a non-string UPI

These have been fixed. There have also other been SHACL speedups.

Bug26839 - Fix SSLCipherSuite directive

A bug fetching the value of the SSLCipherSuite config directive (see Server Configuration and Control) at startup caused it to be ignored. This is now fixed.

Bug26806 - Turtle parser fails on prefixes starting with `base`/`prefix`

AllegroGraph's Turtle/TriG parser used to fail on prefixes that start with directive keywords (base and prefix). For example, all the statements in the following Turtle file

 @prefix : <http://example.org/>.  
 @prefix base: <http://example.org/base/>.  
 @prefix basenames: <http://example.org/basenames/>.  
 @prefix prefix: <http://example.org/prefix/>.  
 @prefix prefixes: <http://example.org/prefixes/>.  
 base:s      :p :o.  
 basenames:s :p :o.  
 prefix:s    :p :o.  
 prefixes:s  :p :o.

would have failed to parse. This has been fixed.

Bug26775 - Turtle parser fails on boolean literals followed by dots

Turtle parser used to signal an error when a boolean literal was used in an object position of a statement and immediately followed by the dot character, for example

@prefix : <http://example.org/>.  
:s0 :p0 true.  
:s1 :p1 false.

bug26765 - Problem with Admin/Audit log page

The audit log was not recording audit events. This is now fixed.

Bug26754 - Load-transform fails on floating point numbers in JSON

Floating point numbers in JSON documents used to produce errors of the form

`0.0d0` is not of type xsd:double

due to conversion operation malfunction. This has been fixed.

Bug26747 - Allow using transform templates with JSON list elements

It is now possible to use tranform templates with elements of JSON lists. For example, assuming JSON source

{"foo": "1", "bar": ["2", "3"]}

command

agtool load <repo> \  
    --tr-prefix 'http://example.org/' \  
    --tr-transform 'bar=http://example.org/bar${foo}-${bar}' \  
    <source.json>

will create two objects for property bar:

_:e0 ex:bar ex:bar1-2, ex:bar1-3.

bug26135 - change in stored queries format in 7.3.0

Stored SPARQL should be preserved when backed up and restored but this failed in 7.3.0 because the name of one of the fields was change from "text" to "query". Now either label can be used.

Bug26755 - Errors when using numeric values in load-transform templates

Populating templates with numeric values (as opposed to strings) would cause an error during load-transform operations, for example, assuming the following JSON document

{ "foo": 1 }

the following load-transform

agtool load <repo> -i json \  
       --tr-transform 'foo=http://example.org/Foo/${foo}' \  
       <source.json>

would signal an obscure error

excl::string2 argument should be a string

because of an internal type mismatch. This has been fixed.

HTTP Client

No significant changes.

SPARQL

Bug26872 - "Not CaaT state" errors with large `OFFSET` and `DISTINCT`

It was possible to get a confusing error of the form

Not CaaT state: nil within set

when running a SPARQL query with DISTINCT modifier and a large OFFSET, e.g.

SELECT DISTINCT ?s ?p ?o { ?s ?p ?o }  
OFFSET 500000  
LIMIT 1000

when the actual number of solutions iterated through exceeded one of the internal thresholds. This has been fixed.

Bug26739 - Type errors on language-tagged literals in SPARQL's REGEX

SPARQL REGEX function used to report a type error when its first argument was a language-tagged string, for example

SELECT (REGEX('Blade Runner'@en, 'Blade') AS ?r) {}

This has been fixed.

AGWebView

bug26762 - cancel query button didn't work in AG WebView

This has been fixed.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

No significant changes.

AllegroGraph 7.3.0

AllegroGraph Server: General Changes

agtool `--output` option replaced by `--output-format`.

--output-format is more descripting. The affected agtool commands include:

catalogs
export
namespaces
query-options
repos
shacl-validate
tokens list

--output can still be used but will trigger a deprecation warning.

Rfe16785: Deprecate `--results-format` in favor of `--output-format`

The command-line interface for agtool query has a new argument, --output-format for specifying the desired output format, replacing --results-format. --output-format is the argument used by all agtool commands. Here is a sample call:

agtool query <repo> query.rq --output-format sparql-json

The old argument can still be used but will trigger a deprecation warning.

Rfe16780 - Support for using AWS session tokens with agtool

New --aws-session-token command-line argument argument was added to the agtool commands that support interacting with AWS S3 (agtool export, agtool load and agtool archive). The session token value can also be read automatically from the configuration files. See Temporary credentials in the agtool document.

HTTP Client

No significant changes.

SPARQL

Rfe16766 - New query option to print query execution error backtraces

The logBacktraceOnQueryFailure causes debugging information to be printed to the log when errors occur when executing a query. While this information is rarely useful to the user, it may be useful when contacting AllegroGraph support.

Rfe16761 - New query options for profiling query execution

New query options profileQuery and profileOutputFormat can cause profile data to be written the the AllegroGraph log. This is useful when a query seems to take longer than expected. See Query execution options in SPARQL Reference.

Bug26683 - Incorrect results with duplicate blank nodes in patterns

SPARQL queries with patterns containing duplicate blank nodes, for example

SELECT * { _:x ?p _:x }

could return incorrect solutions where the values corresponding to the blank nodes were not the same. This has been fixed.

Bug26629 - Errors in GROUP expressions with unknown variables on MJQE

SPARQL queries with unknown variables in a singleton complex GROUP expression, for example

PREFIX franzOption_engine: <franz:mjqe>  
 
SELECT (SAMPLE(1) AS ?foo) { ?s ?p ?o }  
GROUP BY CONCAT(\"bar\", ?bar)

used to fail with the error message

Received signal number 7 (Bus error)

This has been fixed and now the result of evaluating such an expression is assumed to be UNDEF, and the type error warning is reported alongside the unknown variables warning.

Bug26627 - Some SPARQL query warnings may cause obscure print errors

Certain query warnings, for example repeated projected variables like in the following query

SELECT ?s ?s ?o { ?s ?p ?o }

are generated during parsing and trying to send such a query in a parsed form to a remote triple store or a distributed triple store shard may cause errors like

Unable to print  
#<db.agraph.sparql.parser:warn-repeated-projected-variables-ignored  
  @ #x10012f443b2>  
readably and *print-readably* is true.

This has been fixed.

Bug26621 - SPARQL ORDER clause might not work when GROUP present

SPARQL queries with both GROUP and ORDER clauses could return results in an incorrect order because ORDER clause expressions used to have no access to bindings in GROUP and SELECT clauses that defined during aggregation. This has been fixed.

Additionally, ordering now also works with non-aggregate variables: in such cases the order expression is wrapped in an implicit SAMPLE operation and a corresponding warn-implicit-sample-introduced query warning is reported.

Bug26616 - Allow using constant SPARQL expressions as aggregates

It should be possible in SPARQL to use constant expression assignments alongside aggregated variables, for example

SELECT ?s ("constant" AS ?c)  
WHERE { ?s ?p ?o }  
GROUP BY ?s

but assignments likes these used to cause queries to fail with errors like

All selected variables must be aggregates but ?c is not.

This has been fixed.

Changes to the Lisp API

Rfe16175: Unify `run-sparql`'s `rdf-format`/`results-format` arguments

The run-sparql function's arguments rdf-format and results-format are deprecated in favor of the new output-format argument. The old arguments can still be used but will signal a deprecation warning.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

No significant changes.

AllegroGraph 7.2.0

AllegroGraph Server: General Changes

Some TopBraid Composer (TBC) examples no longer work

The document TopBraid Composer Plugin has examples showing how you can connect TBC to an AllegroGraph repository and then directly add triples created by TBC. This worked by using the Sesame2 Remote Repository export format from TBC. Recent versions of TBC (which is now at version 7.x) no longer support that format. To copy triples from TBC to AllegroGraph you must save the triples to a file and load the file into AllegroGraph. The TopBraid Composer Plugin document has been updated to not this change.

New agtool vload command loads materialized triples from a relational database

See agtool and the Creating Virtualized Graphs document for more information. There is an example using the Ontop utility.

New data streaming example

See Graph streaming for an example connecting AllegroGraph a data streaming tool such as Apache Kafka.

New example using Apache Spark

An example of graph analytics using AllegroGraph and Apache Spark can be found here in the franzinc/agraph-examples repo. See the Examples on Github document, which includes examples using Apache Spark, Apache Kafka, Kubernetes, Terraform, load balancing with NGINX, and so on).

New agtool commands query-options and namespaces

The command agtool query-options can be used to add, remove, and list default query options for the user and repo specified in the command line. The command agtool namespaces can be used to add, remove, and list default user namespaces for the user and repo specified in the command line.

New --port and --sslport control options to agraph-control

The agraph-control program now accept --port and --sslport control options, which, if specified, override the values specified for Port and SSLPort in the AllegroGraph configuration file (usually named agraph.cfg). This is convenient if the values specified in the configuration file are already in use. The agraph-control is documented here in the Server Configuration and Control document. The associated agraph also now accepts --port and --sslport arguments.

Handling of numbers in JSON-LD code fixed

Things that looked like numbers in JSON-LD code, including the string ".", were converted to typed numbers, even when that was not wanted. Strings "true" and "false" were also always converted to booleans. This has been fixed. Conversion is now performed only when the appropriate datatype is explicitly specified.

New storage-report option to agtool

The tool shows storage used by a repository. See here in the agtool General Command Utility document. The report includes, along with other information, the percentage of triples stored but deleted.

The Table in the Cluster Managment page modified

The table is illustrated in the Multi-master Replication document. The table now shows the host and port of each instance.

The maximum number of unique strings has increased

From 2^32 to 2^39. (The hash table in shared memory used for locating strings doubled in size.)

Rfe16711 - implement `query-results-limit` user access flag

Users can now be restricted to a maximum number of results from a query. See Limiting results of a query in SPARQL Reference and Limiting the results a user can see in Managing Users. Also see the new QueryResultsLimit configuration directive.

Rfe16709 - XML 1.1 support for rdf serializer

Make RDF serializer output XML 1.1 by default. Previously AllegroGraph wrote version 1.0 XML which is unable to express all of the characters you might find in repository data. The new XMLVersion configuration option can be used to revert back to XML 1.0 if needed.

Rfe16691 - New agtool for automating repository optimization

See the Release Notes description. Also see agtool auto optimize, or AGWebView.

Rfe16639 - Rename agtool command `token` to `tokens` for consistency

The agtool command tree token has been renamed to tokens for consistency with other similar commands. The old token name will still work to ensure backward compatibility but it is deprecated and excluded from the help text.

Rfe16562: Support limiting `agtool query` output to results only

The output of agtool query can now be limited to results only by specifying the --quiet option. When --quiet is set, all log output will be written to the agquery.log file. See agtool query for more information.

Rfe16540 - Add support for creating, deleting, and listing repos and dynamic catalogs with agtool

agtool has new repos and catalogs tools. They can be used, respectively, to create and delete repos (including replicated repos) and dynamic catalogs. The tools can also list existing repos and catalogs. See the agtool document for more information.

Bug26575 - SHACL implicit class targets only worked for rdfs:Class

If a shape is also declared to be a class in the shapes graph then all SHACL instances of this class are a target for the shape (see the SHACL spec here). This only worked if the shape had a rdfs:Class type and failed if it was a subclass of rdfs:Class e.g. owl:Class. This has been fixed.

Bug26473 - Decimal production in SPARQL 1.1

DECIMAL production now uses the SPARQL 1.1 definition, meaning that at least one digit after the dot is required. Previously 1. was read as the DECIMAL 1.0, instead of being correctly read as the integer 1 followed by a period, just like

1 .

This is an incompatible change. The SPARQL 1.0 rule is specified here and the changed SPARQL 1.1 rule is here.

Bug26414: Fix default query memory limit initialization

Default query memory was incorrectly initialized to the 13.3 GB instead of 85% of available system memory, as specified in the documentation. Because of this in some cases the users needed to increase the memory limit manually via the query option

PREFIX franzOption_memoryLimit: <franz:NEW_LIMIT>

where NEW_LIMIT is either a number (of bytes) or a value of the form <number><unit> (e.g. 32G). This has been fixed so default query memory will start at 85% of available system memory.

Bug26221 - Speedup Freetext Index (FTI) phrase match searches

Fixed an inefficiency in FTI phrase matches that caused a slowdown when a word in the phrase appeared in a single triple multiple times. This is now optimized. The speedup seen will vary based on the frequency in which a word from the phrase appears in a given triple. For small string literals and resources, the improvement will be very modest, but for phrase matches in large string literals the speedup can be significant

HTTP Client

No significant changes.

SPARQL

Rfe16530 - Add support for xsd:dateTime subtraction in SPARQL queries

Arithmetic operations in SPARQL's BIND clauses now support xsd:date/xsd:time/xsd:dateTime subtraction that returns xsd:duration values, i.e. query like

SELECT ?x  
WHERE {  
    BIND (("2020-07-29T17:26:08Z"^^xsd:dateTime -  
           "2020-07-28T17:26:08Z"^^xsd:dateTime) AS ?x)  
}

will return a value

"P1D"^^xsd:duration

Note that results of date/time subtraction are restricted to xsd:dateTimeDuration subset of xsd:duration values for simplicity reasons.

Rfe16528 - Remove reorderDuringExecution query option

In practice the pattern estimates are good enough to not have to consider reordering. Using the special query prefix:

PREFIX franzOption_reorderDuringExecution: <franz:yes>

will now give a query warning:

obsoleteQueryOption: reorderDuringExecution

Rfe16510 - Support for SPARQL*

AllegrGraph now supports queries that use the SPARQL* syntax, like:

select * {  
  << ?s ?p ?o >> <ex:key> ?value  
}  
 
select * {  
  bind( << ?s <ex:pred> ?o >> as ?triple )  
}

A reified triple is represented as an IRI that looks like: <http://franz.com/ns/allegrograph/4.0/tripleId#123> which means isIRI(?triple) will return true for reified triples.

There is a new SPARQL filter function: isTriple(?triple) that returns true only if ?triple is a reified triple.

SPARQL* queries are not yet supported on federated or distributed repositories.

Rfe16504: Remove "enableDuplicateReduction" query option

The SPARQL query option enableDuplicateReduction, if set to true, would behave somewhat like specifying reduced. In all cases where it would be relevant to enable this reduction, it is better to specify reduced or distinct in the query. Therefore the setting is removed.

Usage of the franzOption_enableDuplicateReduction query prefix will now lead to a warn-obsolete-query-option entry in the query result metadata:

(run-sparql "PREFIX franzOption_enableDuplicateReduction: <franz:yes>  
             select ...")

has fourth return value:

#<done  
  [warn] obsolete-query-option: enableDuplicateReduction  
  [time] ...  
  ...>

Bug26558 - Fix zero-length property paths with non-existent S/O parts

According to the SPARQL specification, assuming an empty data set, all of these queries should return the same single result :s1:

# query 1  
SELECT ?s {  
  ?s :p1* :s1  
}  
 
# Query 2  
SELECT ?s {  
  VALUES ?s { :s1 }  
  ?s :p1* :s1.  
}  
 
# Query 3  
SELECT ?s {  
  ?s :p1* :s1.  
  FILTER (?s = :s1)  
}

but only the first one returns the correct result, while all others return an empty result set because of a premature optimization. This has been fixed.

Bug26557 - Remove odd zero-length property paths matches on MJQE

MJQE used to return additional results for zero length paths when either subject or object of the pattern was fixed. For example, for the data set

:s1 :p1 [ :p2 :o1 ].  
:s2 :p3 :o2.

the query

SELECT ?o {  
  VALUES ?o { :o2 }  
  :s2 (:p1|:p2)* ?o  
}

returned

-------  
| o   |  
=======  
| :s2 |  
-------

which is only correct when the object of the pattern is not fixed. This has been resolved.

Bug26532 - Query canceling does not work for queries run on MJQE

It was impossible to cancel a query that was being executed on MJQE because of the incorrect job initialization. For example, pressing the Cancel button in WebView had no effect on the MJQE query currently in progress. This has been fixed.

Bug26531 - Fix total query time reporting for queries run via HTTP

When running SPARQL queries through HTTP endpoint (for example, in WebView), the query information sent in x-query-final-info trailer contained incorrect total query time that was approximately twice as big as the real total time. This has been fixed.

Bug26525 - Format selection fails for CONSTRUCT queries run with agtool

When executing queries with agtool query, the --results-format argument was not working properly for DESCRIBE and CONSTRUCT queries with the keywords in upper case. This has been fixed.

Bug26515 - Restore `reorderDuringExecution` query option

The option was removed in v7.1.0 but has now been restored. See Query execution options in the SPARQL Reference document.

Bug26493 - Queries with magic properties may return less results

It was possible for some queries with magic properties and fixed values to return less results because of an incorrect optimization, For example, for the following dataset

@prefix : <http://example.org#>.  
:alice :owns "Galaxy Note".  
:bob   :owns "Galaxy Note".

the following query

PREFIX : <http://example.org#>  
PREFIX fti: <http://franz.com/ns/allegrograph/2.2/textindex/>  
SELECT ?p ?o WHERE {  
  (:bob ?o ?p) fti:match "galaxy".  
}

will return an empty result, while the query without the fixed constant

PREFIX : <http://example.org#>  
SELECT ?s ?p ?o WHERE {  
  (?s ?o ?p) fti:match "galaxy".  
}

returns both results, as expected:

--------------------------------  
| s      | p     | o           |  
================================  
| :alice | :owns | Galaxy Note |  
| :bob   | :owns | Galaxy Note |  
--------------------------------

This has been fixed.

Bug26460: Empty RDF graph was returned by SPARQL's `DESCRIBE * WHERE ...`

The following two SPARQL queries are expected to be equivalent

DESCRIBE * WHERE { ?s ?p ?o . }  
 
DESCRIBE ?s ?p ?o WHERE { ?s ?p ?o . }

but the first one (with the wildcard *) used to return an empty result. This has been fixed.

Bug26452 - MJQE may omit bindings introduced by SPARQL `VALUES` clause

When more then one variable in a pattern has bindings introduced by the VALUES clause, MJQE engine may omit the bindings for one of the variables.

For example, assuming the data

:a :p1 :b.  
:a :p2 :c.  
:b :p2 :c.

the following query

SELECT ?s ?p ?o WHERE {  
  VALUES ?s { :a }  
  VALUES ?p { :p1 :p2 }  
  ?s ?p ?o.  
}

used to return the following result

------------------  
| s   | p   | o  |  
==================  
| --- | :p1 | :b |  
| --- | :p2 | :c |  
------------------

when run on MJQE. This has been fixed.

Bug26451 - Fix occasional errors in queries on security-filtered stores

When security filters were used, it was possible for some queries to fail with an error message of the form

QUERY FAILED: `#<db.agraph.cursor::filtered-cursor ...>' is not  
of the expected type `number'

indicating an internal type error. This has been fixed.

Bug26436 - Fix missing results for multiple property path matches

In cases when there are multiple zero-or-more paths between the subject and object in the restricted pattern, the query engine used to drop the result with multiple paths, for example for the dataset

:s1 :p1 :o1.  
:s2 :p1 :o1.  
 
:s1 :p2 [ :p2 :o2 ].  
:s2 :p3 [ :p2 :o2 ].  
:s2 :p2 [ :p3 :o2 ].

the query

SELECT ?s {  
  # this restricts the pattern in FILTER EXISTS  
  ?s :p1 :o1.  
  # this ends up return less results  
  FILTER EXISTS { ?s (:p2|:p3)* :o2 }  
}

used to return only one result :s1, while it should have returned :s1, :s2. This has been fixed.

Bug26333 - Property paths that join bindings may produce bogus results

It was possible for the SPARQL sequence property paths to produce too many results when joining bindings from VALUES clauses or subqueries. This has been fixed.

Bug26290 - Max when some variable is unbound, must give unbound result value

In AllegroGraph 6.7.0 a SPARQL regression was introduced, where SPARQL MAX aggregator would treat unbound values as bound values (lowest in ordering) and so return a bound value as result.

This was incorrect: when aggregating over an unbound value, the aggregation result should be an unbound value due to a SPARQL type error in the aggregation.

This has been fixed, so that the aggregation result is the unbound value in the described case.

Bug26282 - Stack overflow during OPTIONAL query planning

Certain queries containing an OPTIONAL clause could lead to an error ("stack overflow") during query planning. This is now fixed.

Bug26260 - Fix spurious "emptyBindClause" query warnings

SPARQL queries with a BIND clause at the start of a BGP could generate a spurious "emptyBindClause" warnings saying:

The BINDING form for ?val cannot succeed

This can be an incorrect warning if the BIND expression does a special operation that can deal with unbound variables, like BOUND, COALESCE, IN or IF.

Examples:

select * {  
  bind(bound(?o) as ?val) .  
}  
-> ?val = false  
 
select * {  
  bind(coalesce(?o,1,2) as ?val) .  
}  
-> val = 1

This is now fixed so that no spurious warnings are generated.

AGWebView

Bug26332: Broken bulk namespace input dialog in AGWV

The Add a namespace button in query editor shows a dialog that has a Bulk Input button which in turn should be showing a different dialog that allows to input multiple namespace declarations at once. This dialog was broken and an empty dialog box was shown instead. This has been fixed. See the New Query Page section of the WebView document for more information on the Add a namespace button.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

There is a new AllegroGraph examples document.

This document has links to example locations, either part of the distribution or on our franzinc/agraph-examples repo on Github.

Distributed AllegroGraph

No significant changes.

AllegroGraph 7.1.0

AllegroGraph Server: General Changes

New storage-report option to agtool

The tool shows storage used by a repository. See here in the agtool General Command Utility document.

The Table in the Cluster Managment page modified

The table is illustrated in the Multi-master Replication document. The table now shows the host and port of each instance.

Rfe16562: Support limiting `agtool query` output to results only

Rfe16543 - Replace default index :gospi by :psogi

The set of default triple indices has been modified: gospi has been taken out, and psogi has been added.

Rfe16540 - Add support for creating, deleting, and listing repos and dynamic catalogs with agtool

Bug26221 - Speedup Freetext Index (FTI) phrase match searches

HTTP Client

No significant changes.

SPARQL

Rfe16530 - Add support for xsd:dateTime subtraction in SPARQL queries

Arithmetic operations in SPARQL's BIND clauses now support xsd:date/xsd:time/xsd:dateTime subtraction that returns xsd:duration values, i.e. query like

SELECT ?x  
WHERE {  
    BIND (("2020-07-29T17:26:08Z"^^xsd:dateTime -  
           "2020-07-28T17:26:08Z"^^xsd:dateTime) AS ?x)  
}

will return a value

"P1D"^^xsd:duration

Note that results of date/time subtraction are restricted to xsd:dateTimeDuration subset of xsd:duration values for simplicity reasons.

Rfe16528 - Remove reorderDuringExecution query option

In practice the pattern estimates are good enough to not have to consider reordering. Using the special query prefix:

PREFIX franzOption_reorderDuringExecution: <franz:yes>

will now give a query warning:

obsoleteQueryOption: reorderDuringExecution

Rfe16510 - Support for SPARQL*

AllegrGraph now supports queries that use the SPARQL* syntax, like:

select * {  
  << ?s ?p ?o >> <ex:key> ?value  
}  
 
select * {  
  bind( << ?s <ex:pred> ?o >> as ?triple )  
}

A reified triple is represented as an IRI that looks like: <http://franz.com/ns/allegrograph/4.0/tripleId#123> which means isIRI(?triple) will return true for reified triples.

There is a new SPARQL filter function: isTriple(?triple) that returns true only if ?triple is a reified triple.

SPARQL* queries are not yet supported on federated or distributed repositories.

Rfe16504: Remove "enableDuplicateReduction" query option

Usage of the franzOption_enableDuplicateReduction query prefix will now lead to a warn-obsolete-query-option entry in the query result metadata:

(run-sparql "PREFIX franzOption_enableDuplicateReduction: <franz:yes>  
             select ...")

has fourth return value:

#<done  
  [warn] obsolete-query-option: enableDuplicateReduction  
  [time] ...  
  ...>

Bug26333 - Property paths that join bindings may produce bogus results

It was possible for the SPARQL sequence property paths to produce too many results when joining bindings from VALUES clauses or subqueries. This has been fixed.

Bug26290 - Max when some variable is unbound, must give unbound result value

In AllegroGraph 6.7.0 a SPARQL regression was introduced, where SPARQL MAX aggregator would treat unbound values as bound values (lowest in ordering) and so return a bound value as result.

This was incorrect: when aggregating over an unbound value, the aggregation result should be an unbound value due to a SPARQL type error in the aggregation.

This has been fixed, so that the aggregation result is the unbound value in the described case.

Bug26282 - Stack overflow during OPTIONAL query planning

Certain queries containing an OPTIONAL clause could lead to an error ("stack overflow") during query planning. This is now fixed.

Bug26260 - Fix spurious "emptyBindClause" query warnings

SPARQL queries with a BIND clause at the start of a BGP could generate a spurious "emptyBindClause" warnings saying:

The BINDING form for ?val cannot succeed

This can be an incorrect warning if the BIND expression does a special operation that can deal with unbound variables, like BOUND, COALESCE, IN or IF.

Examples:

select * {  
  bind(bound(?o) as ?val) .  
}  
-> ?val = false  
 
select * {  
  bind(coalesce(?o,1,2) as ?val) .  
}  
-> val = 1

This is now fixed so that no spurious warnings are generated.

AGWebView

Bug26332: Broken bulk namespace input dialog in AGWV

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

No significant changes.

AllegroGraph 7.0.4

AllegroGraph Server: General Changes

Rfe16569 - backup-all (and restore-all) now backs up gruff options

agtool archive backup-all/backup-settings now backs up the settings/gruff/ directory.

Bug26353 - Turtle parser failure on prefixes with upper-case characters

Turtle parser was incorrectly converting prefixes of prefixed names to lower-case if they were at the beginning of the lines and started with the same (ignoring the case) characters as Turtle keywords prefix or base, which caused a namespace lookup failure.

For example, an attempt to parse the following Turtle

 prefix Basic: <http://example.org/basic#>  
 Basic:subject Basic:predicate Basic:object.

failed with the following error

 While parsing /path/to/file.ttl: Turtle parser  
 error on line 2: No prefix expansion defined for 'basic'

This has been fixed.

HTTP Client

No significant changes.

SPARQL

Bug26348 - Fix handling of the FROM clauses of SPARQL ASK queries

Starting with AG 7.0.0, running SPARQL ASK queries with the FROM clauses failed with a verbose error that said something like keyword list ... should only contain ... but unexpectedly found .... This has been fixed.

Bug26346 - Fix 'Funcall of nil which is a non-function' error

It was possible to get a Funcall of nil which is a non-function error when executing a SPARQL query that has a UNION of GRAPH clauses with unsatisfiable patterns. This has been fixed.

Bug26331 - Restriction reasoning did not work in remote Lisp AG client

Both SPARQL and Prolog queries run from a remote Lisp AllegroGraph client would miss inferred results when restriction reasoning is enabled. This has been fixed.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

No significant changes.

AllegroGraph 7.0.3

AllegroGraph Server: General Changes

No significant changes.

HTTP Client

No significant changes.

SPARQL

Rfe16530 - Add support for xsd:dateTime subtraction in SPARQL queries

Arithmetic operations in SPARQL's BIND clauses now support xsd:date/xsd:time/xsd:dateTime subtraction that returns xsd:duration values, i.e. query like

SELECT ?x  
WHERE {  
    BIND (("2020-07-29T17:26:08Z"^^xsd:dateTime -  
           "2020-07-28T17:26:08Z"^^xsd:dateTime) AS ?x)  
}

will return a value

"P1D"^^xsd:duration

Note that results of date/time subtraction are restricted to xsd:dateTimeDuration subset of xsd:duration values for simplicity reasons.

Bug26309 - Missing zero-length property path results if both s and o are collections

It was possible for a query of the form:

select ... {  
  values ?start { ... }  
  values ?end { ... }  
  ?start ( <ex:pred1> / <ex:pred2>* ) ?end  
}

with triple data:

<ex:s> <ex:pred1> <ex:o>

to not return the valid solution:

?start = <ex:a>  
?end = <ex:b>

where was followed zero times, so effectively not followed. This is corrected.

Bug26293: Property path error if object value constrained

In certain cases a property path expresion in which the object value is constrained to a literal, could lead to an error.

For example in this query ?o is constrained to value 2:

SELECT ... {  
  bind(2 as ?o).  
  ?s <ex:p>+ ?o  
}

Depending on the active indices and the exact triples in the repository, this could give the error:

The assertion  
  (loop for x in (list s p o g s-end p-end o-end g-end)  
    always (typep x (quote (or upi null))))  
failed

This is now fixed.

Bug26282 - Stack overflow during OPTIONAL query planning

Certain queries containing an OPTIONAL clause could lead to an error ("stack overflow") during query planning. This is now fixed.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

No significant changes.

AllegroGraph 7.0.2

AllegroGraph Server: General Changes

An AllegroGraph server may restrict itself to the number of licensed cores

If the UseLicensedCores configuration directive is specified true (that is UseLicensedCores true is in agraph.cfg) then the agraph process will automatically reduce the number of cores it uses to the value given in the license file. Otherwise the server will fail to start up if there are more cores on the machine than the license specifies.

Bug26226 - Changing passwords in AGWebView could fail

When a password has expired, changing it in AGWebView could fail. This has been fixed.

Bug26221 - Speedup fti phrase match searches

HTTP Client

No significant changes.

SPARQL

Bug26274 - Insert ..^^xsd:string fails

Queries that insert a triple with an XSD string as object, like:

INSERT {  
  <ex:s> <ex:prop> "2.0.0"^^xsd:string .  
}  
WHERE { }

would fail with the error message: :string fell through a ecase form. This is fixed.

Bug26273 - Do not skip index when rewriting left-joins to joins

In some cases the system would skip the index component of a plan step when constructing a JOIN plan step. This was breaking the assumption that index is non-nil when removing equivalent variables from JOINs. This is now fixed.

Bug26272 - Count DISTINCT of UNION incorrect

It was possible for queries with COUNT DISTINCT of a UNION to give a number that was too high. For example:

SELECT (COUNT (DISTINCT ?s) as ?s)  
{  
  { ?s <ex:p1> <ex:o1> }  
  union  
  { ?s <ex:p2> <ex:o2> }  
}

This is now fixed.

Bug26263 - Type errors when using UNION and FILTER

This is an issue relevant to queries containing UNION in which an unprojected variable occurs both inside the UNION (at most once per UNION subclause) and once outside it, like ?o in:

SELECT DISTINCT ?s {  
{ ?s <ex:p1> ?o } UNION { ?s <ex:p2> ?o }  
FILTER (?o > 3)  
  }

In such cases it was possible for ?o to become unbound in the second occurrence (in the FILTER clause) even though the first UNION clause would bind it. This would lead to incorrect query results.

This is now fixed.

Bug26259 - Property path error: "the value of p is "

Certain property path expression would fail with the error:

the value of db.agraph.sbqe.property-path::p is  
#<db.agraph.utility.dictionary:dictionary ...>  
which is not of type (or null upi real).

This is fixed.

Bug26240 - SPARQL property path that joins bindings misses results

It was possible for property path expressions to miss certain results if the subject or object of the property path was bound by a previous pattern. This is now fixed.

Bug26236 - Errors when using multi-match clause with multiple vars as filter-pattern

There were situations where a query containing a "VALUES" or "FILTER (.. IN ..)" clause, in combination with other patterns, would give incorrect query results. This has been fixed.

Bug26235 - Assertion error :is-ordered-cursor-for-distinct-variable

It was possible for queries with DISTINCT to fail with the error:

the assertion  
  (null (getf (db.agraph.cursor::cursor-properties db.agraph.sbqe::cursor)  
              :is-ordered-cursor-for-distinct-variable))  
failed.

This has been fixed.

Bug26222 - SPARQL FILTER range errs if there are objects with non-overlapping type

If a SPARQL query included a numerical filter, in the presence of triples whose object value is of a numerical type that does not overlap with that range, this would signal an error at query plan time.

For example:

SELECT * {  
  ?s ?p ?o .  
  FILTER (?o > 3 && ?o < 10)  
}

This range (3, 10) of positive integers does not overlap with type xsd:negativeInteger. If the repository contain a triple like:

{ <ex:s> <ex:p> '-123'^^xsd:negativeInteger }

the resulting error would be:

Error: `3` is not of type xsd:negativeInteger, exceeds the maximum: -1  
  [condition type: xsd-datatype-error]

This has been fixed

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

Bug26220 - Fedshard failure for "count distinct" with "group by"

When using FedShard, running a query that combines COUNT DISTINCT with GROUP BY could result in an error. For example:

SELECT ?s (COUNT(DISTINCT ?o) AS ?count) {  
  ?s ?p ?o  
} GROUP BY ?s

This has been fixed.

AllegroGraph 7.0.1

AllegroGraph Server: General Changes

Rfe16481 - Increase triggered checkpoint timeout

Previously while adding a replica of a large repo on a large memory machine an error like the following could occur:

Error: instance add failed with code 400, body "failed with message: Could not trigger a checkpoint: Checkpointer process failed to respond"

This has been fixed.

Bug26176 - SHACL validation would consider shape graph triples

Under certain circumstances, SHACL validation would incorrectly consider shape graph triples for validation, resulting in reporting triples non-conforming when the data graph triples actually conformed. (Reporting triples as conforming when in fact they did not conform never occurred.) This has been fixed. SHACL and SHACL validation are described in the SHACL document.

HTTP Client

No significant changes.

SPARQL

Rfe16452 - Let GROUP BY with no result rows give zero result rows

The results of GROUP BY queries with aggregates in an edge case has been changed to closer follow the intention of the SPARQL 1.1 standard.

Take this query and assume there are no triples with predicate :

select ?s (count(?o) AS ?count)  
where {  
  ?s <ex:p> ?o  
} group by ?s

AllegroGraph would return one result row with ?s unbound, and count zero.

An alternative behaviour that some other triple stores have, is to return zero result rows. The text of the SPARQL 1.1 standard conflicts with the "agg-empty-group2" test case, so that either behaviour is reasonable.

The current consensus is that having zero result rows was intended, and is expected by users. See e.g. https://afs.github.io/sparql-agg-group-empty.html#group-agg-no-rows

AllegroGraph has been updated to follow this consensus. So for GROUP BY queries with aggregates, in case of zero matching pattern in the body, there will be zero result rows.

Bug26190: Incorrect rewrite of OPTIONAL with non-patterns on LHS

In some cases queries of the form:

SELECT .. {  
  VALUES ..  
  ..  
  OPTIONAL { .. }  
}

could be optimized incorrectly, resulting in missing query results. This has been fixed.

Bug26188 - Executing query failed for query with VALUES

Certain queries of the form:

SELECT .. {  
  VALUES ?x { <a> <b> <c> }  
  .  
}

could lead to an error:

Error: the value of ag.sbqe::index is nil,  
       which is not of type ag.sbqe::plan-index-type.

this has been fixed.

Bug26181 - Incorrect :time-total addenda value for query on remote store

When executing a SPARQL query in AGWebView the Information tab shows query execution statistics, including the time it took to plan, execute, and output query results.

It could happen that the total time would be incorrect (too low). This has been fixed.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Distributed AllegroGraph

Rfe16457 - improve shard-local kb support

Previously, when defining a distributed store in the cluster config file, there was limited support for shard-local kb's by including localhost in the KB declaration. That is the following declaration would define a kb on localhost and would inherit whatever scheme/port were the default at the point where the declaration was made:

KB localhost/catalogs/tests/repositories/kb4

If these defaults match the scheme/port used by each AG shard, then the KB will be shard-local. Now, any triple-store-spec that does not include a server part can be used to define a shard-local KB. Thus

KB tests:kb4

no longer will inherit defaults from the environment at the point of the declaration, and will inherit the scheme/port of each shard, even if scheme/port differ across all shards.

See the Distributed Repositories Using Shards and Federation Setup document.

Bug26187 - Do not track servers in KB directives

In Distributed AllegroGraph, the servers that comprise a cluster are derived from the directives that refer to servers in the agcluster.cfg file. This includes server, repo, replica, and --incorrectly-- kb directives.

The servers referred to only in KB directives are no longer considered as part of the cluster. This allows remote KBs which do not reside on a cluster host.

AllegroGraph 7.0.0

AllegroGraph Server: General Changes

agtool load changes

The (undocumented) --blank-node-cache-size argument has been removed. (Even though this argument was undocumented, some users may have been told of its existence.) Blank nodes are no longer cached during data loading.

There is a new -w/--workspace option to agtool load. The value should be an existing directory which will be used for temporary files during data loading, particularly for the temporary storage of a table of blank node information. Absent this option, temporary data is stored in /tmp and if that directory is too small, the load will fail.

agtool now supports a triple-count tool

The triple-count tool returns the number of triples in a repository. See here in the agtool document.

agraph-control now takes a --cluster-config control option

The option allows the path of the cluster configuration file to be specified. This file defines distributed repositories (see Distributed Repositories Setup). The agraph-control program is described in the agraph-control of the Server Configuration and Control document.

Adding a triple with a predicate mapping and a datetype mapping might add two triples

Before this change, when adding a triple, if both a predicate mapping and a datatype mapping applied to the object of the triple, two triples would be added. The new defined behavior is that predicate mappings take precedence over datatype mappings.

index-literals argument passed when creating/modifying a freetext index

When creating a freetext index, the index-literals argument determines if and/or under what circumstances a literal will be indexed. The value passed can be either a boolean or a list containing UPIs, future-parts, or strings.

With this change, when passing a string to index-literals, it is expected to be equivalent to the value returned when calling part->value. That is, the string should not contain any embedded angle-brackets (< or >) or double-quotes (") used to denote the string represents a resource or literal, that are not part of the actual value represented by the string.

Rfe16417 - Deprecate style 2 indices

There are two triple index styles available, "style 1" and "style 2", with different storage structures and thus performance characteristics. In practice "style 1" was generally recommended, and "style 2" has been deprecated. The latter has been removed from the documentation and might be removed in a future release.

Rfe16386 - New BriefBacktrace configuration directive.

See the [administrator note](recent-release-notes.html#admin-notes-700} for more information. See Server Configuration and Control for information on configuration directives.

Rfe16364 - New LicenseWarnInAdvance configuration directive.

See the administrator note for more information. See Server Configuration and Control for information on configuration directives.

Rfe16283 - New configuration directive to control the stale DNS retain time

There is a new configuration directive StaleDNSEntryRetainTime which specifies how long AllegroGraph will retain a stale DNS entry. If the DNS mapping from name to IP address may change while the application is running you'll want to specify a very small value (number of seconds) for this directive. See StaleDNSEntryRetainTime in Server Configuration and Control.

Rfe16244 - Update PARSE-NTRIPLES to support N-Triples 1.1 blank node syntax

When importing N-Triple data, certain blank node formats with dots, dashes or underscores were unsupported. For example importing this line:

 <http://s> <http://p> _:a-b_c.d .

would fail with:

 Error: N-Triples parser error while parsing string at line 1  
       (last character was #\_): Illegal character

Now AllegroGraph supports the blank node syntax as defined in the N-Triples 1.1 technical recommendation.

Rfe16239 - agtool load/export/archive from/to s3

agtool can access files in Amazon S3. To do so, AWS authentication must be specified on the agtool command line. See the section Accessing and operating on files on Amazon S3 in the agtool document. Files in S3 must be prefaced by s3://, like the following:

 s3://bucketname/a/b/c/filename

Rfe14667 - Log THP warning only (don't show in AGWV)

The THP (Transparent Hugepages) warning is now logged to agraph.log only and not the console or AGWebView.

Bug26040 - agtool archive restore to a different repo.

The documentation printed by 'agtool archive restore --help' claimed that if the archive directory contained only one repo backup then you needn't name that backup if you restored to a different repo name. This claim has been removed from the documentation because the behavior isn't supported.

Bug26017 - Literal language in rdf/xml was not properly loaded

Previously, instead of setting the language of the literal object, an additional triple with the object being the language value and the predicate http://www.w3.org/XML/1998/namespacelang was incorrectly added to the triple store.

For example, <rdfs:label xml:lang="fr">Autriche</rdfs:label> loaded as:

<subj> <pred> "Autriche"  
<subj> <http://www.w3.org/XML/1998/namespacelang> "fr"

Bug26011 - Allegrograph licensed cores check prints wrong core count in error message

Previously if the number of CPU cores exceeded the licensed limit, AllegroGraph would generate an error message like the following:

This machine exceeds the CPU core count specified in the license. Maximum licensed cores: X. This machine has: Y.

The value of X would be the license limit but the value of Y would potentially be incorrect (it would report the number of available threads, rather than the number of cores in the machine, which will be different values if hyperthreading is enabled). This has been fixed.

Bug26000 - Fti recovery crashes if stray fti files exist from a prior instance crash

In some circumstances instance recovery could fail with an error message like the following:

opening of database myrepo has failed (The database system tried to  
allocate a new file with the logical name fti-1-chunk-1 which already  
exists in database myrepo)

This has been fixed.

Bug25999 - Crashed instance process can leave service daemon in confused state

Under some circumstances a crashed instance process would not be automatically restarted by the service daemon. This has been fixed.

Bug25986/Bug25997 - Fix agraph-backup tool initialization issues

When the repo spec was used to specify the repository, the credentials and host values from the spec were ignored and the tool always attempted to connect to 127.0.0.1. This commend thus

$ agtool archive backup test:xyzzy@remotehost:10035/test test.bak

could fail with a Connection refused (errno 111) message because the host is not localhost. Specifying the credentials when restoring data also could cause the tool to fail (bug25997):

 # Fails with the message 'Invalid arguments (:params,  
 # :restore-mode-p) found while creating triple-store of class  
 # db.agraph:remote-triple-store. ...'  
 $ agtool archive restore-all test.bak test:[email protected]:10035

Both these issues have been fixed. Note that using deprecated arguments like --host and --port could cause problems if the values specified did not match the values in the REPO-SPEC, even default (and thus not-visible) values in some cases. See Older repository specification arguments in the agtool General Command Utility document for examples.

Bug25959 - Fix ACID isolation violation in some ordered cursor operations.

Fixes a bug introduced in v6.5.0 where some ordered triple cursors could produce triples that were not visible in the current transaction. The scope of the problem was limited to triples committed in concurrent transactions that had yet to be written to disk.

Bug25907 - Document how to pass args to agraph using agraph-control

The ability to use agraph-control to pass arguments to the agraph program (which is called by agraph-control) when starting or restarting AllegroGraph existed prior to this change, but was not clearly documented. How to do so is now documented in the agraph-control --help text and in the description of agraph-control in the Server Configuration and Control document.

Bug25901 - Importing triples bypassed security-filters

It was possible for users with security-filters to import triples into a db that those filters were intended to prevent.

This problem is now fixed.

Bug25880 - Remove whitespace trimming in TRIX literals parsing

To restore the behavior specified by the standard, removed whitespace trimming in the TRIX parser while parsing literals.

HTTP API

No significant changes.

SPARQL

Rfe16433 - Remove full-scan-warning

The query warning type "warn-fullscan" and the query option "fullScanWarningSize" have been removed.

In practice these warnings could occur without there being a problem with the query. It would fill agraph.log with warning lines that would be ignored, defeating the purpose.

Usage of the franzOption_fullScanWarningSize query prefix will now lead to a warn-obsolete-query-option entry in the query result metadata:

(run-sparql "PREFIX franzOption_fullScanWarningSize: <franz:210000000>  
             select ...")

has fourth return value:

#<done  
  [warn] obsolete-query-option: fullScanWarningSize  
  [time] ...  
  ...>

Rfe16324 - Warn on ASK modifiers

An ASK query does not support LIMIT, OFFSET, or ORDER BY. Now a query warning is signaled if any of these is provided, with the exception of LIMIT 1.

Rfe16279 - Improve checking for unknown constants in the query

The check for unknown constants in the query at plan time is now more advanced, resulting in faster query execution.

For example if a resource <s> only occurs in the subject position of triples, a pattern like ?s <p> <s> will now be marked impossible at plan time.

Another example are references to standard IRIs that are present in every repository automatically, like <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>. Previously a pattern like ?s a ?foo would never be marked impossible, but now if there are no triples with such predicate, the pattern will be marked impossible.

This also has effect on magic properties like <http://franz.com/ns/allegrograph/3.0/temporal/intervalBefore> that under the hood expand into patterns with properties like <t:starttime>, <t:startpoint> and <t:time> (with http://franz.com/ns/allegrograph/3.0/temporal/ as t). If a repository only uses starttime, and never the startpoint/time combination, the plan will be optimized accordingly.

Bug26144 - CAAT ignores :pattern-filters of query patterns

Queries executed with the Chunk-at-at-Time (CAAT) strategy (so with a LIMIT, without DISTINCT or aggregators) could give incorrect results, if a specific query pattern was planned to act as filter for another pattern. This has been fixed.

Bug26143 - `Graph <g> {}` gives error

A query with an empty named graph clause, like:

select ?s ?o {  
  graph <ex:g> {}  
  ?s ?p ?o .  
}

would fail with an error: Attempt to access the name field of {g} which is not a symbol. This is now fixed.

The meaning of graph <ex:g> {} as a pattern, is that some triple must exist in the given fixed graph <ex:g>. The meaning of graph ?g {} is that some triple must exist in any named graph.

Bug26137 - Spurious unknown-variables warning for VALUES in inner SELECT

A VALUES clause inside an inner SELECT, like in this query:

SELECT ?region {  
  SELECT ?region WHERE {  
     VALUES ?type { <ex:Region> }  
     ?region a ?type .  
   }  
 }

would give a spurious warning about the VALUES variable:

unknown-variables: Unknown variable used in VALUES clause: ?type_1_1

This is now fixed, so that there will be no more warnings.

Bug26116 - FILTER CONTAINS LCASE gives UPI-NOT-IN-STRING-TABLE-ERROR

Certain SPARQL expressions that create intermediate strings, like:

select ?s {  
  ?s <ex:p> ?o .  
  filter ( contains(lcase(?o), 'something'))  
}

could signal the error: The UPI #(...)is not present in the repository's string table.

This has been fixed.

Bug26093 - GROUP_CONCAT interprets separator arg as format string

If a SEPARATOR argument containing a tilde was supplied to the GROUP_CONCAT aggregator, the resulting string could be incorrect. This has been fixed.

Bug26090 - Boolean value serialized as sparql-json lacks datatype

A boolean value serialized in sparql-json format would look like (here we use the Lisp function sparql:run-sparql but the results would be the same if the query was executed in AGWebView and exported in sparql-json format):

(sparql:run-sparql "select * { bind(true as ?x) }"  
   :results-format :sparql-json)  
 
-> {"head": {"vars": ["x"]},  
    "results": {"bindings": [{"x": {"type": "literal", "value": "true"}}]}}

But the correct way to serialize a boolean is to include the datatype. Thus this:

"x": {"type": "literal", "value": "true"}`

is incorrect and should instead be:

"x": {"type": "literal",  
      "datatype": "http://www.w3.org/2001/XMLSchema#boolean",  
      "value": "true"}

This has been corrected.

Bug26078 - SPARQL FILTER on numeric range excludes UPI types that partially overlap

If a SPARQL query FILTER expression restricts a triple part to a numeric range, and if there is a triple whose part value falls in that range but is of a data type such that the filter range exceeds the data type range, then the triple would not pass the filter, which was incorrect.

For example:

(add-triple "<ex:s>" "<ex:p>" (value->upi 127 :byte))

and query:

select * {  
  ?s ?p ?o .  
  filter ((?o > 120) && (?o < 130)  
}

The query FILTER restricts the triple object to (120, 130). The ?o value 127 should pass this filter. The value 127 is of type :byte, which has range [-128, 127].

The incorrect behaviour is that the FILTER range (up to 130) exceeds the data type range (up to 127) and therefore the value 127 was incorrectly filtered out, meaning the triple would not be returned by this query.

This has been fixed.

Bug26071 - COUNT aggregate on unknown variable should return 0.

If a query aggregates a variable that is not otherwise used, like:

SELECT (COUNT(?bogus) AS ?count) { ?s ?p ?o }

this would be evaluated as if the unknown variable ?bogus was replaced by true. The value of ?count would be the total triple count.

This has been corrected, so that the returned ?count is now zero.

Bug26020 - Invalid OPTIONAL reordering

If a query has an OPTIONAL pattern that binds a variable, followed by a non-OPTIONAL pattern that references that variable, like:

SELECT * {  
  ?s <ex:p1> ?o .    # (1)  
  OPTIONAL {  
    ?t <ex:p2> ?o2 . # (2) possibly binds ?o2  
  }  
  ?s <ex:p3> ?o2     # (3) references (if bound already) or binds ?o2  
}

the query engine could create an incorrect query plan with evaluation order (1) (3) (2). The evaluation order of (2) and (3) would be reversed: variable ?o2 would be bound based on pattern (3). This is now corrected to have evaluation order (1) (2) (3).

Bug26005 - Incorrect SUM and GROUP_CONCAT default values for empty group

If a SPARQL query has no result rows (or if GROUP BY is used, and a specific group has no result rows) a SUM aggregation and a GROUP_CONCAT aggregation would both return an unbound value.

This was incorrect according to the SPARQL 1.1 spec. Now SUM will return 0, and GROUP_CONCAT will return the empty string.

Bug26004 - Queries that have a constantly true HAVING could fail.

A constantly true HAVING expression in a query, like:

SELECT ?s (max(?o) as ?maxO) {  
  ?s ?p ?o  
} GROUP BY ?s  
HAVING (3 < 4)

could err with Received signal number 7 (Bus error). This has been fixed.

Bug25999 - Crashed instance process can leave service daemon in confused state

Under some circumstances a crashed instance process would not be automatically restarted by the service daemon. This has been fixed.

Bug25988 - SPARQL function REGEX should err on non-string

Previously AllegroGraph was liberal in accepting a constant resource as REGEX pattern. For example:

SELECT (regex('ex:foo', <ex:foo>) as ?match) {}

would return "true".

This has been fixed in order to adhere to the SPARQL specification: if the target or pattern is not a string, a type-error is signaled. The above SELECT now returns an unbound value.

Bug25984 - SPARQL function LANGMATCHES should err on non-string

Previously AllegroGraph was liberal in accepting non-strings to function LANGMATCHES and converting them using STR.

For example:

BASE <http://foo/>  
SELECT (langMatches(<ex>, <ex>) as ?a)  
   (langMatches(123, 123) as ?b) {}

was interpreted as:

BASE <http://foo/>  
SELECT (langMatches( STR(<ex>), STR(<ex>)) as ?a)  
   (langMatches( STR(123), STR(123)) as ?b) {}

and would return "true" twice.

This has been fixed in order to adhere to the SPARQL specification: if either argument to langMatches() is not a string, the result is a type-error. The above SELECT now returns two unbound values.

Bug25982: Fix HAVING constraint on implicit single group without aggregates

If a query uses HAVING without using GROUP BY, and no aggregates (like MIN, COUNT), as for example in:

SELECT * { ?s ?p ?o } HAVING (?s = <ex:s>)

then the HAVING clause acts like FILTER:

SELECT ?s ?p ?o { ?s ?p ?o . FILTER (?s = <ex:s>) }

However in the first query AllegroGraph would incorrectly ignore the HAVING clause, meaning results that should have been filtered out were returned. This is now corrected.

Bug25941 - Blank node patterns not supported in subject position of BGP triples

A query pattern with a blank node pattern as subject, and a property path, like:

select * {  
  [ <ex:p2> ?o2 ]  ( <ex:p1> | <ex:p2> )  ?o  
}

would give a parse error at the "(" character. This has been fixed.

Bug25936 - Inefficient join with UNION

The query execution planning for certain queries containing UNION has been improved, leading to faster execution. This impacts also magic properties that are implemented using UNION, like <t:intervalBefore>.

Bug25934 - Problem with BIND (EXISTS { pattern1, pattern2 } as ?x)

If a query binds a variable to an EXISTS expression, where the EXISTS contains multiple patterns, like:

SELECT ?person ?known where {  
  ?person a <ex:Person> .  
  BIND ( EXISTS {  
           ?employee a <ex:Employee> ;  
                     <ex:knows> ?person  
         } as ?known )  
}

then the outcome could be incorrect: the second and later patterns could be ignored, or the query could result in an error like:

Error: Bad variables for set: (?person).

This is now fixed.

Bug25933 - Duplicate patterns in blank node + property path expansion

For query triple patterns with a property path, and a blank node as object, like:

?s  ^<ex:p>  [ <ex:p1> <ex:foo> ;  
               <ex:p2> <ex:bar> ]

the expansion into simple triple patterns could contain duplicates:

1. ?bnode_1 ex:p ?s  
2. ?bnode_1 ex:p ?s  (duplicate)  
3. ?bnode_1 ex:p2 ex:bar  
4. ?bnode_1 ex:p1 ex:foo

The duplicated pattern resulted in superfluous work during query execution. This is now fixed.

Bug25931 - Sending SPARQL query with typed booleans to remote-triple-stores fails

Making SPARQL queries with typed booleans (e.g. true"^^xsd:boolean) would fail with an error indicating that AllegroGraph is unable to convert a boolean UPI into a future part.

This is now fixed

Bug25925 - SPARQL MAX does not handle unbound values correctly

The SPARQL MAX aggregator would return an unbound value as result, if any of the values to aggregate was unbound. This has been fixed.

Bug25920 - SPARQL SERVICE using https without chunking errs

When a SPARQL query was sent to remote SPARQL endpoint that uses HTTPS, and the endpoint sent results not using chunked encoding, then an error would be signaled:

Error: the value of stream is #<excl::ssl-client-stream ..>,  
       which is not of type  
 (or net.aserve::unchunking-stream util.zip:inflate-stream  
           excl::hiper-socket-stream).  
  [condition type: type-error]

This could happen in a SERVICE <https://...> clause, or by proving a :HOST "https://.." argument to function run-sparql.

This has been fixed.

Bug25919 - SPARQL function STRLEN should err on non-string

Previously AllegroGraph was liberal in accepting non-strings to function STRLEN and converting them using STR.

For example:

SELECT (STRLEN (<http://franz.com>) as ?s)  
       (STRLEN (123.45) as ?t) {}

was interpreted as:

SELECT (STRLEN (STR (<http://franz.com>)) as ?s)  
       (STRLEN (STR(123.45)) as ?t) {}

and would return values 16 and 6.

This has been fixed in order to adhere to the SPARQL specification: taking the length of a non-string now gives a type-error. The above SELECT now returns unbound values.

Bug25916 - COUNT DISTINCT of expressions gives error

A query for COUNT DISTINCT where the value is not a variable (like ?o) but an expression (like "strlen(?o)"), for example:

select (count(distinct(strlen(?o))) as ?olen) {  
  ?s ?p ?o  
}

used to signal an error:

Received signal number 11 (Segmentation fault)

These queries are now supported.

Bug25898 - SPARQL error for property multimatch with filtered object range

Queries with a property path with alternatives, that have a filter on the object variable, where the object values are of a type that is hashed to a UPI, like:

select * {  
  ?s ( <ex:p1> | <ex:p2> ) ?o .  
  filter (lang(?o) = 'en')  
}

could result in an error like:

Error: The UPI #(0 0 0 0 0 0 0 0 0 0 0 3) (which is a literal-language)  
       is not present in the triple-store's string dictionary.

This has been fixed.

Bug25821 - SPARQL function SUBSTR should clip LENGTH to string length

If in SUBSTR the length argument is larger than then the string's length, this would result in an error:

SELECT * { bind (substr('abc', 1, 100) as ?s) }  
 
Executing query failed:  
In subseq, `end' (100) is beyond the end of the sequence (3).

This is now fixed, so that the substring up to the end of the string is returned. In this case: abc.

Bug25806 - Fix ORDER BY on non-projected variable

For certain SPARQL SELECT queries that use ORDER BY on a non-projected variable, the results could be ordered incorrectly. This has been fixed.

Bug25694 - Fix simplifying SPARQL= into UPI=

It was possible for a SPARQL equality filter:

FILTER (?a = ?b)

to return false if the values are of different types. For example, if ?a is the integer 2 and ?b the float 2.0. This has been fixed.

Bug25636 - Triple cursor FILTER containing [NOT] EXISTS applied incorrectly

If a query contains a FILTER clause that uses EXISTS or NOT EXISTS, there were cases where that FILTER was evaluated incorrectly and as a result the query would return incorrect results.

Concretely, if in the query plan the filter was integrated in the first triple pattern, it was possible for EXISTS to return true in certain cases where false was the correct answer; and for NOT EXISTS to return false where true was the correct answer.

This has been fixed.

Bug25585 - Support ORDER BY on an aggregate expression

Queries in which the ORDER BY expression refer to an aggregation, for example:

SELECT ?s (MIN(?o) as ?minO) {  
  ?s ?p ?o  
}  
GROUP BY ?s  
ORDER BY (min(?o))

would fail with the error:

:aggregate fell through a etypecase form

This is now fixed.

Bug24815 - Fix MINUS with an empty row on the RHS

A query that uses MINUS, where the second BGP contains only unbound values, like:

SELECT ?x {  
  { VALUES (?x) { (1) (2) (3) (4) } }  
  MINUS  
  { VALUES (?x) { (UNDEF) }  
  }  
}

should have 4 results, namely ?x bound to each integer.

Previously AllegroGraph would return no bindings. This is now corrected.

AGWebView

Bug26010 - Fixed Delete statement confirmation dialog

The Delete statement confirmation dialog in AGWebView is now fixed. Instead of asking for the confirmation, if showed the message: function () { [native code] } and did not allow to confirm the operation. Now, the confirmation dialog is displayed properly and, if the operation is confirmed, it will complete correctly.

Bug25947 - AGWebView doesn't allow all session spec characters

In 6.6.0 AGWebView was too restrictive on the characters allowed in the Session Specification edit box on the main catalog page. This has been fixed.

Bug25856 - Back-up this repository on WebView repo page does not work

Fixed the inner API for remote backup request with target on the server side file system. Fixed the dialog to correctly describe what is expected from user.

Changes to the Lisp API

Improve Materializer support for remote-triple-stores

The keyword arguments :delete-first? and :delete-duplicates-first? are now supported when calling materialize-entailed-triples on a remote triple store via the AllegroGraph Lisp client, and can also be specified via the REST API as well.

Rfe16211 - add IF-DOES-NOT-EXIST parameter to OPEN-TRIPLE-STORE

This change extends the open-triple-store function with an :if-does-not-exist argument. The following values are supported:

:error - throw an error (default);
:create - create new repository.

Rfe12356 - HTTP proxy support for AllegroGraph Lisp client

New :proxy parameter for create-triple-store, open-triple-store and triple-store-exists-p functions of the AllegroGraph Lisp client API allow to specify an HTTP proxy to be used for all HTTP requests issued by client. Proxy settings can be specified in the form

[<user>[:<password>]@]<host>[<port>]

Alternatively, components of the proxy specification can be provided explicitly by :proxy-user, :proxy-password :proxy-host :proxy-port parameters. Additionally, the list of domain name suffix strings, for which the proxy should be bypassed, can be specified via :proxy-bypass-list parameter.

Bug25974 - load-ntriples doesn't work with namestring source on remote-triple-store

Previously calling load-ntriples (or any of the other load-* functions) would fail for remote-triple-stores if the source argument was a string naming a file instead of a pathname. This has been fixed.

Prolog

No significant changes.

Documentation

rfe16275 - Storage and memory document for AllegroGraph

There is a new Memory Usage document which discusses memory and disk space requirements when using AllegroGraph.

nD Geospatial

No significant changes.

AllegroGraph 6.6.0

AllegroGraph Server: General Changes

Rfe16127 - Make AllegroGraph's scripting model permissions more strict

Superuser permission is now required to upload, list, view, or delete both sitescripts and reposcripts.

Additionally, reposcripts can no longer be loaded into shared backends.

Rfe16096 - Simplify AllegroGraph script naming.

Scripts no longer support hierarchical naming, in favor of a simple flat namespace. For example, when uploading a script:

Legal:   http://example.com:10035/scripts/sc.cl  
Illegal: http://example.com:10035/scripts/com/app/module/sc.cl

Rfe16093 - Incoming IP restrictions mechanism for AllegroGraph

New AllowIP configuration directive provides a way to restrict access to AG instance to certain ranges of IP addresses.

Rfe15992 - Add MaximumBackupAge parameter to the agraph.cfg file

One can specify the maximum age in seconds that a backup of the multi-master controlling instance is considered acceptable to use to grow the cluster. See the Instance Settings section of the Multi-master Replication document and also the Top-level directives for multi-master replication clusters section of the Server Configuration and Control document).

Rfe15886, Rfe15470 - UI Improvements to the triple attribute pages.

Merges the 'Define Attribute' and 'Define Static Filter' pages into a single 'Triple Attributes' page that is more streamlined and user friendly. See the Attributes example in the AGWebView documentation.

Bug25833 - Using CorsAllowRegex would break session/backend creation

Setting the CorsAllowRegex directive (see Server Configuration and Control) in your agraph.cfg would cause the creation of dedicated sessions and shared backends to break with a confusing error message in the agraph.log file.

Bug25832 - Only compare CorsAllowOrigin directive against Origin host

Previously, each value specified by a CorsAllowOrigin directive was matched against the full value of the Origin header: scheme, host, and port. Now, each CorsAllowOrigin directive is matched only against the host, per the documentation.

Bug25812 - agtool query --standard-namespaces

If agtool query is given the argument '--standard-namespaces no' then the AllegroGraph set of standard namespace PREFIX definitions will not be prepended to the text of query. Furthermore if the query is going to an AllegroGraph repository that agtool will open itself then the set of namespaces known to AllegroGraph is cleared before the query is run.

Bug25805 - Fix bug in SNA path-finding functions used with remote-triple-stores.

SNA path-finding functions like depth-first-search and all-bidirectional-search-paths would fail when called on a remote-triple-store if either the START or END arguments represented literals. This is now fixed.

Bug25801 - CSV import hangs on surrogate pairs

The CSV importer has been fixed to work with UTF8 surrogate pairs.

Bug25784 - Agtool archive failed when using an https spec.

Trying to use agtool archive with an https spec would fail due to the tool attempting to make regular http requests to the ssl port. This problem is now fixed.

Bug25736 - improved handling of GRAPH in JSON-LD

A JSON-LD error is now signaled when a string which is neither a resource nor URL is passed to as the --graph argument to agtool load. :root is supported as a special value of GRAPH, resulting in using the toplevel subject as a default graph for all triples added for the JSON-LD object with that subject.

Bug25382 - Fix security flaw in AllegroGraph scripting.

Fixes a bug, introduced in v6.3.0, that allowed any file on a filesystem accessible to AllegroGraph to be loaded as a script. Scripts are now only searched for and loaded from their expected locations relative to the SettingsDir directive.

HTTP Client

Bug25820 - CORS does not work via HTTPS

Previously CORS configuration options were ignored for accesses to AllegroGraph made via HTTPS. This has been corrected.

SPARQL

Rfe16157 - Implement logging of slow queries

A new configuration directive SlowQueryLogThreshold, if set, causes SPARQL queries that have total run time in milliseconds greater or equal to the value, to be logged to agraph.log or to a log file specified by SlowQueryLogFile configuration directive.

Bug25831 - Fix optimization of GRAPH with a non-existant value in SPARQL

The query:

select ?s ?p ?o  
{  
  VALUES ?s { <http://s> }  
  VALUES ?p { <http://p> }  
  GRAPH <http://non-existing> {  
OPTIONAL {  
  ?s ?p ?o  
}  
  }  
}

should return 1 result row with ?s, ?p bound and ?o unbound.

However due to an incorrect optimization of unreachable GRAPH clauses there would be no results. This has been fixed.

Bug25803 - Fix augmenting pattern with two IRI equivalence constraints

SPARQL queries that bind a variable multiple times, like:

SELECT * {  
  BIND (<http://ex/p2> as ?p)  
  {  
BIND (<http://ex/p1> as ?p)  
?s ?p ?o  
  }  
}

could be optimized incorrectly, leading to the error message:

Executing query failed:  
 nil is not of the expected type `number'

This has been fixed.

Bug25802 - Join query rewriters lose filters

Certain SPARQL queries where a join happens, and one side contains BIND and FILTER, like:

SELECT ... {  
  ?s ?p1 ?o1  
  {  
    BIND (<http://p2> as ?p2)  
    ?s ?p2 ?o2  
    FILTER (.. ?o2 ..)  
  }  
}

could be optimized wrongly in that the FILTER would be dropped. This has been fixed.

Bug25794 - BIND variables disappear in join

Certain SPARQL queries where a variable was used in a filter and also projected could return results in which that variable was unbound due to an incorrect query optimization. For example:

SELECT * {  
  {  
    bind (4 as ?four)  
    ?s ?p ?o .  
    filter (?o != ?four)  
  }  
  {  
    bind (3 as ?three)  
    ?s ?p ?o .  
    filter (?o != ?three)  
  }  
}

could return rows with ?three and ?four unbound. This has been fixed.

Bug25761 - Whitespace symbols are not escaped properly when writing in TSV format

According to the TSV SPARQL standard, when serializing RDF terms some whitespace symbols like tabs, newlines and carriage returns are encoded in strings as \t, \n and \r respectively. This has not been the case, as these symbols were serialized as-is, leading to some parsing problems later, specifically a tab character being treated as separator for TSV format. This has been corrected.

Bug25756 - Unused variable in complex property path gives errors

Queries of the form:

select ?o {  
  ?s ( ^<http://foo> | <http://bar> ) ?o  
}

in which the property path was not very simple, and also one variable is not otherwise used (here ?s), could lead to an error during execution. This has been fixed.

Bug25752 - Fix missing results for DISTINCT + LIMIT

There could be missing results for queries that uses both DISTINCT and LIMIT due to the DISTINCT condition being applied incorrectly. This has been fixed.

Bug25749 - Filter (str(?var) = '..') marked as impossible

If a SPARQL query had filter pattern comparing a STR() value to a literal string, like:

filter ('http://g' = str(?g))

this would result in no results and the warning:

 Warning: `?g` cannot be a literal so the FILTER ?g is one of 1 thing will always fail

This has been fixed.

Bug25714 - SPARQL IRI() and URI() functions must return valid, absolute results

Previously, the SPARQL IRI() and URI() functions could return IRIs and URIs that were syntactically invalid and/or relative. This has been corrected. AllegroGraph will now signal SPARQL type errors which will be treated as UNBOUND for assignment and as FALSE for FILTER evaluation.

Bug22350 - Fix missing results when using CAAT with DISTINCT

When a SPARQL query containing DISTINCT or REDUCED was executed using the chunk-at-a-time strategy, there could be results missing due to incorrect handling of intermediate query results. This has been fixed.

AGWebView

Rfe16012 - Don't show AllegroGraph Utilities menu unless logged in.

When not logged in, AGWebview displayed the Utilities menu even though none of its menu items work without being logged in. Now, the menu is not presented until a user is logged in.

Changes to the Lisp API

Rfe16071 - Implement breadth-first-search-if functionality

The four breadth-first search SNA functions:

have all been enhanced such that the end argument can be a function of one parameter. If end is such a function, then the search code will call it on each node found during the search and will record a successful search if the function returns true.

Bug25759 - Change load-trix :default-graph argument to :graph

To ensure consistency with other load- functions, :default-graph argument to load-trix has been deprecated and will be removed in the subsequent release. :graph should be used instead.

Bug25738 - AGWebView users are notified when query results may be stale

When reloading queries via the 'Saved' or 'Recent' menus--but also when executing new queries--AGWV will now report when there's a possibility the results displayed are out of date.

Prolog

No changes

Documentation

No changes

AllegroGraph 6.5.0

AllegroGraph Server: General Changes

Rfe15938 - Deprecate --tlog arg to agtool view-tlog

The --tlog argument to agtool view-tlog is deprecated. One need only specify the tlog filename to view.

Rfe15836 - Improve efficiency of dynamic reasoner on rdf:type queries

Increased the speed of the dynamic reasoner when looking for all <SUBJECT rdf:type OBJECT> triples.

Rfe15774 - Make agtools use repo and server specs

All agtool commands are in the process of being modified to use a Repo Spec when referencing a repository and a Server Spec when referencing a server but not a repository. (Repo Specs also identify servers. Server Specs are used when a repository is not relevant to the command, such as with agtool user.) In this release, the following commands have been modified to accept Repo Specs or Server Specs:

archive
create-db
export
generate
materialize
query
replicate
upgrade

These changes are backward compatible. Previous options are supported but deprecated and may cause warnings to be signaled.

Rfe15748 - Add support for backing up a single remote repository

agtool archive now supports remote repository backup. This operation requires a user with AllegroGraph superuser privileges.

Rfe15745 - Check for no-commit mode before creating a new text index

Creating a new text index requires indexing all existing triples and preparing the new index configuration and datafiles for commit.

Previously creating a new text index on a no-commit repository (such as a warm standby replica) would perform the initial indexing and then fail to commit due to no-commit mode. Now the check for no-commit mode is made early to avoid doing work that will end up being discarded.

Rfe15695 - agtool load now reports the filename with buffer-size exceeded errors.

When the length of a line in an input file exceeds the buffer size an error is reported. Previously this error did not specify which file the error resulted from. This has been changed to report the filename.

Rfe15595 - Add AG directives to choose suppored SSL methods and SSL ciphers

It is now possible to specify SSL protocols and ciphers that will be accepted by AllegroGraph when using HTTPS. This can be achieved by using two new config directives - SSLProtocol and SSLCipherSuites. Configuration directives are described in the Server Configuration and Control document.

Rfe15517 - Change agtool load to use repository specifications

agtool load now accepts a repository specification (REPO_SPEC) argument when specifying a repository. A REPO_SPEC names the repository and also its catalog, and the server scheme, host, port, so all associated arguments which specified those values (--scheme, --host, --port, and --scheme) are deprecated.

Rfe15471 - Support For loading JSON and CSV data in agtool load.

agtool load can now load JSON and CSV files. See See the section Loading Raw Data in the agtool load document to learn more about the supported workflows and parameters.

Rfe14214 - Add remote store support to agload

Remote store support has been added to agtool load. It is no longer necessary to run agtool load on the same machine as the AllegroGraph server and as the same user. A username and password must be specified when loading into a remote repository, and that user must have write permission in the repository into which data will be loaded.

Rfe13797 - Reconsider triggering checkpoints on tlog rollover

Automatic checkpoints on tlog rollovers are now disabled.

Rfe10713 - Two phase commit (2PC) support

AllegroGraph can now participate in two phase commit (2PC) operations. See the Two-phase Commit document for details.

Bug25668 - Turtle parser can fail to use same blank node context for all triples in a parse

The turtle parser could fail to unify blank nodes with the same label that appears in RDF lists and outside of those lists. For example, the following data would load with two different nodes instead of one:

prefix : <ex://#>  
:a :p ( _:x1 _:x1 _:x1 ) .  
_:x1 :p 'object' .

This has been corrected.

Bug25653 - Opening of a repl that should be running

A multi-master replication instance repo will remain open until it is explicitly stopped or for the period of time when the AllegroGraph server is not running. If an instance dies for some reason and a request comes in to open its repo then the repo now will open and rejoin the cluster. Previously an explicit start was required to start the instance and open the repo.

Bug25641 - Unhelpful error message from inability to translate a character

The error message that occurred when a data import was unable to translate a character was unhelpful. This has been corrected.

Bug25596 - RDFS++ reasoner does not follow sameAs for domain or range assertions

The dynamic RDFS++ reasoner was not inferring triples that could be derived from rdfs:domain and rdfs:range statements in addition to owl:sameAs statements. This has been corrected.

Bug25589 - Improve domain and range RDFS++ reasoning when subject not supplied

AllegroGraph's RDFS++ dynamic reasoner was failing to find all rdf:type inferences that used rdfs:range or rdfs:domain when the subject portion was not supplied. This has been corrected.

Bug25588 - Rdfs++ reasoner did not infer range/domain via subProperties

AllegroGraph's dynamic RDFS++ reasoner was not making rdf:type inferences when these required rdfs:domain and rdfs:range reasoning through chains of rdfs:subPropertyOf or owl:inverseOf linkages. This has been corrected.

Bug25586 - Error during recovery of partially-created text index

Previously it was possible for a crash to occur when restarting a repository instance if a partially-created text index existed during at the time of the last checkpoint. This has been fixed.

Bug25578 - xsd:integer casting operator does not work on xsd:longs

The xsd:integer() casting operator was not working on values of type xsd:long. This has been corrected.

Bug25574 - Bidirectional search was inefficient when there is no path

AllegroGraph bidirectional search functions did more work than needed when there was no path to be found between two different nodes. This has been corrected.

Bug25511 - Dynamic restriction reasoner requires subclassing unnecessarily

The dynamic RDFS++ restriction reasoning now recognizes owl:Restrictions that are simple rdfs:subClassOf owl:Restriction and also those that are of rdf:type owl:Restriction.

Bug25496 - agtool upgrade will detect a running repo and exit

agtool upgrade will check if a repo is running and if so print a message that an upgrade is not necessary and then exit with a no-error status of zero. (If a repo can be opened successfully, it does not require upgrading. Once a repo is opened in a version of AllegroGraph later than the version in which is was created, it should not be opened in the earlier version again.)

Bug25390 - Anyone can delete roles

A bug that allowed a user to delete roles without logging in has been fixed.

Bug25353 - Fix various issues when using `agtool export --parallel ...'

Fixes a few issues when exporting in parallel (using --parallel or a --workers argument greater-than 1), and using the default --blank-node-handling strategy of `together'

NQX export could be incomplete due to worker threads incorrectly detecting and skipping triples containing blank-nodes. Now, blank-part detection when exporting NQX works correctly.

Only N-Quads export would check for blank nodes in the graph slot, meaning for other quad formats, some blank nodes would not be properly separated into their own file. Now, NQX, TriX, and TriG formats also check the graph part when saving statements containing blank nodes to a separate file.

Bug25201 - agexport output depends on the value of LANG

It was possible for the value of the LANG environment variable to alter the output of agtool export. This has been corrected.

Bug25114 - Fix printing nquad default graphs

When exporting triples in NQUAD format from a federated or distributed triple store, there could be an unintended "default graph" marker like "default-graph1748246237" after the subject, predicate and object. This has been fixed.

Bug24954 - Fix exporting in NQX format of remote stores

Exporting a remote triple store using "agtool export" in "nqx" format:

agtool export -o nqx --user test:xyzzy ...

(with both "-o nqx" and a value for "--user")

would result in the error:

http-triple-cursors cannot change whether or not they return  
extended-triples after they have been created.

This has been fixed.

Bug23424 - Rdfs++ reasoner does not infer subProperties of transitive properties

The dynamic RDFS++ reasoner was failing to infer some triples when they relied on following rdfs:subProperties of owl:transitive properites. This has been corrected.

Bug19476, Bug18483 - restriction reasoning improvements

AllegroGraph's dynamic restriction reasoning was missing some inferences when either the predicate portion of a query was wild or when they required additional rdfs:subClassOf inference on inferred triples. These have both been corrected.

HTTP API

The %D HTTP log directive change

The %D HTTP log directive (meaning response time in microseconds) will now use the high precision clock provided by the operating system rather than the one second resolution clock.

Rfe15876 - Allow specification of Accept-Type in the query parameters

In addition to supporting the normal HTTP Accept-type header, AllegroGraph now also allows an accept parameter to be included in the query portion of an HTTP SPARQL query request. This allows a client to specify the desired content type even when it does not control the headers that it sends. For example, if QUERY is the percent-encoded SPARQL query string, then this sample curl command would return the number of rows that the query produces:

curl "http://REPOSITORY_URL?query=QUERY&accept=text%2finteger"

Rfe15737 - Add config directive to set HTTP keep-alive timeout

Added a new config directive HTTPKeepAliveTimeout to control the HTTP keep alive timeout. See Server Configuration and Control for information on config directives.

Rfe15731 - Add user-agent header to outgoing HTTP requests

To avoid problems with network proxies that refuse HTTP requests without a User-Agent header, this header is now included in outgoing HTTP requests to remote SPARQL endpoints.

SPARQL

Rfe16032 - Add some list and set handling functions to SPARQL

AllegroGraph now supports three new SPARQL functions:

The first two take a list of arguments and associate then with a variable (gensym). The third looks up an RDF list in the current SPARQL dataset and associates the contents with a variable. In each case, this variable can then be used with other SPARQL magic properties and functions to retrieve and manipulate the list or set.

For example:

select ?member {  
  bind(<http://franz.com/ns/allegrograph/6.5.0/fn#makeSPARQLSet>(1, 2, 4, 1, 2) as ?set)  
  ?member <http://franz.com/ns/allegrograph/4.11/sna/members> ?set .  
}

Would return 1, 2, and 4.

Rfe16007 - Evaluate the WHERE clause of SPARQL UPDATE queries in REDUCED mode

If a repository contains duplicate triples, then evaluating the WHERE clause of SPARQL update queries could become very memory intensive. Evaluating these in REDUCED mode ameiliorates this problem.

Rfe15967 - Eager filter evaluation could lead to missing query results

If a SPARQL query was supplied parameter values (e.g., via the :with-variables argument to the Lisp API's run-sparql command) and these values produced type-errors, then the FILTER analysis machinery could become confused and produce an incorrect filter. This has been corrected. As an example, the following query and parameters should produce one result but before this bug was fixed, it would produce no results:

select ?target {  
  optional {}  
  FILTER( ! bound(?target) || ! ( isLiteral(?target) && (lang(?target) = 'en')))  
}

parameters: ((?target . !<ex://test>)). The type-error occurs when the SPARQL lang() function tries to determine that language of the resource !<ex://test>.

Rfe15891 - Provide URI function in SPARQL to parse freetext queries

AllegroGraph now includes a URI function that can be used to see how freetext indexing queries will be parsed. For example:

prefix test: <http://franz.com/ns/allegrograph/6.5.0/>  
select ?query ?expression {  
  values ?query {  
    '"soccer game"'  
    'soccer | f򴢯l'  
    'socc?r ping* '  
  }  
  bind(test:parseFreetextQuery( ?query ) as ?expression)  
}

would output:

---------------------------------------------------------------  
| query           | expression                                |  
===============================================================  
| "soccer game"   | (:phrase "soccer game")                   |  
| soccer | f򴢯l | (:or "soccer " "f򴢯l")                  |  
| socc?r ping*    | (:and (:match "socc?r") (:match "ping*")) |  
---------------------------------------------------------------

which shows how each of the queries would be parsed for use by the fti:match magic property.

Rfe15890 - Enhance fti:match and fti:matchExpression to also bind predicates

AllegroGraph's built in Freetext Index magic properties can now also bind the predicate of any matching triples. For example, a query like:

(?subject ?text ?predicate) fti:match 'sample query' .

Will search all text indices for matches on "sample" AND "query" and then bind ?subject, ?text, and ?predicate to the columns of each matching triple.

Rfe15717 - Query timeouts should only happen during execution not results output

Previously, AllegroGraph would check for query timeouts both during query execution and during result generation. This meant that a timeout could cause partial results to be sent over HTTP. AllegroGraph now only checks for timeouts while the query is executing which means that either no response will be sent or the entire result set will be sent.

Rfe15693 - Optimize some str(?a) = str(?b) filters

AllegroGraph will now execute queries involving str/str equality and inequality FILTERs where the variables in the calls to str are resources more efficiently. For example, if ?a and ?b are resources or plain literals, then this query will be able to execute without needing to look up the string values in the string dictionary:

SELECT * {  
  :entry :has ?a .  
  :reference :contains ?b .  
  FILTER( str(?a) = str(?b) )  
}

Rfe15692 - Speed up query planning for queries with only simple BGPs

AllegroGraph now avoids computing estimates for queries that contain only single pattern BGPs. E.g.,

prefix f: <ex:foo#>  
select * {  
  graph ?g { ?s2 f:record ?o }  
  { select ?s2 { ?s2 f:record ?o2 } }  
}

Rfe15686 - Handle query + filter patterns more efficiently

This change allows the query + filter handling improvements from rfe15628 (implemented in release 6.4.2) to be used more often. In particular, the filter pattern can now be the last pattern in a BGP and the filter pattern can also introduce new bindings.

Rfe15529 - Let run-sparql intern query variables in one package

SPARQL query variables (the third return value from run-sparql) were sometimes located in the query.variables package, and sometimes in the db.agraph.sbqe package. Now they are always in query.variables.

Rfe13652 - Improve performance of some DISTINCT SPARQL sub-queries

AllegroGraph has long evaluated single variable DISTINCT queries using a specialized and more efficient querying technique. This technique did not, however, apply to sub-queries. This meant that these two equivalent queries would perform significantly differently:

# Faster  
SELECT DISTINCT ?p { ?s ?p ?o }

and

# Slower  
SELECT ?p { { SELECT DISTINCT ?p { ?s ?p ?o } } }

The query engine logic has been modified so that the two queries will perform similarly.

Rfe10047 - Basic support for date arithmetic in SPARQL

AllegroGraph now supports adding and subtracting xsd:durations from xsd:dates, xsd:times, and xsd:dateTimes. For example:

## Find subjects modified since yesterday  
SELECT * {  
  bind( now() - 'P1D'^^xsd:duration as ?yesterday )  
  ?s :hasDate ?date .  
  FILTER( ?date > ?yesterday )  
}

AllegroGraph supports adding durations to date/times, adding date/times to durations and subtracting durations from date/times.

Bug25709 - Predicate statistics invalidation is incomplete

The query engine could incorrectly optimize away SPARQL type casting operations. This has been fixed.

Bug25669 - Logical ORs of inequality filters could produce too many solutions

SPARQL FILTERs that logically OR'd multiple inequality constraints together could be incorrectly processed leading to too many solutions. This has been corrected.

Bug25634 - Impossible OPTIONALs can lose solutions in some cases

If a query began with an OPTIONAL clause and that clause found no bindings and the query then required subsequent processing (e.g., via a BIND() clause), then solutions could be lost. This has been corrected.

Bug25564 - SPARQL queries that used constant expressions could fail over HTTP

AllegroGraph v6.4.2 introduced the eager evaluation of constant expressions (like bind(iri('string') as ?value)). This improvement sometimes caused failures when a query plan with these constants was reused because the value of the expression was not being saved by the plan which would lead to UPI is not in string table errors. This has been corrected.

Bug25545 - UUID() and STRUUID() should not be evaluated eagerly

The SPARQL UUID() and STRUUID() functions were being evaluated at plan time rather than query time. This has been corrected.

Bug25501 - Fix bug in t:intervalCotemporal using start and end times

The SPARQL magic property for t:intervalCotemporal worked correctly when using points on the time line (t:startpoint and t:endpoint) but not when using actual times (t:starttime and t:endtime). This has been corrected.

Bug25465 - Constraint analysis could cause queries with SERVICE clauses to fail

AllegroGraph's planning step could fail to process queries with SERVICE clauses that included FILTERs if those FILTERs attempted to query the current repository. This has been corrected.

The problem was introduced in AllegroGraph v6.4.1.

Bug25454 - Problem with SPARQL division by zero

If the operands to a SPARQL divide arithmatic expression were of type xsd:double, then AllegroGraph could incorrectly signal a SPARQL type error if the divisor was 0 instead of returning positive or negative infinity.

Bug25396 - SPARQL parser bug with unnamed aggregate expressions

AllegroGraph v6.4.1 introduced a bug such that the SPARQL parser would fail to parse a GROUP BY expression that did not provide a new variable. For example:

SELECT (count(?s) as ?count) {  
  ?s ?p ?o .  
} GROUP BY str(?o)

Queries that did introduce a variable like:

SELECT ?foo {  
  ?s ?p ?o .  
} GROUP BY (str(?o) AS ?foo)

would parse correctly. This has been corrected.

Bug25391 - SPARQL (count(distinct *) ...) can get the wrong answer

It was possible for SPARQL queries using COUNT(DISTINCT *) to produce the wrong answer. This has been corrected.

Bug25352 - Blank nodes from planning and parsing could incorrectly overlap

If a query used both anonymous blank nodes and property paths, then it was possible for the blank node bookkeeping to incorrectly assign identifiers. For example, a query like:

select * {  
  [ a ?Class ] .  
  ?x :a/:b ?c .  
}

could be parsed as

select * {  
  _:b0 a ?Class .  
  ?x :a _:b0 .  
  _:b0 :b ?c .  
}

instead of

select * {  
  _:b0 a ?Class .  
  ?x :a _:b1 .  
  _:b1 :b ?c .  
}

This problem has been corrected.

Bug25342 - Alternation in Property Paths could return incorrect bindings

If a SPARQL query used alternation in a property path, it was possible that AllegroGraph would bind the subject side of the path to the variable on the object side. This has been corrected.

Bug25334 - SPARQL parser's variable classification was too strict

AllegroGraph was signaling an error if a GROUP BY variable was renamed via a SELECT expression. For example, the following valid query would fail to parse

prefix : <http://example.org/>  
SELECT  (?entry AS ?other) (MIN(xsd:date(?date)) AS ?minDate) {  
  ?item a  :InternalProcedure .  
  ?entry (:logicalSuccessor)+ ?entry .  
  ?entry :eventDate ?date  
} GROUP BY ?event

This has been corrected.

Bug25198 - Variable mismatch for (run-sparql "ask ..") as :alists

There was a mismatch in the return values for:

(run-sparql "ASK .." :results-format :alists)

as the alist (first value) could be:

((?result . t))  
where the variable names (third value) would be: (?found)

Now both use the db.agraph.query.variables::?result symbol.

Bug24650 - Improve SPARQL parser's validation of VALUES clauses

AllegroGraph is now more strict when evaluating VALUES clauses where the number of variables in the clause does not match the size of each row of values.

Bug23066 - XSD casting could fail when used with expressions

If an XSD casting operator was used on the result of an expression, the result could be returned as nil. This has been corrected. For example, the following query used to fail with SPARQL type errors:

select ?value {  
  values (?a ?b) { (1 2) }  
  bind( xsd:double( ?a + ?b ) as ?value )  
}

Bug22548 - Add page rank centrality to Prolog and SPARQL

AllegroGraph now supports computing page rank centrality via the page-rank-centrality Prolog function and also via a new magic property: (link http://franz.com/ns/allegrograph/4.11/sna/pageRankCentrality).

Bug19805 - Dynamic restriction reasoner could fail to infer some triples

The dynamic restriction reasoner was only inferring rdf:type triples when either the subject or the object (or both) was supplied. It would not make inferences when both sides were wild. This has been corrected. Note that inferring all type triples in a query like:

SELECT ?o ?c { ?o a ?c }

can be expensive due to the amount of computation required.

AGWebView

Bug25510 - Bug in reposcript handling in AGWebVeiw

Previously if you saved or loaded a reposcript from the Utilities | Scripts page, the page would change to one which does not show reposcripts at all. This has been fixed.

Bug25336 - 'Cancel' button does not work in 'Import RDF from a text area input' form

The 'Cancel' button on the 'Import RDF from a text area' dialog used not to work unless an upload had been started. This has been corrected.

Bug25164 - Show metadata in WebView for Parallel SPARQL queries

Parallel SPARQL queries run in WebView against a federated triple store would not show query metadata in the "Information" tab. Now the tab shows a list of the query metadata for each shard of the federated store.

Changes to the Lisp API

Rfe15933 - Remove variable get-triples-list-limit

The variable *get-triples-list-limit*, if not nil, used to limit the number of triples returned by function get-triples-list. As get-triples-list already takes an explicit limit keyword argument, the variable has been removed.

Rfe15879 - Remove RUN-SPARQL :ENGINE argument, VALID-QUERY-ENGINES documentation

Since there is only one SPARQL engine in the current release, the :engine keyword argument to run-sparql and the operators get-sparql-query-engines and valid-query-engines have been removed.

Rfe15870 - Error when a query returns a cursor and there is no query environment

If run-sparql returns a cursor, then it is imperative that the code using that cursor be wrapped with the with-query-environment macro. The lack of wrapping can lead to hard to diagnose problems.

Rfe15828 - Remove preserve-strings and return-[non-]encoded-triples arguments

The Lisp API has long included a preserve-strings argument to the various functions that add triples to a repository (such as add-triple, load-ntriples, and others). When true, both a string version and an encoded version of triples whose objects were typed literals were added. Similarly, functions that retrieved triples (such as get-triples) included arguments to request either only the encoded version or the non-encoded version of said triples (return-encoded-triples and return-non-encoded-triples respectively). These three arguments have been removed from the API.

Rfe15603 - Improve behavior of db when closing an encapsulated triple store

The behavior of variable db when closing or deleting a triple store was not always intuitive. The new behaviour is that (close-triple-store :db db) only updates the binding of the variable db if it had db as its value before the call.

The new value of db depends on the type of db:

if db is an encapsulated store (like a reasoning store) then db is afterwards set to the inner store (in a sense unwrapping the encapsulation);
otherwise db is set to nil.

Rfe15282 - Allow warmup of individual index flavors

warmup-triple-store now accepts an indices keyword argument to allow specifying which indices to warm up. When the include-triples is true, indices specifies which indices to warm up. :indices nil or unspecified means warm up all indices.

Bug25553 - optimize-indices on a remote-triple-store ignored the wait-p argument.

In the lisp client, if optimize-indices was called on a remote-triple-store with :wait-p t, a non-True boolean value was passed in the HTTP request generated by the call. As a result, the call would return before the optimization operation had completed.

This is now fixed.

Bug25401 - Fix hanging (get-triples :return-extended-triples t) on remote store

Retrieving triples from a remote-triple-store could indefinitely hang if the number of triples was large (typically tens of thousands). This has been resolved.

Prolog

Bug24993 - Default graph in query results

Prolog queries that return the default graph object in a binding caused an error if CSV, TSV or TTL was used as the output format. Such queries also failed to display results in AGWebView. This has been corrected. Any variable bound to the default graph will be treated as unbound.

Bug22548 - Add page rank centrality to Prolog and SPARQL

Documentation

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.4.6

AllegroGraph Server

Bug25699: Restoring upi-string got illegal value during recovery/replication

AllegroGraph 6.4.0 introduced a bug that could result in the following error message during crash recovery or when using replication:

restoring upi-string got illegal value [xxx]

This could prevent a repository from starting up after a crash. This has been fixed.

Bug25653 - A multi-master repl repo that died will restart the next time it is opened

A multi-master repl instance repo (see Multi-master Replication) will remain open until it is explicitly stopped or for the period of time when the agraph server is not running. If an instance dies for some reason and a request comes in to open its repo then the repo now will open and rejoin the cluster. Previously an explicit start was required to start the instance and open the repo.

AGWebView

Rfe16002 - Move query from URL to body

Previously in AGWebView when running a query or downloading query a result, the query was in placed the request URL. This caused problems with long queries with HTTP proxies that limit the length of URLs. Now the query text is placed in the request body.

AllegroGraph 6.4.5

AllegroGraph Server

Bug25586 - Error during recovery of partially-created text index

Previously it was possible for a crash to occur when restarting a repository instance if a partially-created text index existed during at the time of the last checkpoint. This has been fixed.

SPARQL

Bug25564 - SPARQL queries that used constant expressions could fail over HTTP

AllegroGraph v6.4.2 introduced the eager evaluation of constant expressions (like bind(iri('string') as ?value)). This improvement sometimes caused failures when a query plan with these constants was reused because the value of the expression was not being saved by the plan which would lead to UPI is not in string table errors. This has been corrected.

Bug25545 - Uuid() and STRUUID() should not be evaluated eagerly

The SPARQL UUID() and STRUUID() functions were being evaluated at plan time rather than query time. This has been corrected.

AllegroGraph 6.4.4

AllegroGraph Server: Multi-master replication changes

Rfe15786 - Changes while adding a multi-master replication instance

In earlier releases, when a new MMR instance was added, it could not be removed until the addition completed. This caused problems when the addition took excessive time. Now additions can be removed at any time after they are added. Also, the Replication Cluster Management page in AGWebView would not always display correctly while a MMR instance was being added. That has been fixed.

AllegroGraph 6.4.3

AllegroGraph Server: Multi-master replication changes

Rfe15762 - In repl mgr do timeout for repl/stop http calls

Sending an HTTP request to non-existent machine can sometimes take a long time to timeout. The repl/remove and repl/stop commands for Multi-master Replication now accept a timeout parameter to control how long they wait for the affected host to respond before giving up or moving on.

Rfe15737 - Add config directive to set HTTP keep-alive timeout

The new config directive HTTPKeepAliveTimeout specifies the HTTP keep alive timeout. Configuration directives are described in the Server Configuration and Control document. HTTPKeepAliveTimeout is a top-level directive.

Rfe15712 - Improve replication temporary commit error

When it's not possible to commit at the moment due to replication parameters not being satisfied, an error will be signaled that will not backtrace to agraph.log and that will return a 409 "conflict" error from HTTP.

Bug25397 - Replication restart of two instances at the same time

Previously when two replication instances start up at nearly the same time they can get into a state where they fail to link up. This has been fixed.

Bug25364 - Return x-durability header in commit via HTTP

When an HTTP command results in a commit, "x-durability: missing=N" (where N is the number of of instances that are left to commit to satisfy durability) is supposed to be HTTP response headers but it was not. This has been fixed.

AllegroGraph 6.4.2

AllegroGraph Server: General Changes

Rfe15599 - Disallow creation of repos within the system catalog

Attempts to create repositories in the system catalog will now be met with the following error message:

Repositories cannot be created within the 'system' catalog

Rfe15528 - Increase speed of finding distinct predicates or graphs in a reasoning triple-store

It is now much more efficient to find distinct predicates or graphs when RDFS++ or restriction reasoning is enabled.

Rfe15492: Use triple-store-spec for certain agtools

The following agtools (see agtool) have had their syntax changed to use a triple-store specification instead of the options --port, --user, --password, --host, and --catalog. The affected tools are: cancel-purge-deleted-triples, define-attribute, delete-static-attribute-filter, get-metadata, lookup-attribute-definition, memory-lock, memory-unlock, purge-deleted-triples, purge-rate-limit, set-purge-rate-limit, set-static-attribute-filter. The triple-store specification is

http(s):[//[USER:PASSWORD@]HOST[:PORT]][/catalogs/CATALOG]/repositories/REPO

Items in brackets ([]) may be left out in some circumstances depending on what is required for the tool.

Rfe15482 - Change if-exists default for agtool repl create-cluster

In agtool repl create-cluster --if-exists argument used to default to supersede and now defaults to error. Existing scripts using agtool repl create-cluster may now fail to work until --if-exists supersede is added to the argument list.

Rfe15481 - Improve turtle serializer output for numbers

AllegroGraph's Turtle and TriG serializers now emit integers, decimals and double floats using the more concise numeric representation rather than the full XSD typed literal one.

Rfe14740 - agtool can now manipulate users and roles

Added agtool user and agtool role commands to manipulate user accounts and roles respectively. For more infomation, execute:

agtool user --help  
agtool role --help

See the Managing Users document for further information.

Rfe11183 - Add --optimize option to agload

agtool load now accepts the --optimize option to request full (level 2) index optimization after loading completes. See the agtool load document. Index optimization is discussed in the Optimizing indices section of the Triple Indices document.

Bug25289 - Dropping a text index deletes files too early

Dropping a freetext index could result in instance startup errors if the instance crashed before a checkpoint occurred. This has been fixed.

Bug25288: Free Text Index state updates not recorded in tlog during commit

Starting in AllegroGraph v6.3.0, changes in text index configuration (such as creation and removal) were not stored in the transaction log during commit. This could result in disappearance of an added text index, or reappearance of a deleted text index after a repository crash restart. This problem is now fixed.

Bug25266 - sna:egoGroup function does not find its generator in all cases

The functional form of the sna:egoGroup magic property did not look up and register its SNA generator function if the generator was stored using triples. This has been corrected.

Bug25236 - Turtle serializer could emit invalid qnames

It was possible for the Turtle serializer to produce invalid output if resources contained characters that needed to be escaped. This has been corrected.

Bug25207 - Using Japanese tokenizer can result in errors

Using the Japanese text index tokenizer on a system that does not have the mecab package installed would result in an error like the following:

tagger.cpp(153) [load_dictionary_resource(param)] param.cpp(71)  
[ifs] no such file or directory: /etc/mecabrc

This has been fixed.

Bug25201 - agtool export output depends on the value of LANG

It was possible for the value of the LANG environment variable to alter the output of agtool export. This has been corrected.

Bug25138 - agtool CMD will fail if run from a directory it cannot write to.

If the current working directory is not writable agtool query and agtool materialize would stop running with an error the first time they try to write to their respective log files. Now this behavior has been changed to warn the user that it was unable to write to the log file and then continues processing. It continues logging to the terminal as it would normally do. See agtool General Command Utility for a description of agtool.

Bug25115 - 'agtool export' errors when using --parallel and exporting NQX

Attempting to use agtool export with --output nqx and --parallel mode would result in an error like the following:

error `#<consumer-cursor @ #x1000991a312> does not support extended-triples`  
occurred in worker 11  
interrupting...

This has been fixed.

HTTP API

Bug25246 - Improve HTTP socket reuse for remote repositories

Queries on remote triple stores now reuse the same underlying HTTP socket connection for efficiency.

SPARQL

Rfe15644 - Improve probing decision

AllegroGraph decides at query time whether it is more efficient to scan or probe to evaluate a triple pattern. However, if a pattern had certain constraints on some of its variables, then AllegroGraph was unable to probe. Now AllegroGraph is able to probe in more cases which can greatly improve performance for particular queries.

Rfe15628 - Handle query + filter patterns more efficiently

AllegroGraph now handles SPARQL queries where a pattern in a BGP can act as a filter on the existing set of solutions. As an example, consider this query:

:target :select ?s .  
?s :has ?property .  
?property rdf:type :animal .

Prior to this change, AllegroGraph would first find all <?s :has ?property> triples and only then filter the results using the <?property rdf:type :animal> pattern. Depending on the shape of the dataset, this could generate a large number of intermediate results. AllegroGraph is now able to treat the <?property rdf:type :animal> pattern as a filter while evaluating the <?s :has ?property> pattern which can help speed the query.

Rfe15607 - Sort language tagged literals sensibly

The SPARQL 1.1 standard does not define an ordering on language tagged literals. AllegroGraph now orders language tagged literals by first comparing their languages and then their string values. Plain and xsd:string typed literals come before any language tagged literal.

Rfe15600 - Remove and warn about unused BINDings

AllegroGraph now issues a query warning if variables introduced by a BIND clause are not used elsewhere in the query. For example:

select distinct ?o {  
  bind(23 as ?number)  
  ?s ?p ?o .  
}

Rfe15541 - Warn when a BIND form is extending an empty set of solutions

The SPARQL BIND construct operates by extending an existing set of bindings. If there are no bindings, then there is nothing to extend and so the assignment will uniformly fail. AllegroGraph now warns when it encounters this situation. For example,

SELECT ?ABC {  
  BIND(?s AS ?ABC)  
  ?s ?p ?o .  
}

Previously, AllegroGraph was incorrectly rearranging the above query so that it would return bindings for ?ABC. Now the above query will leave ?ABC unbound and emit a :warn-empty-bind-clause warning.

Rfe15524 - Improve SPARQL property path efficiency

The data structures used to compute SPARQL property paths have been improved to provide better performance.

Rfe15523 - Improve SPARQL dataset efficiency

SPARQL datasets are now slightly more efficient when processing queries.

Rfe15522 - Use existing constant bindings when evaluating subsequent ones

AllegroGraph now uses existing constant BINDings when evaluating subsequent ones during query planning rather than waiting until query execution.

Rfe15515 - Support variable attributes in SPARQL attributes block

AllegroGraph SPARQL now supports variable attributes for SPARQL update operations. For example, the query:

prefix attr: <http://franz.com/ns/allegrograph/6.2.0/>  
insert {  
  attributes ?attr { ?s ex:newPredicate ?o . }  
} where {  
  ?attr attr:attributes (?s ex:oldPredicate ?o }  
}

would replace each triple whose predicate was ex:oldPredicate with one having the predicate ex:newPredicate and the same attributes.

Rfe15501 - Make better use of FILTER constraints in property path queries

AllegroGraph now uses constraints derived from SPARQL property path queries in more cases. This can help to improve the performance of some queries.

Rfe15413 - Improve handling of constant BIND expressions

AllegroGraph now analyzes constant expressions in BIND forms and can use the results to improve the efficency of some queries.

For example, AllegroGraph previously would execute only the first of the following two logically equivalent queries efficiently:

# BIND with a constant  
select * {  
  bind( <http://example.com#Mammal> as ?Class )  
  ?a rdf:type ?Class .  
}  
 
# BIND with a constant _expression_  
select * {  
  bind( iri('http://example.com#Mammal') as ?Class )  
  ?a rdf:type ?Class .  
}

Now both of the queries will execute efficiently.

Rfe15381 - Allow federated geospatial query if one store has no geospatial mappings at all

When executing SPARQL geospatial query on federated-triple-stores an error was thrown if federated stores had different geospatial type mappings. This was an overly strict limitation, so that error is now only thrown when regular type mappings differ.

Bug25271: Fix "agtool query" on SPARQL endpoint

Running a query on a SPARQL endpoint using "agtool query", e.g.

./agtool query http://dbpedia.org/sparql

would, after printing results, signal the error:

Error "http://dbpedia.org/sparql" is not a triple store occurred during setup

This has been resolved.

Bug25268 - Correct handling of query log line length

It was previously impossible to set the query log line length using the logQuery query option. The log length can now be set with the new logLineLength query parameter.

Bug25247 - Some SPARQL sub-queries with external filters could fail.

AllegroGraph could fail to correctly execute SPARQL queries when they involved a sub-query and FILTERs that applied to triple-patterns within the sub-query. For example:

select ?s ?timestamp {  
  {  
    select distinct ?s ?timestamp {  
      ?s allegro:hasCreationDate ?timestamp .  
    }  
  }  
  FILTER (?timestamp >= '2018-04-12T20:32:53.213Z'^^xsd:dateTime>)  
  FILTER (?timestamp <= '2018-04-15T20:32:53.213Z'^^xsd:dateTime>)  
}

This has been corrected.

Bug25170 - Property path analysis of missing constants is too stringent

AllegroGraph was marking some property path queries as if they were impossible when elements of the path were not found in the repository. This led to some queries failing to return results. The problem has been corrected.

Bug25197 - Fix handling 'select *' results from remote endpoints

Queries of the form select * { .. } sent to a non-AllegroGraph endpoint could fail with a "result variables are in wrong order" error even though the query results sent by the endpoint were valid. This has been fixed.

Bug25169 - Impossible NOT EXISTS clauses should not make query impossible

The logic that notices when parts of a SPARQL query cannot succeed and marks them as impossible was incorrectly treating NOT EXISTS filters as if they were EXISTS filters which led to some queries failing to return results. This has been corrected.

Bug25151 - Problem with complex NOT EXISTS filters

Queries that used [NOT] EXISTS in a FILTER could lead to an error if the query inside the [NOT] EXISTS contained language filters.

Bug22482 - Improve property path estimates with fixed subject/object

AllegroGraph now does a much better job of estimating the cost of transitive closure (i.e., zero-or-more and one-or-more) property path queries. These improved estimates can lead to greatly improved plans.

AGWebView

Rfe15410: Add repository name and group columns to the AGWebView cluster table

On the replication manager page in Webview columns were added to display the repository name and catalog as well as the group of each instance. See the Managing Clusters Using AGWebView section for information on the WebView cluster table.

Bug25146: Ok button never reenabled after failed text area load

In AllegroGraph WebView when importing statements from a text area input the OK button didn't reenable after the loading process had failed. This has been corrected.

Changes to the Lisp API

Rfe15547 - Add type checking to the upi function

The upi function now includes type checking so that it will signal a clearer error message if passed an argument of the wrong type. The function is now also defined on UPIs in addition to future-parts.

Rfe15499 - delete-triple improvements

delete-triple now accepts a triple object as argument, as alternative to a triple id.

In either case the return value now indicates whether the deletion was successful.

Rfe15483 - Add part-to-namespace to the Lisp API

The Lisp API now includes part-to-namespace. This function takes a UPI, a future part, or a string representing a URI (with or without angle brackets) and returns the Prefixed Name and Local Part (as multiple values) if a registered namespace can be found to abbreviate the resource. For example:

(part-to-namespace "<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>")  
==>  
"rdf"  
"type"

Rfe14068 - Let run-sparql return query metadata for remote triple stores

Calling the run-sparql function on a remote triple store now returns query metadata like timing information as fourth value, just as for local triple stores.

Rfe8199 - Add a with-open-triple-store convenience macro

The Lisp API now includes with-open-triple-store macro which opens a triple-store, executes the body, and then closes it.

Bug25305 - Export cursor-peek from db.agraph package

The cursor-peek function was introduced in AllegroGraph v6.4.1 but was not exported from the db.agraph package. The symbol is now exported and also available in the db.agraph.user package.

Bug25192 - materialize-entailed-triples was not supported in the remote Lisp client

Both materialize-entailed-triples and delete-materialized-triples are now available in the remote Lisp client.

Bug25168 - Fix leaked http-cursor finalizer clash

Cursors can be leaked, for example by using return-from inside iterate-cursor. These cursors are eventually garbage-collected. There was a finalizer race condition in the collection of HTTP connection based cursors, that would lead to stream-closed-error being signaled to the user unexpectedly. This has been resolved.

Bug25168 - Improve cursor leak message for get-triples on remote-triple-store

If get-triples is called on a remote-triple-store outside with-query-environment, the cursor will open an HTTP socket to the triple store. If the cursor is leaked it will be garbage-collected eventually, but until that moment the socket will be kept open. This can negatively impact performance, therefore a warning will now be logged:

[2018-03-12T09:06:27.250 p13791 W cursor] Cursor created by  
(get-triples) on remote-triple-store "db" was leaked,  
will be closed.

Bug25166 - remote-triple-stores reason on the client rather than the server

AllegroGraph now ensures that the dynamic reasoner will execute on the server rather than the client when using a remote-triple-store. This prevents a large number of get-triples from executing via HTTP when using the RDFS++ reasoner.

Prolog

No significant changes.

Documentation

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.4.1

AllegroGraph Server: General Changes

New agtool commands and some changed arguments

The *agtool command-line utility has new commands available. Further some commands have new arguments and some arguments have changed. See the agtool General Command Utility document for details. Most commands now have a --scheme argument which can be http or https.

agtool repl create-cluster adds --if-exists and --force arguments

agtool repl create-cluster previously would only create new, empty repositories and if an existing repository were specified, it would be deleted and a new, empty repository with the same name would be created. (This behavior differed from the behavior when creating clusters with AGWebView, where a repository had to exist and need not be empty when it was converted to an initial replication instance.) Now agtool repl create-cluster accepts an --if-exists argument with allowable values supersede (the default, meaning delete the existing repository and create a new one with the same name), overwrite (same as supersede), use (convert an existing, possibly non-empty repository), and error (signal an error if the repository specified already exists).

If use is specified, then any encoded id definitions will be examined and if any do not allow for 2^60 distinct strings and so will not work with next-encoded-upi-for-prefix, the conversion will fail and a warning will be printed. If the --force argument with value true is supplied, then the conversion will be done but none of the offending encoded id prefixes must be used again.

The Cluster Replication Manager page now displays statistics

These help users to monitor the state of the cluster. See the Managing clusters in AGWebView once created section in the Multi-master Replication document.

When adding a cluster instance in AGWebView, a progress log is displayed

When adding an instance in Webview you'll see that the log of actions is updated with progress reports describing what is being done. Also the AGWebView replication manager page has new columns and some columns have been renamed. See here for information on using AGWebView to manage clusters.

Rfe15465 - Disallow setting bulk-load-mode in replication instance

In a replication instance one cannot set bulk load mode since replication depends on the transaction logs being an accurate reflection of the changes to the repository and logging is disabled in bulk loading mode. Specifying the --bulk option to agtool load or clicking the Use bulk load mode checkbox when adding triples from an uploaded file in AGWebView will cause the load to fail.

Rfe15451 - Don't fail if unable to write to agload.log

If the current working directory is not writable agload would stop running with an error the first time it tries to write to the log file agload.log. Now this behavior has been changed to warn the user that it was unable to write to the log file and then continues processing. It continues logging to the terminal as it would normally do.

Rfe15426 - Expose and document the --parameter option for agtool load

agtool load now accepts the --parameter option for controlling the configuration of newly created repositories. See the --parameter option in the agtool load document for details.

Rfe15418 - Change StringTableCompression default to 'none'

AllegroGraph now in the default creates new repositories with string compression disabled. This results in faster out-of-the-box operation. String compression can be enabled by using the StringTableCompression directive in agraph.cfg. See the description of StringTableCompress in the Server Configuration and Control document for information.

Note that the string compression type can only been set when creating a new repository. The StringTableCompression parameter is further discussed here in the Performance Tuning document.

Rfe15190 - Remove --repl argument from agtool archive backup and agtool recover

The --repl argument for agtool archive backup and agtool recover is no longer present because it is no longer needed (whether or not a repository is a replication repository is now automatically determined).

Rfe14770 - Error message from agload loses most of the detail

When using agtool load the error messages signaled while loading triples were truncated. This is now corrected.

Rfe14389: AllegroGraph no longer supports Callimachus

AllegroGraph no longer supports Callimachus. Stores that were previously set up to integrate with Callimachus will open normally but they will no longer execute the Callimachus integration code.

Bug25087 - TempDir is not being used for placing temporary query files

Temporary query files were being placed in the repository's directory rather than in the location specified by the TempDir configuration parameter. This has been corrected.

Bug25075 - Bidirectional search can return bad paths

If there were many possible overlapping bidirectional paths between two nodes, AllegroGraph could return incorrect paths. This has been corrected.

Bug25102 - Subscript UPIs did not print correctly.

AllegroGraph was printing the prefix of a subscript UPIs twice. This has been corrected. Previously, they would print as

<http://franz.com/ns/allegrograph/4.14/index#_http://franz.com/ns/allegrograph/4.14/index#_4>

Now they print as

<http://franz.com/ns/allegrograph/4.14/index#_4>

Bug25062 - Fix uninformative error when running agraph-control as root.

When running agraph-control as root when there was no RunAs directive in agraph.cfg would result in the following uninformative error:

Got error while executing plan: `nil' is not of the expected type  
`(array character (*))' for foreign type `(* :char)'

Now the error messages generated for no RunAs user and non-existent RunAs user, respectively, are:

# bin/agraph-control --config lib/agraph.cfg start  
Running AllegroGraph as superuser is not recommended for security reasons.  
It is recommended that another less privileged user be used, specified  
via the `RunAs' directive in agraph.cfg.  
 
# bin/agraph-control --config lib/agraph.cfg start  
RunAs user `UserThatDoesNotExist' does not exist in the passwd database.  
 
Please run the AllegroGraph configuration script or adjust the `RunAs'  
directive in lib/agraph.cfg

HTTP API

Rfe15418 - Allow specifying string table compression method

Added a stringTableCompression parameter to the repository creation HTTP service.

SPARQL

Rfe15421 - Add control over duplicate triple reduction in SPARQL queries

AllegroGraph now has a new query option: enableDuplicateReduction which controls whether or not the SPARQL engine reduces the number of duplicate triples it processes. See SPARQL Query Options in the SPARQL Reference document.

If the store has no duplicate triples (based on subject, predicate, object, and graph) then this option has no effect.

SPARQL is defined on sets of triples (i.e., no duplicates) but AllegroGraph can store duplicates which can lead to duplicate rows of bindings returned by a query.

If this option is set to true, then AllegroGraph will ignore consecutive duplicate triples returned from the storage layer. Since the storage layer makes no guarentees on the order of triples it returns, this can lead to different results from the same query.

Note: Previous versions of AllegroGraph used duplicate reduction by default. This version turns if off by default. If your triple-store has duplicate triples, then this may change the results of your queries.

Rfe15419 - Include query modifiers when printing a SPARQL query plan

AllegroGraph now includes query modifiers like ORDER BY or LIMIT when it prints a query plan.

Rfe15391 - Warn when the variable HOST of a SERVICE clause is not bound

AllegroGraph now warns when the HOST variable of a SERVICE clause is unbound in the query.

For example, both of these queries will produce query warnings:

# Query 1  
select * {  
 {  
   :myService :isAtURL ?url  
 } union {  
   service ?url { ?s ?p ?o }  
 }  
}  
 
# Query 2  
select * { service ?unknown { ?s ?p ?o } }

Rfe15354 - Improve handling of queries containing GRAPH and JOIN clauses

AllegroGraph will now handle queries involving GRAPH and JOIN clauses more efficiently.

Rfe15336 - Sanity check on JSON/XML data returned by SPARQL endpoint

If a SPARQL query includes a SERVICE call, the XML and JSON result data is now subject to more syntactic and semantic error checking. Nevertheless invalid result data from third parties might still lead to unexpected outcomes.

Rfe15273 - Add Javascript API for defining SPARQL extensions

AllegroGraph's server side Javascript engine now supports the following:

sparql.extensionFunctions() - return a list of SPARQL extension function names.
sparql.defineExtensionFunction(uri, function) - define a new extension function associated with uri.
sparql.magicProperties() - return a list of magic property URIs.
sparql.defineMagicProperty(uri, parameters) - define a new SPARQL magic property.

For details, see the Javascript documentation.

Bug25145 - Illegal keyword error when executing some queries

AllegroGraph could signal an Illegal Keyword error when executing some complex queries. This has been corrected.

Bug25113: SPARQL planner stopped optimizing certain queries

If a query has a FILTER like

FILTER( ?var1 = ?var2 )

and it can be determined that both ?var1 and ?var2 are resources, then AllegroGraph can perform certain query optimizations. This feature was inadvertently broken in AllegroGraph 6.4 and has now been reinstated.

Bug25097 - Graph not in dataset warning signaled inappropriately

AllegroGraph v6.3.0 and following signal a query warning if the graph in a GRAPH clause is not in the dataset as in, for example, this query:

select *  
from named <http://example.com#graph> {  
  graph <http://example.org#otherGraph> { ?s ?p ?o }  
}

AllegroGraph was signaling this warning more often than necessary. This has been corrected.

Bug25064 - GROUP BY expressions in a SUB-QUERY causes an error

AllegroGraph was signaling an error if a SPARQL query used GROUP BY expressions in a SUB-QUERY. This has been corrected. An example query that used to fail would be

select * {  
  { select ?strWho (sum(?amount) as ?sum) {  
      ?who ?p ?amount .  
    } group by (str(?who) as ?strWho)  
  }  
}

Bug25057 - Values and OPTIONAL could lead to too many results

If a SPARQL query used a VALUES form inside the query and the VALUES form had only a single variable and a single binding for that variable and the query also used an OPTIONAL form that made use of the same variable, then AllegroGraph could return results with the variable unbound. This has been corrected.

As an example, AllegroGraph previously could have returned results with ?o2 unbound for the following query:

SELECT ?s ?o1 ?o2 {  
  ?s ?p1 ?o1  
  OPTIONAL { ?s foaf:knows ?o2 }  
  VALUES (?o2) { (:b) }  
}

Bug25055 - Improve SPARQL parser error message for bad aggregation expression

AllegroGraph was signaling an error for bad aggregation expressions rather than indicating that the query failed to parse. For example, this query is invalid because it is illegal to GROUP BY an aggreagate:

select (sum(?o) as ?sum) (sample(?s) as ?ss) {  
  ?s ?p ?o .  
} group by (sum(?o) as ?xxx) having (?xxx > 2)

AllegroGraph now signals a SPARQL parser error for queries like this.

Bug25034 - Catch additional SPARQL parse errors

SPARQL is defined such that repeated variables in the projection are ignored. Previously, AllegroGraph would include each repeated variable. Now, it will include only the first instance and will warn if the list includes repetition.

For example, this query:

SELECT ?s ?g ?s { graph ?g { ?s ?p ?o } }

Will project only ?s and ?g and will warn that "The following variable is projected multiple times: ?s. Only the first projection will be included".

The variables in expressions and aggregation specifications may not overlap with other projected variables. AllegroGraph now catches errors like:

SELECT ?o (str(?o) as ?o) {  
  ?s ?p ?o .  
}

SELECT ?s (count(1) as ?s) {  
  ?s ?p ?o .  
} GROUP BY ?s

Previously, the above queries would have passed the parser.

AGWebView

Rfe15395 - Highlight all instances of the token under cursor in the query editor

AGWebView's SPARQL query editor now highlights any tokens that match the one currently under the cursor. This makes it easier to ensure that variable names are not mistyped.

Changes to the Lisp API

rfe15430 Use of next-encoded-upi-for-prefix in multi-master cluster instances

In order to use next-encoded-upi-for-prefix when the repository is a multi-master cluster instance, the template associated with the encoded id prefix must generate exactly 2^60 distinct strings. If it does not, the call to next-encoded-upi-for-prefix will signal an error. As noted just below, the "plain" will work as a template. See register-encoded-id-prefix.

Rfe15466 - Plain format encoded ids

register-encoded-id-prefix will treat the format argument of "plain" (a string) in this special way: The encoded id (the string after the @@) is a variable length integer in the range from 0 to (2^60 -1), much like blank nodes. This is a template suitable for use in multi-master cluster instances. See the Encoded IDs document.

rfe15430 Use of next-encoded-upi-for-prefix in multi-master cluster instances

In order to use next-encoded-upi-for-prefix when the repository is a multi-master cluster instance, the template associated with the encoded id prefix must generate exactly 2^60 distinct strings. If it does not, the call to next-encoded-upi-for-prefix will signal an error. As noted just above, the "plain" will work as a template. See register-encoded-id-prefix.

Rfe15379 - Remove in-line-data argument from the run-sparql

The in-line-data argument to run-sparql is no longer supported. If you need to execute the same basic query with different sets of BINDINGS, then use the VALUES form in your SPARQL query string.

Rfe15373 - Export peekable-cursors

The Lisp API now includes these new functions:

A peekable-cursor is like a regular cursor except cursor-peek can be used to see what cursor-next-row will return without modifying the state of the cursor. See the documentation for details.

Bug25119 - load-turtle not accepting a :base-uri arg in remote lisp client

The remote lisp client did not support the base-uri argument to load-turtle. This has been corrected.

Bug25070 - upi->value uses db when decoded encoded IDs

The upi->value function was using db to decode encoded IDs even when a specific value for db was passed into the function. This has been corrected.

Bug25043 - get-triples-nd-* functions were incompletely documented

The various nD get-triples functions (like get-triples-nd-within-bounding-box) return :part-not-found as a second return value if it can be determined that one of the search parameters is definitely not in the repository. This is now documented.

Bug23635: Make Lisp client :default-graph-uris keyword work with remote-triple-stores

SPARQL queries made against remote-triple-stores via run-sparql, that passed the :default-graph-uris keyword argument, would not return expected results. This has now been fixed.

Prolog

No significant changes.

Documentation

No significant changes.

nD Geospatial

Bug25129: Bus error on geospatial search with a non-existing predicate

The bus error on the server happened when trying to obtain triples from polygon using predicate that was not previously added to the store.

Behaviour for obtaining triples from polygons was modified to properly handle this case.

AllegroGraph 6.4.0

AllegroGraph Server: General Changes

Rfe15302 - Remove support for using rapper as an external parser

The use-rapper argument to the AllegroGraph data import commands has been removed because AllegroGraph now supports all of the standard formats.

The rapper command line argument to agtool load has similarly been removed.

Rfe15219 - Add timeout and limit arguments to the SNA path-finding functions

The SNA path finding functions now have a timeout argument and a limit argument. The first specifies the number of seconds to execute the search whereas the second indicates that searching should stop once that many paths have been found.

The SNA path functions will return (as a second value) the status of the search. This will be either nil if the search was not interrupted, :timeout if the search reached the timeout, or :limit-reached if the limit on the number of paths was reached.

Note that the return format for the REST API to the SNA path functions is now versioned. The desired version if specified via a version parameter in the query string. If the version is 1, then the previous result format of a list of lists representing paths will be returned. Otherwise, the new format will be used. The new format consists of a JSON object with two keys: "paths" and "status". "paths" contains the list of paths and "status" contains the reason that the search completed. It will be one of "complete", "timeout", or "limit-reached".

Bug24980 - Turtle serializer escapes the PNAME_NS incorrectly

The Turtle serializer was escaping both the namespace portion and the local portion of a qualified name using the same logic. This was incorrect as the escaping for the two differ. The problem has been corrected.

HTTP API

No significant changes.

SPARQL

Rfe15337 - Have the NOW() SPARQL function include a timezone

The SPARQL NOW() function now returns dateTimes in GMT. Previously, the dateTimes had no explicit timezone.

Rfe15259 - Add a query option to limit the size of query log entries

AllegroGraph's SPARQL engine now truncates query log message to 512 characters by default. This limit can be changed by adding ":LINE_LENGTH" to the logQuery query option either in the AllegroGraph configuration file or on a per-query basis. For example, this PREFIX line will limit log lines to 100-characters:

PREFIX franzOption_logQuery: <franz:yes:100>

Rfe15258 - Add `maximumValuesCountForService` query option

AllegroGraph uses the SPARQL VALUES clause to pass partial results to SPARQL endpoints. In general this can help the endpoint compute its results more efficiently but when there are many values the cost of transmitting the data can outweigh these benefits.

The maximumValuesCountForService provides a mechanism to limit the total number of VALUES sent.

Rfe15250 - Improve treatment of very large VALUES clauses

The query planner now handles large VALUES clauses significantly faster. In addition, the query executor now decides whether to probe or scan based on the number of VALUES and the expected number of rows that a cursor will need to traverse. These two changes can make some queries significantly faster.

Rfe11137: Support the TIMEZONE() and TZ() functions

AllegroGraph now supports the SPARQL 1.1. TIMEZONE() and TZ() functions.

Bug25003 - FILTERing on an expression like ( ! ?a = ?b ) caused an error

The above expression evaluates as (!?a) = ?b and AllegroGraph was not correctly handling the expression of the inner negation operator. This has been corrected.

Bug24985 - FILTER EXISTS and SUB-QUERY can lose solutions

If a FILTER EXISTS clause started with a sub-query then AllegroGraph could fail to find some solutions. This has been corrected.

Bug24657 - GRAPH filter can be applied to the wrong clause

If a graph variable appeared in the subject, predicate, or object position of a triple-pattern, it was possible for AllegroGraph to apply a GRAPH dataset constraint to it incorrectly. For example, AllegroGraph could have incorrectly determined that the ?g1 :predicate1 ?object pattern in the query below should be evaluated as if it was inside a GRAPH clause which could lead to incorrect results.

select * {  
  GRAPH ?g1 { ?s1 ?p1 ?o1 } .  
  GRAPH ?g2 { ?s2 ?p2 ?o1 } .  
  ?g1 :predicate1 ?object .  
}

This has been corrected.

Bug24151 - Correct a SPARQL FILTER analysis issue

AllegroGraph's SPARQL engine would signal an error if a FILTER contained an expression like

str( ?varOne ) = ?varTwo

This has been corrected.

Bug20593 - Support the SPARQL 1.1 BNODE( identifier ) function

AllegroGraph has long supported the BNODE() function with no arguments. It now also supports the version that takes a single plain literal or XSD string argument.

Bug24076 - Improve handling of DBPedia.org and other Virtuoso servers as SPARQL endpoints.

AllegroGraph works around an apparent problem with Virtuoso.

AGWebView

Rfe11480 - Added Shutdown instance operation to Repository Control section in AGWebView

See the desription of the repository page in the WebView document.

Bug24949 - Overview page broken for remote sessions

Opening a session to a single, remote store in WebView used to produce the following error message:

Loading failed: Service not supported in a session that does not wrap a single local store.

instead of the overview page. This has been corrected.

Bug24929 - Help messages appear out of view.

Help messages were often shown at the top of the Webview page, out of sight of the current page location. They will now be displayed in tooltips.

Changes to the Lisp API

Rfe15325: Make `db` a required thread binding

To help avoid accidental sharing of a repository connection between threads, each new thread will start with *db* bound to nil. Previously, *db* would retain its global value in new threads.

Given this sample code:

(open-triple-store "test") ;; sets global value of *db*  
 
(mp:process-run-function "new thread"  
   (lambda ()  
     ;; The new thread wants its own connection to the repository  
     (open-triple-store "test")))

Prior to v6.4.0, the call to open-triple-store in "new thread" would overwrite the global value of *db*. In v6.4.0, "new thread" will have its own binding of *db* and the global value will not be affected.

Prolog

No significant changes.

Documentation

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.3.0

AllegroGraph Server: General Changes

Catalogs may no longer be named root or system.

The names root and system are reserved as catalog names. root interferes with the default catalog (which is referred to as the root catalog) and the system catalog is used for auditing and other system data. If you have catalogs named root or system, contact [email protected] if you have difficulty upgrading to 6.3.0.

'agtool archive' no longer accepts --catalog, catalog now part of repo name

The agtool archive program (see Backup and Restore) now specifies catalogs and repositories together using the catalog:repository format. The --catalog option to agtool archive is no longer accepted.

Rfe15210 - Log which user starts a session

When a session is started, AllegroGraph now logs the name of the user that started the session. (Although not documented, the AllegroGraph log has always said when a session was started and stopped.) A sample log message:

[2017-09-21T15:59:48.028 p21266 back] Session on port 43704 started by user "test".

Rfe15194 - Support XQuery and XPath Math Functions

AllegroGraph now supports the following XPATH math functions

acos
asin
atan
atan2
cos
exp
exp10
log
log10
pi
pow
sin
sqrt
tan

See http://www.w3.org/2005/xpath-functions/math/ for details.

Rfe15132 - `agtool archive list` can operate on backup directories

agtool archive list now can operate not only on single archive file but can be given backup directory as source. In that case it displays details about every repository inside backup.

Additionally it can be given the optional DB-IN-ARCHIVE argument to display details about single repository in ARCHIVE even if other repositories are present.

New option --summary allows agtool archive list to only list repositories inside ARCHIVE, without printing details about every one.

agtool archive is described in the Backup and Restore document.

Rfe12362 - CORS support

CORS (Cross-Origin Resource Sharing), if enabled, allows scripts run on a web page from one server to make HTTP requests to the (different) server where AllegroGraph is running. CORS is not enabled by default. You enable it with various donfiguration directives described in the CORS directives section of the Server Configuration and Control document.

Rfe12201 - Switching a repository between commit and no-commit modes

This now requires superuser privileges.

Bug24927 - Turtle serializer tried to escape capital A

The turtle serializer could try to escape the letter capital A which could lead to invalid output. This has been corrected.

Bug24910 - Unhelpful Turtle parser error message

Improved the error message the Turtle parser produced when the object of a triple was missing.

Bug24905 - Audit log does not display in AG 6.2.3

The Audit log display feature in AGWebView was inadvertently broken in release v6.2.3. This has been corrected.

Bug24842 - Custom services should not use user namespaces

Custom services were being executed in the context of the current user rather than in the context of the code that defined the service. This has been corrected.

Bug23757 - If a script file generates errors (warnings) while loading, don't start the session.

Previously, any errors or warnings that occurred when compiling a Lisp script were written to the AllegroGraph log file and the session would still start. Now, any problems are still written to the log file but the session will not start and the text of the problem(s) will be returned in the HTTP response.

HTTP API

Bug20845 - /version/info web service removed

The /version/info web service caused the server to report detailed internal information about the server and the machine on which the server was running. It was decided that the information was not in fact useful and could expose security vulnerabilities. The URL now returns 404.

SPARQL

Defining your own magic properties

AllegroGraph lets you define your own Magic Properties using the Lisp API. The Defining Magic Properties Tutorial describes how to do this and provides numerous simple examples.

Rfe15182 - All of AllegroGraph's query warnings are now documented.

See the Query Warnings section of the SPARQL Reference document.

Rfe15152 - Propagate impossible query conditions outward in the plan

When a query contains a triple pattern that cannot possibly match, it will mark the pattern as impossible. The impossible marking will propagate outward (e.g., from the pattern to the BGP to the enclosing JOIN and so on). This allows queries with typos or other errors to fail quickly.

Rfe15135 - Warn if any SPARQL type errors occur during query execution

SPARQL will now log a warning if any type errors are encountered during query execution. For example, it is an error to try and cast the string '1.23e4' to an integer because the lexical representation of an XSD integer consists of "a finite-length sequence of decimal digits (#x30-#x39) with an optional leading sign." (cf https://www.w3.org/TR/xmlschema-2/#integer). Therefore, a query like:

select * { bind(xsd:integer('1.23e4') as ?int) }

will leave ?int unbound. Previously, this type error would have been surpressed. Now, AllegroGraph will issue a warn-sparql-type-errors warning.

Rfe15112 - Improve cancelQueryOnWarnings option

The cancelQueryOnWarnings query option now cancels queries immediately for most plan or query warnings. This option defaults to off but can be turned on by placing

QueryOption cancelQueryOnWarnings=yes

in the AllegroGraph configuration or by prefixing a SPARQL query with the PREFIX

PREFIX franzOption_cancelQueryOnWarnings: <franz:yes>

Rfe12249 - Improve magic predicates for Solr and MongoDB

The magic predicates that interface AllegroGraph's SPARQL engine with Solr and MongoDB have been improved. The three magic predicates:

can now take an optional second parameter in their list of subject arguments. This second parameter will be bound to the matching objects of any triples found by the search in the case of solr:matchID and to the associated linking identifier in the cases of solr:match and mongo:find. For example, the following query will bind ?s to the subjects found and ?text to the object:

prefix solr: <http://franz.com/ns/allegrograph/4.5/solr/>  
select ?s ?text {  
  (?s ?text) solr:matchId 'boatswain' .  
}

Rfe11193 - Improvement performance of SPARQL expression compilation

The SPARQL query expression compiler is now approximately 45 times faster than before. This means that queries that use many expressions (e.g., in FILTERs, BIND, ORDER BY, etc.) will perform more efficiently.

Note that aggregation expressions still use the older and slower compiler.

Rfe11678 - Full-scan warning should take dataset into account

AllegroGraph signals a warning when a SPARQL query needs to perform a full scan and the store contains more than one million triples. AllegroGraph was incorrectly signaling this warning when the query was executed against a SPARQL dataset. This has been corrected.

Bug24898 - Fix problem with magic SNA centrality predicates and neighbor caches

Some queries that used AllegroGraph's SNA centrality magic predicates and neighbor caches could cause an error. This has been corrected.

Bug24861 - Sparql parser confused by "?o<='foo'^^<...> "

The SPARQL parser was incorrectly treating the inequality operator as the start of a URI. This has been corrected.

Bug24681 - Parsing should fail if a blank node label is used in more than one BGP

AllegroGraph did not signal a parsing error if a SPARQL query used the same blank node in more than one basic graph pattern (BGP). This has been corrected.

AGWebView

Rfe15198 - Index management page

An index management page has been added to AGWebView. The new page can be accessed from the Repository Overview page by following a link labeled 'Manage triple indices'. The list of indices has been removed from the overview page and the 'purge deleted triples' link has been moved to the index management page. See The Manage triple indices page for more information.

Rfe14563 - Repository reports

A set of reports describing disk usage and other repository properties has been added to WebView. The reports can be accessed through links on the repository overview page. See the Reports section in the Repository Overview page description for more information.

Rfe13419 - Add warmup triple-store to AGWebView

Add new option Warmup store in Repository Control section in repository overview page. See the description of the Repository Overview Page in the WebView document.

Rfe11915 - Creating freetext index prefills stop words with default ones.

The AGWebView dialog for creating a freetext index now starts with a default selection of stop words (rather than starting with no stop words and the option to add a default selection). See the Free-Text Indices page in the WebView document for more information.

Rfe11256: Import RDF statements directly.

It is now possible to import RDF statements directly by typing or pasting them into a text box in AG WebView. See the Load and Delete Data section of the Repository Overview page description.

Bug24824 - Spurious warning for ASK queries in AGWebView

AllegroGraph WebView was showing a query warning for ASK queries when the "Limit to 1000 results" checkbox was checked. This has been corrected.

Bug24480: "Ignore errors" option was ignored when loading triples from server file

The "Ignore errors" error handling option was not being honored when loading a server-side file using AG WebView. This has been fixed.

Bug23607 - Improve accuracy of query timing results in AGWebView

Previously, the timing results in AGWebView represented the time it took to find the first result. Now, it represents the time it takes to find all results. The total time to display may still be longer as it will also include the time it takes the web browser to render the returned data.

Changes to the Lisp API

Remove :plain datatype mapping

The :plain datatype mapping converted typed string literals into the equivalent plain literal. E.g., it would convert "bear"^^xsd:string into "bear". :plain mappings are no longer necessary because AllegroGraph handles typed strings more efficiently automatically. Mappings are specified with setf and datatype-mapping and predicate-mapping.

Note that any existing datatype or predicate mappings of type :plain will be automatically removed from the store.

Bug24857 - path-finding over HTTP fails when paths include literals

If an SNA path included typed literals, then a remote-triple-store would convert them into the wrong UPI type leading to UPI not in string-table errors. This has been corrected.

Prolog

No significant changes.

Documentation

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.2.3

AllegroGraph Server: General Changes

Rfe15101: Restore tlmgr to tlmgr.restored

The tlmgr file contains information about replication jobs. Since the existence of replication job information can result in transaction log files accumulating, the tlmgr file is no longer restored as is. Now it is renamed to tlmgr.restored during a restore. After a restore, replication jobs must be reestablished by the user.

Rfe12408 - Add graph filtering support to agmaterialize

agtool materialize can now limit materialization to a specified list of graphs via the --graph (short form -f -- note that -g specifies the default graph for inferred triples) command line argument. The inferred graph (specified with --inferred-graph or -g) will always be included in the list. See the Materializer document.

Bug24729 - Spaces in filenames prevent server-side file import.

Importing a server-side file with spaces in its path would fail with an error message about illegal characters in a URI. This issue affected both WebView and the HTTP API, but not agtool load and has been corrected.

HTTP API

Bug20772 - Loading JSON format may fail to commit all triples

Using the 'commit=number' parameter when importing statements through the HTTP API could result in some statements not being committed. The issue only affected JSON imports and has been corrected.

SPARQL

Rfe15077 - Signal a query warning when CONSTRUCT tries to generate invalid triples

AllegroGraph will now signal a query warning if any triples produced by the CONSTRUCT template are invalid. Formerly, these triples were silently discarded.

Rfe15075 - Improve parsing efficiency of large USING [NAMED] datasets

The SPARQL parser now parses UPDATE commands with very large dataset specifications using the USING and USING NAMED clauses more efficiently.

Rfe15074 - SPARQL parser has non-linear behavior for large VALUES clauses

Improved the SPARQL parser performance for queries with 1000s of elements inside of VALUES clauses.

Rfe15061 - Improve efficiency of SPARQL queries with large datasets

AllegroGraph now determines at query time whether it would be more efficient to issue getStatements calls for each graph in a dataset or to issue a single getStatements call and filter the results to include only graphs in the dataset. Depending on the triple-store and the dataset, this can be significantly more efficient.

Rfe15006 - Improve GRAPH and PROPERTY PATH query planning and execution

AllegroGraph now handles property path queries inside of GRAPH clauses more efficiently when the graph is a variable and the clause contains no property path queries that use any of the ZeroOrMorePath, OneOrMorePath, or ZeroOrOnePath operators.

Rfe14450 - Improving cross product warning

The warning issued when AllegroGraph detects a cross product in a SPARQL query is now more descriptive.

Bug24779: Constructed triples should not allow blank nodes in the predicate postion

AllegroGraph was incorrectly allowing CONSTRUCT to produce blank nodes in the predicate position of triples. This has been corrected.

Bug24762 - SPARQL DELETE/INSERT/WHERE and Chunk processing could fail

If the chunk size was too small, a SPARQL DELETE/INSERT/WHERE command could fail to correctly process triples.

Bug24755 - Problems in intervalAfterDatetime and intervalBeforeDatetime

The intervalAfterDatetime magic property could fail to find solutions when using a constant datetime rather than a variable. Additionally, both intervalAfterDatetime and intervalBeforeDatetime could fail to find solutions when the intervals were defined on explicit times rather than using named time points.

Bug24709 - FILTER EXISTS and unbound variables on the RHS

AllegroGraph used to return incorrect results for FILTER EXISTS expressions when the RHS contained any unbound variables. This has been corrected.

Bug24704 - Improve strategy used to find unique values in a store

AllegroGraph uses several techniques to determine all of the unique values stored for subject, predicate, object, or graph. Depending on the quads in the store and the shape of the query, one of these techniques can be much more efficient than the others. It was possible, however, for AllegroGraph to choose the wrong technique in some cases. This has been corrected.

Bug24665 - SPARQL queries with :dateTime inequality filters could lose results

Query results could be lost if a SPARQL query used a non-inclusive inequality filter on xsd:dateTimes or xsd:times. This has been corrected.

Bug24455 - Incorrect results when join variables might be unbound

AllegroGraph would sometimes return incorrect results from joins where a part of the join key was unbound.

This has been corrected.

Bug24215 - Unbound values in join keys

When merging solutions during a join, AllegroGraph would sometimes not produce bindings for variables that are only bound on the RHS of the join and are a part of the join key.

For instance when running this query:

select * {  
  { values(?x ?y) {(1 undef)} }  
  { values(?x ?y) {(undef 2)} }  
}

AllegroGraph would return (?x = 1, ?y = undef) instead of the expected (?x = 1, ?y = 2).

This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

Bug24796 - intern-resource error on future-part with unregistered namespace abbreviation

A call to intern-resource on a future-part whose namespace abbreviation was not defined would signal a bus error. This will now signal a condition of type cannot-resolve-namespace-for-future-part-error.

Prolog

No significant changes.

Documentation

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.2.2

AllegroGraph Server: General Changes

Rfe14926: Reserve parentheses characters

Parentheses are no longer allowed in triple store or catalog names. These characters are reserved for use in triple store specifications.

Bug24700 - Unfriendly error message when session ports are unavailable

Previously, if a request to create a session cannot be completed due to lack of an available session port, an unfriendly error message was reported. This has been fixed and now a clear error message is returned.

Bug24687 - Deleting a tlog when free list is full throws error.

Fixes a bug introduced in v6.1.4 where a tlog that was deleted due to the free list being full would be repeatedly deleted by AllegroGraph despite it no longer existing. This bug could prevent the freeing/deletion of old tlogs, causing them to accumulate.

Log entries that indicate this bug, would be similar to the following:

[2017-05-22T13:49:21.800 p17318 tlarchiver] Store "bug24687", free list is full, will just delete "tlog-2477c96f-0579-ba54-b60b-7083f2293fa8-27"  
 
... later ...  
 
[2017-05-22T13:51:34.996 p17318 tlarchiver] Store "bug24687", free list is full, will just delete "tlog-2477c96f-0579-ba54-b60b-7083f2293fa8-27"  
[2017-05-22T13:51:34.999 p17318 E tlarchiver] Store "bug24687", Attempt to delete tlog-27 gave error There is no file named /dbs/bug24687/tlog-2477c96f-0579-ba54-b60b-7083f2293fa8-27: No such file or directory [errno=2].: No such file or directory [errno=2].

Bug24673 - FTI merger can create empty data file

The FTI merger process sometimes fails and logs an error message similar to the following:

Process 'repo FTI Merger' raised: opening  
#<mapped-file-simple-stream  
#P"/agdata/data/rootcatalog/repo/fti-2-chunk-114-data" mapped  
closed @ #x100054959b2> resulted in error (code 22): Invalid  
argument.

this has been corrected.

Bug24663 - TriG parser uses the wrong graph on the first blank node

In TriG nested blank nodes can be specified in a graph other than the default graph. When the TriG parser encountered the first nested blank node inside a non-default graph the graph for the triples derived from the parsing of that nested blank node would incorrectly be set to the default graph. This has been fixed to use the correct graph.

Bug24642: Incorrect text index query warning

AllegroGraph v6.1.4 added a check and warning for text index queries which could not possibly match. Under certain circumstances this check would warn even though the query could match. This has been corrected.

Bug24623 - SNA path finding in remote-triple-stores did not suppport blank nodes

If a path contained blank nodes, then the SNA path finding functions would signal an error if used on a remote triple-store. This has been corrected.

Bug24615 - Static filter function not re-esablished when attaching to a store

Previously when opening an existing triple store, the static attribute filter was not reestablished. This has been fixed.

Bug24588 - load-nqx does not properly track line numbers

When an error occurs loading extended triples via load-nqx and agload the incorrect line number is reported. This has been corrected.

Bug24548: TransactionLogArchive directive doesn't create catalog and repository directories when tlarchiver is initiated

AllegroGraph did not previously create the TransactionLogArchive directory when starting a database instance. Instead a "no such file or directory" error was logged when the tlarchiver ran. This has been fixed. The directory (and any subdirectories) are now created when a database instance starts, if needed.

Bug24545 - Store specification containing a remote URL could fail

A "missing value for argument db.agraph::user" error would occur when starting a session using a store specification containing a remote URL without authentication information. This has been fixed.

Bug24342 - load-turtle loses default-attributes for RDF collections

Previously the Turtle parser would fail to include attributes when parsing an RDF collection such as:

prefix : <ex://foo#>  
:a :p (1 2 3) .

This has been fixed.

Bug21595 - agload bad error message when given a filespec that does not exist

The agload program reported misleading error messages when attempting to load files that did not exist or contained wildcards. This has been improved by reporting an accurate and more useful error message. See Data Loading for more information on agload.

HTTP API

Rfe14967 - Ensure that the statements REST API uses namespace abbreviations

The statements REST API was not using user defined namespaces to abbreviate RDF resources in formats that supported them like turtle or TriG. This has been corrected.

Rfe14930 - Improve error message when a file cannot be accessed

The agload process for loading a local file via the HTTP interface reported misleading and inaccurate error messages when the local file was inaccessible. This has been improved by returning an accurate and useful error message in the HTTP response.

SPARQL

Rfe14978 - Improve efficiency for SPARQL datasets with many graphs

AllegroGraph now operates on SPARQL datasets with many graphs more efficiently.

Rfe14975 - SPARQL dataset validator is inefficient for large dataset specifications

The SPARQL engine validates any dataset specification before it begins execution. This change improves the performance of the validation which helps query speed especially when the dataset has more than a few thousand graphs.

Rfe14974 - SPARQL parser is inefficient for large dataset specifications

SPARQL queries with thousands of FROM and FROM NAMED clauses in their dataset description now parse much more efficiently.

Rfe14931 - Improve how property path queries implement filtering

SPARQL Property Path queries now handle FILTERs more efficiently.

Rfe14899 - Warn if query LIMIT/OFFSET and externally imposed LIMIT/OFFSET conflict

Pursuant with the changes made for Rfe14597 (described below), AllegroGraph will now warn if a query's internal and external LIMITs differ or if a both an internal and external OFFSET are supplied.

Rfe14597 - Reduce confusion between internal and external query limit and offsets

Previously, any externally imposed LIMIT or OFFSET would override the LIMIT or OFFSET in the query string. This was most evident in AGWebView because the web interface uses a dropdown selector to pick the LIMIT and this LIMIT overrode any specified in the query.

Now, AllegroGraph will:

use the minimum of the internal and external LIMITs.
use the sum of the internal and external OFFSETS.

Bug24684 - Property Paths and MINUS could interact badly

It was possible for queries that used MINUS and Property Paths to generate incorrect answers. This has been corrected.

Bug24610 - Trivial equivalence constraints cause an error.

Queries containing trivial equivalence constraints, such as filter(?x = ?x), used to cause the following error:

missing value for argument db.agraph.sbqe::var2

This has been corrected.

Bug24609 - User attributes set via the SPARQL prefix could be ignored

Depending on how a query was executed, any user attributes set via the userAttributes SPARQL query prefix option could be ignored. This has been corrected.

Bug24608 - Spurious join results.

When performing joins where the left hand side contained unbound values (resulting from the use of OPTIONAL or UNDEF keywords) AllegroGraph could sometimes return the same solution row multiple times. For instance, the following query:

SELECT * {  
  { values (?a ?x ?y) {(0 1 2)} }  
  optional { values (?a ?y ?z) {  
		(0 2 3)  
		(0 undef 4)  
		(undef 2 5) }  
  }  
}

Would return 4 results, including two copies of the (0 1 2 5) row. This has been corrected.

Bug24599 - concatenated-cursors don't support extended triples

When a SPARQL query which uses attributes-related magic properties is issued against a graph-filtered triple store, an error like the following can occur:

Executing query failed: #<concatenated-cursor @ #x1000732dc52>  
does not support fat-triples

This has been fixed.

Bug24538 - Invalid effective boolean value of constant strings in filters

A query like:

SELECT * {  
 ?s ?p ?o  
 FILTER ("bruce")  
}

was returning no results when it should have returned every row in the store. Non-blank strings are supposed to have an effective boolean value of "true". Blank strings have an effective boolean value of "false". This has been fixed.

Bug24530 - SPARQL-JSON result format does not handle encoded boolean properly

Previously, boolean values in SPARQL-JSON results were represented as the strings "0" and "1". Now boolean values are correctly represented as "false" and "true".

AGWebView

Rfe14968 - Remove the deprecated `planner` option from AGWebView

Remove the long deprecated planner option from AGWebView's query page. This option has had no effect for some time.

Rfe13239 - Provide "no limit" option for query results in AGWebView

Subject to the resources of the web browser, AGWebView can now return an unlimited number of results from a query. By default, AGWebView will display 1000 results or use the limit specified by the query (if it is smaller than 1000). Because of this, the More Results button has also been removed. See the WebView document for information on AGWebView.

Bug24616 - N-Triples parser errors mention NQX

When trying to load a .nt or .nq file through AGWebView and a parsing error occurs the error message states that a "NQX" parsing error occurred when it should have stated it was an N-Triples (for .nt) or N-Quads (for .nq) parsing error. This has been fixed to report the correct format type.

Changes to the Lisp API

Bug24669 - serialize-nqx does not work on remote-triple-stores

Serialization of attributed triples was not working correctly for remote-triple-stores. This has been corrected.

Bug24658 - load-nqx fails on remote-triple-stores

An error would occur if no default attributes were passed to load-nqx on a remote-triple-store. This has been corrected.

In order to reduce future confusion, the :attributes argument to the various load functions in the Lisp API has been changed to :default-attributes. Note this is an incompatible change. Code using :attributes will signal an error.

Bug24549: open-store-from-specification does not work

Previously, attempting to open a triple store using a URL could fail like so:

(open-triple-store "http://localhost:10443/repositories/kennedy")  
Error: Invalid character in triple-store name  
       "//localhost:10443/repositories/kennedy".  
The name may not contain these characters /, \, ~, :,  
Space, $, {, }, <, >, *, +, |, [, ].

This has been fixed.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

Bug24680 - Tonativestring is broken on Python 2

The Python client sometimes failed while processing values with non-ascii characters, showing the following error message:

UnicodeEncodeError: 'ascii' codec can't encode characters in position ??: ordinal not in range(128)

This has been corrected.

Bug24443 - getStatements sometimes breaks for raw Python values

The getStatements function can be called with a raw Python value, instead of an RDF term, as its third argument (object filter). For instance, the following call:

conn.getStatements(None, somePredicate, True)

would retrieve all triples with the predicate of somePredicate and the object equal to "true"^^xsd:boolean.

This mechanism did not work correctly if the second argument (predicate filter) was None. In that case an AttributeError exception would be raised.

This has been corrected.

Java Client

Rfe14990 - Provide methods to download query results

The Java client now contains methods to download and save query results to a file. The new methods are:

AGRepositoryConnection.downloadStatements
AGRepositoryConnection.streamStatements
AGQuery.download
AGQuery.stream

Both methods have multiple overloads, allowing for the output path and desired output format to be passed in a variety of ways.

Rfe14970 - Speed up AllegroGraph java client connection pools.

Depending on the connection pool configuration, many unnecessary requests could be made to the AllegroGraph server when borrowing/returning connections to/from a connection pool.

These calls have been cleaned up, resulting in improved connection pool performance.

Bug24648 - inferredGraph parameter missing from AGMaterializer

When materializing triples through the Java API it is now possible to specify the graph into which the generated triples will be added.

Bug24644 - Fix TriG import via the Java client.

The AllegroGraph Java client would fail when attempting to import TriG data due to the content-type "application/x-trig" being unsupported in the server.

AllegroGraph now supports TriG data imported with a content-type of either "application/trig" or "application/x-trig".

nD Geospatial

No significant changes.

AllegroGraph 6.2.1

AllegroGraph Server: General Changes

Rfe14881 - Cross attribute comparisons

AllegroGraph 6.2.0 added a restriction against using an attribute filter operator with two different types. An attempt to perform such an operation would result in a "Comparisons between attributes of different types is not supported" error message.

This restriction has been removed. Note, however, that ordered comparisons between different attribute types is still not allowed since there is no defined ordering between two ordered attributes. Triple attrinutes are discussed in Triple Attributes.

Bug24567 - Initial superuser account file owned by root if configure-agraph run as root

Previously, if AllegroGraph was installed, configured, and started as root, the directories and account file for the initial superuser remained owned by root. This resulted in the following error the first time the superuser made access to AllegroGraph:

mkstemp failed: Permission denied [errno=13]

This has been corrected.

Bug24557 - Fix bug restoring repositories with a non-default stringTableSize.

When restoring repositories with a non-default stringTableSize directive to v6.2.0, a bug in an upgrade step would cause the repo's string table to be set to the default size of 16M. If the stringTableSize was larger than this default, the string table would be truncated, causing data loss and possible corruption.

This bug has been fixed.

Bug24548 - TransactionLogArchive directive doesn't create catalog and repository directories when tlarchiver is initiated

Bug24342 - load-turtle loses default-attributes for RDF collections

Previously the Turtle parser would fail to include attributes when parsing an RDF collection such as:

prefix : <ex://foo#>  
:a :p (1 2 3) .

This has been fixed.

HTTP API

No significant changes.

SPARQL

Bug24538 - Invalid effective boolean value of constant strings in filters

A query like:  
SELECT * {  
 ?s ?p ?o  
 FILTER ("bruce")  
}

Bug24530 - SPARQL-JSON result format does not handle encoded boolean properly

Previously, boolean values in SPARQL-JSON results were represented as the strings "0" and "1". Now boolean values are correctly represented as "false" and "true".

Bug24529 - Some queries using VALUES produce wrong results.

Some queries using the VALUES keyword and joins or unions produced incorrect results. More specifically some rows were missing from the result.

Two examples of such queries are:

select ?x { { values(?x) {(1)} } union { values(?x) {(2)} } }

and

select ?x ?y ?z { { values (?x ?y ?z) {(1 2 3)} }  
                  { values () {()} }}

The first one would produce only a single result (2), while the second one would not produce any results. This has been corrected.

Bug24528 - Wrong join results when using the CAAT strategy

When using the chunk-at-a-time execution strategy some queries that performed left joins with the identity set would produce spurious results. One such query is:

select ?a ?b ?c {  
  { ?a ?b ?c } union { ?c ?b ?a }  
  optional { bind(42 as ?x) }  
}

This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

Bug24545 - parse-remote-store-specification returns inconsistent results

A "missing value for argument db.agraph::user" error would occur when starting a session using a store specification containing a remote URL without authentication information. This has been fixed.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

Rfe14695 - Replace pycurl with requests.

The HTTP library used by the AllegroGraph Python client has been changed from pycurl to requests. This change means that the Python client no longer depends on any libraries that require native extensions. This results in a simpler installation process, as it is no longer necessary to have a working C development environment on the target machine.

Note that pycurl will still be used if it is installed, since it might offer better performance in some scenarios.

Bug24582 - AGRAPH_PROXY variable not used

Proxy settings specified through the AGRAPH_PROXY environment variable used to be ignored.

This has been corrected.

Java Client

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.2.0

AllegroGraph Server: General Changes

Rfe14846 - Revised triple attribute filter operators

There are extensive changes to attribute filter operators, allowing better control when comparing user attributes with triple attributes. New operators include

or, not, empty, overlap, attributes-overlap, subset, superset,  
equal, attribute-set\<=, attribute-set<, attribute-set=,  
attribute-set>, attribute-set>=

The existing operator attribute>= has been renamed attribute-set>= (the old name is still accepted but users are advised to switch to the new name). See Triple Attributes for information on attribute filter operators.

Rfe14806 - Audit attribute and static filter definitions

If auditing is enabled an audit record is made for defining an attribute and for defining the static attribute filter. See the Auditing document for more information on auditing.

The rdf:type of the new records are:

<http://franz.com/allegrograph/4.11/audit-log#defineAttribute>  
<http://franz.com/allegrograph/4.11/audit-log#deleteAttribute>  
<http://franz.com/allegrograph/4.11/audit-log#defineStaticAttributeFilter>

Rfe14773 - Changes to agraph-control

The agraph-control program that starts and stops AllegroGraph has a new command restart, which requests that the server shut itself down, if running, and then start back up again. The reload command, which reloaded the configuration file, is no longer supported. See the agraph-control section of the Server Configuration and Control document.

Rfe14797 - Add permissions for user-attribute specification

Added user permissions user-attributes-header (if true, user can send attributes via HTTP as the value of the x-user-attributes header), and user-attributes-prefix (if true, user can specify the user attributes via the SPARQL prefix franzOption_userAttributes. See the Managing Users section of the WebView document for information on user permissions.

Rfe14733 - String comparison operator for attribute filter

Added the attribute= operator which can be used to compare attributes to other attributes, attributes to strings, or strings to strings. See Triple Attributes for information on attribute filter operators.

Rfe14671 - New configuration option StringTableCompression

The StringTableCompression configuration option allows specifying the compression type (including no compression) for the string table. The default is lzo999 compression. See Server Configuration and Control for information on configuration options.

Rfe14661 - agtool combines many command-line programs

The *agtool program replaces the programs agraph-backup, agmaterialize, agquery, agexport, agload, agraph-recover, agraph-replicate, and convertdb. agtool takes as its first argument the task to perform (usually indicated by a shortened version of the old program name but sometimes quite different -- so agraph-backup becomes archive and convertdb becomes upgrade) and then the various argument and options for that task. For example, in order to load the Ntriples file mytriples.nt into the mystore database in the server listening on port 10035, in earlier releases you ran this command:

% agload --port 10035 mytriples.nt mystore

now you use agtool as follows:

% agtool load --port 10035 mytriples.nt mystore

(Other argument/option combinations will do the same as those shown.) For most tasks, the arguments and options are the same as with the earlier tools, with the following differrences:

agraph-backup (archive), agmaterialize (materialize), The --ext-backup-command is no longer supported. Otherwise arguments and options are the same.

agraph-recover (recover), convertdb (upgrade), these changes: -c accepted for --catalog, -p accepted for --port.

agload (load), this change: -u not accepted as synonym for --base-uri.

agquery (query) and agexport (export), this change: --username is now --user and -u is a synonym of --user.

agraph-replicate (replicate), this change: -c accepted for --catalog, -u accepted for --user.

See the agtool General Command Utility document for more information and links to all the documentation for the individual tasks.

Rfe13254 - Tools for saving and loading metadata

Metadata, that is attribute and static filter definitions, can be saved from a database and loaded into a database. The methods for doing so include using the new --save-metadata option to agtool export (see Repository Export) and the new --metadata option to agtool load (see Data Loading), along with new Lisp functions and REST API, discussed under Lisp API and HTTP API below.

Bug24425 - install-agraph no longer creates a log/ directory

install-agraph would incorrectly create a log/ directory alongside the lib/ and bin/ directories it creates. It no longer does so, leaving the creation of the log directory to tools like configure-agraph, agraph-control, or the agraph server itself.

HTTP API

Rfe13254 - load/save attribute metadata

GET and POST metadata commands allow obtaining metadata (attribute and static filter definitions) from a repo and loading them into another repo. See the Attributes section of the REST/HTTP interface document for details.

SPARQL

Rfe14812 - Support attributes in INSERT DATA and INSERT/WHERE

The syntax of SPARQL INSERT DATA and DELETE/INSERT/WHERE commands has been extended to support the fine-grained specification of attributes on a per-triple basis. See Triple Attributes for more information on attributes.

Rfe14726 - New prefixes for attributes

The defaultAttributes SPARQL query option specifies the attributes to be associated with each triple added via the INSERT DATA, DELETE/INSERT/WHERE, or LOAD commands. The userAttributes query option specifies the user attributes to be used when processing the query. See Triple Attributes for more information on attributes.

Rfe13305 - Old-style SPARQL GEO queries are no longer supported

Because AllegroGraph's magic properties support a richer range of geospatial features than the old-style SPARQL GEO syntax, the old syntax was deprecated in 2014 and has now been removed from the code. Any queries that use the old-style syntax will need to be rewritten.

Magic properties support both the deprecated 2D geospatial encodings and the newer nD ones.

Bug24481 - Wrong results for VALUES with a single variable

When processing a VALUES clause in a SPARQL query AllegroGraph would return incorrect results if there was exactly one variable bound by that clause and the list of values contained either:

An UNDEF value: it would be silently ignored (but other values would still be used).
Duplicates: each value will be matched exactly once, regardless of the number of times it appears inside VALUES.

This has been corrected.

Bug24470 - Spurious results for SELECT DISTINCT on a bound variable

When executing a query which tries to find unique values of an already bound variable like in the example below:

select distinct ?s WHERE {  
  bind(ex:A as ?s) .  
  ?s ?p ?o .  
}

AllegroGraph would ignore the binding and return all results that match the pattern. This error could be observed only if the single-set execution strategy was used.

This has been corrected.

AGWebView

Rfe14576 - Make deleting namespaces easier on AGWebView Query page

You can now delete namespaces on the AGWebView Query page.

Rfe14569 - New vmstat-like server performance charts added to WebView

A new menu item in the WebView Utilities menu brings up a page of performance charts on the server machine. The several charts show approximately the same machine load data printed by the vmstat command, charted for the past five minutes. AGWebView is documented in the WebView document. The new Server Load page is documented here in that document.

Changes to the Lisp API

Rfe14797 - New user-attributes-prefix-permission-p keyword argument to run-sparql

The function run-sparql now have a user-attributes-prefix-permission-p keyword argument. It must be given a true value in order to run a SPARQL query including "PREFIX franzOption_userAttributes".

Rfe14754 - Allow #\: to be used as the catalog and repository delimiter

In addition to using the slash character (#\/), the colon (#\:) can now be used to designate a triple-store in a different catalog. I.e., the following are equivalent:

(open-triple-store "test" :catalog "production")  
(open-triple-store "production/test")  
(open-triple-store "production:test")

Note that the format AllegroGraph uses to display triple-stores will now be catalog:name rather than catalog/name.

Rfe14705 - The triple-store-user (db.agraph.user) package uses the sparql package

The triple-store-user package, also named the db.agraph.user package, now uses the sparql package which means that symbols in the sparql package like run-sparql can be used more easily as they do not need to be package qualified.

Rfe14703 - :return-fat-triples keyword argument is now :return-extended-triples

The return-fat-triples keyword argument to get-triples, get-triples-list, and freetext-get-triples has been renamed return-extended-triples. The new functions extended-triple-triple and extended-triple-attributes return the triple and the attributes (respectively) of an extended triple. The new function copy-extended-triple makes a copy of an extended triple.

Rfe13254 - New functions to load/save attribute metadata

The functions get-metadata. set-metadata, load-metadata, and save-metadata allow retrieving metadata (attribute and static filter definitions) from a store and adding metadata to a store. The metadata can be transmitted as a string or written/read from a file.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.1.6

AllegroGraph Server: General Changes

Rfe14785 - Allow default session idle timeout to be set in agraph.cfg

Two new configuration parameters have been defined for controlling the default and the maximum allowed session idle timeout. DefaultSessionTimeout specifies the default timeout and MaximumSessionTimeout specifies the maximum timeout. See the description of those directives in the list of top-level directives in Server Configuration and Control for details.

Bug24519 - Path-finding broken on remote-reasoning-triple-store

Path-finding operations (such as depth-first-search) were broken for remote-reasoning-triple-stores since v6.1.4. This has been fixed.

Bug24389 - Bus error when adding triples with unspecified graph to security-filtered-triple-store

Previously, adding a triple to a security-filtered-triple-store could result in a bus error. This has been fixed.

Bug24221 - String table configuration mismatch during restore.

An error would occur when trying to restore a database that was backed up in AllegroGraph prior to version 6 if the catalog it was restored to had a string table size different than the default 16M entries.

The error was:

> String table configuration mismatch: Configured number of slots X  
 does not match recovered number of slots (16777216)

This has been corrected.

HTTP API

No significant changes.

SPARQL

Bug24463 - Binding produced index out of range error during query

Some queries using BIND used to fail with the above error message. This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

Bug24427 - Javadocs missing from agraph server distribution

Previously the Javadocs for the AllegroGraph Java client were missing from the installed documentation set. This has been fixed.

Python Client

Rfe14803 - Support for decimal literals

Literal objects constructed from Python's Decimal values will now have the default datatype of xsd:decimal.

Rfe14694 - Drop support for cjson

The Python client no longer uses the cjson library. Python 2.6 users should install the simplejson library instead to avoid a significant degradation in performance. Other versions of Python are unaffected.

Bug24428 - Creating literals from time objects with timezone

RepositoryConnection.createLiteral used to raise an error when passed a datetime.time instance that contains time zone information. This has been corrected - such calls will now return literals of type xsd:time with time value converted to UTC.

Bug24405 - Non-ASCII characters in addData()

The Python client used to raise an exception (specifically a UnicodeDecodeError) if the data passed to addData was a Unicode string containing non-ASCII characters.

This has been corrected.

Bug24322 - Python clients reads imported file into memory

When uploading data to the server the Python client used to read the whole file into memory before sending it. This could cause out of memory errors for larger files. To avoid this issue, the client now sends files in chunks. This method requires only a limited amount of memory.

Java Client

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.1.5

AllegroGraph Server: General Changes

Bug24442 - Fix intermittent query failures that report a "broken pipe" error

A bug was introduced in v6.1.3 where requests could fail inappropriately under certain conditions, with a "broken pipe" error message appearing in agraph.log or reported in AG WebView. This has been fixed.

Bug24412: agload incorrect line number reporting on invalid character

The agload program (see Data Loading (agload)) could report incorrect line numbers if errors were encountered when loading large N-Triples and N-Quads files in parallel. This problem could only occur if the files were larger 4MB. This has been corrected.

HTTP API

No significant changes.

SPARQL

Bug24352 - In FILTERs can be handled very inefficiently

SPARQL queries that constrain a particular binding to a set of values could operate very inefficiently in some circumstances. Such queries include ones that use the IN FILTER or that use the VALUES clause to supply a list of possible matches. The inefficiency has been corrected.

AGWebView

Bug24423 - WebView hangs if HTTP tracing is enabled.

Previously AGWebView would hang on page load if HTTP tracing was enabled. This has been fixed.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.1.4

AllegroGraph Server: General Changes

Rfe14558 - If possible increase the file descriptor limit at startup

If the agraph server is started as root it will set the number of file descriptors agraph can use to 65536 thus not being limited by the default operating system setting (which is usually quite low).

If the file descriptor limit is raised you'll see a line like the following in agraph.log just after the server starts:

[2016-11-08T17:07:23.956 p13419 daemon] Set fd limits soft: 65536, hard: 65536

Rfe11277 - Warn when a freetext query tries to match on things that cannot possibly match

AllegroGraph now generates a query warning when a freetext search expression cannot possibly find matches. For example, the default stop words in an AllegroGraph freetext index include the and an so it is impossible to search for matches on the expression (or 'an' 'the'). This means that a query like

select * { ?s fti:match 'the || an' . }

will return no results. Previously, AllegroGraph would have returned no results without generating a warning. Now, AllegroGraph will warn that The freetext expression the || an cannot succeed.

Rfe11046 - Release fds for free text index disk tries

Making use of a free text index will require far fewer file descriptors than before.

Bug24395 - TriG parser errors could print that they were from the Turtle parser

Some TriG parser errors incorrectly indicated that the current parser was using Turtle syntax rather than TriG. This has been corrected.

Bug24349: Transaction log archiving to same filesystem broken

Transaction log archiving was broken in v6.1.2 when archiving to the same filesystem. This problem is now corrected.

Bug23354 - Check SSL cert when frontend starts

If the SSL certificate information you put in your configuration file is not correct AllegroGraph will log an error and exit during server startup. Previously you didn't find out about SSL certificate errors until the first https request came to the server.

Bug24332 - AllegroGraph doesn't fail gracefully if it cannot claim desired ports

Previously, if AllegroGraph was unable to bind the TCP port specified by the Port configuration directive, it would leave some lingering processes after printing the error message. This has been fixed.

Bug24297 - <http://www.w3.org/2005/xpath-functions#:translate> was broken

The XPath translate function could not be used between AllegroGraph 6.0.2 and 6.1. This has been corrected.

If a query attempted to use the translate function, it would get a parse time error.

HTTP API

No significant changes.

SPARQL

Rfe14722 - Improve planning speed for SPARQL queries that use IN filters

The code generated for evaluating SPARQL IN queries has been optimized for some common cases. This will result in faster plan times for such queries.

Bug24383 - Sparql Update can fail if the WHERE part uses string literals

If the WHERE portion of a SPARQL DELETE/INSERT/WHERE command used string literals that were not already interned in the triple-store in FILTER expressions, then it was possible for the UPDATE to fail. This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

Rfe14736 - Add graph parameter to SNA path finding functions

The various SNA path finding functions like all-depth-first-search-paths and bidirectional-distance all take a new graph/s parameter. This can specify either a single graph or a list of graphs that will be used to restrict path search to only triples from these graphs.

Rfe14721 - Path finding functions accept lists of predicates as generators

The Lisp API to AllegroGraph's Social Network Analysis path finding algorithms has been extended. Previously, the functions took a generator that must have been previously defined. Now, you can pass a list of predicates in the place of a generator and these will be used to create a generator on the fly. Note that this added flexibility does come at the cost of between 5 and 10 percent slower performance so predefined generators are to be preferred.

The syntax of the predicate list is as follows. Each element in the list must be one of

predicate URI
(:undirected predicate URI)
(:subjects-of predicate URI)
(:objects-of predicate URI)

For example, we can find the paths between ex:A and ex:B using undirected links on either ex:knows or ex:likes via the following:

(all-breadth-first-search-paths  
  !ex:A !ex:B  
  '((:undirected !ex:knows) (:undirected !ex:likes)))

This change also adds a new REST API to the path finding algorithms via sna/paths. See the HTTP reference for details.

Bug24293 - Some plain literals could be printed incorrectly

Changes in AllegroGraph 6.1 made it possible for some plain literals to print incorrectly in some situations when using remote-triple-stores. For example, the literal a/b/c could print as a/b"/c.

This has been corrected.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

Rfe14693 - Allow data import from strings

A new method named addData has been added to the connection class. This method makes it possible to add RDF data stored in a string to the store. All RDF data formats supported by addFile can be used.

Rfe14692 - Add more RDFFormats

The only formats supported during data import (addFile) used to be N-Triples and RDF/XML. It is now possible to use other formats, namely Turtle, TriG, N-Quads, NQX and TriX. The addFile method is now capable of determining the correct format based on the filename. When using this capability it is also possible to import gzip compressed files.

bug24379: HTTPS in Python does not work with Basic auth

The Python client used to only allow HTTPS connections if a client certificate was used for authentication.

It is now possible to connect to HTTPS servers by passing either a host value starting with 'https://' or a protocol value of 'https' when creating the server object.

Java Client

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.1.3

AllegroGraph Server: General Changes

Rfe14363: Add binary-only, non-interactive modes for install-agraph

Two command-line options have been added to install-agraph to support non-interactive installation:

--no-configure: do a binary-only install. Skips the configure-agraph step.
--supersede: skip interactive prompt and overwrite installation directory if it exists.

Installation and configuration are described in the Server Installation document.

Bug24253 - Triple-store statistics could become stale when new datatypes were added

It was possible for the statistics AllegroGraph keeps on each predicate to become stale if triples were added with heretofore unseen XSD datatypes. This has been corrected.

Bug24245 - Text index word filters and wildcard search do not work together

A freetext index that used word filters like drop-accents or the stemmer could fail to find matches when using wildcard searches. This has been corrected.

HTTP API

No significant changes.

SPARQL

Bug24273 - The SPARQL RAND() function was being evaluated at plan time

AllegroGraph was mistakenly evaluating the SPARQL RAND() function at plan time rather than query execution time. This could lead to incorrect results and has been corrected.

Bug24214 - Querying using the direct reification magic property returns at most one match

AllegroGraph's direct reification magic property only returned the first matching triple. This has been corrected so that all matches are returned.

AGWebView

Rfe14205: Maintain and provide information on live HTTP requests

A new page, Admin -> Requests, was added in AGWebView. It shows all the currently running and recently finished HTTP requests handled by AllegroGraph and its subprocesses.

Bug24256 - Regular users cannot see catalogs

The catalogs list displayed on the main WebView page used to contain only the catalogs for which the current user has 'read' access to all repositories therein.

The list now contains catalogs for which the user has 'read' access to at least one repository therein.

Changes to the Lisp API

No significant changes.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

Bug24300: Python shutdown hang

Previously if a session was created but not closed before Python was ready to exit, the shutdown of Python could be delayed by a very long time. This problem has been fixed. However, it is always best to call closeSession() on a session when it is no longer needed.

Bug24290 - Changes to Python tutorial

The Python tutorial is now configurable using environment varaibles which provide information about the catalog, username, password, etc. See the Python Tutorial for more information.

Java Client

Bug24238 - Java API Unicode issues

The Java client used to incorrectly process non-ascii characters in results of CONSTRUCT queries. Such characters were replaced by question marks. A similar problem could be observed with results of some other calls (e.g. evalInServer) on platforms where the default encoding used by Java is not UTF-8. A notable example is Windows, where the encoding tends to be CP-1252 (although that depends on the language).

Both issues have been corrected.

nD Geospatial

No significant changes.

AllegroGraph 6.1.2

AllegroGraph Server: General Changes

Rfe14561 - Remove the deprecated --encoding option from agload

Previously, the -C/--encoding option was used to allow agload to process N-Triples and N-Quads files encoded in a character encoding other than UTF-8.

This option has been removed. All N-Triples and N-Quads files with non-standard encoding must be converted with a separate tool, such as iconv, prior to being processed by agload. agload is described in the Data Loading (agload) document.

Bug24233 - Install-agraph.sh errors on tutorial directory when overwriting

When installing AllegroGraph to an existing installation directory, an error would occur when attempting to install the tutorial directory. This is now fixed.

HTTP Client

No significant changes.

SPARQL

No significant changes.

AGWebView

Rfe14407 - Use RDF syntax for displaying namespaces in AGWebView

The display style of namespace abbreviations in AGWebView was changed so that they can now be copied and pasted into a SPARQL query.

Bug24194 - Free text search URI incorrectly parsed slashes in WebView

In WebView, URLs of freetext search and script pages were incorrectly parsed if their arguments contained slashes. This has been corrected.

Bug24172 - Namespace sorting and duplicate inconsistencies in WebView

In AGWebView, namespace abbreviations on the Namespaces and Query pages are now sorted alphabetically. Also, adding a namespace with a duplicate prefix is now forbidden. To modify a namespace, delete it and add it, modified, again.

Bug22807 - WebView error when asking for the plan of a SPARQL update command

AllegroGraph signaled an error when asked to display the plan of a SPARQL Update command. This has been corrected.

Changes to the Lisp API

Bug23492 - with-query-environment is not documented

Documentation has now been added for with-query-environment. That macro ensures all cursors created in its body are discarded when the body completes.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

Javascript Client

Bug24207 - Javascript client library could leak cursors

It was possible for scripts using the Javascript client library to leave cursors in an active state. Although these cursors are eventually freed, they use resources and can cause lock contention. The problem is now corrected: Javascript scripts will always close all cursors automatically.

nD Geospatial

No significant changes.

AllegroGraph 6.1.1

AllegroGraph Server: General Changes

Rfe14493 - Print progress messages during long restores.

During a database restore by agraph-backup a progress report is periodically printed to stdout, showing the fraction of the data file which has been processed, the recent processing rate, and an estimate of the completion time. See Backup and Restore.

Bug24168 - Wrong triple count after a delete with duplicate suppression

Deleting an uncommited triple could cause AllegroGraph to report an incorrect triple count (possibly negative) if duplicate suppression was active at commit time.

This has been corrected.

Bug24153 - owl:sameAs interferes with owl:transitiveProperty during materialization

The presence of owl:sameAs assertions could lead to AllegroGraph failing to infer all owl:transitivePropery triples during materialization. This has been corrected. (Note that this problem only occurred using the materializer and not the dynamic RDFS++ reasoner).

Bug24147 - Support "null" as a synonym for the default graph

The remove-graph-uri, insert-graph-uri and using-graph-uri parameters of the SPARQL and Prolog HTTP endpoints now allow the default graph to be referred to by using the string "null" as the URI.

This resolves a problem with the Java client in which SPARQL Update queries issued through the Jena API did not work as expected. Specifically, a WHERE clause would fail to match any triples unless a graph was specified explictly in the query.

HTTP Client

No significant changes.

SPARQL

No significant changes.

AGWebView

Bug24062 - AGWebView uses "Repository" and "Store" interchangeably

In WebView, both "repository" and "store" (and also "triple store) were used to describe repositories. Now, "repository" is used in all places. The documentation has been updated to reflect the change,

Bug24173 - TypeError when going back to query view with saved query result in AGWebView

In AGWebView, going back to a recent query view with cached result caused a harmless error to appear on the browser console. This has been corrected.

Changes to the Lisp API

Bug24179 - Unicode issues in the remote Lisp client.

The remote Lisp client in AllegroGraph 6.1 used to fail in an unpredicatable manner if non-ascii characters were present in query results.

This has been corrected.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

nD Geospatial

No significant changes.

AllegroGraph 6.1

Server Changes

Rfe14282 TBC 5.1.4 plugin

AllegroGraph plugin for TopBraid Composer has been updated to be compatible with TBC 5.1.4. See TopBraid Composer Plugin for more information.

Rfe14279 - agraph-backup should shut down instances

Previously, agraph-backup would leave instances open and lingering after performing a backup or restore operation. This could result in hangs due to memory exhaustion when running backup-all or restore-all operations across many repositories. The likelihood of hang would depend on a combination of repository size and the value of the InstanceTimeout directive.

Now, agraph-backup will attempt to shut down the instance process of all repositories it operates on, as soon as the operation has completed. agraph-backup is described in the Backup and Restore document.

Rfe14273 - Reduce repeated query log messages

AllegroGraph was repeating some messages unnecessarily when logging query execution. This repetition has been reduced.

Bug24130 - Access to the system initfile should be protected

It was possible for any user to access the contents of the AllegroGraph init file. This has been corrected. Now only users with superuser privileges will be allowed to view the file.

Bug24128 - Blank nodes in RDF/XML collections

When parsing an RDF/XML file containing a collection with a blank node as one of its elements, AllegroGraph used to generate an invalid triple (using "li" as the predicate, instead of one of 1, 2, _3, ...).

This has been corrected.

Bug24126 - RDF/XML parser incorrectly handles empty elements.

The RDF/XML parser used to incorrectly handle empty literals encoded as property elements with empty bodies. The parser would generate a blank node instead of an empty literal.

This has been corrected.

Bug24079 - agload with '--error-strategy ignore' and unrecognized file types

agload will now print a warning before ignoring a file of an unrecognized type regardless of the value of the --error-strategy argument. agload is described in the document Data Loading. Example:

agload --error-strategy ignore repo wrong-file  
Cannot guess the format of wrong-file. Use --input to specify it.  
Load finished 0 sources in 00:00:01 (1.00 seconds).  No triples added.  
Terminating agload processes, please wait...

Compare with previous behavior:

agload --error-strategy ignore repo wrong-file  
Load finished 0 sources in 00:00:01 (1.00 seconds).  No triples added.  
Terminating agload processes, please wait...

Bug24066 - Sessions did not close their databases after idle timeout expires

When a session expired after a given timeout, it would not close its database until another session was started. This has been corrected.

Bug23964 - Incorrect escaping of IRIs in Turtle output

AllegroGraph used to incorrectly escape some characters when using the Turtle format. Furthermore, spaces and a few other characters that are not legal in IRIs were not escaped at all. Other characters, such as double quotes, were escaped using an invalid syntax (\" instead of \u0022).

Qualified names were affected by a similar problem.

This has been corrected - all characters are now properly escaped in both IRIs and qualified names.

Bug23960 - gz file import is broken

Importing .gz or .bz2 files using single-threaded import used to cause an error as the file was not correctly uncompressed.

This has been corrected.

Bug23828 - CSRF vulnerability in file upload

The import dialog in AGWebView was not adequately protected against Cross-Site Request Forgery (CSRF) attacks. When a logged-in user visited a malicious web site, an attacker could import arbitrary triples into any store the user can access.

This has been corrected - a CSRF attempt will now result in a password prompt.

Bug23820: Prevent repository access when a restore is in progress

Previously, an attempt to access a repository while it was in the process of being restored could result in random errors and problems, including being unable to close a repository. This change improves the way in which AllegroGraph notes that a repository is being restored, and improves the check for restore-mode so that these issues are prevented.

Bug23705 - A triple-store must be open to define SPIN magic properties

Changes in AllegroGraph 6.0 required that a triple-store be open before any SPIN magic properties could be defined. This has been corrected.

Bug23339 - Canonical escaping in N-Triples

AllegroGraph used to produce invalid escape sequences in URIs when printing triples in the N-Triples format. Some reserved characters, such as double quotes, were escaped with a backslash. Others were not escaped at all.

This has been corrected. All reserved characters are now properly escaped using the \u notation.

In addition, escaping of strings in N-Triples has been modified to make the output match the canonical form described in the standard. The most notable change is that non-ascii characters are no longer escaped.

Bug23181 - Consistency bug when backing up deleted bitmap

If a triple was deleted while backup was in progress, that triple would also be marked as deleted in the restored store. This has been fixed, backup now represents a consistent state of the store at the time it was started.

Bug22115: Fix configure-agraph handling of passwords.

configure-agraph could crash when entering passwords that contain certain non-alphanumeric characters in them. This bug has now been fixed.

Bug21655 - serialize-rdf/xml can emit illegal xml characters

In XML, control characters (e.g. NUL, BEL, STX, ...) cannot be represented in any form. The RDF/XML serializer (with the Lisp function serialize-rdf/xml, used by agexport when passed --output rdfxml) used to silently output such characters, producing files that could not be parsed.

This has been corrected - if the serialized data contains any invalid characters an error will be reported, as in the following example:

bash-4.2$ agexport -o rdfxml store output.rdf  
open the triple-store store on port 10035 and export to ...  
Warming up the store  
Error Unable to escape character U+0008 (Backspace). occurred during processing  
logging error to /.../agexport.log

Bug20220 - Fix bug when validating the --loaders argument to agload

The --loaders argument to agload accepts as a value a positive integer between 1 and 32. On machines with a large number of cores, agload would default to a number of loaders greater than 32 and then error due to the value being too large. This validation bug is now fixed.

HTTP Client

Rfe14445 - Add REST API for [catalogs/CATNAME]/respositories/REPONAME/exists

AllegroGraph's REST API now includes

[catalogs/CATNAME]/respositories/REPONAME/exists

which can be used to determine whether or not the named repository exists. It will return a 404 code if the repository does not exist and a 200 response with the body true if it does.

Bug23942 - Handle HTTPS scheme for SPARQL UPDATE LOAD command

The SPARQL UPDATE LOAD command would fail when using the HTTPS URL scheme. This has been corrected.

Bug23817 - Running a query that used MINUS repeatedly via REST could error

Under certain circumstances, if a SPARQL query that used MINUS was evaluated more than once using the HTTP REST API, AllegroGraph could signal an error. This has been corrected.

SPARQL

Rfe14278 - Improve SPARQL SERVICE efficiency for AllegroGraph endpoints

AllegroGraph now uses a more efficient binary data format when communicating with AllegroGraph-based SPARQL endpoints. This can lead to large performance improvements for SPARQL SERVICE calls.

Rfe14257 - DISTINCT|REDUCED mode should be passed on to SERVICE calls

When the main query uses DISTINCT or REDUCED, any use of SPARQL federation will pass on the modifier to endpoints. This can greatly reduce overall IO.

Rfe14254 - SERVICE should remove duplicates from any VALUES it passes along

If the main query is using the DISTINCT or REDUCED modifier, then the SERVICE requests should remove duplicate rows from the list of VALUES that is passed to the endpoint. This can make the query much more efficient.

Rfe14086 - New SPARQL functions for performing haversine calculations.

These SPARQL functions accept various combinations of latitude/longitude coordinates and nD encoded literals. They compute distances on the Earth's surface.

haversineLatLonLatLon
haversineLatLonLoc
haversineLocLoc

The individual names indicate the required argument types: a Lat or Lon must be a number interpreted as degrees. A Loc must be an nD literal that encodes a latitude and a longitude. Its nD encoding must be registered in the open database, but is not needed in any of the function calls themselves.

Each of these functions accepts an optional additional argument specifying the units of the computed result. It may be :km (default) or :miles expressed as a resource in the keyword namespace, i.e., <http://franz.com/ns/keyword#km> or <http://franz.com/ns/keyword#miles>.

See the SPARQL functions list for a list of SPARQL function with links to further documentation.

Rfe11390 - Modify SPARQL 1.1 lexer/parser to match changes to standard

The SPARQL parser used to fail when the input contained any:

Qualified names containing backslash escapes in the local part, such as ns:something\#\.
Qualified names containing percent escapes in the local part, such as ns:something%23%2E
'.' not followed by whitespace, for example: ?s ?p ?o.filter(?o > 42)
Some expressions containing longer products, such as: select ?x {bind (1+1*1*1 as ?x)}

All mentioned cases are now parsed correctly, in accordance with the SPARQL grammar.

Bug24161 - Correct possible segmentation faults during SPARQL DELETE operations

Under some circumstances, a SPARQL DELETE command could cause a segmentation fault. This has been corrected.

Bug24103 - Correct problem in nD magic property SPARQL queries

It was possible to get an error when executing a query that combined nD magic properties and other patterns. This has been corrected.

Bug24030 - An empty IN filter is ignored if there are any other IN filters

If a SPARQL query contained two or more IN filters that were associated with the same triple pattern and one of the IN filters was empty, then both IN filters were ignored. This has been corrected.

Bug24026 - Fix query failure when using DISTINCT, SERVICE and magic predicates

It was possible for a query to fail during planning if it used multiple SERVICE clauses, the service clauses used magic predicates, and the query asked for DISTINCT results. This has been corrected.

Bug24020 - Property path queries in saved plans with Chunk-at-a-Time engine can lose solutions

It was possible for some solutions to be lost on subsequent runs of a saved query plan using the Chunk-at-a-Time execution engine if the query used SPARQL OneOrMorePath or ZeroOrMorePath property path expressions. This bug sometimes appeared when running queries in AGWebView because it directs AllegroGraph to use saved plans. This has been corrected.

Bug23988 - Fix query failure in federations of reasoning and non-reasoning stores

Some queries could fail with an error in a federated triple-store composed of a mix of reasoning and non-reasoning leaf stores. This has been corrected.

Bug23949 - Filtering on multiple language tags could fail

The FILTERing machinery in AllegroGraph's SPARQL engine could fail to correctly process collections of language tag checks. For example, a query with a filter like:

FILTER( lang(?x) = 'fr' || lang(?x) = 'en' )

could fail. This has been corrected.

Bug23897 - Query warnings are incorrectly cached

Query warnings saved in SPARQL plans were not being updated to take account of changes in the underlying store.

Bug23893 - Federated-triple-stores and statistics caching could lead to poor plans

The caching of store statistics could lead to incorrect plans when querying a federated-triple-store. This has been corrected.

Bug23889 - Correct breakage in solR magic property queries

The SPARQL magic properties for solR queries were broken in AllegroGraph 6.0. This has been corrected.

Bug23884 - Constant pattern only CONSTRUCT could fail when executed via a saved plan

If a CONSTRUCT query template contained only constant patterns (i.e., no variables), and there was a limit on the query, and the query was being executed via a saved plan, then only the first execution would return results and subsequent executions would return nothing. This has been corrected.

Bug23882 - Algebraic query combines some BGPs too aggressively

The query planner was sometimes combining BGPs and inadvertently producing cross-products which could lead to slower query performance. This has been corrected.

Bug23881 - FILTER clause could be lost in complex SPARQL queries

It was possible for a FILTER clause to be lost during query planning. This has been corrected.

Bug23873 - Neither limit nor planner should require eval permission

Users without 'eval' permissions were unable to choose a planner or limit for a query. This has been corrected.

Bug23870 - Restriction reasoning could cause error during SPARQL query

Depending on the way it was invoked, a SPAQRL query could signal a error when restriction reasoning was used. This has been corrected.

Bug23841 - Do not allow triples in query results to be deleted

It was possible to remove triples from the database by clicking the X icon next to CONSTRUCT or DESCRIBE query results. This possibility has been disabled. CONSTRUCT and DESCRIBE queries do not return enough information to uniquely identify a triple in the triple-store. Offering to delete them is error prone and ambiguous. To delete triples, either use a SPARQL DELETE statement or the node view (click any part of a triple in the results).

Bug23755 - Problem with alternating property path in sub-queries

A sub-query that used a property path pattern with alternation could incorrectly bind the predicate of matched triples to output variables. An example of the problem would have been:

select ?s ?o {  
  { select ?s ?o {  
     ?s :hasChild | :hasParent ?o .  
    } }  
}

This query would bind the predicate of matched triples to ?o. This has been corrected.

AGWebView

Rfe14371 - Allow the number of loaders to be selected in AGWebView

It is now possible to specify the number of loaders when invoking agload through AGWebView.

Rfe13428 - Display query information for CONSTRUCT, DESCRIBE and ASK queries

The query information tab is now displayed alongside results of CONSTRUCT, DESCRIBE and ASK queries.

Bug24047 - WebView: download duplicates ignores mode

The Export duplicate statement command in Store Control ignored the choice of SPO/SPOG. This has been corrected.

Bug23948: Fixed paged behavior after using the browser's back button

Previously, the Download and Load more buttons would function incorrectly if used after clicking the browser's back button. This has been corrected.

Bug23888 - Upload progress indicator not showing

AGWebView will now display progress information when uploading a file. This was previously not visible due to a bug. Progress is reported as a text line similar to this:

Uploading file triples.nt(1475 Kb uploaded)

Bug23887 - Automatic scrolling in the AGWebView view server log pane broken

The server log in AGWebView would not auto-scroll when new log data appeared. This has been corrected.

Bug23857 - Load more does not work in Free Text search

Loading the next page of Free Text search results did not work. This has been corrected.

Bug23856 - Repeated text search breaks navigation

Page navigation would sometimes break when a user entered the same query twice in the Free Text search field. This has been corrected.

Bug23840 - Wrong even/odd row coloring when odd limit chosen

Row coloring could appear incorrect when using an odd limit and paging in the query view. This has been corrected.

Bug23839 - Planner selector grayed-out after language detection

It was possible to disable the planner selector in the query view if language autodetection was used. This has been corrected.

Bug23827 - Process list in WebView crashes on Firefox

The process and job list views of AllegroGraph WebView did not display any processes or jobs on Firefox. This has been corrected.

Bug23811 - AGWebView import fails for RDF/XML

Multi-core file import in AGWebView would fail if the chosen format was RDF/XML. This has been corrected.

Bug23810 - Importing via file upload no longer works in AGWebView

The multi-core file import feature in AGWebView was not working unless the file type was set directly. This has been corrected. In addition, AllegroGraph now recognizes application/n-triples and application/n-quads as media-types for the import of N-Triples and N-Quads (respectively).

Bug23800 - Query progress spinner only appears on every other query in Agwebview

The "Loading..." indicator was not displaying the first time that a new SPARQL query was executed via AGWebView's query editor. Now the loading indicator is shown every time.

Bug23774 - CONSTRUCT queries not displaying results in AGWebView

Results from CONSTRUCT queries were not being displayed in Safari under OS X. This has been corrected.

Bug23752 - Agwebview Lisp editor fails when digits are at the end of a line

The Lisp and Prolog script editors would become unresponsive if the last token on a line was a number. The JavaScript console showed the following error:

TypeError: undefined is not an object (evaluating 'stream.peek().toUpperCase')

This has been fixed.

Bug23141 - AGWebView's CONSTRUCT doesn't show the MORE button

Previously, if the number of results of a CONSTRUCT query exceeded the limit set by the user, there was no way to show more results. Now, there is a MORE button at the bottom of the results which functions similarly to the one used for SELECT queries.

Bug22314 - Federated-triples-stores, encoded IDs and AGWebview don't mix

Attempting to use AGWebView to query a federated triple-store whose leaf nodes used encoded IDs led to an error. This has been corrected. Note: Correct behavior of a federation of stores that use encoded IDs assumes that each leaf node has the same definitions for encoded ID prefixes.

Changes to the Lisp API

Rfe11567 - Support store specifications in the Lisp client

In addition to using the catalog keyword argument, AllegroGraph now accepts store names in the form CATALOG-NAME/STORE-NAME. E.g.,

(open-triple-store "general/test-data")

would open the store test-data in the catalog general.

You can also supply a triple-store specification in calls to open-triple-store. This will build a store instance from the specification, bind it to *db* and return it. In this case, the store instance will be given a name like store12341.

Finally, you can query store specifications directly using any of the HTTP clients by using a URL encoded version of the specification in place of a simple store name. E.g.,

curl -X GET -v "http://localhost:11040/repositories/%3Ca%3E%2B%3Cb%3E?query=select%20*%20%7B%20%3Fs%20%3Fp%20%3Fo%20%7D%20limit%205"

would send the query

select * { ?s ?p ?o } limit 5

to the federation of triples stores a and b (as specified by <a>+<b>).

Note that in this case, AllegroGraph will not start a new session so Prolog functors or other things defined in the servers initialization file will not necessarily be present.

Bug24105 - Deprecate serialize-rdf-n3 in favor of serialize-turtle

AllegroGraph can serialize data in turtle format but not in N3 so it is better to use the correct name. The now deprecated function serialize-n3-rdf will be removed in a future release and should no longer be used. serialize-turtle should be used instead.

Bug24006 - pcache Lisp module conflicts with AllegroGraph

A package conflict made it impossible to load AllegroCache and pcache in the same Lisp execution as AllegroGraph (without tiresome package manipulation). The problem has been fixed in this release.

Bug23846 - warmup-triple-store does not work in remote-lisp-client

Calling warmup-triple-store on a remote store used to have no effect. The call did not fail, but no operation was performed. The remote client will now correctly pass the request to the server it is connected to, preparing it for subsequent operations.

Prolog

Bug23877 - Prolog select exposes a security hole

There was a remote code execution vulnerability in Prolog query handling code where users without eval permissions could execute arbitrary code on the server. Executing Prolog queries now requires eval permissions. (This bug was fixed in various maintenance releases including version 6.0.2. We repeat it here because of its importance.)

Documentation

Bug23874 - Documentation search in local AllegroGraph docs doesn't work

The documentation search field in the upper left corner of AllegroGraph documentation pages now uses Google to search for the given phrase within:

franz.com/agraph/support/documentation/<version>/

where <version> is vX.Y[.Z] (v6.1 in the case of the current release). This means that even if you have a local copy of the documentation the search is done in the web copy on the Franz Inc. website.

Python Client

RepositoryConnection.createCoordinate arguments renamed

The lat and long arguments of RepositoryConnection.createCoordinate have been renamed latitude and longitude.

Bug23901 - RepositoryResult.asList() always returns None

The asList method in the RepositoryResult class was buggy and always returned None. It now correctly returns all statements from the result object.

Java Client

No significant changes.

Geospatial

Rfe12345 - Unnecessary geospatial functions removed

The functions geospatial-subtype-index-in-this-image and geospatial-subtype-index-in-db are neither useful nor necessary in the geospatial API. They have been unexported and removed from the documentation.

Bug23933 - Geospatial subtypes could be mistranslated on remote stores

It was possible for encoded geospatial data to be mistranslated when connecting to AllegroGraph via a remote-triple-store in the Lisp environment. This has been corrected.

Bug23848 - nd:inCircle :units incorrectly defaults to :miles, not :km

The :units argument to get-triples-nd-within-radius in Lisp and the similarly named functor in Prolog, and nd:inCircle in SPARQL, incorrectly defaulted to :miles instead of :km. This has been fixed so the behavior now agrees with the documentation. Since a kilometer is less than a mile, the change may cause previous queries that did not specify :units now to return fewer results. The extra results before the change were bogus.

Bug23767 - automate-nd-datatype-mappings could fail under parallel load

Spurious metadata conflict errors could be signaled if a store using automatic nD-datatype mappings was being accessed from multiple clients (for example, via agload). This has been corrected.

Bug23584 - nD query support for ISO6709 time

In an nD query, the min and max limits for a :time type ordinate may be expressed as a real number, as a dateTime UPI, or as a string literal in ISO6709 syntax (either as an xsd:dateTime typed string or an untyped string). These are supported in both SPARQL and Lisp/Prolog queries. Previously ISO6709 was not accepted in all contexts.

AllegroGraph 6.0.2 user-visible changes

Prolog

Bug23877 - Prolog select exposes a security hole

There was a remote code execution vulnerability in Prolog query handling code where users without eval permissions could execute arbitrary code on the server. Executing Prolog queries now requires eval permission. User permissions are discussed in the section Managing Users in the WebView document.

AllegroGraph 6.0.1

Server Changes

Bug23796: Disk-chunks-search.*.tmp files leaked when closing database

Starting in v6.0, files matching the pattern disk-chunks-search.*.tmp would remain open after closing a database. If a database was opened and closed frequently, this would eventually exhaust all available file descriptors in the process. This has been fixed.

Bug23715 and Bug23748 - FTI bugs

Bug23715: Creating a text index in which a word occurred in more than two million triples could sometimes crash the server.

This resulted in a log message similar to the following:

Process 'db-name FTI Merger' raised: opening #<...> resulted  
in error (code 22): Invalid argument.

Bug23748: Indexing a very long document might cause a crash.

An error would sometimes occur when trying to add a triple with very long text to a text index. The error looked like this:

Error: opening #<mapped-file-simple-stream ...>  
       resulted in error (code 22): Invalid argument.

Both bugs have been corrected.

Bug23010 - Rolling back a freshly staged text index leaves a file around

When a text index was deleted or its creation rolled back, a relatively large (~5MB) file used to be left behind and only deleted at the next checkpoint. This has been changed. The file is now deleted immediately as long as there are no live sessions that could still "see" the index before deletion.

HTTP Client

Bug23589 - Sending a SPARQL Update command to the Prolog REST API generates a backtrace

Sending a Prolog query that is not a valid list (e.g. because it is actually a SPARQL query) to the Prolog REST API caused the system to produce a backtrace and return error 500 with the following message:

Attempt to take the cdr of db.agraph.query.prolog:select  
which is not listp.

This has been corrected: the server now answers with code 400 and the following message:

MALFORMED QUERY: Prolog queries must be lists.

SPARQL

Bug23789, Bug23788 - issues with nD geospatial SPARQL queries

SPARQL n-Dimensional geospatial magic properties were signaling an error when used inside of a GRAPH clause. They were also signaling an error when the magic property was used with no variable bindings (i.e., when all of the arguments in the pattern were fixed). Both of these issues have been corrected. (Repeated under nD Geospatial below.)

AGWebView

Bug22920 - Agwebview result list has dynamic height

The window with SPARQL query results in AGWebView used to have a height of 300 pixels. The height is now dynamically adjusted to fill the available viewport.

Bug22604 - Improve handling of invalid user files

Previously, an invalid user file (e.g., one that was zero length or corrupted) would cause AGWebView to become unusable. This has been corrected. Now, invalid files will be ignored. Note that this means they will not be displayed in AGWebView's user list.

Changes to the Lisp API

Bug21770 - Some old SPARQL functionality now deprecated

Using either of the extendedp and permitted-verbs arguments to run-sparql will now signal a warning. These parameters will be dropped completely in a future release of AllegroGraph. This change also removes the special variable sparql:*use-extended-sparql-verbs-p*.

Prolog

Bug23745 - Improve future-part handling in Prolog select queries

It was possible for a Prolog select query to fail to find results if future-parts were used inside lisp expressions in the query. This has been corrected.

Bug23726 - Prolog select queries that involved type mapping could fail

Under some circumstances, a Prolog select query that used constants requiring type mapping (e.g., '1234'^^xsd:integer) could fail to return results. This has been corrected.

Documentation

No significant changes.

Python Client

Bug22303 - python-client: missing not-equal method for URIs

Trying to test if two URI objects in the Python API are different used to be impossible (i.e. code such as uri1 != uri2 raised an exception). The missing comparison method has been implemented. In addition, it is now possible to order URIs, literals and BNodes. Comparing values of two different types yields an arbitrary but consistent result under Python 2 and causes an exception under Python 3. This follows conventions established by Python's built-in types.

Java Client

No significant changes.

nD Geospatial

Bug23789, Bug23788 - issues with nD geospatial SPARQL queries

AllegroGraph 6.0

Server Changes

Use agraph for the Lisp client fasl name

AllegroGraph 6.0 now uses agraph.fasl instead of agraph[major-version-number].fasl for the name of the Lisp fasl file. Both the direct Lisp client fasl file and the various remote Lisp client fasl files (on the various platforms suitable for use as remote clients) are named agraph.fasl. Users need to manage the locations of the various individual agraph.fasl files to be sure the correct one is loaded when used.

Rfe13969 - Add HDFS input support to agload

agload now supports loading files from HDFS sources. For this to work, there must be a working "hdfs" program in your PATH and HDFS input files must be specified on the agload command line using the hdfs:// prefix. For example:

agload repo hdfs:///user/bruce/lubm-50.nt.gz

agload is described in the Data Loading document. (This change was also in release 5.1.1, as noted below, but we repeat it here as some users may not have installed version 5.1.1.)

Rfe13826 - Improve estimates on federated remote stores

Cardinality estimation of query result sets on remote stores has been greatly improved which leads to more efficient query plans on federated remote stores.

Rfe13807 - Added deleted triple purge operation

When triples are deleted they are not immediately removed from indices. They must remain in all indices until all transactions which may potentially need to see those triples (such as transactions that started before the triples were deleted) have completed (committed or rolled back). A triple that cannot possibly be accessed by any live transaction is referred to as an inaccessible triple. The existence of inaccessible triples in indices wastes space and impacts query performance. Therefore AllegroGraph now provides a facility to purge these triples from indices. See Purging Deleted Triples.

Rfe13806 - Configuration option to control index style

[Not relevant to current release.] AllegroGraph v6.0 adds a new style of index. The old style is called style 1 and the new style is called style 2. Style 2 indices are faster than style 1 when performing point queries where all of s/p/o or s/p/o/g are fixed (where the result is typically a single triple), but these indices take longer to construct and to optimize. Style 1 indices are better for queries which end up scanning many triples and typically produce results with many triples.

The Style2Indices catalog configuration directive can be used to control which index flavors should use style 2. The directive value can be a comma- or space-separated list of index flavor names, or the value none. The default value is none. Configuration options are discussed in Server Configuration and Control.

Examples

Style2Indices none  
Style2Indices posgi  
Style2Indices spogi, posgi

The Lisp API function add-index and the REST interface command PUT /repositories/[name]/indices/[type] allow for specifying the index style as well.

Rfe13641 - Change default for temporaryFileDiskSpace

AllegroGraph now defaults to using no more than 25% of the available filesystem space for temporary query files (at the time the query begins). This setting can be changed using the temporaryFilesystemSpaceLimit query option either in the AllegroGraph configuration file or via a PREFIX option to a SPARQL query. If a query attempts to use more space than this limit allows, it will be canceled. temporaryFilesystemSpaceLimit is a new name for the temporaryFileDiskSpace option. The old name has been deprecated. See agcontrol.

Rfe12000 - Minor improvements in how typed string literals are stored

AllegroGraph now stores typed string literals like "hello"^^xsd:string more efficiently.

Rfe11116 - Support SHA384 digest function in SPARQL

In addition to MD5, SHA1, SHA256, and SHA512, AllegroGraph's SPARQL engine now supports SHA384.

Rfe11078 - Improve agload verbosity behavior

Previously, passing -v or --verbose to agload (see Data Loading) could actually decrease its level of verbosity. This has been fixed. The new verbosity levels are:

0 (-v not supplied): Only report periodic load rate information.  
1 (-v)             : As above, plus print the job option summary  
		 before starting the operation.  
2 (-v -v)          : As above, plus print the name of every file  
		 that has been processed.

(This change was also in release 5.1.1, as noted below, but we repeat it here as some users may not have installed version 5.1.1.)

Bug23660 - AuditEventsToEmail parser too strict

Previously the events list in an AuditEventsToEmail configuration directive had to be comma-separated with no intervening whitespace. This has been relaxed.

Bug23654 - agraph-backup restore fails if StringTableSize specified

An error would occur when trying to restore a database if its StringTableSize did not match the one in the catalog into which it was being restored. The error was:

Fetching user information failed: String table configuration mismatch:  
Configured number of slots (X) does not match recovered number  
of slots (Y).

This has been corrected. agraph-backup is described in Backup and Restore.

Bug23604 - Improve bidirectional search speed when no paths are found

AllegroGraph's bidirectional search algorithm was expanding more nodes than necessary when no paths between the start and end were found. This has been corrected.

Bug23598 - Error when accessing a suspended account

Previously, if an account was suspended but AccountUnsuspendTimeout wasn't configured, the following error would be signaled when accessing the suspended account:

nil is not of the expected type number

This has been corrected.

Bug23585 - Audit dialog not showing any results

It was possible for the Audit Log query in AGWebView to fail to display results. This has been corrected.

Bug23526 - Fixed size deleted triple bitmap

Previously an error would occur if a triple with id greater than 34,359,738,367 was deleted. This has been fixed.

Bug23512 - Signal an error when attempting a freetext query and no freetext indices exist

Previously, AllegroGraph would simply return no results if a freetext query was executed and there were no freetext indices defined. Now, AllegroGraph will signal an error. This also modifies the behavior of list-freetext-indices for federated triple-stores. Previously, a federated store would return the intersection of its freetext indices. Now, it returns the union. This makes it easier to use the freetext indexing facility with them.

Bug23442 - Full filesystems can lead to truncated user files

In some cases a full filesystem could cause AllegroGraph user preference files to be truncated. This has been corrected.

Bug23441 - Retry tlog creation if filesystem is full

Previously, if the filesystem ran out of space while allocating a new tranasction log file during a checkpoint or commit, the instance would crash. Now the affected process will block, log a message to agraph.log, and periodically retry the operation until it succeeds. To avoid confusing hangs, failure to preallocate transaction log files on database creation will still result in an immediate error.

Bug23370 - N-Triple, N-Quad and Turtle parsers should handle UTF8 files with a BOM

AllegroGraph's parsers did not correctly parse UTF8 files that started with a byte order marker (BOM). This has been corrected.

Bug23270 - flush-in-memory-chunks deadlock

Previously, when performing duplicate triple deletion, it was possible for the AllegroGraph instance to become deadlocked and hang. This has been fixed.

Bug23361 - N-Triples parser could incorrectly import non 7-bit ASCII data

Unless a UTF-8 encoding was explicitly specified, the AllegroGraph N-Triples parser could fail to handle Unicode N-Triples data correctly. This has been corrected.

Bug23238 - Fix 'UPI has not been added to string dictionary' bug

Previously, certain queries involving large intermediate result sets could fail with a UPI has not been added to string dictionary error. This bug has been fixed.

Bug23233 - Fix bug in agraph-backup progress output.

Some garbled output showing a "Printer error" could be produced when running a backup operation using agraph-backup. This has been corrected.

Bug23231 - Prevent spurious output from agraph-replicate

Previously, agraph-replicate would emit spurious output when exiting. This output has been suppressed.

Bug23229 - agraph-replicate --stop would not stop replication.

Previously, agraph-replicate --stop would not actually cause replication to stop. This has been corrected. See Replication for information on agraph-replicate.

Bug22490 - Fix potential deadlock when creating/modifying freetext indexes.

With proper timing, it was possible hang the server when adding or updating a freetext index. This has been corrected.

Bug22489 - agraph-backup fails due to missing tlog file

Under certain circumstances agraph-backup could fail to back up a database due to not finding a transaction log file. This error has been corrected.

Bug22165 - commit-time duplicate suppression does not operate under lock

Previously, commit-time duplicate triple suppression did not properly consider triples that were added in a concurrent transaction, possibly resulting in duplicate triples being added. This has been corrected.

Bug19831 - xsd:booleans are now stored as encoded UPIs

Previously, xsd:booleans were stored as strings. They are now stored as encoded UPIs. This will make some queries slightly more efficient.

HTTP Client

Rfe13941 - Allow specification of chunk style when adding an index

The PUT command

PUT /repositories/[name]/indices/[type]

now accepts a style query parameter which may be 0, 1, or 2. See Triple Indices for a discussion of index styles.

Rfe13750 - Log errors during writing the HTTP response

While using the REST API, which most clients do, serving a request usually has a computation phase that's followed by an output phase. If an error happens in the computation phase, the appropriate HTTP status code is returned. The tricky case is with chunked encoding (for example, streaming query results back) when some results were already written to the socket. Then upon encountering an error, it is not possible to change the HTTP status code and return a meaningful error message. For this reason, errors should normally be detected during the computation phase.

With this change, any errors (except those caused by the client end of the socket being closed) encountered during the output phase are logged to make debugging easier.

Rfe13711 - Add query PREFIX option to cancel queries when there are warnings

AllegroGraph has a new query PREFIX option: cancelQueryOnWarnings. If true, then a SPARQL query will cancel itself if it encounters any warnings during query planning or execution. Typical warnings include:

unknown or out of scope variables in FILTER expressions

SELECT * { ?s ex:Predicate ?o . FILTER( ?missing > 2 ) }

constants that are not in the underlying store:

SELECT * { ?s ex:Predicate ex:thisResourceIsNotInTheStore . }

unknown variables in an ORDER BY clause:
```
SELECT * { ?s ?p ?o } ORDER BY ?age 
```
cross products:
```
SELECT * { ?a ?b ?c . ?d ?e ?f } 
```

If the option is set to no, then warnings will be logged but query execution will continue.

Bug23546 - Problems adding very long objects with JSON upload

An error could occur if triples with very long strings were imported using the REST API and the application/json content-type. This has been corrected. Other content-types were not affected by this bug.

Bug23324 - Some text index properties could not be viewed

If an AllegroGraph freetext index specified that it should only index literals with one or more particular datatypes, then that index would signal an error when its properties were read using the REST API. This has been corrected.

SPARQL

Rfe14076 - Optimize RDF serialization of CONSTRUCT and DESCRIBE queries

RDF serialization formats like RDF/XML or Turtle were spending more time than necessary trying to find namespace abbreviations. This has been improved.

Rfe13998 - agquery can now interact with SPARQL endpoints

AllegroGraph's command-line agquery tool can now send queries to SPARQL endpoints and retrieve the results. agquery is described in the Querying using agtool document.

Rfe13975 - Streamline internal AllegroGraph merge operations

Query operations that require merging of multiple result streams have been optimized. Examples include the inference of types and properties in the dynamic reasoner and computing distinct SPARQL results.

Rfe13519 - General low-level SPARQL query optimizations

Modified the low-level cursor operations in the SPARQL query engine so that they perform between 3 and 5% better for most queries.

Rfe13387 - Improve performance of SPARQL replace function

The SPARQL replace function is now significantly faster when called with a constant regular expression.

Rfe13327: Improve parsing of diskChunkRowCount and other query parameters

It is now possible to specify query parameters like diskChunkRowCount and other query parameters using sizes like 100m or 1g rather than needing to write out the numeric value as an integer.

Rfe12143 - Improve analysis of constant BIND and VALUES forms

AllegroGraph now does a better job of analyzing and using constant BIND values and single variable VALUES forms. When possible, these forms are pulled into the triple-patterns in the BGPs rather than being evaluated at run-time. For example, both of these queries will now run much more efficiently:

select * {  
    bind( 'test' as ?test )  
    ?s :predicate ?test .  
}

and

select * {  
  values ?test { 'test' 'two' 'three' }  
  ?s :predicate ?test .  
}

Rfe12131 - Remove support for the old SPARQL 1.0 query engine

AllegroGraph no longer supports the older and long deprecated SPARQL 1.0 query engine. The newer 1.1 engine is to be preferred in all cases. In particular, this means that you can no longer force the use of the 1.0 engine via the queryEngine PREFIX.

Bug23696 - Improve query plan for some property path queries in sub-queries

AllegroGraph could fail to optimize a simple alternation property path query when it appeared inside of a sub-query. This has been corrected.

Bug23653 - Remove spurious cross-product warning for (optional identity A)

AllegroGraph was signaling a cross-product warning for SPARQL queries that contained an OPTIONAL with no left hand side. For example:

select * {  
  optional { ?s a :Animal }  
}

has a query algebra like:

(left-join identity BGP)

Though the warning is technically correct, the construct above is harmless and the warning is unnecessary.

Bug23632 - SPARQL parser has trouble with escaped backslashes at the end of a literal

The SPARQL parser would fail to correctly interpret literals with an escaped backslash as their final character, e.g., 'abc\'. This has been corrected.

Bug23616 - COUNT aggregation counts unbound values from unknown variables

If a COUNT aggregration expression included variables that were not projected from the underlying query, then AllegroGraph was treating them as if they were bound and returning a count. This has been corrected.

Bug23582 - SPARQL could lose results when a sub-query used limit and distinct

It was possible for solutions to be overwritten and lost when a sub-query used both LIMIT and DISTINCT. This has been corrected.

Bug23581 - Optional mis-analysis in sub-queries could drop solutions

It was possible for AllegroGraph to mis-analyze whether a variable in a SPARQL sub-query was optional or not and therefore to lose solutions. This has been corrected.

Bug23524 - Property path queries could lose results when there is circularity

Under some circumstances, a SPARQL query that used property paths could lose results that arose from circular paths through the triple-store. This has been corrected.

Bug23519 - SPARQL sub-query and GRAPH clauses could interact badly

It was possible for a query that used sub-query and GRAPH clauses to lose solutions. This has been corrected.

Bug23489 - Queries that started with a SERVICE clause could lose bindings

If a SPARQL query started with a SERVICE clause and contained a cross-product then it was possible for some bindings to be lost. This has been corrected.

Bug23452 - Some temporal magic properties could fail to produce results

Some of AllegroGraph's temporal magic properties could produce plans that introduced filters that could not succeed. This has been corrected.

Bug23448 - Fix problem with SPARQL t:pointBeforeDatetime and t:pointAfterDatetime

The two SPARQL magic properties t:pointBeforeDatetime and t:pointAfterDatetime could fail to find bindings if the dateTime supplied was a constant not present in the triple-store. This has been corrected. SPARQL magic properties are listed here in SPARQL Magic Properties.

Bug23443 - Remote SPARQL queries cannot handle really large results

Returning very large query result sets (i.e., Gigabytes) from remote SPARQL endpoints could lead to resource exhaustion. This has been corrected.

Bug23373 - Improve SPARQL efficiency for small ORDERed or DISTINCT result sets

SPARQL queries with small result sets that use ORDER BY or DISTINCT will now be slightly faster.

Bug23344 - SPIN magic properties don't work if they use other SPARQL functions

A query-time error would be signaled if a SPIN magic property used other SPARQL functions like CONCAT or SUBSTR in its definition. This has been corrected.

Bug23341 - remote-triple-stores handle limit and offset in SPARQL queries incorrectly

If a query specified an OFFSET as part of the query string, then any externally imposed OFFSET (i.e., from the HTTP query parameters) was ignored. This has been corrected. If a query specified a LIMIT as part of the query string, then the smaller of that limit and the externally imposed one would be used instead of just using the external one. This has been corrected. Note that this change means that specifying a limit in a query in AGWebView will have no effect as the limit is controlled via the dropbox selection box.

Bug23329 - Use SPARQL/JSON for SERVICE calls to avoid problems with embedded control characters

AllegroGraph now uses SPARQL/JSON rather than SPARQL/XML to receive results from SPARQL endpoints. This makes it possible to return results that contain control characters like ^Z. The format is also more concise and easier to parse which makes processing the transmissions approximately twice as fast.

Bug23321 - Nested sub-queries and BIND forms can fail to compute correct results

In rare cases, it was possible for a query involving nested SPARQL sub-queries and BIND expressions to fail. This has been corrected.

Bug23311 - Requesting application/nquads from a CONSTRUCT query leads to error

If an application requested application/nquads from a CONSTRUCT query via the HTTP interface, a string-table lookup error would result. This has been corrected. Although version 1.1 of SPARQL CONSTRUCT can only return triples, it still makes sense to ask for N-Quads. In this case, the N-Quads will simply have the graph place left off.

Bug22674 - DESCRIBE and CONSTRUCT queries on end points could fail

SPARQL CONSTRUCT and DESCRIBE queries against remote SPARQL end points could fail when trying to parse the returned results. This has been corrected.

Bug22672 - Queries where every FROM NAMED graph did not exist could signal an error

If every FROM NAMED graph in the dataset did not exist, then some queries that used the GRAPH clause in their body could signal an error rather than returning results. This has been corrected.

Bug22515 - Increase responsiveness of query canceling during result generation

AllegroGraph did not check for query canceling or timeouts during result post-processing (e.g., while sorting the final result or removing duplicate entries from a DISTINCT result set) or result generation. This has been corrected.

Bug22292 - tripleid magic property should handle fixed triple IDs

The SPARQL magic property for AllegroGraph tripleId-based reification would not function for fixed triple IDs. This has been corrected.

Bug22268 - Sub-queries that used Aggregation and limit could lose solutions

If a sub-query had a limit and used aggregation, then it was possible for some solutions to be lost during processing. This has been corrected.

AGWebView

Bug23690 - Fixed bug breaking the AGWebView View Statements dialog in IE.

A bug in AGWebView that prevented the View Statements dialog from fetching and displaying statements when brought up in Internet Explorer has been fixed. The symptom was a partially loaded page with the spinning "Loading..." icon remaining permanently on the page.

Bug23521 - More... button in AGWebView is broken for Prolog select queries

The button to show additional results in the AGWebView query window was broken for Prolog select queries. This has been corrected.

Bug23358 - AGWebView doesn't always display Unicode consistently

Under some conditions, AGWebView would display literals and resources containing Unicode using the \u escaping mechanism of the N-Triples syntax rather than just displaying the characters directly. This has been corrected.

Bug23315 - Running backup from within AGWebView does not work

The Backup this Store dialog in AGWebView was broken. This has been corrected.

Changes to the Lisp API

Rfe13941 - Allow specification of chunk style when adding an index

add-index now accepts a style keyword argument which can be 0 or nil (either meaning use the default chunk style for the flavor based on the Style2Indices configuration parameter), 1, or 2. See Triple Indices for a discussion of index styles.

Bug23600 - remote-triple-stores CONSTRUCT queries now correctly returns cursors

In the Lisp client, a SPARQL CONSTRUCT query should return a cursor that yields arrays of bindings but the remote-triple-store was instead yielding lists of bindings. This has been corrected.

Bug23597 - remote-triple-store CONSTRUCT queries return nil bindings

Many of the programmatic RDF formats (such as :lists or :arrays) for SPARQL CONSTRUCT queries on remote-triple-stores returned nil bindings rather than the actual values. This has been corrected. Note that this bug did not effect text based output formats like N-Triples or Turtle.

Bug23510 - remote-triple-stores did not support load-trig

Lisp remote-triple-stores did not support loading the TriG format. This has been corrected.

Bug23509 - Altering restriction-reasoning-enabled-p could lead to an internal error

Altering the value of restriction-reasoning-enabled-p using the setf method after the reasoner had been applied and prepared could lead to an error. This has been corrected.

Bug23404 - It was impossible to create an anonymous remote session

AllegroGraph would signal an error that This session is owned by another user when trying to create an anonymous remote session in the Lisp client. For example, the following would fail:

 (open-triple-store "remote"  
    :triple-store-class 'remote-triple-store  
    :server "www.example.com"  
    :specification "<sp2b-5e4>[rdfs++]"  
    :port 9002)

This has been corrected.

Bug23320 - The Lisp direct client now ensures that UPIs are interned when calling add-triple

The Lisp direct client was not checking that UPIs were interned which could lead to a UPI-not-in-string-table error when there were bugs in customer code. This has been corrected. The check does add a small cost to programmatic triple insertion. The while-not-validating-that-upis-are-in-string-table macro can be used to temporarily disable the checks once you are sure that the code is correct.

Bug22844 - The SPARQL datatype function failed to handle nD-geospatial literals

The SPARQL datatype function was unable to handle nD-geospatial literals. This has been corrected.

Bug22793 - Problem with remote-reasoning-triple-stores and SNA

On the Lisp client only, calling Social Networking (SNA) functions on a remote-reasoning-triple-store instance could lead to a stack overflow. This has been corrected.

Bug22791 - get-triple was broken for remote-reasoning-triple-stores

On the Lisp client only, calls to get-triple on a remote-reasoning-triple-store could fail to return results. This has been corrected.

Bug22587 - copy-triples doesn't support remote triple-stores

The copy-triples function did not work on remote-triple-stores. This has been corrected. Note that copy-triples on remote-triple-stores requires that

the triples are all valid RDF
no blank nodes are used.

Bug22586 - Improve copy-triples

The copy-triples function was not behaving correctly with respect to the source-db argument. This has been corrected.

Bug22447 - Fix cursor memory management

Under rare circumstamces, especially in a multi-threaded environment, cursors used in the Lisp client could end up being used in an unsafe way leading to various errors. With the AllegroGraph server, shared backends experiencing high load were at the risk of such failures. This bug has been fixed.

Also, in the Lisp client with a direct connection, any code that involved the creation of cursors (such as queries and calls to get-triples) had to be wrapped in a with-query-environment form. While that's still the recommended approach to take in production, with this change when there is no enclosing with-query-environment, cursors holding critical resources get finalizers which makes it more convenient to experiment at the REPL by reducing the risk of resource exhaustion.

Bug20852 - remote-triple-stores do not support SPARQL update

Lisp remote-triple-stores now support SPARQL update.

Prolog

Rfe13968 - Add :offset to Prolog select

AllegroGraph's Prolog based select query language now supports an :offset option. This can be used to skip the first offset results. For example,

(select (?class)  
  (:offset 10)  
  (q ?s !rdf:type ?class))

would skip the first 10 results and return the rest of the bindings for ?class.

Documentation

No significant changes.

Python Client

Rfe12828 - Support Python 3

AllegroGraph now supports both Python 2 and Python 3. The sources for Python 2 continue to be in the src2 directory whereas those for Python 3 are in the new src3 directory.

Java Client

No significant changes.

nD Geospatial

Rfe13978 - Validate arguments to ndfn:ordinateValue at parse time when possible

AllegroGraph now validates constant arguments to ndfn:ordinateValue at parse time. See the N-dimensional Geospatial Usage Guide and Example document.

AllegroGraph 5.1.2

Server Changes

Bug19296 - AllegroGraph can stop responding because all threads seem busy

Under rare circumstances it was possible for AllegroGraph to stop responding. Such a hang was accompanied by "All threads busy, pause" log messages. This has been corrected.

AllegroGraph 5.1.1

Because there are only a few changes in release 5.1.1, only headings with actual changes are listed.

Server Changes

Rfe13969 - Add HDFS input support to agload

agload repo hdfs:///user/bruce/lubm-50.nt.gz

agload is described in the Data Loading document. HDFS loading is described in the section Loading files from HDFS filesystems in that document.

Rfe11078 - Improve agload verbosity behavior

Previously, passing -v or --verbose to agload could actually decrease its level of verbosity. This has been fixed.

The new verbosity levels are:

0 (-v not supplied): Only report periodic load rate information. 1 (-v) : As above, plus print the job option summary before starting the operation. 2 (-v -v) : As above, plus print the name of every file that has been processed.w

agload is described in the Data Loading document.

AllegroGraph 5.1

Server Changes

Rfe13570 - Support the TriG format for serialization

AllegroGraph can now serialize data using the TriG format. For example, you can use the output format trig in agexport or an Accept type of application/trig from the REST API.

Rfe13501 - Implement HTTPNoProxy configuration directive

This change adds a new global configuration directive, HTTPNoProxy. HTTP requests to domains that match one of the suffixes specified with HTTPNoProxy are never proxied. HTTPNoProxy can be specified multiple times.

HTTPNoProxy localhost  
HTTPNoProxy mydomain.com

Requests to 'localhost' and '127.0.0.1' are never proxied. Top level directives are described in Server Configuration and Control.

Rfe12010 - Support the TriG format for data import

AllegroGraph now supports the TriG data format for loading data. See http://www.w3.org/TR/trig/ for details on the format. See Data Loading.

Rfe11730 - Update Turtle parser to conform to the specification

AllegroGraph's turtle parser now passes the W3C conformance tests (http://www.w3.org/TR/turtle/). The parser is also significantly faster.

Bug23077 - Rare thread safety issue in shared back-ends

Previously, when multiple requests for the same repository were being executed concurrently on the same shared back-end a very rare thread safety issue could cause the backend to die. This bug has been fixed.

Bug22944 - Freetext indexer could mishandle encoded literals

AllegroGraph's built-in freetext indexer was trying to index encoded literals (numbers, dates, times and dateTimes) and would sometimes misinterpret them leading to extremely slow indexing. This has been corrected by preventing the indexing of these literals. This means that freetext indices will not include values for encoded literals.

Bug20556: Double-fault of trie-full condition

Previously, when adding a word with many hits to a text index a "trie-full" error could occur. This has been fixed.

HTTP Client

Bug23064 - Post requests to the statements REST API could fail

It was possible for POST requests to the statements REST API to corrupt the stream of bytes being sent. This has been corrected.

SPARQL

Rfe13683 - Improve variable count tracking during query execution

AllegroGraph keeps better track of the estimated cardinality of bindings for variables during query execution which improves the execution engine's overall efficiency.

Rfe13601 - Improve handling of property path queries that use simple alternation

AllegroGraph can now apply its constraint machinery to simple property path queries that use only alternation. For example:

select ?o {  
  ?s :a | :b | :c ?o .  
  filter( ?o > 10 )  
}  
This means that queries like the above can operate significantly faster.

Rfe13599 - Improve query pattern constraint logic

AllegroGraph now does a better job in choosing how to constrain variables in SPARQL triple-patterns during execution.

Rfe13476 - Add more logging information to SPARQL SERVICE queries

AllegroGraph now includes more logging during the execution of SPARQL SERVICE queries.

Rfe13426 - Provide more control over federated SPARQL queries

The new query option serviceRequestSize provides control over the size of the POST requests used during SPARQL federated query. The default size is one megabyte. A smaller size may cause additional trips to the remote server. Some servers may reject requests that are too large. SPARQL query options are listed in the SPARQL Query Options of the SPARQL Reference.

Rfe13213 - Allow BIND to float during query analysis

AllegroGraph's SPARQL query analyzer now does a better job rewriting query patterns involving BIND. In particular, the BIND clause now floats around JOIN, GRAPH and FILTER clauses. This can help make these queries perform significantly better.

Rfe12269 - Improve query planning with respect to cross products

AllegroGraph's SPARQL query planner now does a better job of detecting cross products at plan time and modifying the query plan to avoid the cross product when this is possible.

Bug23125 - Improve query planning for some property path queries

SPARQL Property Path queries using the zero-or-more operator (i.e., *) could sometimes choose a sub-optimal plan because AllegroGraph was misestimating the size of the triple-patterns involved. This has been corrected.

Bug23095 - Correct internal error that occurred during some Property Path queries

Some Property Path queries could fail to run to completion because of an internal bookkeeping error. This has been corrected.

Bug23081 - Certain SPARQL queries could cause cursor data-structure leakage

Certain SPARQL queries could fail to correctly clean up all of the cursors that they used which could lead to backends becoming unresponsive if they were under heavy load. This has been corrected.

Bug23027 - SPARQL was not ordering reified triple IDs correctly

The SPARQL query engine was not correctly sorting URIs representing triple IDs. This has been corrected.

Bug23026 - Some SPARQL SERVICE calls could mangle their query

It was possible for a SPARQL SERVICE call to lose space characters and thereby send the wrong query to the remote end point. This has been corrected.

Bug23005 - RDF/XML serializer could emit an invalid namespace abbreviation

If there was a namespace mapping between the empty string and some URI, then the RDF/XML serializer would emit an invalid namespace abbreviation. This could occur, for example, when running a CONSTRUCT query using a PREFIX abbreivation like

PREFIX : <http://example.org#>

This problem has been corrected.

Bug23004 - Sparql XML parser could fail to parse some literals

AllegroGraph's SPARQL XML parser is used when making federated queries against remote endpoints. This parser was incorrectly handling literals with ampersands (or other escaped characters) in them. This has been corrected.

Bug22999 - When embedded in a GRAPH clause freetext SPARQL queries could fail

Depending on the data and the query, a freetext match or matchExpression magic predicate could fail to find all results when it was embedded inside of a GRAPH clause. This has been corrected.

Bug22994: Fix bad interaction between SPARQL sub-query and federation

If a SPARQL query used a sub-query inside of a federated query, AllegroGraph could lose track of the variables involved and create inadvertant cross-products. This has been corrected.

Bug22966 - SERVICE clause in a sub-query could fail

Depending on the variables projected, a SPARQL query that used a SERVICE clause inside of a sub-query could fail during execution.

Bug22937 - GROUP BY returned an empty row when there are no aggregates projected

If a SPARQL aggregation query projects only variables (i.e., it does not use any aggregation functions) and the query has no results, then AllegroGraph would return a single row of unbound bindings rather than returning no results at all. This has been corrected. An example of the query in question would be

select ?anything {  
  ?s <ex://missingPredicate> ?anything .  
} group by ?anything

Bug22915 - SPARQL STR function is incompatible with aggregation

Queries that used the SPARQL STR function on the result of an aggregation would fail. This has been corrected. An example of such a query would be:

select ( str( count(1) ) as ?count) {  
  ?s :testPredicate ?o .  
}

Bug22904 - Bad query bookkeeping could cause performance problems

Given the right data and the right query, the SPARQL query engine could be led to evaluate triple patterns twice: once with constraints and once without which would lead to severe performance problems. This has been corrected.

Bug22899 - Query options could not always be used with SPARQL SERVICE

When using the SERVICE clause, AllegroGraph's query PREFIX options were not always being sent to endpoints in a way that they could be read. This has been corrected.

Bug22874 - N-Dimensional magic property queries could fail

Depending on the data and the query, AllegroGraph could choose an incorrect plan ordering for an n-Dimensional SPARQL magic property. This would lead to the UPI not in string table error. This has been corrected.

Bug22845 - A query timeout of zero now means "no timeout"

Previously, a query timeout of zero was interpreted literally to mean that the query should terminate immediately. Now, a timeout of zero means that the query should run as if there is no timeout specified. This is especially helpful when a default timeout is included in the AllegroGraph configuration file.

Bug22751 - Error using one-or-more property paths in SERVICE clause

AllegroGraph was signaling an error if a query tried to use a one-or-more property path query inside of a SERVICE clause. This has been corrected.

Bug22680 - Correct SPARQL parsing problem for property paths in object lists

The SPARQL Parser was not correctly handling property paths embedded inside object lists like ?s <ex:predicate> [ <ex:Path>* ?o ]. This has been corrected.

Bug22593 - Use the correct count estimates during query execution

Depending on the query and the store, AllegroGraph could sometimes use the wrong variable count estimates which would lead to sub-optimial performance. This has been greatly improved.

Bug22388: Chunkprocessingmemory defaults to a larger value than memoryLimit

The chunkProcessingMemory size defaulted to a larger size than the memoryLimit which meant that queries could fail when using chunk processing. This has been corrected.

Bug22212 - Magic property for t:intervalStarts logic was wrong

The magic property for t:intervalStarts was incorrectly behaving the same as t:intervalCotemporal. This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

Rfe13593 - The compiler is needed in direct lisp clients that perform queries

This change will likely not affect any users since the compiler is normally in Lisp images, but it is possible to build an image without the compiler and without the compiler being available to load (though, again, it is unusual to do so). Such images serving as direct Lisp clients cannot perform queries.

Bug22973 - Fix Lisp direct client sigsegv when rolling back a closed store

Previously, in the Lisp client when using a direct connection to a triple store, calling rollback on a closed store would produce a SIGSEGV error. With this fix, an informative error condition is signaled.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

nD Geospatial

Bug22786, Bug22850: improve nD string writers and parsers

nD data encoding and decoding supports various human-readable formats (ISO6709 and ISO8601) when such a format can be applied unambiguously to a particular nD encoding. If a human-readable format applies, it will be preferred on decoding (output) and accepted on input (encoding). Previously some correct externalization formats were rejected on input or not used on output.

nD now correctly supports encodings with multiple ordinates of the same specialized type, i.e. lat, lon, alt, and time. Ordinates must of course all have different names. Use of such encodings will necessarily suppress human-readable externalizations since they are ambiguous.

Error detection during nD encoding/decoding is improved along with error message text.

AllegroGraph 5.0.1

Server Changes

Bug23016 - Incorrect triple-count and garbage triples

Under certain circumstances (particularly when delete-duplicate-triples is used), crash recovery or replication can result in an incorrect triple count and garbage triples being added to the database. This has been corrected. We recommend that users using replication and duplicate triple deletion recreate their replicas from a fresh backup. Replication is described in the Replication document.

HTTP Client

Bug23040 - Performing the same ASK query more than once under REST could error

Under certain circumstances, if an ASK query that used joins was evaluated more than once under the HTTP REST API, then AllegroGraph would signal an error. This has been corrected.

AllegroGraph 5.0

Server Changes

Rfe13506 - Minor query optimization for triple pattern filters

AllegroGraph now performs triple pattern filtering slightly more efficiently when the current set of solutions has only a single member.

Rfe13405 - Warn if transparent hugepages are enabled

Due to the potential for negative performace impact, AllegroGraph now logs a warning into agraph.log on startup if Transparent Hugepages are enabled. See this section of Performance Tuning for information on the transparent hugepage issue and its effect on performance.

Rfe13376 - Improve hashing performance

AllegroGraph's internal hashing functions have been further optimized leading to small general performance improvements.

Rfe13346 - configure-agraph should allow for specification of port

The configure-agraph script now prompts for the Port configuration parameter and accepts the --port command line argument. See Running the configuration script in Server Installation.

Rfe13265 - Update N-Triple and N-Quad parsers

AllegroGraph's N-Triple and N-Quad parsers now support the full W3C specifiation and pass all tests. The performance is also slightly improved. Note that this change means that AllegroGraph will no longer load invalid N-Triple and N-Quad source data unless the relax-syntax argument is used. See Data Loading, the Lisp API functions load-ntriples and load-nquads, and [PUT/POST /repositories/name]/statements in REST/HTTP interface.

Rfe13172: Allow setting size of strings hash table at database creation time

A new database configuration parameter StringTableSize is now available. It can be used to configure the number of hash table slots used for the string storage hash table and setting it properly can enhance performance. See Catalog Parameters that Improve Performance in Performance Tuning for more information on performance-enhancing options for details.

Rfe13146 - Improve suboptimal index scan performance

The efficiency of low-level cursor operations has been improved when an optimal index for the cursor is not available.

Rfe12872 - Remove ag3-export / ag4-import

The obsolete ag3-export and ag4-import utilities have been removed. Serializing data to nquads in AllegroGraph 3 and loading it into AllegroGraph 4 is the recommended alternative.

Rfe11580: Added --no-daemonize flag to agraph binary

The agraph binary is the actual program which starts AllegroGraph. It is not typically invoked directly, but rather using the agraph-control script, but it can be called directly. When it is and --no-daemonize is supplied, the service daemon will remain in the foreground. See agcontrol.

Bug22837 - Text index incorrectly indexing the string "nil"

Since version 4.13 the text indexer was incorrectly indexing the word "nil" for triples that were ineligible for text indexing. This resulted in unnecessary work being performed and space being consumed. This has been fixed. We recommend that you drop and recreate any existing text indexes.

Bug22835 - Fix numa info parsing failure

At startup, information about numa (non-uniform memory access) topology is collected and logged. Previously, parsing failed with an error if a cpumap file in /sys/ contained a comma separated list of numbers instead of just a single number. This bug has been fixed.

Bug22767 - Use agraph5 for the Lisp client fasl name

AllegroGraph 5.0 now uses agraph5.fasl instead of agraph4.fasl for the name of the Lisp fasl file.

Bug22754 - Concurrent duplicate triple deletion can result in inaccurate triple count

Under some circumstances duplicate triple deletion transactions which occur concurrently with other deletion transactions could result in an inaccurate triple count. This has been corrected.

Notes:

Though the triple count could become inaccurate, the integrity of the triples in the database remained intact.
The triple count on already-affected databases will remain inaccurate until they are recreated.

Bug22566 - Encoded ID validation was too weak

In some cases, a URI representing an invalid encoded ID could be accepted by AllegroGraph instead of signaling an error. This has been corrected.

Bug22564 - Data import could fail to determine the correct base URI

When importing a compressed RDF/XML file, AllegroGraph could fail to determine the correct base URI. This has been corrected.

Bug22559 - RDF/XML parser should validate the base URI if supplied

Previously, the RDF/XML parser failed to validate the base URI it was given which could lead to invalid data. This has been corrected.

Bug21867 - Ensure text index queries are called using strings

AllegroGraph sometimes called lower level text index query functions with invalid arguments. This has been corrected.

Bug21460 - Polygons data cannot be exported and imported

AllegroGraph now converts resources of the form

<http://franz.com/ns/allegrograph/4.14/index#_N>

(where N is replaced by a whole number) into encoded UPIs rather than hashed ones. These URIs are used to serialize AllegroGraph polygon data which means that polygons can be efficiently exported and imported. Note that the Lisp API upi->value will return the numeric representation for the UPI rather than the string representation. Use part->string if the string representation is needed.

Bug20497, Rfe11047 - Fix fti too many open files error

Previously, when constructing a new text index for a database with existing triples, there was potential for the operation to fail with a "Too many open files" error. This has been fixed.

HTTP Client

Rfe13304 - Support relaxed load syntax for remote triple-stores

The HTTP REST API for loading statements now includes the relaxSyntax argument. When true, N-Triple and N-Quad data are not validated as stringently. For example, blank node names may include underscores. See PUT/POST statements in REST/HTTP interface.

Rfe13249 - Augment optimize-indices API to allow for single index optimization

The HTTP API's indices/optimize service now takes an additional index query parameter which can be specified multiple times. If omitted, then all of a store's indices will be optimized. If supplied, each parameter value should be the name of an index whose optimization is being requested. See [POST /repositories/name]/indices/optimize in REST/HTTP interface.

Rfe11547 - Support Sesame query timeout parameter in HTTP

AllegroGraph's SPARQL HTTP REST interface now supports an optional timeout parameter. If specified, it must be a number greater than zero which denotes the number of seconds to limit SPARQL query execution time. If unspecified, then there is no limit on the length of a time a query may run. See [GET/POST /repositories/name in REST/HTTP interface.

Rfe11221 - General type-mapping improvements

The interface for setting up predicate- and datatype-mappings has been simplified. The same HTTP REST API calls can be used to setup regular XSD datatype mappings and geospatial ones (including cartesian, spherical and n-Dimensional). In each case, the datatype must be specified using its URL. E.g., the 10x10 cartesian mapping with resolution 1 would use the URL:

<http://franz.com/ns/allegrograph/3.0/geospatial/cartesian/0.0/10.0/0.0/10.0/1.0>

See Type mappings in REST/HTTP interface.

Bug22576 - POST|PUT statements service should not require eval permission

AllegroGraph would not allow client-side files to be imported into a store unless the user had eval permission. This has been corrected. Of course, eval permission is still required to import server-side files.

SPARQL

Rfe13510 - Improve performance of the SPARQL query REST API

AllegroGraph now automatically caches query information across REST API calls to improve overall query performance.

Rfe13323 - Provide API to retrieve only the ID part of an encoded ID

There is now a SPARQL function, , that retrieves the non-prefix portion of an encoded ID. For example, if the encoded ID has IRI http://example.com/person@@3489a, then the function will return "3489a". In a query, this would look like:

select * {  
  _:b ex:identifier ?id .  
  bind( <http://franz.com/ns/allegrograph/fn#encodedIdId>(?id) as ?value )  
}

Rfe13305 - Deprecate old-style SPARQL GEO syntax

Because AllegroGraph's magic properties support a richer range of geospatial features than the old-style SPARQL GEO syntax, the old syntax is being deprecated and will now signal a warning when it is used. See N-dimensional Geospatial Overview.

Rfe13297 - Reduce memory consumption for distinct aggregation in SPARQL queries

AllegroGraph now better manages memory when evaluating distinct aggregation queries like:

select ?dept (count(distinct ?student) as ?sc) where {  
  ?dept a :Department .  
  ?student :memberOf ?dept .  
} group by ?dept

Rfe13289 - When logging, use query supplied namespaces for abbreviations

Previously, AllegroGraph would ignore SPARQL query supplied namespaces when displaying resources during query logging. It now uses these namespaces to improve the readability of the display.

Rfe13171 - Improve query checks for unknown variables

AllegroGraph now warns more consistently when a valid SPARQL expression uses variables in unreasonable ways. For example, this query is perfectly valid but the ORDER BY variables are not bound by the query so it is unlikely to be what was intended:

select ?a ?year ?label {  
  ?a a :Car .  
  ?a :modelYear ?year .  
  ?a rdfs:label ?label .  
} order by ?autoYear

Rfe12470 - Apply multiple constraints to the same triple pattern

Previously, AllegroGraph could only apply a single logical FILTER expression to any given triple pattern. This meant that queries like:

select * {  
  ?s ?p ?o .  
  filter( ?p = ex:pred1 && ?o > 10 )  
}  
would optimize the query plan with either the ?p constraint or the ?o  
constraint but not both. This has now been improved so that queries  
like the above will use both constraints simultaneously and thereby  
run much more efficiently.

Bug22803 - Fix bug in authorizationBasic query option

The authorizationBasic query option was not being parsed correctly which meant that queries that tried to use it would fail. This has been corrected.

Bug22802 - Support userinfo in SPARQL service request URIs

URIs used in SPARQL service requests did not allow the inclusion of userinfo information so that queries like this would fail:

select * {  
  service <http://user:[email protected]:7001/repositories/sp2b-1e4> {  
    ?s ?p ?o  
  }  
} limit 5

This has been corrected.

Bug22750 - Correct an error in SPARQL property path queries

In some circumstances, a SPARQL property path query could signal an error when trying to find paths between fixed endpoints. This has been corrected.

Bug22735 - SPARQL federated query could fail to recognize geospatial data

SPARQL federated query (via the SERVICE) clause could fail to transform returned results using AllegroGraph geospatial encoding. This would lead to query errors and has been corrected.

Bug22735 - SPARQL federated query could fail to recognize geospatial data

SPARQL federated query (via the SERVICE) clause could fail to transform returned results using AllegroGraph geospatial encoding. This would lead to query errors and has been corrected.

Bug22734 - Query options should be passed along to the HOST of any SERVICE clauses

Previously, any custom query options set using the PREFIX notation were not passed along to the remote server when using SPARQL Federated query. Now, these options are passed to the remote server which provides more control when that server is running AllegroGraph.

Bug22732 - Correct errors for SERVICE clauses that used blank node labels

SERVICE clauses that used blank node labels (or property paths that might be rewritten using blank node labels) could lead to an error when AllegroGraph tried to merge these bindings with the containing query. This has been corrected.

Bug22731 - Error when using property paths in a SERVICE query with distinct results

If a query asked for DISTINCT results and had a property path query inside of a SERVICE clause, then AllegroGraph would signal an error during query planning. This has been corrected.

Bug22730 - Some uses of UNION and VALUES could lead to an error

Some uses of UNION clauses and SPARQL in-line data via the VALUES clause could lead to a run-time query error. This has been corrected.

Bug22727 - EXISTS filters sometimes performed more computation than necessary

SPARQL EXISTS (and NOT EXISTS) filters were performing query computation even when the main result set was empty (and hence had nothing to filter). This has been corrected.

Bug22660 - Temporal magic properties introduce new variables

Temporal magic properties like were introducing new variables into queries so that a query like

prefix m: <http://di.uoa.gr/~charnik/music_db#>  
prefix time: <http://franz.com/ns/allegrograph/3.0/temporal/>  
select * {  
  m:p_1939-08-19 time:pointBefore m:p_1980-12-04 .  
}

was returning bindings instead of empty rows. This has been corrected.

Bug22588 - Further improve handling of multiple range filters on the same variable

AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like

select * {  
  ?s ex:test ?age .  
  filter( ( ?age >= 10 && ?age <= 20 ) ||  
          ( ?age >= 60 && ?age <= 70 ) ||  
          ( ?age >= 80 && ?age <= 90 ))  
}

will now execute as the combination of two range queries rather than a single scan. This change is similar to the change for bug22535 but more general.

Bug22569 - Improve canceling of SPARQL SNA generators

Previously, neither the administrator jobs list nor the query canceling API functioned properly if a query used an SNA generator that was implemented via a separate SPARQL query. This has been corrected. The SNA generator SPARQL queries will no longer appear in the administrator jobs list and they will now inherit any timeout from the query that they are running in service of. This means that canceling the parent query will cancel these child queries.

Bug22541 - Improve handling of invalid typed literals

Previously, the SPARQL parser would generate a general error and log a backtrace if a query included illegal XSD datatypes like "2014-Mar-13"^^xsd:date. These are now treated as SPARQL parse errors.

Bug22541 - Improve clause reordering for some SPARQL GRAPH queries

AllegroGraph's statistical BGP reordering could lead to sub-optimal query plans for certain queries that involved GRAPH queries and FILTERs on ranges. This has been corrected.

AGWebView

Rfe13530 - Rearrange some of AGWebView's settings

The "show long parts" and "show contexts" settings are global to the browser session but they were only available on the query viewer. These settings are now also accessible via the new Utilities | Settings menu. See the AGWebView document.

Rfe13504: Add a "large operations warning" to AGWebView

AGWebView's site settings now include a place to enter text that will appear before it executes an operation that potentially exposes a large amount of repository data. Note as well that the site settings are now available via the new Utilities | Settings menu.

Rfe13183 - Agwebview uses Unicode rather than \u escaping to display data

AGWebView now presents non-ASCII characters using Unicode instead of their \uXXXX escape sequences.

Rfe12523 - Improve data import dialogs in AGWebView

The data import dialogs in AGWebView now support AllegroGraph's relaxed syntax and continue on error options. Relaxed syntax allows the RDF importers to accept data that it not strictly correct but is still easily understandable. For example, a blank node label is not allowed to include the - (dash) character but many source files use it regardless. Normally, AllegroGraph will not allow these files to be imported but if relax syntax is turned on, it will. Continue on error allows AllegroGraph to continue loading data even if it encounters a bad triple or quad. Note, this option only affects line based formats like N-Triples or N-Quads. As ignoring errors is an all or nothing affair, this option should be used with caution: it will not be possible to determine how many triples were skipped or even if any errors were encountered. These new options correspond to the --error-strategy and the --relax-syntax options to agload.

Rfe11466 - Use double-clicks to navigate the file picker in AGWebView

AGWebView's file picker for selecting files on the server side used single-clicks rather than double clicks to change directories. This has been corrected.

Bug22800 - Fix regression in v4.14 that breaks proxying through frontend

There was a regression in v4.14 that partially broke the use of AGWebView over https. This is fixed. Additionally support is added for proxying requests to https sessions through an http frontend, or to http sessions through an https frontend.

Changes to the Lisp API

Rfe13249 - Augment optimize-indices API to allow for single index optimization

The Lisp API's optimize-indices function now takes an additional indices argument. This can be nil (the default) or :all to optimize all of the triple-store's indices, a single index name (like :psogi) to optimize one index or a list of index names to optimize each of them.

Rfe13184 - Check for uninterned UPIs when adding triples

Previously, AllegroGraph did not ensure that the UPIs in a triple being added were already interned in the triple-store's string-table. This led to the possiblity of adding triples that could not be re-serialized. The default is now to check UPIs unless the code doing the adding is wrapped in the new while-not-validating-that-upis-are-in-string-table macro.

Rfe6878 - open-triple-store can take a triple-store instance

The open-triple-store function can now be called using an existing triple-store instance. In this case, it will return a new instance of the same underlying store.

Bug22811 - Fix use of direct Lisp client when AllowHTTP is false.

Previously, the direct Lisp client would be unable to access an Allegrograph server if it was started with the config directive AllowHTTP set to false. This bug is now fixed.

Bug22772 - Include inferred-triple-p in the remote Lisp client

The inferred-triple-p function was inadvertently left out of the remote Lisp client. This has been corrected.

Bug22703 - Add relax-syntax argument to load-nquads|ntriples-from-string

The funtions load-nquads-from-string and load-ntriples-from-string were missing the relax-syntax argument. This has been corrected.

Prolog

No significant changes.

N-dimensional geospatial

Rfe13364 - Add automate-nd-datatype-mappings switch to agload

agload now supports AllegroGraph's automatic N-Dimensional geospatial type mappings. If the new --automate-nd-datatype-mappings option is specified on the command-line, then any geospatial datatypes in a source file will be automatically recognized and added to the triple-store. See Data Loading.

Rfe13350 - Add additional standard namespace abbreviations

The following namespace abbreviations have been added to AllegroGraph:

nd - <http://franz.com/ns/allegrograph/5.0/geo/nd#>
ndfn - <http://franz.com/ns/allegrograph/5.0/geo/nd/fn#>
keyword - <http://franz.com/ns/keyword#>

These are helpful in using the new N-Dimensional geospatial functionality.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe12921 - Upgrade to sesame-2.7.11 and jena-2.11.1 jars

With this change, the Java Client has been upgraded to use Sesame 2.7.11 and Jena 2.11.1 jars. Previously, the Java Client used Sesame 2.6.8 and Jena 2.7.2 jars.

Rfe12920 - Tutorial for connection pooling using Sesame

Example24 has been added to the Sesame tutorial examples to create and manage connection pools using Sesame. Java Sesame tutorials in general are discussed in Java Sesame Tutorial. Example24 does not have a writeup and is in the file TutorialExamples.java discussed in Java Sesame Tutorial.

Bug22854 - TutorialExamples.java signals warnings in java client

With this change, Tutorial examples run with out any runtime warnings. Also some warnings are fixed in callimachus tests

Bug21911 - Implementation of Jena QueryExecution methods

Previously, the AGQueryExecution class had no implementation for following methods:

execConstructTriples()
execDescribeTriples()
getQuery()
setTimeout(long timeout),
setTimeout(long arg0, TimeUnit arg1)
setTimeout(long arg0, long arg1)
setTimeout(long timeout1, TimeUnit timeUnit1, long timeout2, TimeUnit timeUnit2)

This has been corrected. An example (example24) for a Jena Graph Union has been added to the Jena examples. Jena tutorials in general are discussed in Java Jena Tutorial. Example24 does not have a writeup and is in the file JenaTutorialExamples.java.

AllegroGraph 4.14.1

Server Changes

Rfe13193 - Parameterize email notifications

This change builds on rfe13104 and makes it possible to specify for which audit events emails should be sent and to what address. The body of the email contains the triples added for the audit event in pretty printed Turtle format.

The AdminEmail and AdminEmailSMTPHost configuration directive were removed in favor of the new AuditEventsToEmail directive. (Note this is a backward-incompatible change. See the Auditing document.)

The following rule asks for emails to be sent to [email protected] via the gmail smtp host (defined with SMTPHost directive) when any of the expirePassword, addUser or deleteUser audit events trigger:

AuditEventsToEmail to="[email protected]", smtphost="gmail", \  
  events="expirePassword,addUser,deleteUser"

If there is only one smtp host defined then it can be omitted:

AuditEventsToEmail to="[email protected]", \  
  events="expirePassword,addUser,deleteUser"

Note that auditing must be enabled for this to work. See the Audit documentation for more information.

Rfe13201 - Restrict the range of backend ports

This change adds the BackendPorts global configuration directive to restrict the range of ports on the loopback interface to be used by shared back-ends. Note that the maximum number of shared back-ends is limited by the number of ports in the specified range even if the Backends directive is higher. Configuration options are described in Server Configuration and Control.

Bug22675 - Simple queries may not show up in the jobs list

This is a continuation of the work done for bug22536 and increases the number of queries that can be controlled in the jobs list.

Bug22666 - run-fti-merge doesn't fail fast

Previously, if the text index merger process encountered an error during merging, it would log the error and a backtrace to agraph.log and then continue operation. If the error condition persisted, this logging could continue indefinitely, resulting in agraph.log growing to excessive size. This has been fixed.

Bug22641 - Listing and cancelling queries

Previously, job management (available via /jobs REST service and also admin/job in AGWebView) had a bug which caused concurrent requests to fail with:

the assertion (endp (intersection  
(db.agraph.spawn::receiver-for db.agraph.spawn::r)  
(db.agraph.spawn::receiver-for db.agraph.spawn::receiver))) failed

This bug has been fixed.

Bug22641 - Listing and cancelling queries

Previously, job management (available via /jobs REST service and also admin/job in AGWebView) had a bug which caused concurrent requests to fail with: the assertion (endp (intersection (db.agraph.spawn::receiver-for db.agraph.spawn::r) (db.agraph.spawn::receiver-for db.agraph.spawn::receiver))) failed This bug has been fixed.

Bug22636 - Accumulation of datafile mappings during unshuffling

For very large databases it was possible for index optimizers to exceed address space limitations and throw errors during large optimization operations. This has been fixed.

Bug22624 - Level 2 optimization should stop after single round

Previously, if a level 2 optimization was requested and, while it was running, additional triples were loaded into the database, the level 2 optimization would run again, redoing most of the work that it had done before. This has been fixed.

HTTP Client

Rfe12546 - Implement get-duplicate-triples.

Add support for retrieving the set of duplicate triples in a repository via the following REST/HTTP service call:

GET /repositories/[name]/statements/duplicates

See Deleting Duplicate Triples.

SPARQL

Bug22692 - Improve error logging for SPARQL update commands

AllegroGraph now prints additional information to the query log when errors are encountered during SPARQL update processing.

Bug22601 - SPARQL INSERT WHERE could fail to encode typed literals

When adding triples using the SPARQL DELETE/INSERT/WHERE update command, it was possible for typed literals to be saved as unencoded UPIs (stored in the string table) rather than as the more efficient encoded UPIs (stored directly with the triples). Depending on other settings, this could lead both to inefficent queries and missing results. The problem has been corrected.

Bug22588 - Further improve handling of multiple range filters on the same variable

AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like

select * {  
  ?s ex:test ?age .  
  filter( ( ?age >= 10 && ?age <= 20 ) ||  
          ( ?age >= 60 && ?age <= 70 ) ||  
          ( ?age >= 80 && ?age <= 90 ))  
}

will now execute as the combination of two range queries rather than a single scan. This change is similar to the change for bug22535 but more general.

Bug22540 - Handle failures of WHERE parts of update queries

Previously, if the query in the WHERE part of an update query failed (it timed out, ran out of memory, or was cancelled), the update failed with a bus error. This bug has been fixed.

Bug22334 - SERVICE doesn't deal with encoded IDs

SPARQL Federated query was not working correctly with AllegroGraph encoded IDs. This has been corrected.

AGWebView

Rfe12546 - Add Export Duplicate Statements link in AG Webview

On the Repository Overview page there is now an option to have the browser download the set of duplicate triples in the repository. Use the adjacent dropdown box to choose the method by which duplicates are determined (SPO or SPOG). See Deleting Duplicate Triples.

Changes to the Lisp API

Rfe12546 - Implement get-duplicate-triples.

Add support for retrieving the set of duplicate triples in a repository with the following Lisp API get-duplicate-triples, which returns a cursor of all duplicate triples visible inside the current transaction. See also Deleting Duplicate Triples.

Prolog

No significant changes.

Documentation

New document Deleting Duplicate Triples discusses the issue of duplicate triples in a store.

Python Client

Rfe12546: Add getDuplicateStatements to AGRepositoryConnection

Adds a new method to the RepositoryConnection class called getDuplicateStatements, that returns the set of duplicate triples in the repository. The method signature is:

def getDuplicateStatements(mode)

Duplicate triples are determined by the mode argument, which can be specified as either "spo" (compare as ntriples) or "spog" (compare as nquads). See also Deleting Duplicate Triples.

Java Client

Rfe12546 - Add getDuplicateStatements to AGRepositoryConnection

Adds a new method to the RepositoryConnection class called getDuplicateStatements, that returns the set of duplicate triples in the repository. The method signature is:

def getDuplicateStatements(mode)

Duplicate triples are determined by the mode argument, which can be specified as either "spo" (compare as ntriples) or "spog" (compare as nquads). See also Deleting Duplicate Triples.

AllegroGraph 4.14

Server Changes

Rfe13144 - Add `bulk mode` switch to agmaterialize command line tool

agmaterialize now supports a --bulk command-line option. If specified, then materialization will be run in bulk mode. This mode provides approximately 30% faster materialization. Note that bulk mode is global to the store and means that transactions will not be durable (in particular, will not be logged). Therefore, the switch should generally be used only when the store is not being used for other operations. See the Materializer document for information on agmaterialize.

Rfe13116 - Audit events for AGWebView login, logout, timeout

With this change, if auditing is enabled, then logIn, logOut, logOutWithTimeout events are added to the audit log whenever a user logs in, logs out, or when a login session times out. See Auditing.

Rfe13108 - Allow restricting superuser access to triples data

This change adds a new global configuration directive: SuperUserCanAccessAllData. If it is enabled (the default), then users with the Superuser permission will have read/write access to all repositories (just like in previous AllegroGraph versions). If it is turned off, then the superuser needs to be granted access to repositories. This is most useful when auditing is enabled and any change to user permissions is logged, as superusers will be able to grant themselves any permissions they want and auditing must be enabled if the fact that a superuser granted him/herself permission is to be recorded.

Rfe13104 - Notify admin by email of user account status changes

This change adds three new global configuration directives: SMTPHost, AdminEmail, and AdminEmailSMTPHost. With SMTPHost one can associate login, password, port, etc with a server:

SMTPHost gmail \  
  server="smtp.gmail.com", ssl=true, starttls=false,\  
  from="[email protected]", login="[email protected]", \  
  password="somepassword"

If AdminEmail is defined, then emails for user account status changes events are sent to it:

AdminEmail [email protected]

If there are multiple SMTPHost definitions, then one can explicitly select one of them:

AdminEmailSMTPHost gmail

See the documentation in Security Implementation for more information.

Rfe13094 - Add EvalAllowed configuration directive

This change introduces the EvalAllowed global configuration directive. If it is yes (the default), then the "Evaluate arbitrary code" permission bits are effective. If it is no, then arbitrary code evaluation is disabled for all users (including superusers), regardless of the value of a user's actual "Evaluate arbitrary code" permission. Configuration directives are described in Server Configuration and Control.

Rfe13091 - Password and account expiration, suspending, disabling

This change introduces a number of parameters to control passwords and accounts. All are described in Server Configuration and Control. The new options are:

PasswordExpiry: The time since the last password change after  
which the password will be expired. One cannot login with an  
expired password, it can only be used to change the password.  
 
PasswordExpiryGrace: The time since password expiry after which  
the account is disabled. Only the administrator can reenable  
accounts. This option does not affect users with superuser  
permission.  
 
MaxFailedLogins: The number of failed logins in a row after which  
the account is suspended.  
 
AccountUnsuspendTimeout: The time after which suspended accounts  
are unsuspended.  
 
AccountExpiry: The time since the last authenticated activity of a  
user after which the account is deleted. This option does not  
affect users with superuser permission.

Rfe13058 - Control password complexity

This change adds four new global configuration directives: PasswordMinLength, PasswordMinUppercaseChars, PasswordMinDigitChars, PasswordMinSpecialChars to allow the administrator to specify password complexity thresholds. All are described in Server Configuration and Control.

Rfe12941 - Allow use of more than 64GiB of query memory

Previously, the start address of the heap holding the query data structures has been such that when the heap grew beyond 64GiB it was likely to collide with shared libraries mapped there which made allocations fail. This change moves the heap so that collision is less likely and much larger heaps are possible.

Rfe12929 - Add changePassword audit event

With this change, when a user's password is changed and auditing is enabled, an audit event of type <http://franz.com/allegrograph/4.11/audit-log#changePassword> is logged.

Rfe12915 - Improve reporting of shared memory allocation errors

Previously, if allocation of shared memory used by processes serving a database instance failed, the error message was an opaque 'end-of-file'. With this change, the error message expicitly mentions the allocation failure.

Rfe12884 - Include failed login attempts in the audit log

With this change, when auditing is enabled, a failedLogin event is written to the audit log on authentication failures.

Rfe12848 - Removal of query temporary files

Sometimes queries write intermediate results to disk when they will not fit in memory, for instance with an ORDER BY query. If a query failed, some of its temporary files were not removed in a timely manner. With this change, temporary files are always deleted whether the query finishes or fails.

Bug22527 - Sorting UPI maps

Previously, on certain filesystems sorting of UPI maps could fail, which broke the repository instance. The only indication of this bug was the log message "Sort command failed". This bug has been fixed.

Bug22507 - Include id of deleted user in audit report

Previously, the audit report did not include the user id of the affected user associated with deleteUser events. This bug has been fixed.

Bug22466 - Setting geospatial type mappings did not register the subtype

Previously, setting a geospatial datatype or predicate mapping did not also add the geospatial subtype to the triple-store. This has been corrected.

Bug22465 - In the turtle format, QNames are a superset of XML QNames

AllegroGraph's turtle parser required QNames (like rdf:type) to be XML QNames but the turtle standard approved in February 2014 allows additional flexibility. In particular, the first character of the local part of a QName may start with a digit. Previously, a QName like rdfs:0 would have signaled a parse error. Now, it will be accepted.

Bug22450 - RDF/XML serializer could re-use node abbreviations

For efficiency, the RDF/XML and RDF-N3 serializers maintain a mapping from URI to prefix abbreviation (e.g., from to rdf:type). It was possible for the map to become corrupted during serialization leading to different nodes being printed as if they were the same. This has been corrected.

Bug22422 - Federated triple-stores cannot use geospatial magic properties

The check for whether or not a triple-store could use a geospatial magic property was too stringent. This has been corrected so that federated triple-stores can be queried with geospatial magic properties. Note that every store in the federation must agree on any geospatial predicate type mappings used in the query. If they do not, then an error will be signaled during query planning.

Bug22371 - Fix race condition when using free-text indices.

Previously, a race condition existed where a client A would fetch the list of free-text indices on a repository, but have the set of indices modified by a client B, thus rendering client A's list out of sync. This could result in unexpected errors in certain situations, such as when opening repositories. Now, attaching to free-text indices is synchronized so that this race condition is avoided.

Bug22363 - Fix log rotation

A bug was introduced in v4.13 where a SIGHUP sent to the Service Daemon would not cause the log file to be closed and reopened. This is now fixed.

Bug22345 - Use customer namespaces in remote clients

Similar to rfe12679 (under SPARQL below), AllegroGraph now uses customer namespaces when serializing output in RDF/XML for all clients. Previously, temporary namespaces were used.

Bug22319 - Materializer not inferring rdfs:subClassOf reflexivity

The materializer was not always correctly inferring that classes were rdfs:subClasses of themselves. This has been corrected. This change also improves the performance of AllegroGraph's OWL 2 RL materializer significantly.

Bug22300 - Fix error when system time is in the past

Previously, if the system time in the environment of deployment was before the build time of the AllegroGraph, then queries would fail with an error saying:

Attempt to do an array operation on nil which is not an array.

This bug has been fixed.

Bug22297: ensure-not-lingering should latch

Previously, if a request to delinger a store could not be honored because it was still in use by another process, the request would be ignored. Now, the request is deferred until the store is no longer in use.

Bug22293 - Make agraph-backup support dynamic catalogs.

Previously, backup and restore operations would fail when operating on repositories in dynamic catalogs. Backup operations were unable to locate the main directory of the repository. Restore operations were unable to locate the main directory of dynamic catalogs.

Now, backup and restore operations support repositories created in dynamic catalogs. They follow the same rules as other repositories, most notably that when performing a restore operation, all dynamic catalogs must already have been created, or the restore will fail.

See Catalog definitions in Server Configuration for a description of dynamic catalogs.

Bug22259 - Ensure proper transaction log stream alignment on restart

Previously, under certain unusual circumstances, the transaction log stream could be left in a state that prevented a database from reopening properly. This has been fixed.

Bug22243 - Less fragmentation in the shared memory

With this change, the shared memory allocator is more aggressive about coalescing freed regions which reduces the tendency of shared memory to fragment.

Bug22229 - Geospatial type mappings could cause duplicate triples

If a triple-store registered a predicate and a datatype geospatial type mapping that applied to the same triple, then that triple would be added to the store twice. This has been corrected.

Bug22228 - Rare error when opening a triple store

Under extremely rare circumstances, opening a triple store could result in the following error:

The assertion (integerp db.agraph.storage::chunk-size) failed.

This has been fixed.

Bug22226 - More efficient memory usage

With this fix, the query engine does a more thorough job of giving freed memory back to the operating system which enables more efficient operation under high concurrency workloads.

Bug22211 - Trix parser did not treat Return (ASCII 13) as whitespace

The TriX parser was not treating Return characters (ASCII 13) as whitespace which sometimes led to parse errors. This has been corrected.

Bug22208 - Concurrent queries in shared backends

When the system is under load, a shared backend may execute several queries concurently. Under rare circumstances, the queries involved could change each other's query options and produce incorrect results. This bug has been fixed.

Bug22198 - Listing freetext indices should not error for federated triple-stores

Previously, federate-triple-stores would signal an error when asked to list their freetext indices. They now return an empty list.

Bug22141 - Attempt to close instance during agraph-backup --supersede restore operations.

Previously, using agraph-backup to restore into an open triple-stores would fail. Now, when the --supersede option is specified, such restores can succeed. agraph-backup is described in Backup and Restore.

HTTP Client

Rfe12963 - SERVICE URIs with query strings

Previously, AllegroGraph dropped the query string in a SERVICE URI when making a request. In this example:

SERVICE <http://sparql.bioontology.org/ontologies/sparql?apikey=XXX>

the 'apikey' was not sent to the remote web server. This has been fixed.

Rfe12952 - Add global configuration directive UseMainPortForSessions

Normally, HTTP requests to a dedicated session go directly to the process serving that particular session. This has low overhead, but it can make firewall management more difficult because more ports have to be open. With this change, the new UseMainPortForSessions directive may be enabled in the configuration file which makes all communication be proxied through the front-end. Previously, similar functionality existed only in the Java client controlled by the UseMainPortForSessions property.

Rfe12798: Implement repo-specific scripts

The existing scripting HTTP API (sitescripts) manages scripts that can be used with any repository. An HTTP API has been added for managing scripts that are local to a given repository, called "reposcripts". Reposcripts can only be loaded and run against the repository under which the script was loaded. See Scripting the server in HTTP Protocol for more information.

Bug22524 - HTTP protocol should not send partial responses

Previously, queries that used Chunk-at-a-Time (CaaT) processing over the HTTP REST interface would begin streaming results back as soon as the first chunk was processed. If a query timeout was in effect, this meant that AllegroGraph could issue a 200 (OK) response and then only send back only a partial response body. To prevent this, AllegroGraph now computes the entire query result before sending any HTTP response when there is a time out active.

Bug22476 - Unauthorized response for javascript eval

Previously, if the user had no eval permission, attempts to execute javascript failed with HTTP status 400 and a "MALFORMED PROGRAM" message. With this change, the coreect HTTP status 401 'unauthorized' is returned.

Bug22445 - Don't percent-decode path component of HTTP requests twice.

Previously, path components of HTTP requests sent to AllegroGraph were sometimes being percent-decoded twice. For example, the URI

http://localhost:10035/repositories/test%252frepo/statements

could be seen by AG as either

http://localhost:10035/repositories/test%2frepo/statements

http://localhost:10035/repositories/test/repo/statements

depending on the URI being requested. This bug isn't likely to have been encountered unless you were attempting to include special characters in repository or catalog names. It is now fixed. Users with applications that make direct HTTP requests should check their request URIs to ensure that they do not double-percent-encode any path components, as these may stop working or have unexpected behavior. Percent-decoding of URI query arguments is not affected.

Bug22369 - Fix server-side javascript http-request property values.

Previously, the property values of javascript http-request objects (method, url, body, etc.) could raise errors due to returning invalid objects. This bug has been fixed, and return values should match what is described in the documentation.

Bug22329 - Fix handling of HTTP 'Expect: 100-Continue'

Previously, AllegroGraph did not send the '100 Continue' response to requests with 'Expect: 100-Continue' headers when the request was forwarded internally to another component. This happened, for instance, when a request to a shared backend or a dedicated session was forwarded via the frontend and also when a requested made directly to a dedicated session was forwarded to the frontend. With this change, '100 Continue' is sent as appropriate.

SPARQL

Rfe13029 - Use POST instead of GET for remote SPARQL queries

AllegroGraph now uses HTTP POST requests instead of HTTP GET requests when talking to remote SPARQL endpoints. This allows for larger query strings to be sent.

Rfe13028: Improve use of partial results when making SERVICE queries

AllegroGraph now makes more use of partially computed results when making federated SPARQL queries using the SERVICE clause. This results in more efficient queries against SPARQL endpoints.

Rfe12911 - Improve estimates for magic geospatial properties

AllegroGraph's size estimates for geospatial queries that use SPARQL magic properties are now more accurate. This improves query clause reordering and can greatly improve query performance.

Rfe12891 - Improve efficiency of some SPARQL geospatial queries

Some queries that use SPARQL magic properties to use AllegroGraph's geospatial engine will now operate more efficiently.

Rfe12822 - Add temporaryFileDiskSpace query option

Sometimes queries write intermediate results to disk when they will not fit in memory, for instance with an ORDER BY query. With a huge query it is possible for a such temporary files to fill the filesystem. In order to prevent this, the temporaryFileDiskSpace query option may be set. For example:

PREFIX franzOption_temporaryFileDiskSpace: <franz:10G>

As with all query options, it is also possible to change the default value globally in the AllegroGraph configuration file. For example:

QueryOption temporaryFileDiskSpace=10G

The default is to have no limit. Query options are described here in the SPARQL Reference; the AllegroGraph configuration file is described in Server Configuration.

Rfe12788 - Raise error for invalid use of SPARQL geospatial functions

AllegroGraph now signals an error at parse time when a geospatial function is used with a predicate that is not associated with a geospatial type-mapping.

Rfe12750: SPARQL Property Path cache improvements

SPARQL Property Path computations use a cache to improve the efficiency of transitive closure calculations. The data structures for this cache have been rewritten to be more memory and time efficient. Secondly, the heuristic that the query engine uses to determine whether or not the cache should be aggressively or lazily filled have been improved.

Rfe12284 - Signal an error if a SPARQL GEO pattern uses a constant object or graph place

The SPARQL GEO Clause uses its first pattern to link with AllegroGraph's geospatial system. The object place of this pattern must be left unbound. Previously, AllegroGraph would silently return no results when the place was bound. Now, it will signal an error. For example, the following query would have previously run but returned incorrect results but now it will signal an error.

prefix s: <http://franz.com/ns/allegrograph/3.0/geospatial/spherical/km/-180.0/180.0/-90.0/90.0/>  
select distinct ?event ?coord ?date ?platform ?EventType {  
  GEO SUBTYPE s:1.0  
  BOUNDINGBOX( POINT(-084,34), POINT(-082,36)) {  
    # this is an error because the first clause is the one  
    # that connects the query with AllegroGraph's geospatial  
    # sub-system  
    ?event a <http://UCore-SL-Taxonomy-OWL-DL#ObservationEvent> .  
    ?event mo:hasGeospatialCoordinate ?coord .  
  }  
  WHERE { }  
}

Rfe12240 - Improve bound variable analysis when using magic properties

The SPARQL 1.1 query planner now recognizes that variables set via assignment using the BIND clause can be treated as bound when determining whether or not a magic property can be used. This means that queries like

select ?event ?coord {  
  bind(44.30 as ?xMin) .  
  bind(33.40 as ?yMax) .  
  (?event ?coord) franzGeo:inBoundingBoxXY (swp:startLatLong ?xMin 44.50 33.30 ?yMax)  
}

can now be used.

Rfe11880 - Improve handling of some SPARQL GRAPH clauses

AllegroGraph's analysis of when and how it can execute SPARQL GRAPH clauses has been improved. Depending the query, this enhancement can speed execution by a factor of 10 or more.

Rfe10132 - Improve performance of find unique values for a column queries

SPARQL queries that ask for the DISTINCT values of a particular column are now sometimes significantly faster. Previously, they performed linearly in the size of the triple-store; now, they perform linearly in the size of the number of values returned. In particular, for stores with a small number of predicates or graphs (relative to the size of the store), queries like these will be much, much faster:

SELECT DISTINCT ?p { ?s ?p ?o }  
SELECT DISTINCT ?g { graph ?g { ?s ?p ?o } }  
SELECT DISTINCT ?class { ?s a ?class }

Note that aggregate queries still require a full-scan of the store, so a query like:

SELECT ?p (count(1) as ?pCount) {  
  ?s ?p ?o .  
}

will still be expensive.

Bug22529 - Improve use of intermediate information during query

AllegroGraph's set based SPARQL 1.1 query engine sometimes used an intermediate result set when it would have been more efficient to compute the solutions directly. This has been corrected.

Bug22535 - Improve handling of multiple range filters on the same variable

AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like

select * {  
  ?s ex:test ?age .  
  filter( ( ?age >= 10 && ?age <= 20 ) ||  
          ( ?age >= 60 && ?age <= 70 ))  
}

will now execute as the combination of two range queries rather than a single scan.

Bug22485 - Improve pattern size estimation logic

AllegroGraph's estimation of the size of triple-patterns in a BGP has been improved. This in turn improves the automatic BGP re-ordering and leads to better query plans when using the statistical clauseReorderer.

Bug22430 - Cross products with unbound variables could lose results

Depending on the query algebra and the order in which clauses were evaluated, it was possible for AllegroGraph to lose some solutions with unbound variables when computing a cross product. This has been corrected.

Bug22391 - Make SPARQL rand() results more independent

Due to an unintended sharing of random number generator state, SPARQL rand() calls in queries running on the same backend can be produce the same stream of pseudo random numbers.

Bug22389 - The chunkProcessingMemory query option could not be specified in a query

The chunkProcessingMemory could only be set using the AllegroGraph config file rather than being able to also specify it in using the PREFIX notation in a SPARQL query. This has been corrected.

Bug22480 - Range queries could cause other filters to be ignored

If a query had multiple FILTERs that applied to a single pattern and one of these FILTERs created a range query, then it was possible for the other FILTERs to be ignored. This has been corrected.

Bug22372 - SPARQL JSON results format was using literal-typed instead of typed-literal

The SPARQL JSON results serializer was emitting literals that had an associated datatype as

"name" : {"type":"literal-typed", "datatype":" D ", "value":" S "}

rather than

"name" : {"type":"typed-literal", "datatype":" D ", "value":" S "}

This has been corrected.

Bug22348: EXISTS and NOT EXISTS FILTERs could fail

If an EXISTS or NOT EXISTS FILTER used variables from the outer query in a FILTER (rather than a BGP), then too many solutions could be returned. E.g., a query like:

PREFIX : <http://example.com/>  
SELECT ?x ?n WHERE {  
  ?x :p ?n  
  FILTER NOT EXISTS {  
    ?x :q ?m .  
    FILTER(?n = ?m)  
  }  
}

uses the bindings for ?n in the NOT EXISTS FILTER and ?n is not one of the variables that the NOT EXISTS FILTER binds itself. Previously, AllegroGraph will leave ?n unbound and therefore return too many results. This has been corrected.

Bug22336 - Allow the use of magic properties in more situations

The SPARQL 1.1 query planner ensures that a magic property can only be used when all of its required variables are bound. Previously, however, the test for bound variables was overly strict and did not, for example, take BIND or VALUES into account. This has been corrected. For example, this query would previously have been disallowed:

prefix sna: <http://franz.com/ns/allegrograph/4.11/sna/>  
prefix pred: <http://www.testData.com/predicate/>  
prefix : <http://ex#>  
select * {  
  values ?start { <http://www.testData.com/graph/10444> }  
  ?graph sna:nodalNeighbors (:graphSubClass ?start) .  
}

because AllegroGraph would not have seen that ?start is bound by the VALUES clause. Now this query will function as expected.

Bug22347 - Query options were sometimes ignored in SPARQL update commands

Previously, AllegroGraph PREFIX options were ignored during the WHERE clause of a SPARQL DELETE/INSERT/WHERE command. This has been corrected.

Bug22321 - Use other sets more in many more cases

The set-based SPARQL query engine now makes better use of partially computed result-sets during query execution. This can greatly accelerate many queries.

Bug22272 - Correct unbound variable handling in MINUS

In rare circumstances variables that were unbound in the right-hand side of a SPARQL MINUS clause could be incorrectly matched with bound variables on the left hand side. This has been corrected.

Bug22270 - Correct BGP estimation for predicates not in the store

If a query clause had either a fixed subject or object and a predicate that was not in the triple-store, then AllegroGraph's statistical estimator could return a positive number rather than zero. This has been corrected.

Bug22253 - Combining reorderDuringExecution and Property Path queries could lose filters

Under rare circumstances, the query engine could lose track of FILTERS in queries that used SPARQL Property Paths and the AllegroGraph reorderDuringExecution option. This has been corrected.

Bug22246 - Catch argument count mistakes in query functions

SPARQL queries that contained functions being called with an incorrect number of arguments would silently fail. These errors are now detected and reported at query time. For example, a query like:

prefix gf: <http://franz.com/ns/allegrograph/3.0/geospatial/fn/>  
select * {  
  bind( gf:cartesianDistance(1, 2, 3, 4) as ?b )  
}  
would have previously returned a single unbound result because the  
`cartesianDistance` function should only be given two arguments. Now,  
AllegroGraph will signal an error.

Bug21941 - Magic properties could lose track of FILTERs

In rare circumstances a query that used magic properties and FILTER on some of the variables in the properties could incorrectly filter out too many solutions. This has been corrected.

AGWebView

Rfe13159 - Enhance AGWebView query GUI

AGWebView's SPARQL query user interface now supports query log viewing, query plan inspection and reports additional details about query execution such as query time and warnings. The HTTP REST API for query has a new parameter, returnQueryMetadata. If supplied and true, then the HTTP response will include a JSON payload containing information about the query in an HTTP header named x-query-info. The revised query page is described here.

Rfe13103 - Reposcripts added to AGWebView

Access to the new Reposcripts API has been added to AGWebView (see the Rfe12798 release notes entry, which has links to other documentation).

Rfe13068 - Login timeout for AGWebView

The new global configuration directive LoginTimeout controls the expiry of inactive AGWebView login sessions. See the description of that option in Server Configuration and Control for the details.

Rfe12925 - Only run AGWebView queries when the Execute button is pushed

Previously, AGWebView executed recent and saved queries as soon as they were selected. Now AGWebView only executes queries when the execute button is clicked.

Rfe12579 - Allow users to change their own passwords

This change introduces the PasswordChangeAllowed global configuration directive. If it is yes (the default), then users can change their own passwords via AGWebView. This is a change in default behavior (there was no facility to allow users to change their own passwords in earlier releases). If you set the value of the option to no, the old behavior will be restored. Configuration directives are described in Server Configuration and Control. User passwords are described in Security.

Rfe12275 - Add query cancellation interface to AGWebView's query pane

In addition to the existing job canceling screen, AGWebView now provides query cancellation directly from the SPARQL query interface. The revised query page is described here.

Bug22536 - Simple queries may not show up in the jobs list

Previously, simple queries like

select  { ?s ?p ?o . filter( fn(?o) ) }

could fail to appear on the query jobs list in AGWebView. AllegroGraph now checks for query timeout more often which means that even these queries will appear and be cancelable.

Bug22260 - Fix AGWebView audit log date range

Previously, when the start date or the end date was specified in the audit log query (at /#audit), then the query failed with a type error. This bug has been fixed.

Bug20820 - AGWebView's multi-core data import could not specify a graph

When using multi-core loading, AGWebView was incorrectly formatting the graph it sent to AllegroGraph's parallel data import tool. This meant that it was impossible to specify a graph. This has been corrected.

Changes to the Lisp API

Rfe12862 - Extend Lisp geospatial functions with subject, object and graph arguments

Add s, o and g arguments to the Lisp API geospatial functions:

The new arguments act as additional filters on the triples returned by the functions. g can only be used when the use-g argument is nil and o can only be used when the use-g argument is non-nil.

Rfe12830 - Warn when a query has constants that are not in the store

In the Lisp API, the fourth return value of run-sparql now contains information about any warning that occurred during query planning and execution. This includes a warning when a query contains a constant that is not in the store. For example:

select * { ?s <ex://#notHere> ?o }

would generate a warning regarding <ex://#notHere>. Additional warnings will be added in the future.

Bug22322 - remote-triple-store does not honor default-prefixes argument to run-sparql

The default-prefixes argument to the Lisp run-sparql function was being ignored for remote-triple-stores. This has been corrected.

Bug22225 - Memory corruption in the Lisp direct client

Previously, if an unclosed database object became garbage, memory corruption could occur which lead to unpredictable errors such as segmentation faults under extremely rare circumstances.

Bug22167 - Fix flaws in delete-duplicate-triples.

delete-duplicate-triples is documented as preserving the triple with the lowest possible triple-id from a group of duplicate triples. Flaws in the algorithm used to delete duplicate triples made it possible that the triple preserved would not necessarily have the lowest possible triple-id, and could depend on the order in which the duplicates were sorted. This change corrects that problem.

Prolog

Bug22179 - Allow safe Prolog query requests in shared backends

Previously, AllegroGraph only allowed Prolog queries when a user had eval privileges for the store. Now, eval privileges are required only for Prolog queries that make use of functors that escape to Lisp.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe12952 - Update Java client for handling of useMainPortForSessions

When the server configuration option UseMainPortForSessions is enabled, all requests to sessions are multiplexed through the frontend port. In this case, the Java system property useMainPortForSession is redundant and its value is not consulted. This means that one can turn on this feature on the client if it's not enabled in the server config, but cannot turn it off.

Bug22392 - Hang with useMainPortForSessions

Previously, when the Java client's useMainPortForSessions feature was enabled, a request returning a large response could hang. This bug has been fixed.

Bug22323 - Fix UseMainPortForSessions when using SSL

Previously, when the com.franz.agraph.http.useMainPortForSessions property was set to true, the Java client arranged for all communication with a session be funneled through the front-end. This fix makes this feature work with SSL connections. Also, with this change it is required that SSLPort is set in the configuration file, otherwise using SSL connections to dedicated sessions with fail.

AllegroGraph 4.13.2

The only change was to replace the SSL library with one that avoided the heartbleed bug.

AllegroGraph 4.13.1

Server Changes

Bug22255 - Concurrent full index optimization requests

Previously, if index optimization was requested with a full index optimization already in progress, the database instance would log an error and hang. This bug has been fixed.

AllegroGraph 4.13

Server Changes

Rfe12645 - Add a command line interface to AllegroGraph's materializer

AllegroGraph now includes a command line tool to materialize the inferred triples in a triple-store. The tool is named agmaterialize. It can currently be run only on the same machine where the AllegroGraph server is running. The command line interface is described in the Materializer documentation.

Rfe12642: Improve handling of datafile free space information

Previously, datafile free space information was maintained in a fixed sized data structure which could run out of room after a certain level of datafile fragmentation was reached. This has been corrected.

Rfe12590 - Shared backend scheduling improvements

Previously, when all shared backends were busy, further requests were held until a backend became available. Such pending requests were scheduled for execution in basically random order with latency up to 1s.

With this change, pending requests are scheduled rather than essentially random. The scheduling is basically FIFO (first in, first out, meaning the oldest request is processed first), but FIFO may be violated for efficiency reasons. (Strict FIFO behavior is undesirable, because it greatly reduces AllegroGraph's ability to schedule requests optimally under stress.)

Also with this change, scheduling latency is now minuscule and shared backends are now allowed to execute slightly more queries in parallel if all backends are busy, which allows greater efficiency in most cases.

Rfe9523 - Logging improvements

This change adds a centralized logging facility to the AllegroGraph daemon. The new logger guarantees no interleaving of messages and is able to deal with full filesystem conditions gracefully. Lisp direct clients have now the option of sending log messages to an AllegroGraph server with the with-log-routed-to-server macro. See the documentation for more. Previously, log messages were sent to the AllegroGraph server belonging to the first direct connection opened in the client.

Bug22220 - Race condition while determining last written transaction log id

Previously, a rare error could be encountered when starting replication or while switching roles, indicating that replication failed due to a missing transaction log that had not yet been written to disk. This problem has been fixed. Now, only transaction logs on disk are used to start replication.

Bug22184 - Instance shutdown and replication

In rare circumstances, when the instance belonging to a replicated repository was shut down, writing the transaction log could fail. This bug has been fixed.

Bug22156 - date-string-to-upi doesn't handle zero time zones consistently

The date-string-to-upi functor was not treating a Zulu time zone specified as "Z" consistently with the rest of AllegroGraph's date and time parsers. This has been corrected.

Bug22111 - Database names may not contain special characters

Because it is impossible to reliably use databases (also called repositories) in session specifications if their names contained characters that are used in those specifications, the list of characters which must not be used in names has been expanded. The following characters are now disallowed in database (repository) names:

\ / ~ : Space $ { } < > * + [ ] |

Previously, only the following four characters were disallowed in database (repository) names:

\ ~ / :

Note that this change only effects new databases (repositories); existing databases (repositories) with any of the newly invalid characters in their names will continue to be accessible. It may, however, result in difficulties when attempting to use them in more complex session specifications.

Bug22100 - More strict, faster parsing of some xsd datetimes

The parsers for xsd:date, xsd:dateTime and xsd:time used to be lenient and accept things like:

20131118         # missing separator  
2013-11-18T12:27 # missing seconds

and many other convenient but non-standard formats. The new parser only accepts what the XML Schema specification mandates which also allows it to be faster. See XSD Date and Time datatypes for more information on these datatypes.

Bug22086 - Materializer deletes triples with the same subject, predicate and object

The initial version of AllegroGraph's materializer deleted all triples with the same subject, predicate and object. This limitation has now been lifted: the materializer now only deletes triples with matching subject, predicate, object and graph.

Bug22032 - Control whether or not external references are followed during RDF/XML import

Previously, AllegroGraph ignored external references in RDF/XML source files. The Lisp API now includes a new keyword argument to load-rdf/xml:

:resolve-external-references : if true, the external references will be followed. The default continues to be nil.

agload also has a new command-line switch. Supplying either of -x or --external-references will cause agload to follow external references. See Data Loading.

Bug21953 - Allow import of triples to reasoning-triple-store

A bug was introduced in v4.12 that prevented import of triples via load-* routines (e.g. load-rdf/xml) on reasoning-triple-stores, when the source argument was passed a URI. This problem has been fixed. This change also provides a better error message when attempting to import statements to a federated-triple-store, which is not allowed.

Bug21656 - Cleaner daemon shutdown

When the AllegroGraph daemon was stopped, despite the claim of doing a 'clean shutdown' in agraph.log, the shutdown of the instances was not clean and the corresponding repositories would potentially require crash recovery the next time they were opened. Also, such a shutdown lost any index optimization work performed since the most recent checkpoint.

With this change, instance processes perform a checkpoint if necessary which makes shutdown slower, but startup faster and it makes sure the index optimization results persist.

Bug20773 - Support federations of RDFS++ stores

Previously, queries against a federation of RDFS++ stores would cause an error during query planning. This has been corrected.

HTTP Client

Rfe12763 - Update HTTP API for the materializer

The REST API for materializeEntailed now includes an inferredGraph argument that can be used to specify which graph newly entailed triples are added to (or deleted from). The materializer is described in Materializer.

SPARQL

Rfe12767 - Improved the efficiency of single group aggregate queries

The code generated for single group aggregate queries is now slightly more efficient.

Rfe12762 - Optimizing level 2 index optimization

With this change, level 2 index optimization (the default) no longer tries to optimize access for triples added since optimization was started. This allows index optimization to finish even if triples are added continuously at a high rate. The new code is faster on average, has the same worst case disk usage (about two times the original size), but it has a higher average disk usage.

Rfe12760 - Improve efficiency of some SPARQL Property Path queries

AllegroGraph now builds some property path data structures as needed rather than computing all of them up front. Depending on the data, this can greatly enhance the efficiency of some queries.

Rfe12757 - Improve efficiency of some property path queries

In some circumstances, AllegroGraph could choose the less efficient of two strategies when evaluating a property path query. AllegroGraph now ensures that it makes the better choice.

Rfe12717 - Handle RDFterm-equality of strings consistently

Previously, AllegroGraph implemented sameTerm() using a strict interpretation of RDF literal equality when comparing plain literals and typed literal strings but implemented = using a more relaxed interpretation. These tests are now done consistently using the relaxed interpretation. In particular,

'hello' is RDFterm-equal to 'hello'^^xsd:string

This is in accordance with both common usage and the RDF 1.1 specification.

Rfe12699 - Improve efficiency of SPARQL equality tests on resources

The efficiency of equivalency tests in SPARQL filter expressions has been improved.

Rfe12681 - Optimize interprocess communication

The performance of message passing between different AllegroGraph server processes has been slightly improved.

Rfe12679 - Use query local namespace abbreviations in result serialization

AllegroGraph now uses the namespaces defined by a SPARQL query's prefixes when serializing output in RDF/XML. Previously, temporary namespaces were used instead.

Rfe12670 - Allow /SPARQL access to AllegroGraph as a SPARQL endpoint

Previously, access to an AllegroGraph triple-store as a SPARQL endpoint was via sending a query to

SERVER:PORT/respositories/NAME?query=...

This REST API has been augmented so that sending a query to the triple-store's URL with a trailing /sparql also works. E.g.,

SERVER:PORT/respositories/NAME/sparql?query=...

This is not required by the SPARQL standard but is in conformance with many other existing endpoints.

Rfe12657 - More efficient handling for the VALUES clause

AllegrGraph now rearranges queries so that VALUES clauses are evaluated first when possible. This improves query efficiency as VALUES clauses are generally small.

Rfe12656 - Improve execution of some nested joins

AllegroGraph now evaluates nested joins more efficiently.

Rfe12498 - Improve full-scan warning heuristics

AllegroGraph warns when a SPARQL query executes a full-scan. Heretofore, this meant that a query like

select * { ?s ?p ?o } limit 100

could have produced a full-scan warning even though it is benign. AllegroGraph now saves its warnings for when they may actually indicate a problem with the query.

Rfe11981 - New query option chunkProcessingMemory

Previously when chunk-at-a-time processing was enabled (see query option chunkProcessingAllowed), the query option chunkProcessingSize controlled memory consumption by limiting the number of intermediate results stored in memory. The total memory consumption depended on the number of variables and the complexity of the query, so it was rather difficult to know how much a query will actually use. With this change, chunkProcessingSize is deprecated and chunkProcessingMemory can be specified instead. The unit of chunkProcessingMemory is a byte but the usual abbreviations are also accepted ("8G" (the default), "2M").

Rfe11515 - Fully support the SPARQL 1.1 HTTP Protocol

AllegroGraph now fully supports the five different HTTP request types specified in the SPARQL 1.1 HTTP Protocol. These are:

Query
- via GET with query parameters
- via POST with urlencoded parameters
- via POST with a plain query string (using the content-type of application/sparql-query).
Update
- via POST with urlencoded parameters
- via POST with a plain update request string (using the content-type of application/sparql-update).

In addition, AllegroGraph continues to support SPARQL update using POST where the update command is encoded in the query portion of the HTTP request.

Bug22350 - Distinct queries could produce duplicate results in rare cases

Under rare circumstances it was possible for the query engine to produce duplicate results when using chunk-at-a-time (CaaT) processing. This has been corrected.

Bug22214 - Encoded IDs failed to compare properly in SPARQL queries

Inequality comparisons between encoded IDs were always returning true. This has been corrected.

Bug22204 - decimal to decimal cast loses precision

When in a SPARQL query an xsd:decimal was cast to xsd:decimal it lost precision as it was converted to a single float internally. This bug has been fixed.

Bug22203 - Fix decimal parsing in SPARQL queries

In SPARQL query strings, decimals in the (-1,0) interval were parsed with the wrong sign. For example, -0.9 was parsed as 0.9. This bug has been fixed.

Bug22153 - Some wild card property path queries could generate incorrect paths

In rare circumstances, the data structures used to cache property path information could be invalidated leading to too many path results. This has been corrected.

Bug22147 - Add support for ObjectId's in the _id field.

The SPARQL MongoDB magic predicate now works with documents containing ObjectId's in the _id field.

Bug22142 - Expressions on aggregates fail for empty group

If a query wrapped an aggregate with an expression like

select (xsd:float(avg(?age)) as ?avgAge) ...

and the result set contained an empty group (i.e., a group with no members), then AllegroGraph would signal error. This has been corrected.

Bug22134 - SPARQL aggregates on expressions with errors could compute incorrect results

Other than COUNT, a SPARQL aggregate on an expression should return unbound if that expression evaluates to a type error during computation. AllegroGraph was incorrectly returning the aggregate of any bindings produced after the last type error. This has been corrected.

As an example, if AllegroGraph needed to compute the MIN( ?p * 2 ) and ?p was bound to 4, 7, 15, "hello", 6, and 1, then AllegroGraph would previously have returned 2 (the minimum of 12 and 2). Now, it will return no binding for that expressions.

Bug22123: Empty construct template will cause an error

If a SPARQL CONSTRUCT had an empty template like

CONSTRUCT {} WHERE { ?s a ?o }

then AllegroGraph would signal an error. This has been corrected.

Bug22094 - Correct `sparql<` and `sparql>` for date and dateTime comparisons

Because there is no defined operator mapping for inequality comparisons between xsd:date and xsd:dateTime, comparing them should throw a SPARQL type-error. AllegroGraph was incorrectly returning nil instead. This has been corrected. See XSD Date and Time datatypes for more information on these datatypes.

Bug22074 - DELETE templates with unknown variables caused an error

AllegroGraph would signal an error if a SPARQL DELETE/WHERE command's DELETE clause contained variables that were not bound by the WHERE clause. This has been corrected. An example of such a command would be:

DELETE { graph ?unknownGraph { ?s ?p ?o } }  
WHERE { ?s ?p ?o }

Bug21561 - Remote-triple-stores do not support the table results format

Remote-triple-stores did not support returning tables from SPARQL queries. This has been corrected.

AGWebView

Bug22161 - Adding security filters in Firefox

Previously, with Firefox, clicking the 'add' button in the security filters table of the user management page (Admin/users in the menu) of AGWebView did not submit the form. This bug has been fixed.

Changes to the Lisp API

Rfe12632: Improvements in Lisp API to AllegroGraph materializer

Materialization previously occurred only for triples in the default-graph of the triple-store. It now occurs for all of the graphs in the store. Materialization on subsets of the graphs can be accomplished by creating a graph-filtered triple-store and calling materialize-entailed-triples on it.

Materialization previously placed inferred triples into a special graph that could not be utilized by the rest of AllegroGraph (the special graph behaved similarly to but not exactly like the usual AllegroGraph default-graph). It now places inferred triples into a graph that can be specified in the call to materialize-triples using the materialized-graph argument. It defaults to http://franz.com/ns/allegrograph/4.12/materializer/defaultGraph.

Two new arguments were added to delete-materialized-triples. These are:

db - which controls the triple-store that is modified.
materialized-graph - which controls the triples that are deleted. It defaults to http://franz.com/ns/allegrograph/4.12/materializer/defaultGraph.

Materialization now logs its start and end to the AllegroGraph log file. The overall efficiently of the materializer has also been improved.

Bug22132 - calling run-sparql with the :with-variable argument with a timezoned upi

When the Lisp function run-sparql was used in conjunction with a :with-variable argument that bound a variable to an encoded xsd:dateTime or xsd:time, then in some cases comparisons involving that variable failed to match equivalent representations of the same xsd:dateTime or xsd:time with a different timezone. This bug has been fixed.

Prolog

Bug22171: Select0 should return UPIs when it can

Previously, the Prolog select0 query macro could return future-parts rather than UPIs. For example, this query would return a future-part.

(select0 (?a) (= ?a !rdf:type))

This could cause problems in the unlikely event that namespace abbreviations changed when running the same query multiple times. select0 now returns UPIs so that the above behavior is not possible. Note that if a future-part represents a triple that is not currently in the triple-store, then a query can return UPIs that cannot be printed (because the UPI will not be in the string dictionary).

Bug22157 - Temporal functors could signal an error when used with all bound values

Previously, AllegroGraph used real numbers to represent times, dates and dateTimes. Now, the internal representation includes time-zone information which means that comparing two values can no longer use simple inequalities. The Prolog temporal functors like point-before-datetime had not been updated to reflect these changes. This oversight has been corrected. This change means that the Prolog functors will use the same XSD logic used by the SPARQL date and time functions.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

AllegroGraph 4.12.2

Server Changes

Rfe12734 - More aggressive level 2 index optimization

In some cases, level 2 index optimization (the default) left a small part of the index unoptimized. With this change, the entire index is optimized in all cases.

Rfe12726 - Index optimization increasing database size on disk

Space in some garbage on-disk data structures is now reclaimed in a more timely manner which makes the increase in database size during index optimization less severe.

Rfe12725 - Faster level 2 index optimization

Level 2 index optimization (which is the default level) has been made faster.

Rfe12636 - Make config-file argument to agraph-backup backup-settings optional.

Previously, the agraph-backup backup-settings command required a config-file argument. This argument has been eliminated in favor of an optional --config option that can be used in place of the --port argument. If both --port and --config are supplied, then the port argument takes precedence. Also, agraph-backup could fail in the past if there were discrepencies between a --port argument specified on the command line, and a Port directive contained in a config file. Now, if the --port argument is specified, it will take precedence over any Port directives read from a config file. See Backup and Restore.

Rfe12635 - Major revision to agraph-backup operating modes.

The various backup and restore modes of agraph-backup have been overhauled.

Previously, backup mode created an archive file containing only repository data. Now, it creates an archive directory that contains both repository data and settings. This archive can be restored via restore or restore-all. Previously, restore mode operated on an archive file. Now, it can also operate on an archive directory created by backup or backup-all. In either case, it will restore a single repository (data and settings) from the archive.

The layout of archive directories created by backup-all are slightly changed. The restore-all mode will now fail if repositories being restored to already exist. Use the --supersede option to force an overwrite of any such repositories.

Previously, the backup-settings mode created an archive containing both repository and system settings. It also supported the --ext-backup-command option. Now, it only archives system settings and the --ext-backup-command option is no longer supported. Adds a restore-settings option that restores system settings, overwriting any existing settings. It operates on any valid archive directry (though ones created by backup will likely not have any system settings data).

Please refer to the updated Backup and Restore documentation for full details.

Rfe12621 - xsd:time improvements

Zoneless xsd:times are no longer converted to zero timezone when added, and timezone offsets roundtrip to and from the database. Also, with this change xsd:times and xsd:dateTimes are no longer comparable. This puts xsd:time on equal footing with xsd:dateTime. See the date and time documentation for more information.

Rfe12580 - Don't create repo if client/server uids don't match

Previously, if the uid of the direct Lisp client and the AllegroGraph server differed, then a create-triple-store call would fail with a `server-pid-and-client-pid-must-match-error` indicating that the creation failed. However, the repository would actually be created, possibly overwriting an existing repository. Now, the uid check is performed earlier in the create repository process, so that the repository is not created when the uids do not match.

Rfe12456 - Time zone roundtrip for xsd:dateTime and xsd:time

AllegroGraph previously represented xsd:times and xsd:dateTimes with Zulu (i.e. Greenwich Mean) time internally and converted data to that time zone as it was imported. On output, any xsd:time or xsd:dateTimes would be displayed in Zulu time and would lose the original time zone information. For example,

"2014-02-23T08:18:59+05:00"^^xsd:dateTime

would become

"2014-02-23T03:18:59Z"^^xsd:dateTime

AllegroGraph now keeps the original data's time zone information. Note that the additional computation involved with coercing time information during query means that queries involving xsd:dateTimes or xsd:times may run slightly (a few percentage points) more slowly. There is also a new query option, presentationTimeZone, that controls the display of time zones in the output of SPARQL queries. For example, adding

PREFIX franzOption_presentationTimeZone: <franz:-04:00>

will cause all xsd:times and xsd:dateTimes to be printed at the -04:00 time zone. See the documentation for more information. Since the database format has been changed, the usual upgrade process involving a backup/restore cycle (see Backup and Restore) is required. Please note the following:

1: There are no changes for xsd:dates. They still support what the XML Schema describes as 'recoverable timezone'.

2: "2014-02-23T08:18:59+08:00"^^xsd:dateTime is DISTINCT from "2014-02-23T00:18:59Z"^^xsd:dateTime.

3: Joins can be made over dates that represent the same point on the same timeline even if they have different timezones. Which (equivalent) representation is returned by the join is unspecified.

See the date and time documentation for more information.

Rfe11303 - Fix partial repo creation on agraph-backup restore operations

When performing an agraph-backup restore operation, if you passed a directory name as the archive argument, the restore would fail, but not before a repository was created. Further, the repository would remain in 'restore' mode, so it could not be opened, and its existence would prevent a subsequent valid agraph-backup restore operation from succeeding. agraph-backup now first checks that the argument filename names an existing file which is not a directory before creating any repositories. Also, error messages when agraph-backup** is passed invalid archive file arguments have been improved. See Backup and Restore.

Rfe11044 - Add -p short form of --port to agraph-backup.

With agraph-backup, users may use either -p or --port to specify the port on which an AllegroGraph server is running. See Backup and Restore.

Rfe10629 - Add --supersede argument to agraph-backup

agraph-backup now accepts a --supersede option, which (if specfied) allows overwriting of existing archives (for the backup command) and directories (for the backup-all command), and databases (for the restore and restore-all commands). See Backup and Restore.

Bug22099 - Ensure UUIDs are unique during agraph-backup restores.

Previously, when restoring archive backups to an AllegroGraph server, it was trivially possible to create multiple repositories with the same uuid. Now, agraph-backup will error if a restore operation will cause a uuid conflict. See Backup and Restore.

Bug22090 - Timezone minutes in xsd types

Specifying timezone minutes is no longer optional in xsd:dateTimes and xsd:times, so the following examples are invalid:

"1978-01-03T00:00:00+05"^^xsd:dateTime INVALID  
   Correct form: "1978-01-03T00:00:00+05:00"^^xsd:dateTime  
"00:00:00+05"^^xsd:time INVALID  
   Correct for: "00:00:00+05:00"^^xsd:time

Previously, only xsd:dates checked for the presence of timezone minutes. See the documentation for more information.

Bug22083 - xsd:date and xsd:dateTime years

AllegroGraph now correctly parses both positive and negative dates with more than four digit years. It also signals an error if the date is too large to be encoded into a UPI. Previously, dates like these were silently truncated. See the date and time documentation for more information.

Bug22073 - ORDER BY and non-encoded decimals

When non-encoded decimals were being ordered, they were first converted to IEEE 54 double floats (which have a precision of 15-17 significant digits) which potentially caused them to be sorted incorrectly. This bug has been fixed.

Bug22072 - Validate xsd:decimals

AllegroGraph now signals an error if an xsd:decimal's exponent in scientific notation is outside of the range [-63, 63] because these values cannot be encoded into a UPI. Previously, xsd:decimals like these were silently truncated.

Bug22068 - Reasoning and constant triple patterns

With reasoning enabled, some queries involving triple patterns without variables failed with 'Bus error'. This has been corrected.

Bug22066 - Printing of NaN, INF and -INF

Serialization of xsd single and double float NaN, INF and -INF was broken (for instance, INF became "#.inEinity-double"). This bug has been fixed.

Bug22057 and Bug22054- agraph-backup backup-settings works as documented.

The agraph-backup backup-settings command was incorrectly backing up data as well as settings rather than just backing up settings. Now it just backs up settings. Also, the backup-settings command now can work by finding the settings location from the config file (so the server need not be running). See Backup and Restore where a note describes all the agraph-backup revisions.

Bug22045 - 14 hour datetime comparison logic

For two xsd:dateTimes, one with and one without a timezone, all comparisons must return false if they differ by less than 14 hours. Similar logic applies to xsd:dates and xsd:times. Previously, support for this 14 hour comparison logic was spotty. It worked for some combinations (such as comparisons in a FILTER with a literal) but didn't work for others. With this fix, comparisons for all three types follow the XML Schema specification. See the date and time documentation for more information.

Bug22044 - Fix xsd:date xsd:dateTime comparisons

AllegroGraph previously compared xsd:dates and xsd:dateTimes by converting the date to a dateTime at its first instance. Because dates are actually 24-hour intervals, this logic is suspect so AllegroGraph now signals a type error if this comparison is made (i.e., FILTERs will not longer match and BIND would leave its variable unbound). The previous behavior can still be had by using an explicit cast from date to dateTime. See the date and time documentation for more information.

Bug22041 - Comparisons to decimal zero

Due to a bug in the encoding, decimal 0 appeared to be greater than all decimals less than 1.0. The usual upgrade process involving a backup/restore cycle (see Backup and Restore) is required. All misencoded decimal 0s will be converted to the correct value. If any such corrections are made during the upgrade, then indices are resorted and background index optimization is begun. The index optimization may continue even after the upgrade process has completed. The resulting indices are then as optimized as if all the triples were just imported into the database. This means if the backup was of a fully optimized repository, then after the upgrade running level 2 (the default) index optimization is necessary to get back to that state.

Bug22039 - agload character encoding fixes

Previously, the --encoding argument to agload was ignored. This has been corrected. See Data Loading for information on agload.

Bug22034 - part->concise on encoded dates and decimals

For xsd:dates, xsd:dateTimes, xsd:times and xsd:decimals, the Lisp functions part->concise and part->terse returned the stringified internal representation (like "172839/500" for decimals, and "( . )" for datetimes) which didn't fulfill the promise of being in a "human readable form". With this change, the above functions will return the value in a readable string format without the type. An example:

(part->concise (value->upi "2013-01-03T10:34:00-05:00" :date-time))  
=> "2013-01-03T10:34:00-05:00"

See XSD Date and Time datatypes.

HTTP Client

No significant changes.

SPARQL

Rfe12624 - Standardize some low-level SPARQL equality tests

The efficiency and conformance of low-level tests used to compute RDFterm-equality and SPARQL ordering has been improved. One non-backwards compatible change is that AllegroGraph previously allowed ordered comparisons between plain literals that looked like numbers and typed numeric literals (e.g., "6.21" and "6.23"^^xsd:integer). This comparison now correctly throws a SPARQL type error.

Bug22092 - Fix for SPARQL queries that use the STR() function with a constant

If a SPARQL query contained a FILTER that used the STR() function applied to a constant, then the query could signal an error at planning time. For example, a FILTER like

FILTER( STR( <http:://example.com> ) = ?var )

would not succeed. This has been corrected.

Bug22078 - SPARQL insert did not always encode typed literals

Triples added to a triple-store using SPARQL inserts that used computed values did not always have their typed literals encoded into UPIs. For example, an insert like:

insert { <ex:a> <ex:b> ?c . } where { bind( '1'^^xsd:byte as ?c ) }

would store the literal in string form rather than encoding it directly. This has been corrected.

Bug22077 - Improve efficiency of certain SPARQL sub-queries

Bound variable information could be lost while processing certain SPARQL sub-query forms which would then result in less efficient query execution. This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

Bug21608 - Include ag-call in remote Lisp client.

Previously ag-call was omitted from the remote Lisp client. This has been corrected.

Prolog

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

AllegroGraph 4.12

Server Changes

Rfe12576 - Improve RDF/XML parser efficiency

Improve the lower-level efficiency of AllegroGraph's RDF/XML parser so that it is between 5 and 10% faster.

Rfe12522 - Global configuration directive for query options

A new global configuration directive named QueryOption was added. It can be used multiple times. Each specification is equivalent to a query prefix option. The global configuration directive "QueryOption NAME=VALUE" is the same as the query prefix option "PREFIX franzOption_NAME: ". See Server Configuration for information on configuration directives.

Rfe12429 - More flexible parsing for InstanceTimeout

With this change, InstanceTimeout configuration parameter values in the configuration file can be specified as times rather than only as a number of seconds. For example, you can use 12s, 2m or 3h to specify 12, 120 or 10,800 seconds. The configuration file is described in Server Configuration.

Rfe11010 - Require the `i` index for freetext queries

AllegroGraph's native freetext index requires the presence of the triple-id (i) index in order to operate efficiently. A common problem was to try to use freetext queries when there was no i index. This change automatically adds an i index whenever a freetext index is created and also signals an error if a freetext query is attempted when there is no such index.

Bug22024 - Serialization of decimals

xsd:decimals with zeroes right before the decimal point were serialized with the decimal point misplaced. This bug has been fixed. The database contents were not affected.

Bug21978 - serialize-* functions in Lisp client default to UTF-8.

Use UTF-8 as the external-format when a filename is passed to one of the serialize-* functions in the Lisp client instead of the default external format, which can vary depending on the system's locale.

Bug21976 - Complex GRAPH queries could produce incorrect graph bindings

Depending on the dataset, queries with deeply nested structure that used EXISTS or NOT EXISTS FILTERS with embedded GRAPH clauses could generate incorrect bindings for the graph variable. This has been corrected.

Bug21975 - Increase AllegroGraph Load speed

Low-level efficiency enhancements improve AllegroGraph load speed by roughly 10%.

Bug21966 - Possible "UPI not in string table" problem with remote-triple-stores

It was a possible for a remote-triple-store to lose track of the strings cached on the client after a rollback operation. This could lead to a "UPI not in string table" error. This has been corrected.

Bug21956 - Improve memory usage for zero-length property path queries

A zero-length property path query connects all subjects and objects on the right and left hand sides of the path to themselves. If the store is large and the path produces many matches on one or both sides, AllegroGraph was using more memory than necessary. This has been greatly improved though some queries may experience a slight performance degradation.

Bug21954 - Problem generating soundexes for phrases in Callimachus

If a store was operating in Callimachus mode and a phrase was soundexed that had words in it with no soundex (e.g., a number) then AllegroGraph would generate garbage soundex data. This has been corrected

Bug21949 - Improve bookkeeping of already processed variables during query execution

AllegroGraph was not updating the list of bound variables when evaluating a BIND clause during query execution. This did not change the final results of a query but did mean that AllegroGraph would not always evaluate the query in the most efficient manner. This has been corrected.

Bug21937 - Zoneless xsd:dateTime roundtrip

Adding a triple with an xsd:dateTime without a timezone (e.g. "2013-07-30T23:59:59") was stored as if it had a timezone of zero. With this bug fix, zoneless dateTimes are stored and serialized without timezone. This is in agreement with the XML schema specification and similar to how xsd:dates behave.

Bug21934 - String table mutex not reinitialized after recovery

A mutex used in the string table data structure was not being reinitialized after database restart. Under certain circumstances this could result in a database hanging after restart. This problem has been corrected.

Bug21920 - Soundex should ignore punctuation

AllegroGraph's soundex function (which is used by Callimachus) was not ignoring punctuation properly. This has been corrected.

Bug21915 - Make upi->value on xsd:dates return integers

Due to an undocumented change in v4.11, upi->value on a upi encoded as :date returned a cons of the integer representing the instant and a flag indicating whether the timezone was present. With this change only the integer part is returned (like before v4.11). The flag is returned as the second value by upi->values.

Bug21914 - xsd:dates on the first day of the month

Adding a triple with an xsd:date with a positive timezone that falls on the first day of a month resulted in error "illegal date 0". This bug has been fixed.

Bug21909 - agraph-backup could fail to generate unique UUID on restore

When using the --newuuid option, it was possible for agraph-backup to create multiple triple-stores with the same UUID. This has been corrected. See Backup and Restore for information on agraph-backup.

Bug21903 - Free text indexing with chunk-at-a-time processing

Depending on the execution plan, a query that used both fti:match and chunk-at-a-time processing (see franzOption_chunkProcessingAllowed) could produce too few results. This bug has been fixed.

Bug21899 - Shared memory server fails to clean up

Under low memory conditions, the shared memory server can fail to allocate blocks of shared memory on behalf of a database instance. Under certain circumstances the shared memory server would fail to clean up from partial allocations, thereby exacerbating the low memory condition. This problem has been fixed.

Bug21873 - Link to docs in rpm install

When AllegroGraph was installed from an rpm package, the documentation links in AGWebView were broken. This bug has been fixed.

Bug21872 - agload will write output to agload.log when in --debug mode

Along with the other effects of using the --debug command line argument to agload, all output will be logged to agload.log. See Data Loading.

Bug21869 - Fix problem with registering cartesian predicate mappings

There was a problem which prevented the shortened form of cartesian predicate mappings from working. This has been corrected.

Bug21853 - Connection Pool session port connection failure

Under certain circumstances, the AllegroGraph Java client would throw an exception like the following:

com.franz.agraph.http.exception.AGHttpException:  
Possible session port connection failure

Setting the SessionPorts configuration parameter is discussed here in Server Installation and also in Server Configuration. But this exception could occur even with a proper SessionPorts setting. This problem has been resolved.

Bug21846 - Long running federated queries could fail

The query canceling interface introduced in AllegroGraph 4.11 could lead to query failures when using federated triple-stores. This has been corrected. Note that it is not currently possible to cancel queries running on a federated store. This will be improved upon in a future release of AllegroGraph.

Bug21816 - Fix session multiplexing file descriptor leak

When requests to sessions are multiplexed through the server's main port, the frontend didn't close the http socket used to communicate with the session process. This bug has been fixed. Note that session multiplexing can be enabled for the java client by setting the system property:

com.franz.agraph.http.useMainPortForSessions=true

Bug21784 - Safer auto-close for garbage db's in the Lisp direct client

The db's are now marked for closure when it is noticed they are no longer used and closed at a later specific time.

Bug18813 - Reasoning and subclasses of owl:TransitiveProperty

With this fix the reasoner is able to infer that a property whose type is a subclass of owl:TransitiveProperty is a transitive property. That is,

:p rdfs:subClassOf owl:TransitiveProperty .  
:q rdf:type :p .  
:a :q :b .  
:b :q :c .

entails

:a :q :c .

HTTP Client

No significant changes.

SPARQL

Rfe12537 - Improve zero length property path clause estimation

This change improves the heuristics AllegroGraph uses to estimate the number of rows that a zero length property path query can generate which improves the way AllegroGraph processes a BGP.

Rfe12520 - The SNA members and size magic properties work with RDF lists

You can now use the http://franz.com/ns/allegrograph/4.11/sna/members and http://franz.com/ns/allegrograph/4.11/sna/size magic Properties to query RDF lists stored in your triple-store. For example, if points to the head of the RDF list "1", "2", "3", then this query would return the three elements:

prefix sna: <http://franz.com/ns/allegrograph/4.11/sna/>  
select ?item {  
  <ex:list> <ex:pointsTo> ?list .  
  ?item sna:members ?list .  
}

Would return "1", "2" and "3". See SPARQL Magic Properties for SNA.

Rfe12506 - Improve handling of some queries that use GRAPH clauses

SPARQL is defined such that a query like

GRAPH ?g { A }

should behave as if A was evaluated with ?g bound to each graph in the dataset in turn and then the results of all of these evaluations were unioned together. While correct, this implementation can be very inefficient. AllegroGraph uses various heuristics to determine if a given GRAPH clause can be evaluated more directly and does so when it can. This change improves these heuristics so that more queries will be able to use the more efficient implementation.

Rfe12481 - Improve SPARQL DELETE efficiency

This change greatly increases the efficiency of the DELETE portion of a SPARQL DELETE... INSERT... WHERE... command. The actual increase is dependent on the data and the delete template but it can be more than 40 times faster.

Rfe12442 - Efficiency improvements for SPARQL aggregation

Optimize the code used for SPARQL aggregate queries in general and for count aggregates in particular.

Rfe12400 - Minor property-path query optimization

Improve the underlying efficiency of some property-path queries.

Rfe12398 - Clarify semantics of the identity clause reorderer

AllegroGraph has two clause reorderers which try to preserve the ordering of triple-patterns in a BGP (basic graph pattern). Previously, these were named:

:identity - try to preserve ordering but still prefer to augment the current set of bindings rather than start a new set.
:strict-identity - use the given ordering directly without changing it.

To avoid future confusion, these options are being renamed such that the first method is called :weak-identity and the second is called :identity.

The default query clause reorderer continues to be statistical so queries that do not specify a reorderer will behave exactly as they did before this change.

Rfe12392 - Optimize certain complex join combinations

The query optimizer now converts a query with an algebra like:

(:join (:join A :bgp1) :bgp2)

into the equivalent:

 (:join A :bgp1-and-bgp2)

This BGP (basic graph pattern) merger improves AllegroGraph's ability to reorder queries for efficient execution. AllegroGraph also converts the expression:

(:join (:join :bgp1 A) :bgp2)

into

(:join :bgp1-and-bgp2 A)

unless the two BGPs do not overlap (in which case the transform would produce a cross-product).

Rfe12385 - Improve BGP reordering in some cases

In some cases, the SPARQL query planner now does a better job reordering the triple-patterns in a BGP (basic graph pattern).

Rfe12380 - Improve efficiency of some DISTINCT or REDUCED queries

AllegroGraph now collects fewer intermediate results for some DISTINCT or REDUCED queries.

Rfe12377 - Add intervalContainsTime magic property

Added an additional temporal magic property that can be used to determine if a time is contained within an interval. See SPARQL Magic Properties.

Rfe12372 - Make BGP reordering more intuitive

Previously, AllegroGraph would switch the execution of patterns in a BGP when the BGP contained only two patterns and these patterns had the same statistical clause estimates. E.g., in a query like:

?a ?b ?c .  
?c ?d ?e .

AllegroGraph would execute the second pattern first. AllegroGraph now executes the first pattern first in the case of ties which is more intuitive.

Rfe12368 - Use constraints to limit bindings for GRAPH queries

Improve AllegroGraph's use of FILTERs in limiting the graphs considered in GRAPH clauses with variable graphs. For example,

GRAPH ?g { A }  
FILTER( ?g = <ex:test> )

will now be evaluated more efficiently since AllegroGraph will only consider the graph <ex:test> rather than every graph in the dataset.

Rfe12352 - Provide a sort order for SNA path and group gensyms

Social Network Analysis (SNA) Magic Properties can create node identifiers to keep track of paths and groups. These act like blank nodes and so had no sort order defined for them. This was inconvenient because it made it impossible to, for example, list all of the nodes of a path together. This change adds a sort order to these node identifiers so that they are easier to use in queries.

Rfe12192 - Optimize some wild-card property-path queries

AllegroGraph now computes property-path queries that use the * or + operators more efficiently.

Rfe12032 - Improve property-path expansion

AllegroGraph now expands more property-path queries into their equivalent BGPs (basic graph patterns). This can improve the performance of the query optimizer. For example, AllegroGraph previously expanded

?s :a/:b ?o .  
into ?s   :a :b0 . :b0 :b ?o .

but did not expand paths with non-simple operators like wild-cards or alternation. It now expands these as well so that a path like

?s :a/:b/(:c|:d)/:e/:f ?o

becomes

?s   :a _:b0 .  
_:b0 :b _:b1 .  
_:b1 (:c|:d) _:b2 .  
_:b2 :e _:b3 .  
_:b3 :f _?o .

Rfe11948 - Add PREFIX option for enabling reasoning in a SPARQL query

AllegroGraph's SPARQL 1.1 engine now supports the inferenceMode query option to control the kind of inference used by an individual query. The mode is selected using the query option PREFIX notation. For example, the following PREFIX would cause a query to be evaluated using AllegroGraph's RDFS++ reasoner:

PREFIX franzOption_inferenceMode: <franz:rdfs++>

The following modes can be supplied:

rdfs++ - use RDFS++ reasoning,
false - use no inference,
restriction - use RDFS++ and the restriction reasoner.

Rfe11625 - Improve algebraic optimizations of joins and left-joins

Improve the logic used in the algebraic transform of

(:join (:left-join A B) C) into (:left-join (:join A B) C)

Which is the textual equivalent of moving the optional clause down in a query like

A  
optional { B }  
C

so that it becomes

A  
C  
optional { B }

which can be evaluated more efficiently unless A and C share no bindings.

Rfe11146 - Handle simple OPTIONAL clauses more efficiently

AllegroGraph now handles OPTIONAL clauses that contain only a single triple-pattern more efficiently. This means that queries with clauses that look like

 ?s :predicateA ?o .  
 OPTIONAL { ?s :predicateB ?b }  
 OPTIONAL { ?s :predicateC ?c }

will be processed more efficiently.

Rfe11068 - Improve literal equality constraint handling

AllegroGraph now handles filter expressions involving literal equality more efficiently. For example, in a query like

select * {  
  ?a :predicate ?b .  
  filter( ?b = 'literal' )  
}

AllegroGraph previously would scan all triples with the given predicate and apply the filter afterwards. Now AllegroGraph determines the set of possible matches for 'literal' and queries for them directly.

In addition, AllegroGraph now performs a similar operation if the filter is a literal equality on an expression using the SPARQL str function. For example:

filter( str(?variable) = 'searching' )

As with regular literal equality constraints, such an expression is converted into the equivalent tests for specific values.

Bug22030 - ORDER BY expressions with non-query variables could cause errors

If a SPARQL query used an ORDER BY expression that contained variables that were not found anywhere in the query, then an error could result. An example of this would be a query like:

select * {  
  ?s a ?o .  
} order by desc( str( ?notFoundInQuery ))

This issue has been corrected.

Bug22021: Correct SPARQL TSV printing problem for encoded UPIs

When encoded UPIs were serialized from a SPARQL query using the SPARQL TSV (tab sepated values) output format, they were printing without the enclosing quotes. This has been corrected.

Bug22016 - Correct SPARQL 1.1 bug with IN FILTERs

It was possible for the query engine to produce incorrect results when using FILTER with either the SPARQL IN operator or multiple equality filters on the same variable. This has been corrected.

Bug22015 - Fix possible error when using the SPOGI cache and SPARQL 1.1

In rare circumstance it was possible for a SPARQL 1.1 query to signal an error when the SPOGI cache was enabled. This has been corrected.

Bug21986 - Using distinct aggregation with GROUP_CONCAT and a separator could fail

If a SPARQL query used the GROUP_CONCAT aggregator along with both the DISTINCT and SEPARATOR modifiers, then the solution bindings for that variable could be left empty. This has been corrected.

Bug21980 - Too many results in DISTINCT and REDUCED queries

Under extremely rare circumstances, queries involving DISTINCT or REDUCED could return too many results. This bug has been fixed.

Bug21974 - Assertion failure involving 'n-valid-partials'

When chunk-at-a-time processing was enabled (see query prefix franzOption_chunkProcessingAllowed, described here in the SPARQL Reference), the code keeping track of the number of intermediate results used to run into an assertion failure under very rare circumstances (when two consecutive intermediate results were the same). This bug has been fixed.

Bug21962 - Improve estimates for certain query patterns

AllegroGraph was mis-estimating the sizes of query patterns that resulted from the inlining of filters that used the IN expression. For example,

?s ?predicate ?object .  
FILTER( ?predicate IN ( ex:a, ex:b, ex:c )

This could result in sub-optimal BGP re-ordering. This has been improved.

Bug21961 - Incorrect results with EXISTS and NOT EXISTS

Some queries involving EXISTS or NOT EXISTS filter expressions returned too few results. This bug has been fixed.

Bug21955 - Sorting on unbound variables could lead to a string dictionary error

If a query used an ORDER BY expression involving a computation and some of the variables in the expression were unbound, then the expression could signal a UPI not in string dictionary error. This has been corrected.

Bug21948 - Improve bookkeeping of bound variables during sub-query

When executing a sub-query with a limit, AllegroGraph failed to update the list of variables known to be bound. This prevented some queries from executing as efficiently as possible. This has been corrected.

Bug21947 - Improve efficiency of VALUES in some cases

AllegroGraph now does a better job of using the information in a VALUES clause to restrict alternate solutions. This can greatly increase the speed of some queries.

Bug21941 - Queries with magic properties could fail in a remote-triple-store

Queries that used magic properties could cause an error when run from a remote-triple-store. This has been corrected.

Bug21940 - Using magic properties with filters could lose solutions

If the circumstances were just right, a query that used a magic property that produced multiple bindings and that used a filter on those bindings could fail to return all solutions. This has been corrected.

Bug21933 - Externally introduced variables could fail in nested sub-queries

If a query used externally introduced variables (e.g., via the Lisp APIS's with-variables argument or via the variable bindings in an HTTP REST request), and these variables were used in FILTER clauses in a nested sub-query, then the query could fail to find any matches. This has been corrected.

Bug21921 - BGP triple filters with Chunk-at-a-Time processing

In rare cases, a filtering triple-pattern could fail to remove some solutions when using Chunk-at-a-Time query processing. This has been fixed.

Bug21902 - Fix incompatibility between magic predicates and the identity reorderer

If the identity BGP clause reorderer was being used in a query with magic predicates, then the magic predicates were being sent to the end of the BGP. This has been corrected.

Bug21901 - Minor improvements to some range query clause orderings

Improve the statistical estimates used to reorder BGP patterns that use range queries.

Bug21892 - Query engine selection failed to take OFFSET into account

AllegroGraph's decision on whether to use the single-set or the chunk-at-a-time query execution engine relied only on the LIMIT of the query rather than the LIMIT and the OFFSET. This has been corrected.

Bug21880 - Using with-variables in SPARQL describe caused an error

Using SPARQL describe with an external variable binding would lead to an error at query execution time. For example, this query would fail:

 describe ?x

(where ?x was bound externally). This has been corrected.

Bug21865 - Comparisons of xsd:dateTimes with fractional seconds

Comparisons (including ORDER BY) of xsd:dateTimes with fractional seconds were broken. This bug has been fixed.

Bug21862 - Some FILTERs and BGP orderings could lead to empty bindings

In rare cases, AllegroGraph could fail in its bookkeeping of which variables were bound while it executed a query. This could lead to the introduction of spurious rows with null bindings. This has been corrected.

Bug21858 - Distinct, SERVICE and sub-query could lead to planning failure

If a query used a sub-query in a SERVICE clause and asked for unique results, then AllegroGraph would signal an error during query planning. This has been corrected.

Bug21852 - Improve bound variable tracking

AllegroGraph should assume that any variable supplied via external settings (e.g., via the with-variables parameter to run-sparql) is bound. Failure to do so could cause some queries to fail. This has been corrected.

Bug21828 - VALUES could fail to correctly handle OPTIONAL bindings

Using the VALUES clause could incorrectly drop solutions whose rows contained unbound values. This has been corrected.

Bug21815 - Fix property-path memory leak

Some queries involving property-path expressions could run out of memory, because the intermediate result set was not freed as soon as it was consumed. This bug has been fixed.

Bug21803 - Exclusive ranges of encoded non-integer types

Previously, filter expressions on all non-integer types (floats, decimals, etc, excluding xsd:date and xsd:dateTime) involving exclusive ranges ('<' and '> ') behaved as if an inclusive range ('<=', '> =') had been specified. This bug has been fixed.

Rfe12591: Improve efficiency of some Basic Graph Pattern evaluations

AllegroGraph now makes better use of intermediate result sets when evaluating basic graph patterns (BGPs) in nested groups of joins and left-joins.

AGWebView

Bug21997 - Agwebview and sessions

Due to a thread safety issue, AGWebView's 'Create Session' or any subsequent action in that session could produce garbage results ranging from 'Not found' to garbled output. This bug has been fixed.

Changes to the Lisp API

Rfe12562 - upi->value round-trip

For dates and date-times a complete representation is a (<universal-time> . <timezone-information>) cons. While this would make a good candidate for the return value of upi->value, to maintain backward compatibility by default upi->value must return a single number representing the universal time.

However, this means that during the upi->value/value->upi round-trip the timezone information is lost.

This change adds a new keyword argument :complete? (defaults to nil). With :complete? t, upi->value returns the complete representation and the roundtrip works in the sense of:

(upi= (value->upi (upi->value upi :complete? t) (upi-type-code upi))  
      upi)

Currently, for UPIs other than dates and date-times, :complete? has no effect.

Rfe12557 - run-sparql can now use a UPI or future-part for the default base

Previously the Lisp function run-sparql only allowed a string to specify the default-base. It can now take a string or a UPI or future-part.

Rfe12508 - normalizing with-variables should convert $var to ?var

The with-variables parameter in the Lisp API to run-sparql allows for variable names to be identified as strings or symbols. The name can be a plain string like "person" or a string whose first character is #\? or #\$ like "?person" or "$person". All three of these representations refer to the same query variable: ?person. Prior to this change, AllegroGraph could misinterpret variable names that used $ as the first character. This has been corrected.

Bug21953 - Support server-side loading via the Lisp client.

When using the Lisp client and connected to a remote-triple-store, passing a URI or URI-like string to one of the bulk-load functions (e.g. load-ntriples) will cause the resource to be loaded directly by the server instead of being delivered via the client.

Bug21891 - db.agraph.log:log-output captured in dumplisp

If dumplisp (a function used to save a running image) was used after opening a triple store, the stream used for logging would be captured. Upon restore of the dumped image, subsequent logging attempts (performed by internal AllegroGraph code) could result in confusing errors or file corruption. This has been fixed.

Bug21860 - Make Lisp clients find bundled liblzo2.so.2

Previously, Lisp clients resorted to using the liblzo library located in the acl directory ("sys:") or somewhere the OS dynamic library loader can find it. With this change the bundled liblzo2.so.2 is loaded.

Bug21841 - db-get-triple leaks sockets

The Lisp function db-get-triple didn't close the socket it used for communication with a remote triple store which resulted in leaking sockets in CLOSE_WAIT state. This bug has been fixed.

Bug21848 - load-turtle could use the wrong triple-store

The load-turtle command could fail to respect the value of the db parameter and instead load triples into the store currently bound to db. This has been corrected.

Bug21810 - Polygon-vertexes did not work with future-parts

If called with a future-part rather than a UPI, the polygon-vertexes function would return no results. This has been corrected.

Bug21611 - Load patches in Lisp remote client

With this fix the Lisp remote client loads patches and enables pretty printing of parts and triples just as the direct client does.

Bug21070 - Fix corruption hazard in direct Lisp client.

Prevent possible database corruption that could occur when interrupting a direct Lisp client app that is modifying the transaction log.

Bug21045 - create-freetext-index function failed when called with UPIs

The create-freetext-index function (in the Lisp API) would fail if called with UPIs (rather than, e.g., future-parts). This has been corrected.

Prolog

Rfe12222 - AllegroGraph Prolog unifies UPIs and future-parts

When AllegroGraph is loaded, the semantics of Prolog unification is extended such that:

Non-eq UPIs that are equivalent (byte-for-byte the same) will unify.
Non-eq future-parts that resolve to the same UPI will unify.
Similarly, a UPI and a future-part that resolves to that UPI will unify with each other.

This enhancement does not change the capabilities of Prolog queries in AllegroGraph, but allows simpler, less-verbose coding that looks a little more declarative and less procedural. For example, the following

(select (?manager ?employee)  
  (member ?employee (!acme:joe !acme:jill !acme:jim !acme:jerry))  
  (q ?employee !acme:hasManager ?manager)  
  (member ?manager (!acme:bill !acme:bob !acme:betty !acme:benjamin)))

would not work as intended before this change because ?manager would be unified with a UPI, and a UPI would not unify the future-parts in the last clause. The last clause would have had to be coded as something like

(member ?manager1 (!acme:bill !acme:bob !acme:betty !acme:benjamin))  
(part= ?manager manager1)

Using Prolog is AllegroGraph is documented here in the Lisp Reference.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Bug21953. Fix generation of some POST method requests.

AGHTTPClient would write query parameters to the body of POST requests when the content-type was not application/x-www-form-urlencoded. This can result in http servers not finding the query parameters and processing the request incorrectly. This bug corrects the problem by writing POST request query parameters to the URL, unless the request has no body and the content type is application/x-www-form/urlencoded. AGRepositoryConnection.load() (see Javadocs) now works for importing server-side statements.

AllegroGraph 4.11

Server Changes

Rfe12309 - Improve algebraic query manipulation

AllegroGraph now notices patterns of the form

(:join (:join A (:bgp B)) (:bgp C))

and transforms them into the equivalent

(:join A (:bgp B C))

(I.e., it merges the two BGPs). This transform brings more triple-patterns together which lets AllegroGraph order the query operations better.

Rfe12305 - Relative paths in the configuration file

Relative file or directory paths in the configuration file were always interpreted relative to the directory containing the configuration file. In this rfe, a BaseDir configuration directive was added that allows another path to be specified for this purpose. See the documentation in Server Configuration for more.

Rfe12280 - Simplify geospatial type-mapping API

Geospatial subtype mappings can now be set up with the regular datatype and predicate mappings rather than needing to use the specialized functions. The old functions still work but are now unnecessary. They will be removed in some future release.

Rfe12260 - Automatically map temporal predicate data to xsd:dateTimes

Triple-stores created in this version of AllegroGraph will include the three AllegroGraph temporal predicates in the list of automatically mapped predicates. I.e., the objects in triples with these predicates:

<http://franz.com/ns/allegrograph/3.0/temporal/endpoint>
<http://franz.com/ns/allegrograph/3.0/temporal/startpoint>
<http://franz.com/ns/allegrograph/3.0/temporal/time>

will be converted into encoded xsd:dateTime literals. This encoding makes range queries against them faster and makes it easier to use AllegroGraph's temporal functionality.

Rfe12259 - Optimize certain combinations of JOINs and FILTERs

The query engine now recognizes certain patterns of JOINs and FILTERs and rewrites them to be more efficient. For example, a pattern like

(:join A (:filter B f))

can often be better evaluated as

(:filter (:join A B) f)

Rfe12182 - More restrictive umask

With this change, users who aren't the owner of the file, or a member of the group of the file (that is, "other" users in chmod manpage terms) have no read, write or execute permission on files created by AllegroGraph.

On existing installations, it's recommended, but in no way enforced, to change permissions the same way with 'chmod -R o-rwx <directories>' where <directories> is the list of directories holding AllegroGraph data (LogDir, SettingsDirectory, catalog directories, etc).

Rfe12175 - Audit log

When the 'Auditing' global configuration directive is set to 'yes' (it defaults to 'no'), events that have implications on security or performance will be logged to the new system database in the new system catalog. Note that regardless of permissions only superuser can read or write the system catalog.

See Auditing for more information.

Rfe12151 - Remove support for loading N-Triples into a federated triple-store

AllegroGraph 3 introduced some techniques that optimized loading of N-Triples into a federated-triple-store. These have been superseded in AllegroGraph 4 because agload is significantly faster and easy to use.

Rfe12147 - Better manage memory especially for large solutions

AllegroGraph now manages large query result sets (with or without DISTINCT or ORDER BY) more efficiently.

Rfe12138 - Agload: reduced memory footprint and other enhancements

Reduced memory consumption of agload processes. Particularly in the case where there are many input sources.

Agload now analyzes input sources on the fly. This means you will not necessarily encounter errors about invalid input sources prior to modification of a triple-store. Use --dry-run command line argument if you wish to pre-analyze an agload job.

Rfe12125 - Improve performance of unordered queries with small limits

AllegroGraph now uses chunk-at-a-time query processing for queries with small LIMITs that do not use ORDER BY more frequently. In particular, the chunkProcessingAllowed query option can now take three values:

yes - use chunk-at-a-time mode whenever possible,
no - always use single-set mode, or
possibly - choose the execution mode that makes the most sense for the current query.

The default value for chunkProcessingAllowed has been changed from no to possibly.

Rfe12111 - Improve memory checking efficiency

Improve the efficiency of AllegroGraph's memory tracking code slightly.

Rfe12089 - Improve cursor resource management

Improve AllegroGraph's low-level cursor memory management.

Rfe12063 - Allow in-memory-triple-stores to use blank nodes

The in-memory-triple-store implementation for the remote-lisp-client was missing several methods required for full blank node support. This has been corrected.

Rfe11973 - Improve usage of UPI-map

Improved the way AllegroGraph manages the use of secondary indices (UPI-maps).

Rfe11955 - :commit argument added to delete-duplicate-triples

The :commit argument was added to delete-duplicate-triples to allow for the deleting a very large number of duplicates.

Rfe11886 - Use statistics more often when reordering query clauses

The query engine now uses store statistics in more places when working to reorder the clauses in a query's basic graph patterns (BGPs). This should produce better plans more often with less need for customer query tweaking.

Rfe11585 - Some queries unnecessarily accumulated partial results

AllegroGraph was sometimes accumulating results unnecessarily. For example, the query

select (count(*) as ?total) where {?s ?p ?o}

was accumulating the actual results even though only the summary was necessary.

Rfe11511 - Encode xsd:decimals

xsd:decimals used to be stored as strings. With this change, they are automatically encoded directly in the UPI which potentially brings significant savings in storage space and computation in queries. The most significant 20 digits are preserved by the encoding.

Note that this change only affects the encoding of new triples. To convert old triples and take advantage of the more efficient storage, the database contents must be reloaded from a serialized format. We recommend exporting in nquad format and then reimporting it with agload. Restoring from a backup is not sufficient.

Rfe11197 - Serialize triple IDs as resources rather than literals

Previously, a triple ID would have been serialized as a typed literal like

"13"^^<http://franz.com/ns/allegrograph/4.0/triple-id>

This format prevented the use of triple IDs in the subject of a triple so we have changed the serialization to a URI like

<http://franz.com/ns/allegrograph/4.0/tripleId#13>

Any triple IDs already in a triple-store will remain the same (though they will print differently). This change does mean, however, than any files that contain triple IDs in the old format will need to be converted in order to be re-loaded (note, however, that the IDs assigned to triples are not part of any serialization format which means that they cannot reliably round-trip in any case).

Rfe10292 - Handle filename limit for unix domain sockets

Linux has a limit of 107 characters for filenames of UNIX domain sockets. With this change, when bumping into that limit, a meaningful error message is printed and the workaround of setting the UnixDomainSocketDirectory or the SettingsDirectory configuration directive to a shorter path is suggested.

Bug21793 - Fix race in inter process communication

Under extremely rare circumstances, multiple threads could end up listening on the same communication channel which can lead to errors and deadlocks. This was duly detected by an assertion that appeared in the log as:

the assertion (endp (intersection  
                     (db.agraph.spawn::receiver-for  
                      db.agraph.spawn::r)  
                     (db.agraph.spawn::receiver-for  
                      db.agraph.spawn::receiver)))  
failed.

This bug has been fixed.

Bug21782 - Potential double close of cursors

An inadvertent double close of cursors was discovered that could potentially result in improper sharing of a data structure. This has been corrected.

Bug21779 - Fix low-level cursor resourcing bug

Under heavy load, the secondary index (UPI-map) creation function could leak a cursor leading to potential resource problems. This has been corrected.

Bug21761 - Prolog `q-` functor could be treated as `q`

If a reasoning triple-store can use secondary indices (UPI-maps) then it was possible for AllegroGraph to treat some q- functors as q functors in a given Prolog select query. This has been corrected.

Bug21745 - Fix crash related to aggregate functions

Under rare circumstances, queries involving aggregate functions may cause the backend to crash. This bug has been fixed.

Bug21744 - Some Prolog select queries could return duplicate results

If using secondary indices (i.e., UPI-maps), it was possible for Prolog select queries that returned raw UPIs to overwrite one solution with another. This problem could only occur with UPI only queries and never with queries that returned strings. The problem is now fixed.

Bug21740 - Rare hangs under high load

The message broker (called the "Hub" process) may get into a state where forwarding a message blocks, preventing all other messages from being handled. This bug has been fixed.

Bug21735 - Corrupted distinct results

Under rare circumstances, queries involving DISTINCT could produce results with the bindings assigned to the wrong variables. This bug has been fixed.

Bug21702 - XSD casting in a BIND could convert to xsd:integer incorrectly

Using an XSD cast in a BIND could loss the datatype and switch it to xsd:integer. E.g.,

BIND(xs:long(?test) AS ?testLong)

would bind ?testLong to a value whose datatype was xsd:integer rather than xsd:long. This has been corrected.

Bug21693 - Avoid forking too many blank node servers in agload.

Improves agload performance under default conditions when loading ntriples or nquads files which are small and numerous.

Bug21691 - Reasoning over subproperties of symmetric properties

AllegroGraph's RDFS++ reasoner partially failed to infer triples from sub-properties of a symmetric property. This has been corrected.

Bug21680 - Some BIND forms could be evaluated incorrectly.

A BIND form at the start of a BGP could fail to get values from earlier in the query evaluation.

For example, this form

?a a ?b .  
{ BIND( ?b as ?otherB  
  ?otherB a ?c .  
}

could lose the binding for ?otherB. This has been corrected.

Bug21670 - FILTER and EXISTS could interact incorrectly.

Depending on the shape of the rest of the query, it was possible for a FILTER clause to be applied to the inside of an EXISTS or NOT EXISTS filter rather than being applied to the main body of the query. This has been corrected.

Bug21663 - Redo of new index creation confuses instance process

If a database with a newly-created index has been shut down uncleanly, restart of the database may crash. This has been corrected.

Bug21651 - Prolog ?? and reasoning

On a reasoning triple store, the prolog ?? construct in the predictate position used to fail with a type error. This bug has been fixed.

Bug21642 - Agload buffer overflow

Due to a buffer overflow, agload could corrupt memory and potentially crash. This has been fixed.

Bug21640 - Race condition when reaping abandoned sessions

Previously, if concurrent processes performed the operation to reap abandoned sessions, shared memory state could be corrupted, resulting in hangs or crashes. This has been fixed. (This bug was actually fixed in version 4.10.)

Bug21625 - Large result sets could produce bogus triples with INSERT and CONSTRUCT

In cases where there were more than 100,000 results, the machinery that prevents INSERT and CONSTRUCT queries from producing duplicate results could create invalid answers. This has been corrected.

Bug21596 - Serialize-rdf/xml could fail on some predicates

AllegroGraph's RDF/XML serializer would fail if a predicate ended with something other than a letter, a digit, an underscore or a dash. This conformed to an older version of the Extensible Markup Language but not to the most recent version. This has been corrected.

Bug21586 - Possible divide by zero error in index optimizer

Under some circumstances an index optimizer could attempt to perform a division by zero when logging statistics to agraph.log. This would result in a non-fatal error message being logged instead of the statistics. This has been corrected.

Bug21580 - Improve external variable binding bookkeeping

The values for externally supplied variables could be lost for some complex interactions of EXISTS (or NOT EXISTS) filters and UNIONS.

This has been corrected.

Bug21570 - Fix 'No session ports available' error message

When trying to start a dedicated session and no more free ports were available, AllegroGraph failed with an internal error (calling @UUID on NIL) instead of producing an informative error message. This bug has been fixed.

Also, the http error code was changed from 400 ("Bad request") to 503 ("Service unavailable") to indicate a temporary condition.

Bug21553 - Fix MemoryCheckWhen without MemoryReleaseThreshold

If the memoryReleaseThreshold configuration directive was not specified but memoryCheckWhen was, then query execution failed with type errors trying to compare memory footprint to NIL. With this fix, AllegroGraph refuses to start in these circumstances.

HTTP Client

Rfe12136 - Optimize front-end http server

The timeout for reading headers was lowered to 5s which makes the front-end http server make better use of its available resources and more resilent when clients with http keep-alive don't close the connection in a timely manner.

Also, a warning is logged if the number of idle http worker threads falls below 2, because it can easily cause a performance drop.

SPARQL

Rfe12296 - Support AllegroGraph direct reification in SPARQL

AllegroGraph now supports direct reification in SPARQL queries using the <http://franz.com/ns/allegrograph/4.0/tripleId> magic property. This property allows you to bind the ID of a triple to a SPARQL variable which can then be used in reification. For example, the following query first adds two triples and then adds four additional triples that describe the first two:

prefix : <eh://>  
prefix franz: <http://franz.com/ns/allegrograph/4.0/>  
insert data {  
  :gary :likes :dogs .  
  :gary :likes :cats .  
} ;  
 
insert {  
  ?id :mentioned ?now .  
  :gary :believes ?id .  
} where {  
  ?id franz:tripleId (:gary :likes ?what) .  
  bind( now() as ?now )  
}

See the documentation for additional details on direct reification and its advantages and limitations.

Rfe12246 - Enhancements to SPARQL freetext queries

The fti:match and fti:matchExpression predicates can now optionally retrieve the object of the matching triples and select the freetext index to use. (These enhancements are currently available only for AllegroGraph's native freetext indices. Support for Solr and MongoDB will be forthcoming).

The new features rely on SPARQL magic properties to provide a syntactic shortcut for list patterns. As an example, this query retrieves the subject (into ?subject) and text (into ?text) for the freetext query that matches "nepal":

select ?subject ?text {  
  (?subject ?text) fti:match 'nepal' .  
}

This query retrieves only subjects and uses only the freetext index named comments.

select ?item {  
  ?item fti:match ('upgrade' 'comments')  
}

More examples and information can be found in the documentation.

Rfe12243 - Support AllegroGraph Social Networking Analysis in SPARQL

AllegroGraph's SPARQL engine now supports a set of magic properties to enable Social Networking Analysis including centrality measures, path finding and cliques. See SPARQL Magic Properties for more details.

Rfe12231 - Improve SPARQL geospatial integration

AllegroGraph now supports several magic properties to make SPARQL / geospatial integration easier. For example, this query finds all of the points within a circle centered at 10, 2:

select ?who ?where {  
  (?who ?where) geo:inCircleXY ( ex:location 10 2) .  
}

The following magic properties are provided:

geo:inBoundingBoxXY  
geo:inBoundingBox  
geo:inCircle  
geo:inCircleKilometers  
geo:inCircleMiles

(where geo is the usual Geospatial namespace abbreviation for <http://franz.com/ns/allegrograph/3.0/geospatial/>). See SPARQL Magic Properties for additional details on using these properties.

Rfe12171 - Improve variable GRAPH clause handling in some situations

SPARQL specifies that a graph clause like

 GRAPH ?g { ... }

must be implemented as if it iterated over each graph in the dataset and took the union of the resulting bindings. There are sometimes more efficient ways to get the same results and AllegroGraph uses these when it can. AllegroGraph was not, however, using information about any constraints on the variable ?g that were provided by other parts of the query. For example, a query like:

?s :p ?g .  
GRAPH ?g { ... }

might produce a single binding for ?g but AllegroGraph was still iterating over all of the named graphs of the dataset. This has been optimized so that AllegroGraph will use the additional information when it can determine that the number of bindings for ?g are smaller than the number of graphs in the dataset.

Rfe12057 - Manage disk and memory resources better for CONSTRUCT queries

When there are many results, SPARQL CONSTRUCT queries buffer their results to disk to conserve memory. The cutoff value for when to buffer to disk was unnecessarily small which meant that CONSTRUCT went to disk more often then necessary. This has been corrected.

Rfe12035 - Improve handling of some SPARQL queries with a GRAPH clause

By definition, SPARQL must handle a GRAPH clause as if the query engine evaluated the clause once for each graph in the dataset. However, it is often possible to evaluate the clause more efficiently.

This change improves the heuristics AllegroGraph uses to determine when the more efficient approach will achieve the correct results which means that some queries with a GRAPH clause will be executed more quickly.

Rfe11319 - Support SPARQL federated queries (the SERVICE clause)

AllegroGraph now supports SPARQL federated query using the SERVICE clause. The initial implementation is functional but not highly optimized.

Rfe8063 - Support AllegroGraph temporal reasoning in SPARQL

AllegroGraph's SPARQL engine now supports a set of magic properties to enable temporal reasoning. See SPARQL Magic Properties for more details.

Bug21757 - SPARQL geospatial queries could fail

Depending on the set up of a triple-store's geospatial subtype mappings, it was possible for a SPARQL query using geospatial extensions to fail to find results. This has been corrected.

Bug21742 - SPARQL parser failed on quad patterns with collections

The SPARQL parser could lose track of triple-patterns that arose from abbreviations using either nested blank nodes or collections. This has been corrected.

Bug21725 - Correct potential error when SPARQL Update touches too many rows

If a SPARQL DELETE ... INSERT ... WHERE request changes many rows, then AllegroGraph caches the results of the WHERE expression to disk to reduce memory pressure. In some cases, this could result in a error when the cached results were re-read. This only occurred for Update requests that touched more than 100,000 rows. This has been corrected.

Bug21718 - Unused variable optimization could produce incorrect results

AllegroGraph analyzes SPARQL queries that use DISTINCT in order to remove the collection of bindings for variables that are not needed to produce the final result. In some cases, it was possible for this analysis to cause the execution of a code path that added solutions incorrectly. This has been corrected.

Bug21573 - SPARQL queries could intern strings unnecessarily

SPARQL queries that used the FROM or FROM NAMED clause always tried to intern the graphs into the triple-store's string table. This was both unnecessary and could cause an error if the triple-store was read-only or a federation.

This has been corrected.

Bug21011, Bug21571 - handle nested SPARQL BGP syntactic sugar correctly

Some SPARQL triple-patterns that involved nested brackets and object lists were being parsed incorrectly. This has been corrected. Some examples of queries that work correctly now but that failed without this change include:

PREFIX : <ex:>  
SELECT * {  
    :x :y [:z (:b [:d (:e :f),(:g :h)])]  
}  
 
PREFIX : <ex:>  
SELECT * {  
    ?psi :a :b, [:c :d], [:e :f] .  
}

and

PREFIX : <ex:>  
SELECT * {  
    ?psi :a :b, [:c [:d :e]] .  
}

Improve Pattern estimation where there is no :p index

Improved query pattern estimators for triple-stores with no predicate index (e.g., :posgi, :psogi, etc.). This will help reorder query clauses more intelligently.

AGWebView

Rfe12325 - Add audit log query interface to AGWebView

There is now an audit menu item in the admin menu. This provides access to the audit log query page which lets you filter audit log events by user, class and date.

Changes to the Lisp API

Bug21708 - Fix some low-level Lisp API geospatial functions

Some low-level Lisp API functions had not been enhanced to account for AllegroGraph's geospatial encoding. This has been corrected.

Bug21689 - Lisp client printer variable leakage

On a remote store, the run-sparql function failed with an 'Illegal sharp character' error if print-length was set to a small value. This bug and more generally the leakage of printer variables into run-sparql has been fixed.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe12118 - AGRepository with connection pooling

With this change, an AGRepository can now be configured to use a connection pool via the setConnPool method, enabling Sesame apps to transparently benefit from connection pooling. See the javadoc for setConnPool for details.

Rfe10513 - Configurable blankNodesPerRequest

With this change, the Java client supports configuring the number of blank nodes automatically fetched per request to the server by the AGRepository's AGValueFactory. The default remains at 100, but it can be configured by setting the property:

com.franz.agraph.repository.blankNodesPerRequest

Applications can also configure it directly, see the javadoc for AGValueFactory#setBlankNodesPerRequest for details.

Rfe9869 - Support SPARQL/Update from Jena

With this change the Jena adapter supports an execUpdate method in AGQueryExecution for executing SPARQL Updates. Jena Tutorial example13 has been augmented to demonstrate its use.

AllegroGraph 4.10

Server Changes

Rfe12004 - Ag4 plugins for TBC 4.0.0

With this change, the AG plugins have been updated to work with TBC 4.0.0. For installation details, see Top Braid Composer Plugin.

Rfe11986 - Improve efficiency of some property-path queries

AllegroGraph now evaluates property path queries whose initial or final endpoints come from a small set of values more efficiently. For example, a query like

prefix gnO:<http://www.geonames.org/ontology#>  
select ?id ?par ?parname {  
  bind (URI('http://sws.geonames.org/104515/') as ?id)  
 
  ?id rdf:type gnO:Feature .  
  ?id gnO:parentFeature+ ?par .  
  optional {  
    ?par gnO:name ?parname  
  }  
}

will be evaluated much more quickly than before because the subject of the path query is fixed.

Rfe11937 - Agload: remove blank node cache size guessing code

In agload, blank node cache size guessing code did not work very well. It was removed and the blank node cache size defaults to 1,000,000 which works reasonably well on small and large memory machines.

This change will only be noticed when loading ntriple or nquad files with blank nodes.

Rfe11839 - Improve eager evaluation of impossible query constraints

AllegroGraph tries to evaluate FILTER constraints as early as possible. In particular, a constraint may become impossible or a tautology before the query even runs if external variable bindings are supplied. AllegroGraph was already handling tautologies eagerly but was not doing the same for impossible constraints. This has now been remedied.

Rfe11827 - Improved memory footprint of joins with unbound variables

Previously, queries involving joins had a worst case memory footprint of O(2^N) where N is the number of join variables. This is now O(1), but the time complexity is O(N). In practice, this means that the client gets a timeout instead of a memory exhaustion warning.

Rfe11233 - Simplify property path queries when possible

Property path queries that do not use the zero or more or one or more operators and that do not have complex alternation can be expanded directly into simpler forms. This lets them be executed using the regular query machinery which can often be faster than the specialized path query machinery.

Rfe9699 - Dynamically grow string table files

Prior to this change string tables files were large sparse files. The apparent size of these files frequently led to user confusion. In addition, full filesystem errors were not handled gracefully when modifying the sparse files. Now string table files start at a minimum size and grow as needed. If growth of a string table fails due to a full filesystem error, a complaint will be logged to agraph.log and the operation will be retried periodically until it succeeds.

Bug21592 - Improve cursor resource management

Some cursors were failing to close all of their file handles which could lead to errors. This has been corrected.

Bug21583 - see Bug21575 below.

Bug21582 - Complex nested unions could fail to return some bindings

Given the right data, a complex query involving multiple nested unions could misapply an optimization meant for one part of the union to a different part. This could lead to some variables in the query being unbound.

This has been corrected.

Bug21575, Bug21583 - property path optimizations and better bookkeeping

Improve the way that AllegroGraph implements some property path queries.

Correct a problem with property path queries and over-zealous resource management.

Improve the bookkeeping of unused variables when using external bindings.

Bug21570 - Fix 'No session ports available' error message

Bug21550 - Complex FILTER EXISTS expressions could cause parsing errors

Complex FILTERs that used EXISTS (or NOT EXISTS) could signal errors during parsing. This has been corrected.

Bug21549 - Nested sub-queries could fail to project variables correctly

A query with nested sub-queries that used * to return all variables in some of the sub-queries could incorrectly reason about variable scope which could cause queries to fail. This has been corrected.

Bug21534 - Garbled error message on memoryCheckWhen parsing failure

When the memoryCheckWhen configuration directive had an unsyntactic value ("memoryCheckWhen junk:1"), the error message was wrong. This bug has been fixed.

Bug21506 - Correct possible error when mixing UNION and GRAPH

Under some circumstances, a query with both UNION and GRAPH clauses could signal an error during execution. This has been corrected.

Bug21505 - Unused variables from a sub-query could confuse AllegroGraph

AllegroGraph removes query variables that are not used if a query has either the DISTINCT or REDUCED modifiers. The unused variables logic was not correctly handling the case of a sub-query whose projected variables were not used elsewhere. This could lead to an error during query processing. This has been corrected.

Bug21498 - Do not signal error when all ORDER BY expressions are redundant

AllegroGraph removes ORDER BY expressions that do not depend on the bindings of a row because they cannot have any bearing on the outcome. For example, the query:

SELECT * { ?s a ?o } ORDER BY ?pineapple

is equivalent to the query

SELECT * { ?s a ?o }

Unfortunately, AllegroGraph was incorrectly signaling an error when all of the expressions in an ORDER BY clause were irrelevant and therefore removed.

This has been corrected.

Bug21497 - Filters could be misapplied in some situations

AllegroGraph attempts to move FILTERS closer to where they apply in order to increase query speed. If a FILTER was applicable on both sides of a UNION and also applied to single triple-patterns then it was possible for AllegroGraph to think that the FILTER only needed to be used on the left-hand side of the UNION. For example, a query like this could fail to apply the FILTER to ?c on the right-hand side:

select * {  
  { ?a ex:b ?c . } union { ?a ex:c ?c . }  
  filter (?c*2 = 2)  
 }

This has been corrected.

Bug21496 - Improve memory management for CONSTRUCT queries

AllegroGraph now tracks memory more accurately and manages it more efficiently for CONSTRUCT queries.

Bug21493 - Bind is too aggressive in assuming single values

AllegroGraph was over-optimizing assignment (BIND) such that assignments to the same variable name in different pieces of a UNION could fail to work correctly. For example, a query like

SELECT * {  
  { BIND( :person1 as ?person ) }  
  UNION  
  { BIND( :person2 as ?person ) }  
}

would return only one binding for ?person rather than two.

This has been corrected.

Bug21486 - Correct handling of some IN LINE data syntax

AllegroGraph was not correctly reading all of the supplied values when using the abbreviated simple-variable syntax. E.g., a query like

select ?person (concat(?first, ' ', ?last) as ?name) {  
  values ?person { ex:person1 ex:person2 ex:person10 }  
  ?person ex:first-name ?first .  
  ?person ex:last-name ?last .  
}

was processing only the first supplied value. This has been corrected.

Bug21476 - Some blank nodes could fail to print in a 32-bit Lisp client

It was possible for blank-nodes to raise an error when AllegroGraph tried to print them in N-Triples format. This has been corrected.

Bug21470 - Corrected Turtle parser relative URI normalization

AllegroGraph was incorrectly applying sections 6.2.2 and 6.2.3 of RFC3986 when it merged a base URI with a relative one. This has been corrected.

Bug21458 - Standardize AllegroGraph exit code

AllegroGraph's license check can be resolved in three ways:

Normal start up - a license is present and correct.
Free start up - no license is present so the free limit is used.
Limited start up - only partial license information is present so AllegroGraph runs in free mode.

In all of these cases, the exit code returned will be zero (0).

If the license code is invalid or if there are any other problems with server start up, then AllegroGraph will fail to start and return a non-zero exit code.

Bug21454 - URI parsing was too strict

AllegroGraph was being stricter than necessary when parsing BASE URIs such as foo:bar:/baz. This has been corrected.

Bug21379 - Reasoning triple-store confusion about unspecified graph fixed

A reasoning triple-store failed to distinguish between a query where the graph was left off (i.e., left as a wild card) and a query where the graph was explicitly set to the default-graph of the current triple-store. This could cause it to infer incorrect triples.

This has been corrected.

Bug21365 - Fix lookup of user access rights when authorizing access to repositories in the root catalog.

AllegroGraph would deny authorization requests on repositories in the root catalog under some conditions, despite specific user access right allowing such (read/write access on catalog "/", repository ""), or global rights (read/write access on catalog "", repository "*"). This has been corrected.

Bug21353 - Correctly handle aggregation on empty datasets

AllegroGraph was failing the Data Access Working Group (DAWG) "agg empty group" test because it was not returning a row for a query asking for aggregation on an empty dataset. This has been corrected.

Bug21158 - Catch errors during filter analysis

AllegroGraph was signaling an error during query planning if the eager analysis of a FILTER expression resulted in a error. The correct behavior is to treat the FILTER as an impossible constraint.

This is now handled correctly.

HTTP Client

Rfe11609 - Support text/table content-type in the HTTP protocol

AllegroGraph will now return a simple ASCII table if an HTTP request uses the content-type text/table. For example, a query might return a table like:

  -----------------------------------  
  | person    | name     | location |  
  ===================================  
  | mo:Bob    | "Bob"    | ---      |  
  | mo:Robert | "Robert" | 23       |  
  -----------------------------------

Bug20622 - "Mode" was not optional for DELETE /statements/duplicates

The mode parameter, which is used to indicate which components of each triple must be equivalent to count as duplicates of each other, is now optional and defaults to "spog" for the DELETE /repositories/[name]/statements/duplicates HTTP protocol request.

SPARQL

Rfe11920 - Signal error when aggregate expressions are used in the wrong place

AllegroGraph did not signal a useful error when aggregate expressions were used incorrectly in a SPARQL query. E.g.,

SELECT * {  
  ?a ex:foo ?value .  
  BIND( SUM(?value) AS ?sumValue )  
}

This has been corrected.

Rfe11494 - Update semantics of SPARQL property path queries

AllegroGraph's property path queries now match the updated semantics of the SPARQL 1.1 specification. In particular, wild-card operators like * and + no longer count their matches. AllegroGraph still supports the now deprecated counting operations, e.g.,

{2,4}

but no longer counts the unique paths found. I.e., you can use these non-standard operators to find if there is a path but not to count the number of paths.

Note that AllegroGraph now passes all of the Data Access Working Group (DAWG) tests with the exception of

 (pp37) Nested (*)*

This test uses the path

((:p)*)*

and AllegroGraph incorrectly returns some duplicate results.

Bug21599 - Variable GRAPH queries that used the graph could lose results

A SPARQL GRAPH query that used a variable graph and included the graph variable in the body of the query could under some circumstances lose results. For example, a query like

SELECT * {  
  GRAPH ?g {  
    ?g a <FoodGroup> .  
    ?g <covers> ?r .  
  }  
}

could possibly have returned too few results depending on the data in the store.

This has been corrected.

Bug21581 - Improve bookkeeping when using property paths

If AllegroGraph can determine that a variable in a SPARQL query is not used, then it will enable certain optimizations. For example, if a query is returning DISTINCT results and a variable is not used in any joins nor returned by the query, then its values do not matter.

This logic was not correctly rewriting property paths to account for the unused varaibles.

This has been corrected.

Bug21562 - Insert & remove-graph-uri(s) should not override WITH

If insert-graph-uri(s) or remove-graph-uri(s) were passed as query parameters to a SPARQL DELETE...INSERT...WHERE statement, then AllegroGraph was using these parameters even if the statement used WITH to specify a GRAPH. This is now corrected. SPARQL Update statements that use WITH will not be overridden via insert or remove graph URI(s).

Bug21551 - Deletes via SPARQL Update did not correctly handle encoded UPIs

SPARQL DELETE DATA and DELETE...INSERT...WHERE commands could fail to remove some triples due to AllegroGraph's internal representation of typed literals. The failure would occur only if the literal was specified directly (i.e., when the literal was a variable, AllegroGraph behaved correctly).

Bug21548 - Nested unions with fixed patterns could miss results

If a SPARQL query contained nested unions and these unions contained fixed triple-patterns (i.e., patterns with no variables or patterns where all of the variables are bound via some external binding), then some results from these unions could be lost. This has been corrected.

Bug21546 - Correct off-by-one error in some SPARQL constraints

Depending on the data and the query, some SPARQL constraint analysis could lead to a range error. This has been corrected.

Bug21515 - Improve SPARQL parser speed

Compliance changes in the SPARQL parser of AllegroGraph v4.9 reduced the speed of parsing significantly for very large queries and update requests.

This has been corrected so that parse speed is now equivalent to what it was in AllegroGraph v4.8.

Bug21514 - Fix possible error caused by nested sub-query

The data structures used for SPARQL sub-queries were sometimes being cleaned up too early which could lead to a bus error. This has been corrected.

Bug21507 - Fix problem with property path query clause estimation

AllegroGraph was incorrectly estimating the size of clauses involving property paths in SPARQL queries which could lead to poor plans. This has been corrected.

Bug21478 - Fix spurious parsing error when mixing sub-query and aggregation

AllegroGraph's SPARQL parser could become confused if a query mixed sub-query and aggregation and the sub-query used * to refer to all variables.

This has been corrected.

Bug21469 - Relative URIs were not being correctly normalized by the SPARQL parser

Section 5.2 of RFC3986 specifies how "." and ".." path components of relative URIs should be normalized when merging with a BASE URI. The original SPARQL parser implementation skipped this step in the interests of efficiency. This has now been corrected.

Bug21464 - Some combinations of nested UNION and OPTIONAL could fail

The SPARQL 1.1 engine could fail to correctly keep track of solution bindings for queries with certain patterns of UNIONs and OPTIONALs. This has been corrected. #### Bug21501 - Queries with cross-products and empty RHS could emit incorrect solutions

If a query had a cross-product and the right-hand side of the cross-product was empty, then it was possible for AllegroGraph to not discard the solutions on the left-hand side. This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

Bug21535 - Db registry memory leak in the Lisp client

If a triple store object isn't closed properly and becomes garbage, the finalizer is supposed to close it. Fixed a bug that rendered this mechanism ineffective, causing leakage of memory.

Bug21512 - With-variables not handled consistently with sub-queries

In some cases, external bindings imposed from with-variables could be lost when using sub-queries. This has been corrected.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe11527 - Support sessions that use the main server port

With this change, the Java client can now be configured to have sessions use the main server port rather than a dedicated port. By default, a dedicated port will be used. Setting the system property com.franz.agraph.http.useMainPortForSessions=true will configure the client to use the main port for sessions.

Rfe8384 - Limiting the duration of a query

With this change, the Java client now supports limiting the duration of a query. When Sesame applications use the AGQuery#setMaxQueryTime method, the AG server will timeout the query appropriately during evaluation and clean up server resources, and the client library will throw a QueryInterruptedException back to the client application.

AllegroGraph 4.9

Server Changes

Rfe11922 - Add materiazer. See Materializer for details.

Rfe11879 - Improve error message on invalid license

AllegroGraph now prints information to the console when there are problems with the license settings in the configuration file.

For example, if the configuration file is missing the LicenseExpires parameter, then AllegroGraph will print:

AllegroGraph Server Edition 4.9, built on October 01, 2012 10:04:33 GMT-0700  
Copyright (c) 2005-2012 Franz Inc.  All Rights Reserved.  
current-time: Monday, October 01, 2012 10:43:09 AM  
 
Daemonizing...  
Server started: LicenseExpires parameter missing. Running with triple-limit of 5,000,000.

Rfe11845 - Optimize blank node printing

Blank nodes now print slightly faster and more efficiently than they did before.

Rfe11827 - Reduce memory footprint a bit

Result arrays are now freed right when they become unused. Previously, garbage accumulated until a certain limit was reached. This should have a small positive effect on high water-mark memory footprint.

Rfe11784 - Release memory back to the operating system after query failure

Prior to this change, AllegroGraph would continue to claim memory used for queries even after the query would complete. AllegroGraph will now return unused memory back to the OS when a query memory failure occurs.

Rfe11762 - Improve the efficiency of queries with unbound results

Queries that use UNION or OPTIONAL can have unbound columns in the result set. This change improves the way AllegroGraph does joins when there are unbound columns in a result set.

Rfe11756 - Improve the chunk processing behavior of ORDER BY and LIMIT queries

The chunk-at-a-time processor automatically reduces the size of the chunks being processed based on the limit (it makes no sense to process a chunk of 500,000 rows if you are only interested in the first one!). In the case of ORDER BY queries, however, it makes sense to keep the larger limit since all of the results must be examined. This is now the case.

Rfe11728 - Improve handling of large DISTINCT or ORDERed results

This enhancement improves the efficiency of the on-disk merging operations so that the size of ORDERed or DISTINCT result sets is limited only by disk space and not by available RAM.

Bug21531 - Non-simple EXISTS FILTERs could cause query failure

AllegroGraph was incorrectly processing non-simple graph patterns in an EXISTS or NOT EXISTS FILTER. This meant that queries like

SELECT * {  
   ...  
   FILTER( EXISTS { ... } ||  
           EXISTS { ... FILTER( ... ) } )  
}

could cause an error rather than being correctly processing. This has been corrected.

Bug21415 - Queries could return invalid solutions

It was possible for solutions with no bindings for some of their variables to escape from the right-hand side of a join. This would cause too many solutions to be returned from the query.

This has been corrected.

Bug21410 - Duplicate suppression not handling deleted triples correctly

AllegroGraph was handling deleted triples improperly when duplicate suppression was enabled. This has been corrected.

Bug21407 - An error in a SELECT expression prevented processing of subsequent ones

An error in a SELECT expression would prevent processing of the rest the expressions in the query for the current solution. For example, depending on the data, in a query like:

SELECT  
        ?s  
        ?str  
        (STRAFTER(?str,"b"@cy) AS ?abcy)    # could cause a type error  
        (STRAFTER(?str,"ab") AS ?aab)  
WHERE {  
        ?s :str ?str  
}

The evaluation of STRAFTER() with a second argument of "b"@cy could signal a type-error. AllegroGraph was stopping further computation at that point which meant that the binding for ?aab could be left empty when it should not have been.

This has been corrected: now, each expression is computed separately.

Bug21404 - Config file EOL format

Previously, non-unix newlines (CR+LF, LF+CR) in the license section of the config file could prevent the server from starting up. This has been fixed and any mix of Unix, Mac and DOS style newlines is accepted.

Bug21398 - Variables in EXISTS filter could be incorrectly bound

It was possible for variables that appeared only in an EXISTS (or NOT EXISTS) filter to be incorrectly bound during query evaluation which could lead to incorrect results.

This has been corrected.

Bug21390 - N-Triples serializer does not handle embedded angle brackets

Even though valid URIs are not allowed to contain embedded angle brackets, it is still possible to add them to a triple-store. For example, the valid N-Triples file:

<ex://h> <ex://p> <ex://invalid\u003E>

would add a triple whose object is ex://invalid>.

AllegroGraph was serializing such URIs without performing the necessary \u escaping which meant that the data exported could not be re-read by AllegroGraph's parsers.

This has been corrected.

Note that this was only a problem for data with invalid URIs.

Bug21377 - Updates with both DELETE and INSERT

Due to a race condition in a shared backends, updates that involved both DELETE and INSERT could produce bogus results. This bug has been fixed.

Bug21376 - Wrong results with ORDER BY

Due to a rare race, in shared backends, queries with ORDER BY could produce bogus results. This has been fixed.

Bug21367 - Expressions in ORDER BY could cause query failure

Queries with expressions in an ORDER BY could cause query failure during string look up. This bug was introduced in AllegroGraph v4.8 and is now corrected.

Bug21349 - Eager evaluation of boolean FILTERS could lead to failures

If a query used FILTERS consisting of boolean combinations of simpler filters and also used external variable bindings such that some of the FILTER clauses could be evaluated immediately, then it was possible for the FILTER evaluator to signal an error.

This has been corrected.

Note that bug21342 corrected this problem for the simpler case of negation. This bug fixes the more general case.

Bug21345 - Possible problem with BIND inside of an OPTIONAL

If a query included a BIND clause that stood by itself and there was a join, it was possible to lose the value of the binding. In particular, a query like

SELECT * { ?a ?b ?c . OPTIONAL { bind('hello' as ?x } }

could leave ?x unbound. This has been corrected.

Bug21344 - Bind could fail to use external variable bindings

If a query used the BIND form and the value of the expression depended on variables set via an external binding (e.g., via the with-variables argument in the Lisp API or via HTTP parameters in the HTTP interface), then it was possible for the BIND expression to ignore those values. This has been corrected.

Bug21342 - Eager evaluation in FILTERs can lead to problems using variables

If a query uses a FILTER and also variable bindings such that an expression in the FILTER could be evaluated before actually running the query and if the expression was used in other boolean combinations (e.g., with not, and, or or), then it was possible for AllegroGraph to signal an error. For example, if the query was

select * { ?a ?b ?c . FILTER( ! bound( ?d )) }

and if ?d was bound using a variable binding, then AllegroGraph would signal an error instead of correctly realizing that the FILTER could never succeed.

This has been corrected.

Bug21341 - Non-simple IF statements could cause an error

It was possible for non-simple IF statements in an expression to lose track of their variables which led to an error at query planning time.

This has been corrected.

Bug21326 - Geospatial query failure

Fixed a bug that, under rare circumstances, caused geospatial queries to fail with bus error.

Bug21325 - Lost results in sub-queries and some joins

When an internal buffer holding intermediate results was grown in a sub-query or in some joins, results could have been lost. This bug has been fixed.

Bug21272 - No results with BIND

Sometimes, as in the query below, instead of no results, a single result was returned when BIND was involved.

select * {  
  bind('o' as ?c)  
  ?a ?b ?c .  
  ?d ?e ?f .  
  filter( ?a < ?d && ?d < ?a )  
}

This bug has been fixed.

Bug21270 - Sub-queries with ORDER BY can fail to bind results

Depending on the data and what other queries have been evaluated, a query that used an ORDER BY clause in a sub-query could fail to correctly bind the result variables. For example, a query like:

select ?age {  
  { select ?age {  
        ?who <http://people.example/age> ?age .  
    } order by ?age  
  }  
}

could return results but leave the bindings for ?age unbound.

This has been corrected.

Bug21260 - XPath Constructor Functions fail on strings to numbers

The XPath Constructor Functions should convert typed string literals into the requested numeric type when the value of the string can be so coerced. For example:

xs:int('2'^^xsd:string) ==> 2

AllegroGraph was raising a type error in this case. This has been corrected.

Bug21256 - Parser problem with some queries that mixed aggregation and sub-query

AllegroGraph was incorrectly signaling a parser error on some queries that mixed aggregation and sub-query. For example,

select (max(?a) as ?foo) {  
  select distinct ?a ?b {  
    ?a a ?c .  
    ?c a ?b .  
  }  
}

This has been corrected.

Bug21245 - Allegrograph's turtle parser performs URI normalization incorrectly

Percent encoded URIs should not be unescaped when loading data in the Turtle format. AllegroGraph was incorrectly doing so. This has been corrected.

Bug21243 - Non-QNameable URIs cause turtle serializer error

If a triple contained a URI that could not be expressed as a QName, then the turtle serializer would signal an error that it could not split the URI rather than just emitting the entire URI and continuing. For example, a triple containing the URI <<ttp://example/> would cause this error:

Cannot split <http://example/> into XML QName.

This has been corrected.

Bug21226 - agload max blank node cache size not obeyed in all cases

Agload could over-allocate memory in a certain case. This has been fixed.

Bug21215 - Improve cursor resource usage of Prolog SELECT queries

AllegroGraph previously closed some of the cursors created while running a Prolog query lazily. It now closes them when the query completes which improves overall resource utilization.

Bug20906 - Remove duplicate triples fails on a store with zero triples

An error would be raised if duplicate deletion was attempted on a store with zero triples.

This has been corrected.

HTTP Client

Rfe11755 - Add support for insert-graph-uri, remove-graph-uri, etc.

Version 6 of the Sesame HTTP Protocol includes several graph parameters that can be used to control the behavior of SPARQL Update statements. These are:

remove-graph-uri - the graph from which triples are deleted in a DELETE/INSERT/WHERE clause.
insert-graph-uri - the graph into which triples are added in a DELETE/INSERT/WHERE clause
using-graph-uri and using-named-graph-uri - these specify the dataset (similar to using-graph-uri and named-graph-uri in the HTTP protocol and to the from and from-named keyword arguments in the Lisp API's run-sparql command.

These four parameters are now supported by AllegroGraph

Rfe11743 - Add baseURI to the SPARQL HTTP Protocol

The SPARQL HTTP protocol now accepts baseURI as an HTTP query request parameter. This can be used to shorten the rest of a SPARQL query.

Bug21408 - Retrieving JSON statements over HTTP

Fixed a bug that caused the statement list to be truncated when an 'application/json' encoded response was requested via the HTTP protocol.

Bug21280 - Query failures cause spurious errors

AllegroGraph 4.8 modified the internal API used to signal query failures (e.g., due to timeout or memory limits). Unfortunately, the code in the HTTP layer of AllegroGraph was not modified to accommodate the change which led to spurious bus errors when a query failed. This has been corrected.

SPARQL

Rfe11862 - Match default dataset behavior for SPARQL Update and Query

SPARQL Update was using a default dataset that contained only triples with unnamed graphs (the default graph in AllegroGraph terminology) in the FROM portion and put triples with graphs into each of their graphs in the FROM NAMED. This was different from the default dataset for SPARQL Query which put all of the triples into the FROM portion and treated the FROM NAMED portion similarly.

AllegroGraph now uses the same default dataset behavior for both Query and Update. This dataset is as described for query above. Note that this means that an Update command like

DELETE DATA { <eh://a> <eh://b> <eh://c> }

will delete the triples

{ <eh://a> <eh://b> <eh://c> }

from every graph in which it appears whereas the previous behavior would have been to delete it only from the default graph.

Note that triples added during an Update command which do not specify a graph will be added to the triple-store's default graph.

Rfe11817 - SPARQL parser should signal an error on badly formed requests

Previously, the SPARQL parser would silently accept nonsense queries like:

<http://www.foo.com> <http://www.bar.com> 'banana' .

because it would silently ignore tokens that came before the SPARQL prologue (i.e., before PREFIX and BASE) and convert such a query into an empty UPDATE request (i.e., a non-operation).

These queries will now signal a SPARQL parse error. Because the parser is now more strict, some queries that previously parsed may now signal an error. For example, this invalid query was previously accepted and now will not be:

# WRONG: bad prefix because it ends with a period (".")  
PREFIX ex: <http://example.com> .  
SELECT * { ?s ex:testsFor ?o . }

Rfe11793 - Improve handling of numeric and date literal equality filters

Improve the SPARQL 1.1 query planner's handling of numeric and date literal equality filters. In particular, AllegroGraph will now optimize queries with FILTER clauses like:

?s ex:p1 ?o .  
FILTER( ?o = 3 )

which would previously have required a scan of all triples with predicate ex:p1.

Rfe11787 - Support SPARQL 1.1 UUID and STRUUID functions

Adds support for the SPARQL 1.1 UUID() and STRUUID() functions.

Rfe11779 - Support SPARQL 1.1 change to inline data syntax

SPARQL 1.1 recently switched from a syntax for inline data that used BINDINGS to one that uses VALUES. The standard also made it possible to use inline data in more places. AllegroGraph now fully supports this feature and passes all of the Data Access Working Group (DAWG) tests for it.

Note that the BINDINGS syntax is still supported in this version of AllegroGraph. It will be deprecated and removed in some future release.

Rfe11757 - Provide select-N strategy for ordered results with small limit

Typically, a SPARQL query with an ORDER BY clause must accumulate all of the results before sorting them. If, however, the query has a small LIMIT, then it can be more efficient to retain the top LIMIT results and only sort those.

Rfe11555 - Manage memory used to ensure unique results from CONSTRUCT and UPDATE

The data structures AllegroGraph uses to ensure that results of a SPARQL CONSTRUCT or UPDATE are unique have been modified so that result sets can be processed correctly regardless of size (up to the amount of available disk space).

Bug21401 - SPARQL Update must intern strings from variable bindings

If variable bindings are used in a SPARQL Update query and these bindings are part of the template being used to insert new triples into the store, then the strings from the variable values must be added to the triple-store's string table. This was not being done which could lead to a UPI not in string table error.

This has been corrected.

Bug21391 - Queries against in-memory-triple-stores could return incorrect results

The Lisp SPARQL API allows for the creation of in-memory-triple-stores from CONSTRUCT and DESCRIBE queries. These suffered from a bug such that queries that specified more than two of subject, predicate, object and graph fields could return incorrect triples. As examples, a query like:

ASK { ex:foo ex:bar ?o }

would work correctly whereas a query like

ASK { ex:foo ex:bar ex:baz }

could return yes when it should have returned no.

This has been corrected.

Bug21372 - Possible error when requesting `sparql-xml` output for Prolog `select`

An error could be signaled if the default query engine was sparql-1.0 and a results format of sparql-xml was requested for a Prolog select query.

This has been corrected.

Bug21371 - IF confusion with free variables

SPARQL queries with free variables could produce incorrect results when using IF. For example, the following query

SELECT ?result {  
  BIND( IF(!bound(?dataType), "first", "second") as ?result)  
}

would return second instead of first. This occurred because the query engine was not correctly analyzing the free variable ?dataType (this variable is free because the query provides not binding for it).

This has been corrected.

Bug21361 - Improve SPARQL BASE URI handling

AllegroGraph was not always correctly merging base and relative URIs. For example, a query like

BASE <http://localhost:8080/>  
SELECT * { ?a ?b </> }

was being incorrectly run as if it was

SELECT * { ?a ?b <http://localhost:8080//> }

This and other similar issues have been corrected.

Bug21357 - SPARQL blank-node parsing was too strict

The SPARQL parser was not allowing the full range of blank node names to be used. Thus, queries like:

SELECT * {  
  ?a ?b _:123 .  
  _:123 ?c _:fv-0 .  
}

would fail to parse correctly. This has been corrected.

Bug21354 - Update STRAFTER() and STRBEFORE() to match changes in the standard

Recent changes to the SPARQL 1.1 standard changed the output of STRAFTER() and STRBEFORE() when no match is found. Previously, they were specified to return an empty literal with language or datatype that matched the first argument. Now they should produce an empty literal with neither language nor datatype. AllegroGraph now matches the standard.

Bug21294 - Sparql XML results for ASK queries missing `head` clause

The SPARQL XML generated by AllegroGraph was missing the <head/> clause. Since this clause is generally empty, only systems that validate the results against an external DTD were effected by the missing clause.

This has been corrected.

Bug21277 - Improve cursor resource usage of SPARQL 1.1 queries

AllegroGraph previously closed some of the cursors created while running SPARQL 1.1 queries lazily. It now closes them when the query completes which improves overall resource utilization.

Bug21266 - Queries with only a geospatial clause could fail to gather solutions

If a SPARQL query contained only a geospatial clause, then the SPARQL 1.1 engine could fail to add the solutions it found to the answer set. This has been corrected.

Bug21249 - Incorrect return value of COPY GRAPH <X> TO GRAPH <X>

A SPARQL Update statement like

COPY DEFAULT TO DEFAULT

should have no effect on the triple-store and should succeed. AllegroGraph implemented the behavior correctly but was returning failure instead of success. This has been corrected.

Bug21239 - Concurrent queries in shared backends

With the sparql-1.1 engine, queries running in the same shared backend concurrently had unexpected interactions that lead to various errors. This has been fixed.

Bug21237 - Computations in aggregate expressions could fail

If a complex expression was used inside of a SPARQL aggregation function, the value could be left unbound. A work-around was to express

(min(2 * ?x) as ?smallestDoubled)

(min(?x) as ?smallest) (2*?x as ?smallestDoubled)

This is now corrected and the work-around is no longer necessary.

Bug21233 - SPIN functions could fail when used with constants

SPARQL queries that used SPIN functions could fail if the arguments to the functions were constants not already in a triple-store's string-table. This has been corrected.

Bug21222 - Failing SPARQL UPDATE HTTP request should error

Previously SPARQL UPDATE requests that failed would return false in SPARQL/XML format. In order to be better conform with other SPARQL endpoints, AllegroGraph now returns a 412 status code (precondition failed) instead.

Bug21163 - Add new default-dataset-behavior: rdf

SPARQL leaves things up to the implementation when no explicit dataset is specified (i.e., when no FROM or FROM NAMED clauses are part of the query). AllegroGraph now provides three choices for dataset behavior:

:all - All triples will be in the FROM portion of the dataset. Triples with a non-default graph will be in that graph. Note that this means that triples with non-default graph slots will appear in the dataset twice: once in a named graph (named by their graph slot) and again in the default graph of the dataset.
:default - Only triples whose graph is the default graph of the triple-store will be in the default graph of the dataset. No triples will be in the named graph portion of the dataset.
:rdf - Only triples whose graph is the default graph of the triple-store will be in the default graph of the dataset. Triples with a non-default graph slots will be in the named graph portion of the dataset (each non-default graph triple will be in a graph named by the graph of that triple).

AllegroGraph's standard behavior continues to be :all. You can change this on a per-query basis using the defaultDatasetBehavior query option:

PREFIX franzOption_defaultDatasetBehavior: <franz:rdf>

AGWebView

Rfe11910 - Add HTTP handlers for materialization

Added the materializeEntailed service.

A PUT request will materialize entailed triples based on the with, without, useTypeSubproperty, and commit query parameters.

A DELETE request will remove the materialized triples.

Rfe11856 - WebView - new UI Beta for Graph

In WebView next to the Documentation menu is a link to "WebView Beta" a new experimental UI for Query and Graph. A link at the top will returns to the classic WebView.

A link at the top will open the new Graph View UI.

Also Node queries have a View menu in the results to open the node in Graph view.

Administration features of WebView are not yet included within the new UI.

Rfe11676 - Webview: in Scripts editor, use CodeMirror JavaScript

The WebView Scripts editor now supports syntax highlighting and indentation by including the CodeMirror JavaScript mode.

rfe11596 - WebView: new UI Beta for Query

In WebView next to the Documentation menu is a link to WebView Beta a new experimental User Interface (UI) for Query. The new UI provides a modern style and easier navigation. A link at the top will returns to the classic WebView.

All existing features of WebView Query are implemented:

select repo
query sparql
query prolog
free text search (pattern or expression)
namespaces
save and delete any of the above to user or repo areas
recent queries
example queries
click node in results goes to new query screen
download query results
sessions (and functors)

Administration features of WebView are not yet included within the new UI.

Rfe11582 - Webview: upgrade Query Editor to CodeMirror v2.32

The query editor for SPARQL and Prolog, implemented by CodeMirror, has been upgraded from v2.24 to v2.32.

Bug21250 - Webview: remove limit from download

The download link for Query results no longer uses the limit parameter in the query UI.

Changes to the Lisp API

Rfe11454 - `get-triples-list` (Lisp API) returns cursor only on request

The Lisp get-triples-list function previously returned the cursor it created as a second value. Now, however, it returns a cursor if either one was passed in initially (using the cursor keyword argument) or if the new keyword argument return-cursor is true.

Bug21381 - (Lisp API) SPARQL introspection functions required engine argument

The functions get-allowed-results-formats and get-allowed-rdf-formats returned nil if no query engine was provided. This has been corrected.

Bug21378 - (Lisp API) sparql-1.1 engine does not implement destination-db

The Lisp API allows SPARQL Update commands to specify the triple-store that they modify. This feature was not included in the sparql-1.1 engine. This has been corrected.

Documentation

No significant changes.

Python Client

Rfe11911 - Add materializer calls to python client

Added materializeEntailed and deleteMaterialized calls to RepositoryConnection.

Java Client

Rfe11906 - Java support for the materializer

With this change, the Java client now supports materialization, allowing inferences to be generated and added to a repository in a transactional fashion. See the Javadoc for AGMaterializer#newInstance() to acquire a configurable materializer, and AGRepositoryConnection#materialize to run the materializer.

Rfe11771 - Upgrade embedded JavaScript implementation

Fixes several small bugs in the way uploaded JavaScript scripts are interpreted. The most visible one was that the !== operator was acting like the < operator.

Rfe11241 - Support more query/update protocol params

With this change, the Java client now sends HTTP protocol params for update, baseURI, timeout, remove-graph-uri, insert-graph-uri, using-graph-uri, using-named-graph-uri, and logQuery.

AllegroGraph 4.8

Server Changes

Rfe11711 - Improve cursor resource usage of Geospatial Prolog queries

AllegroGraph previously closed cursors created while running a Prolog geospatial query lazily. It now closes them when the query completes which improves overall resource utilization.

Rfe11653 - Rm requests can be processed out of order

The order in which file management requests were processed caused unexpected errors under heavy load. The bug is now fixed.

Rfe11649 - Manage memory for ORDER BY and DISTINCT queries

Previously AllegroGraph needed to accumulate all results in memory for queries that used DISTINCT or ORDER BY. This could greatly increase the overall memory footprint. AllegroGraph now handles large result sets in available memory.

Rfe11631 - Improve query optimizer when inference is enabled

One part of query optimization is reordering the triple-patterns in each BGP. This reordering is based on the statistics collected by the store and these statistics do not take inference into account. This meant that the plan produced could be quite bad.

This has been corrected by modifying the estimation code to take RDFS++ into account.

Rfe11416 - Provide greater control of query memory usage

Added a memoryLimit query option that controls how much memory a query can use. If a query needs to allocate more memory than this value, it will fail.

Bug21235 - Resource manager sync-files effectively ignores msync errors

If the filesystem on which a database resided filled up during a checkpoint, this fact was logged but the checkpoint process would continue running as if the error didn't happen, which would result in database corruption. This bug has been fixed. Now when a full filesystem is detected during a checkpoint, AllegroGraph stops operating in order to protect committed data.

Bug21224 - Crashes while processing queries with DISTINCT

Under rare circumstances, the internal data structures used while processing DISTINCT queries could get corrupted leading to segmentation faults. This bug has been fixed.

Bug21214 - Correct scope of BINDings introduced on the LHS of union

The binding of a variable on the left-hand side of a UNION should not influence its value on the right-hand side. For example, the following query should produce two rows with ?a bound to 1 in one of them and left unbound in the other.

select ?a {  
  { bind(1 as ?a) } union { bind(2 as ?b) }  
}

AllegroGraph was retaining the binding on the LHS across to the RHS of the union and so was answering with two rows each of which had ?a bound to 1. This has been corrected.

Bug21187 - Stack exhaustion with shared cursor

With lots of deleted triples a query could run out of stack when creating cursors. This has been fixed.

Bug21182 - Unbound values in distinct aggregrates could lose results

If a variable binding used in a distinct aggregate was unbound, it could cause other aggregrates in the same row to fail to update properly. For example, a query like:

SELECT (count( distinct ?taste) as ?tasteCount ) (count(?who) as ?whoCount) {  
	   ?who <http://people.example/age> ?age .  
	   OPTIONAL { ?who <http://people.example/taste> ?taste . }  
}

could lose data for the ?whoCount bindings if the ?taste binding from the OPTIONAL was unbound. This has been corrected.

Bug21145 - agload: message corrected for insufficient buffer size

The error message for insufficient ntriples/nquads buffer size was reporting buffer as size -1. This message has been corrected and now reports actual buffer size in order to abate confusion about the condition.

Note: This condition is usually triggered when an ntriple of greater than 4MB is encountered while using the default dispatch strategy.

Bug21136 - Free BGP patterns can lead to very large memory allocation requests

Queries with free patterns (like ?s ?p ?o) in a BGP on the right-hand side of a query could lead the query size estimator to believe that it should reserve a very large space to save the putative result set. For example, a query like

SELECT ?s WHERE {  
    { ?s ?p ?o }  
    MINUS  
    { ?other ?p ?o . ?other ?someProp ?s }  
}

would lead the query engine to try and allocate 1024-Petabytes. This has been corrected.

Bug21124 - Zero length path between non-matching constants

AllegroGraph was incorrectly finding a path between two different constants which led to queries like the following to return a single (empty) result when they should have returned no results at all.

select * {  
  ex:one ex:connectsTo{0} ex:two .  
}

This has been corrected. Other zero-length path queries were behaving correctly.

Bug20853 - Updated the soundex algorithm

AllegroGraph's freetext indexing word filter was using an older variant of the Soundex algorithm that did not match the latest version (cf. http://www.archives.gov/research/census/soundex.html). The algorithm has now been updated.

Freetext indices created with the soundex word-filter should be dropped and re-built.

HTTP Client

Rfe11681 - HTTP: Allow statements queries using the POST method

Add a [store]/statements/query URL that corresponds to /statements, except that it isn't limited to the GET method, and thus can be used to do complex queries without building up overlong URLs.

Rfe11680 - HTTP: Add /unique/[column] service

Add a [store]/unique/[column] URL that allows clients to fetch the set of unique parts that occur in a column in the database, optionally restricting other columns.

SPARQL

Rfe11682 - Improve performance of SPARQL OPTIONAL queries

If a query has the form A OPTIONAL B (where A and B share variables so that it is a join) and the A part has no rows, then it is not necessary to evaluate the B part at all.

The SPARQL 1.1 query engine now recognizes this situation and skips the evaluation of the right-hand side.

Rfe11591 - Modified the Lisp run-sparql API to better handle query failure (e.g., for when a query runs out of time or other system resources).

Previously run-sparql would return (as multiple values):

the results of the query (or nil if the results were streamed)
the verb of the query (e.g., :select or :construct)
the list of variables

Now run-sparql returns four values. In normal execution, the first three values are the same and the fourth value will be nil. When a query fails, the results will be:

a keyword indicating why the query failed (e.g., :timeout)
the verb
the variables
a property list of additional information (the properties will vary depending on the failure).

Code that requests a cursor should be prepared to handle a condition of type db.agraph.query.base:cancel-query-condition.

Rfe11581 - handle repeated Prologues in SPARQL UPDATE queries

SPARQL 1.1 UPDATE allows the Prologue (BASE and PREFIX definitions) to be repeated between each UPDATE command. Previously, AllegroGraph only allowed BASE and PREFIX to be used at the beginning of a SPARQL statement. This is now corrected.

Rfe11559 - Add logQuery parameter to the HTTP SPARQL protocol

SPARQL queries can now use the logQuery parameter in their HTTP requests. The parameter can be set to one of:

false - no logging is reported
true - log information is sent back in the body of the response (using text/plain).
log - log information is added to the AllegroGraph log file.

The log output can be cryptic; please contact Franz Inc. for support in interpreting the messages.

Bug21220: SPARQL Update LOAD should require only write permission (not eval)

A user would need to have evaluate permission on a triple-store in order to use SPARQL Update's LOAD command to load data from a file. This has been changed so that having write permission is sufficient.

Bug21203 - SPARQL Update insert data fails to treat blank nodes correctly

The INSERT DATA and MODIFY SPARQL Update commands failed to use the small blank node identifier in the INSERT template. This caused multiple blank nodes to be created when there should have been only one.

Bug21192 - Upi not in string table error when mixing BIND, SPARQL Update and FTI

SPARQL Update queries that produced new strings (e.g., using BIND) on a triple-store with free-text indexing turned would lead to UPI not in string table errors. Note that in this case a triple would have been added to the store but its strings would not be added to the string table which could cause subsequent queries to fail until the errant triple was removed.

This has been corrected.

Bug21168 - Support SPARQL 1.1 COUNT(DISTINCT *)

The SPARQL 1.1 query engine was not correctly handling the COUNT(DISTINCT *) form. This has been corrected.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Documentation

No significant changes.

Python Client

Rfe11697 - Added RepositoryConnection.prepareUpdate

Added RepositoryConnection.prepareUpdate for SPARQL 1.1 Update requests. The call to evaluate on the returned UpdateQuery object returns True or False.

Java Client

Rfe11748 - Upgrade to sesame 2.6.8

With this change, the Java client has been upgraded to use openrdf sesame 2.6.8 jars.

Rfe11243 - Support X.509 authentication

With this change, the Java client now supports X.509 server and client authentication. For set up details, see the javadoc for AGServer.

Bug21225 - With no requestEntity, post using setRequestBody

Previously, AGHTTPClient post method used setQueryString to pass params when there was no requestEntity provided. This burdened the server when a param became very large. Now params are passed using setRequestBody when no requestEntity has been provided.

AllegroGraph 4.7

Server Changes

Rfe11587 - Change InstanceTimeout defaults

The configure-agraph script now supports instance timeouts in interactive mode and via the --instance-timeout command line option. It defaults to one week.

Also, the default value the InstanceTimeout configuration directive (effective when there is no global or catalog level InstanceTimeout) has been changed from 0 (= disabled) to one hour.

Rfe11536 - Transitive inference queries could be slow

The RDFS++ reasoning for transitive queries could be slower than necessary for some datasets. The code has been reimplemented to be significantly faster.

Rfe11285 - Improve query constraint analysis

AllegroGraph now does a better job of applying filter constraints throughout a query. For example, previous versions of AllegroGraph would not have transformed either of the two BGPs (on either side of UNION) into range queries (which are more efficient).

SELECT * {  
  ?a ex:d ?c .  
  { ?a ex:b ?c . } UNION { ?a ex:c ?c . }  
  FILTER (?c > 2)  
}

Now it will.

Bug21114 - Instances lingering too long

Instances with non-zero InstanceTimeout only timed out when there was activity involving the service daemon. This has been corrected. Instances with an InstanceTimeout of zero (the default) are not affected.

Bug21094 - Queries against federations could signal UPI-not-in-string-table errors

It was possible for queries against federated triple-stores to lead to UPI-not-in-string-table errors. This has been corrected.

Bug21089 - agload division by 0

Progress reporting failed with division by 0 if the operation completed within 1ms.

Bug21077 - Expressions can be analyzed too eagerly

It was possible for constant expressions to be analyzed to their final values too quickly which could lead to an error if the expressions were used in a BIND. For example, this query would fail:

select * { bind(str('hello') as ?x }

rather than producing a single binding of ?x to the string hello. This has been corrected.

Bug21069 - A warm-standby client now exits non-zero if error occurs during commit replay

Replication secondary (warm-standby client) now exits non-zero if error occurs during commit replay. This is to ensure transactional behavior in the replica.

Bug21059 - Improve efficiency of all predicates cache when RDFS++ is enabled

The query engine keeps track of the predicates in the triple-store to optimize queries which contain clauses where the predicate position is a variable. If RDFS++ was enabled and the cache was not already fresh, then AllegroGraph would try to find all of the store's predicates with reasoning enabled. Since new predicates cannot be inferred, this could be hugely inefficient. This has been corrected.

Bug21057 - Fix log rotation causing crashes

When the SIGHUP handler closes a log stream, attempts to log could fail with a 'stream closed' error potentially crashing the server. This bug has been fixed.

Bug21040 - Turtle data import should use UTF-8 encoding

The Turtle specification mandates that Turtle files are encoded using UTF-8. AllegroGraph was not enforcing this in all cases which could mangle Unicode data on import. This has been corrected.

Bug21022 - Group expressions with unbound elements could be dropped

AllegroGraph could lose rows in a aggregation query if the GROUP BY expression resulted in unbound columns. This has been corrected.

Bug21008 - Correct XSD datatype for certain division operations

When performing division of two xsd:integers or two xsd:decimals, AllegroGraph was emitting the result as an xsd:float rather than an xsd:decimal. This has been corrected.

Bug21007 - Error creating a session with non-default graph for inferred triples

The syntax for starting a reasoning session that uses a different graph for its inferred triples is

<store>[rdfs++#<graph>]

AllegroGraph was failing to parse this specification correctly and signaled an error instead. This has now been corrected.

HTTP Client

Rfe11601 - HTTP and WebView: MongoDB interface config

Added resources to HTTP protocol for MongoDB parameters:

GET /repositories/[name]/mongoParameters  
POST /repositories/[name]/mongoParameters

See Mongo Interface, The HTTP Protocol, GET mongoParameters, and POST mongoParameters.

SPARQL

Rfe11558 - Add query logging support

To add in query debugging and tuning, SPARQL queries using the 1.1 query engine can now be logged to the AllegroGraph log file. To log a query, use the prefix option notation by pre-pending this prefix to your query:

prefix franzOption_logQuery: <franz:yes>

The log output will go directly to the AllegroGraph log. In this release, the output is relatively cryptic. Please consult with Franz ([email protected]) to interpret the data.

Rfe11492 - Add support for SPARQL CSV and TSV

Added CSV (comma separated values) and TSV (tab separated values) as native SPARQL output formats. Also corrected problems with using CSV output on result sets with unbound values.

Rfe11486, Rfe11538 - improve SPARQL 1.1 engine geospatial integration

The SPARQL 1.1 query engine did not implement the POINT extension for SPARQL FILTER functions so that queries like:

SELECT * {  
   ...  
} ORDER BY geo:haversine-km(?loc,POINT(105.842328, 21.0184040))

would fail. This has been corrected. Additionally, the geospatial query analyzer has been improved so that more computations are carried out at query analysis time rather than at query execution time. This can speed up some geospatial queries tremendously.

Bug21117 - Some SPARQL 1.1 path queries could parse incorrectly

The SPARQL 1.1 parser could combine some combinations of alternation and sequences incorrectly. For example, this query

SELECT ?xxx ?yyy {  
    ?xxx (dc:aaa|dc:bbb/dc:ccc) ?yyy .  
}

would have been parsed as if it read (dc:aaa|dc:bbb|dc:ccc). This has been corrected.

Bug21093 - SPARQL UPDATEs could fail to save newly created strings

If run with chunkProcessingAllowed, a SPARQL UPDATE query that generated new strings could fail to save some of these new strings in the underlying triple-store. This has been corrected.

Bug21082 - Optional cross-product with no bindings could fail

If the final step of a query was a LEFT-JOIN and that LEFT-JOIN was also a cross-product (i.e., the right hand side shared no variables with the left hand side) and the right hand side produced no bindings of its own (e.g., it was all constant or its variables could be elided because they were not used anywhere else in the query), then it was possible for AllegroGraph to incorrectly remove answers. This has been corrected.

Bug21078, bug21083 - Handle sub-query SELECT expresions and filters correctly

SELECT expressions in sub-queries were not being handled correctly by AllegroGraph. This has been corrected so that queries like

SELECT ?a (?a * 2 as ?b) ?c ?d {  
  BIND(1 as ?a)  
  { SELECT ?c (2* ?c as ?d) {  
    BIND(20 as ?c)  
  } }  
}

now perform properly.

It was also possible for AllegroGraph to incorrectly optimize FILTER expressions in sub-queries that returned all variables (*). This too has been corrected.

Bug21072 - unused variable check could remove variables in EXISTS filters

The SPARQL 1.1 query planner removes variables that are not used in the query from the set of bindings maintained during query execution. Unfortunately, the check to see whether or not a variable was used could fail to properly descend into EXISTS (and NOT EXISTS) filters. For example, the following query would have failed:

PREFIX :    <http://example/>  
PREFIX  rdf:    <http://www.w3.org/1999/02/22-rdf-syntax-ns#>  
SELECT DISTINCT (?s1 AS ?subset) (?s2 AS ?superset)  
WHERE  
{  
        ?s1 rdf:type :Set .  
        ?s2 rdf:type :Set .  
        ?s1 :member ?x .  
        FILTER ( ?s1 = ?s2 || NOT EXISTS { ?s2 :member ?x . } )  
}

This has been corrected.

Bug21064 - Unneeded graph variables with DISTINCT query could error

If the variable in a GRAPH clause was not needed by the query (because, e.g., the query asks for DISTINCT results and the graph variable is not used anywhere else in the query), then the query planner will remove the variable from the plan. Unfortunately, the query executor could still try to assign a value to the graph variable which would lead to an error at query execution time. This has been corrected.

Bug21046 - Multiple MINUS clauses could lead to query execution error

A SPARQL query with multiple MINUS clauses could cause the SPARQL 1.1 query engine to incorrectly determine the variables on which to join the clauses. For example, this query would fail:

SELECT (?s1 AS ?subset) (?s2 AS ?superset) {  
    # All pairs of sets  
    ?s2 rdf:type :Set .  
    ?s1 rdf:type :Set .  
 
    MINUS {  
        ?s1 rdf:type :Set .  
        ?s2 rdf:type :Set .  
        # Assumes ?s1 has at least one member  
        ?s1 :member ?x .  
        # If we want to exclude A as a subset of A.  
        # This is not perfect as "?s1 = ?s2" is not a  
        # contents based comparison.  
        FILTER ( ?s1 = ?s2 || NOT EXISTS { ?s2 :member ?x . } )  
    }  
    MINUS {  
        # If we don't want the empty set being a subset of itself.  
        ?s1 rdf:type :Set .  
        ?s2 rdf:type :Set .  
        # Choose the pair (empty set, empty set)  
        FILTER ( NOT EXISTS { ?s1 :member ?y . } )  
        FILTER ( NOT EXISTS { ?s2 :member ?y . } )  
    }  
}

This has been corrected.

Bug21037 - Construct queries with Turtle results format can cause string lookup error

If a SPARQL CONSTRUCT query used expressions to create strings that were not already in the underlying triple-store and if the RDF format requested with Turtle, then a UPI not in string table error could be generated. This has been corrected.

Note that the other RDF formats do not exhibit this behavior.

Bug21031 - Remove unstable optimization

The SPARQL 1.1 query engine attempted to reduce the amount of work needed to evaluate certain joins. Unfortunately, it was possible to create queries that exposed a conflict between this optimization and other optimizations which would lead to an error at query evaluation time. This optimization has been temporarily removed. It will be added again once the conflicts it exposed have been resolved.

Bug21023 - Correct datatype of the seconds() function

The SPARQL 1.1 seconds() function should return a value of type xsd:decimal but AllegroGraph was returning xsd:float. This has been corrected.

The date and time functions (like month() and seconds()) were also not behaving correctly on literals of type xsd:time. This has been corrected.

Bug21019 - Filter EXISTS with fixed triple patterns could fail

If a SPARQL query used FILTER (NOT) EXISTS and the EXISTS clause contained fixed triple-patterns, then AllegroGraph could signal an error. For example, this query would fail:

SELECT * {  
  ?a a ex:Fruit .  
  FILTER EXISTS { ex:apple a ex:Plant . }  
}

This has been corrected.

Bug21016 - UPDATE with blank nodes could lead to error

A SPARQL UPDATE query that included blank nodes in the INSERT template could lead to an error. This has been corrected.

Bug20994 - Correct (in)equality test between plain and typed literals

Some comparisons between plain literals and typed literals could return false rather than signaling a SPARQL type error. This meant that a FILTER that tested for (in)equality could succeed where it should have failed. This has been corrected.

Bug20840 - Improve comparisons between xsd:dates and xsd:dateTimes

The SPARQL operator mapping table is quite strict in the sort of comparisons that are allowed. AllegroGraph was allowing inequality comparisons between xsd:dates and xsd:dateTimes even though these datatypes are not strictly comparable. These sort of comparisons will now signal a type error leading FILTER conditions to fail. I.e., a query like

... FILTER ( ?aDate > ?aDateTime ) ...

will now fail (correctly) even though it may have resulted in results previously.

Bug20736 - Federated-triple-stores can fail to find equivalent literals

In SPARQL filters, typed-literal equality is based on the value of the literal, not its lexical form. I.e., these literals are all the same:

'4'^^xsd:byte
'4'^^xsd:long
'4'^^xsd:unsignedByte

To reduce the amount of work during query execution, the SPARQL engine checks to see whether any triples exist with literals of these (encoded) types. This check was being handled incorrectly for federated triple-stores which could lead to answers being left out of federated result sets. This has now been corrected.

Bug20770, Bug21020 - improve SPARQL 1.1 aggregation support

SPARQL 1.1. allows assignment in a GROUP BY clause. For example:

... GROUP BY (str(?name) as  ?theName) ...

AllegroGraph did not allow for this syntax and required the work-around of:

... BIND(str(?name) as ?theName) ... } GROUP BY ?theName ...

This omission has been filled.

Secondly, AllegroGraph did not allow aggregate expressions in the HAVING clause or SELECT expression involving more than one aggregate. For example, these queries would fail:

SELECT (avg(?value) as ?avg) ((min(?value) + max(?value)) / 2 as ?half {  
  ...  
}

and

SELECT * {  
  ...  
}  
group by ?value  
having(sum(?value) > 20)

Both of these omissions have also been corrected.

Bug20165 - Unnecessary variable in a BIND clause could cause error

If the variable in a BIND clause was not needed by the query (because, e.g., the query asks for DISTINCT results and the variable is not used anywhere else), then the query planner will remove the variable from the plan. Unfortunately, the query executor could still try to assign a value to the variable which would lead to an error at query execution time. This has been corrected.

AGWebView

Rfe11601 - MongoDB interface config

Added support in WebView for setting MongoDB parameters, in the Repository Overview page. See Mongo Interface.

Rfe11557 - WebView: serve doc files

AllegroGraph documentation, a collection of HTML files installed with the server, is now served by WebView, and the Documentation menu in WebView links to these local files in addition to those on franz.com.

Rfe11474 - Execute query without changing text

In the Query editor, a query may now be executed without editing the text.

Rfe11398 - Update query editor to CodeMirror v2.24

The query editor for SPARQL and Prolog, implemented by CodeMirror, has been upgraded from v2.15 to v2.24, including javascript and css.

Bug21085 - Webview: Recent queries should not add items redundantly

WebView Recent Queries now keeps the most recently used queries. If a previous query (including the same planner and limit) is repeated, that query is moved to the top of Recent Queries, with the number remaining the same. Limit and planner are displayed on the Recent Queries page.

Bug21062 - Recent queries in multiple browser tabs

Using WebView in multiple tabs of a browser caused a javascript error leading to a blank screen below the menu. This has been fixed by consistently using localStorage.

Note for using multiple browser tabs/windows: the last tab to execute query will overwrite localStorage of "recent queries" that will be seen if a new browser tab is opened.

Bug21050 - WebView: CodeMirror Query editor steals arrow keys

When the cursor is in Query Editor, Alt-LeftArrow and Alt-RightArrow will now perform the default browser behavior instead of navigating within the editor page.

Bug21036 - Editor empty after 40 recent queries

WebView holds up to 40 recent queries, but incorrectly restored the text when more than 40 were made. This is fixed.

Note that logout and login will clear the list of recent queries.

Bug20998 - Webview: failures in IE7

Many links to navigate between WebView pages did not work in Internet Explorer 7 (IE7) due to usage of browser history. In the Query editor page, limit and planner options did not take effect in IE7. These problems have been fixed.

Bug20982 - Query limit and planner selections ignored

In the Query editor screen, the limit and planner selections were ignored. Now, they are both used for Execute and Download. Also, they will remain set after Execute, and following a link from the Recent Queries page will remember these settings for that query (but they are not stored for a Saved Query).

Changes to the Lisp API

Bug21042, Bug21043 - standardize remote-triple-store SPARQL cursors

The cursors returned by run-sparql for remote-triple-stores behaved differently from ones returned for other triple-store classes. In particular, SELECT queries could not request a results-format of :cursor and the rows returned by cursors were lists of bindings rather than arrays of bindings.

The cursors for remote-triple-stores are now standardized to behave in the same way as cursors in the rest of AllegroGraph. This item is repeated above under AllegroGraph Server: General Changes, where it is also pointed out this applies to Lisp clients only.

Bug20067 - Make session ping threads more resilient

Remote clients ping their sessions to keep them from timing out on the server. Intermittent network errors could cause the ping thread to fail. This fix makes pinging attempts continue until the session timeout passes without successful pings.

Documentation

Bug21080 - Doc layout bad in IE intranet-mode

AllegroGraph docs now use html5 doctype and style. For Internet Explorer (IE), the highest mode available will be used because IE7 layout is not optimal.

Python Client

Rfe11244: Python Support for client X.509 certificate authentication

To support X.509 certificate authentication with SSL, AllegroGraphServer objects now have cainfo and sslcert keyword arguments for indicating the client security information. verifyhost and verifypeer can also be used.

These arguments correspond to the pycurl.CAINFO, pycurl.SSLCERT, pycurl.SSLVERIFYHOST, and pycurl.SSLVERIFYPEER settings on the Curl object if you want to read about them in the pycurl documentation for more information.

user and password keyword arguments can be omitted when using certificate authentication.

See src2/franz/openrdf/tests/tests.py's test_ssl for example use.

Java Client

Rfe11523 - Upgrade to Sesame 2.6.5 jars

With this change, the Java client has been updated to use the Sesame 2.6.5 jars.

Rfe11358 - Upgrade Clojure to 1.4.0

The Clojure client API now uses Clojure 1.4.0 and Leiningen 1.7.1.

Bug21099: java pool holds sockets open

Problem: the AGConnPool used one MultiThreadedHttpConnectionManager per AGRepositoryConnection, but only closed it with the pool rather than with the repo. This used multiple connections to the main AGraph port and left http socket connections open for them and a session port after the AGRepositoryConnection was closed. Session ports are not recycled by the AGraph server if SessionPorts is not specified.

Fix: use a SimpleHttpConnectionManager for each AGRepositoryConnection and close both at the same time. This minimizes socket connections and avoids the secondary pool within MultiThreadedHttpConnectionManager.

Bug21092 - Clojure: errors in tutorial

Enhanced repo-federation to work without rcon-args as was used in the tutorial. This fixes tutorial example16.

Enhanced value-factory to work with either Repository or RepositoryConnection. This reduces reflection warnings in the tutorial.

AllegroGraph 4.6

Server Changes

Rfe11449 - Timeout for database instances

Support has been added for making database instances linger for some amount of time after their last close. This allows for reducing the likelihood that a subsequent access to an otherwise idle database will require a long startup time. The time to linger is determined by the global and per catalog InstanceTimeout directive whose default is 0s.

InstanceTimeout is from the instant the last session or backend that has the store open closes it.

New function ensure-not-lingering of the same argument as open-triple-store.

close-triple-store has a new keyword arg :ensure-not-lingering.

Rfe11445 - Queries with large intermediate result sets could generate segfaults

Queries with very large intermediate result sets could generate segmentation faults. This has been corrected.

Rfe11440 - Remove trailing slash from server URL

With this change, AGServer now removes any trailing slash from the specified server URL. Previously, an errant trailing slash on the server URL resulted in an obscure error message about "No suitable response format available" that was difficult to diagnose.

Rfe11418 - improve efficiency of AllegroGraph's turtle parser

Improved the memory usage and speed of AllegroGraph's turtle parser making it more than twice as fast as previous versions.

Rfe11404 - Optimized opening stores

By reducing inter-process communication latency, opening triple stores has been made significantly faster.

Rfe11177 - Agload --relax-syntax flag

agload now uses --relax-syntax flag instead of --relax-for-btc flag. This flag now allows for underscores and dashes to be used in anonymous node strings for N-Triples and N-Quads. agload is documented in Data Loading.

Rfe11126 - SSL client certificate and CRL support

New https-verification keyword argument to open-triple-store and create-triple-store.

Rfe8881 - Optionally suppress duplicates at commit time

AllegroGraph now has a switch duplicate-suppression-strategy, which will prevent new duplicate triples (both duplicates already in the store, and duplicates within the same commit) from being committed to the store.

Bug20997 - Solve latency issue while starting replication.

Fixed a bug where transaction logs (tlogs) were repeatedly (and unnecessarily) examined when performing replication. Now each tlog is examined once. This results in the time required to be O(n) instead of O(n*n) (that is, time is linear with the number of tlogs rather than increasing by the square of the number of tlogs).

Bug20984: Turtle serializer emitted invalid blank node identifiers

The rdf-n3 (turtle) serializer was emitting blank node identifiers without prefixing them with ":_". This has been corrected.

Bug20981 - Exhausted cursor recycling happens too early

Exhausted cursors were being recycled too early which could sometimes result in program crashes. This has been fixed.

Bug20968 - Improve string dictionary cache performance

When heavily used, AllegroGraph's string dictionary cache could become bogged down in bookkeeping resulting in very poor performance. This change prevents this excessive bookkeeping and provides the expected near-constant cache performance.

Bug20943 - Free text index bug under replication

When a new free-text index was created while replication was enabled, it caused an error in the standby process. This is now fixed.

Bug20913: Xsd date and time coercion functions could signal error

If given invalid input, the type coercion functions for dates, times and dateTimes could signal an error other than a SPARQL type error. This would lead to a query error rather than a filter failure. For example, this query:

prefix xsd: <http://www.w3.org/2001/XMLSchema#>  
select ?ds ?d {  
  bind( '2012-12-04 is my favorite day' as ?ds )  
  bind( xsd:date(?ds) as ?d )  
}

would signal an error rather than returning a single result for which the binding for ?d is empty.

This has been corrected.

Bug20910 - Queries with variable predicates could lose results

If AllegroGraph finds a query pattern with variable predicates and distinct results (and certain other conditions that are data dependent are also true), then it will optimize the pattern by converting it into an iteration over all of the distinct predicates in the store. A typical query of this form is

select distinct ?p {  
  ?s a ex:Animal .  
  ?s ?p ?o  
}

Because computing all of the distinct predicates is expensive, AllegroGraph caches this information. Previously, AllegroGraph could fail to update the cache correctly when new triples were added. This could lead to queries returning too few results.

This has been corrected.

Bug20896 - Hang when max chunks for an index reached

As part of their operation, the automatic index optimization processes will sometimes merge some number of input chunks into a single output chunk in order to keep the total chunk count below a certain threshold. However, under some circumstances, some indices would never undergo the optimization process so this chunk reduction would not occur. Eventually the chunk limit would be reached and the database would hang. This problem has been corrected.

Bug20887 - AllegroGraph XML parsers should not follow external references

AllegroGraph's RDF/XML and TriX parsers could try to follow external references in input files and fail. This has been corrected.

Bug20877 - xs:date could fail on some short plain literals

The xs:date constructor function could fail to correctly create a date if the argument passed to it was a small plain literal. E.g., constructs like

BIND (xsd:date( concat(str(?date), '-01-01')) as ?begin_date)

could pass a string whose length was less than 10 to the xs:date function and this could lead to the failure.

This has been corrected.

Bug20833 - openFederated should handle remote stores

AllegroGraphServer.openFederated now works on remote stores. Previously it always attempted to open supplied arguments as local stores.

The openFederated API still has some limitations. To open a federated session with all federation minilanguage features such as reasoning and graph filtering, pass a properly formatted spec string to AllegroGraphServer.openSession directly.

Bug20822 - Cascading error can occur if database create/open fails

Previously, if there was an error during database open/create, another error could occur during the cleanup process, obscuring the actual problem. This has been fixed.

Bug20819 - problem with some combinations of multiple OPTIONAL clauses

Some query patterns with OPTIONALs whose left-hand side was empty could lead to incorrect results. An example of a query that could have failed was:

select distinct ?value ?type ?name {  
  OPTIONAL {  
    ?value <http://example.com#isa> ?type .  
  }  
  OPTIONAL {  
    ?value <http://example.com#name> ?name .  
  }  
  FILTER( REGEX(STR(?name),'Sa') && BOUND(?type) )  
}

This has been corrected.

Bug20814 - Make session urls unique

In order to avoid accidental sharing of sessions, include a unique session id in the session-url. Behind the scenes, what previously looked like http://localhost:55555 is now something like: http://localhost:55555/sessions/7e8df8cd-26b8-26e4-4e83-0015588336ea.

Bug20806 - fix loading of initfile and sitescripts

Initfiles and sitescripts longer than 4096 characters were presented with 4096 character long blocks in reversed order. This bug is now fixed.

Bug20799 - turtle-parser was case sensitive when parsing numbers

AllegroGraph's turtle parser was failing to parse floating point numbers that used an upper-case E before the exponent. This is now corrected.

Bug20793 - RDFS++ queries with free predicate could infer invalid triples

If a query involving RDFS++ reasoning used an unbound predicate, the reasoner could use inverse property reasoning to infer invalid triples. This has been corrected.

Bug20757 - Problems when starting agraph while another agraph is running

Previously if you tried to start AllegroGraph when it was already running, the second start would fail but it would cause the already-running AllegroGraph to stop operating correctly. This has been fixed.

It has always been acceptable to run two or more AllegroGraph servers simultaneously as long as they, at the very least, used different settings for the following parameters in their configuration files:

Port
SettingsDirectory
GlobalShmServerDirectory (defaults to the value of SettingsDirectory)

The operating system already enforced a unique Port setting. This change adds enforcement of unique SettingsDirectory and GlobalShmServerDirectory settings. Therefore some (dangerous) side-by-side AllegroGraph configurations that used to work may no longer start up. An instructive error message will be displayed in this case.

Bug20628 - Changes to warmstandby API

PUT /repositories/[name]/warmstandby

If one attempts to start a replication task that already exists, an http 409 Conflict response will be returned.

Bug20533 - Agload --rapper flag documentation

The --rapper flag is now specified in the group "Other options". Previously it was not given a group in the command line help. agload is documented in Data Loading.

Bug20508 - Remote triple store can't use the RDFS++ reasoner

Added support for apply-rdfs++-reasoner on a remote client triple-store.

HTTP Client

Rfe10812 - Improve http tracing

HttpTrace is now a global configuration directive of the same meaning as the --http-trace command line option. The global directive HttpTraceOptions and the command line option --http-trace-options were added, the value of which is a comma separated list of options. Options starting with the character + turn on the corresponding log category. Those starting with - turn them off. Use the max-message-size=<N> option to truncate overly long messages. The default is +log,max-message-size=1000.

Bug20798 - Improved http error handling

Previously, if the client that was performing an http request died and the backend couldn't transmit the response, then the backend was killed and the request was retried on another backend that also failed because the client socket was closed which wasted resources. The new behavior is to let the backend live on (without retrying), if the error is really from the client side.

SPARQL

Rfe11469: add line number to SPARQL parser error messages

The SPARQL parser will now indicate the line at which an error is encountered. The error message has also been reformatted to make it easier to understand the parsing problem.

Rfe11357 - enhance SPARQL 1.1 engine for changes to standard

The SPARQL 1.1 standards committee recently made several small changes to the standard. To accommodate these, AllegroGraph's SPARQL engine has:

added support for the functions STRBEFORE, STRAFTER and REPLACE
removed support for the function SHA224
relaxed the type validation for REGEX so that language tagged literals and typed literals of type xsd:string can be searched without resorting to the STR function.
returns rdf:langString as the datatype of a language tagged literal
accepts solution modifiers for ASK queries

There are several other changes to the standard which AllegroGraph will support in a future release. These include:

Grammar: Backslash character escapes in prefixed names
Grammar: Fix: Allow ASK to take solution modifiers
Grammar: Fix: AdditiveExpression: '?' => '*'
Allow aggregates in ORDER BY clause

For additional information, please see the SPARQL release notes.

Bug20951 - SPARQL could fail to match plain and typed strings

SPARQL specifies that a plain literal (like "hello") is equivalent to a literal with type xsd:string (like "hello"^^xsd:string). The SPARQL 1.1 query engine was failing to provide this entailment in some cases. This has been corrected.

Bug20939 - CLEAR only recognized if it's the first verb

The SPARQL 1.1 UPDATE verb CLEAR was only recognized correctly if it was the first verb in a SPARQL statement. This has been corrected.

Bug20936 - OFFSET 0 and LIMIT did not mix

If a SPARQL query used an explicit OFFSET of zero, the LIMIT would be ignored. I.e., a query like:

SELECT * { ?a a ex:Animal . }  
LIMIT 10 OFFSET 0

would return all ex:Animals rather than just 10. Leaving the OFFSET off entirely would produce the correct behavior. This has been corrected so that OFFSET 0 is treated as expected.

Bug20912 - Inequality filters could signal an error

Inequality filters in SPARQL queries can be transformed into fast range queries. If a query used a filter like

FILTER( ?a < 0.0 )

then the SPARQL 1.1 query planner could attempt to decrement an unsigned number below zero which would lead to a segmentation fault.

This has been corrected.

Bug20911 - SPARQL could lose precision on very small numbers

SPARQL numeric conversion routines could treat double-floats and decimals as single-floats which would lose precision. For example, a query with a filter like

FILTER( ?x > 4.5e-45 )

would have been interpreted as

FILTER( ?x > 0 )

This has been corrected.

Bug20898 - Complex queries could fail on federated triple-stores

Because of the way statistics were calculated on federated triple-stores, SPARQL queries with many clauses in a basic graph pattern (BGP) could fail when the query engine attempted to reorder them. This has been corrected.

Bug20891 - Improve SPARQL comparator selection

SPARQL specifies difference comparison operations depending on the type of the things being compared. The SPARQL 1.1 query engine could select the wrong operation if both the left and right hand sides were variables. This has been corrected.

Bug20884: improve Lisp client remote-triple-store SPARQL interface

The remote-triple-store implementation of the Lisp client was unable to execute SPARQL queries with non-trivial filter clauses. This has been fixed. (Repeated in the Lisp API section below.)

Bug20867: REPLACE function in SPARQL 1.1 used incorrect syntax for matches

The SPARQL 1.1 REPLACE function was implemented using a different syntax for pattern replacement than that specified by the standard. Replacements are now correctly specified using the $<number> syntax.

Bug20863 - Some SPARQL XSD numeric computations could fail

Depending on the datatypes of the numbers involved, it was possible for a SPARQL numeric computation to signal a type error rather than correctly returning a value. This has been corrected.

Bug20859 - Some filter combinations could cancel each other out

If a SPARQL query used IN (or several FILTERS like ?uri = constant) and if there were other filters on variables that overlapped with the variables in the IN construct, then it was possible for AllegroGraph to lose track of the additional filters in some circumstances. This happened due to two separate query optimizations handling the filters inconsistently. This problem has been corrected.

Bug20852 - remote-triple-store and SPARQL UPDATE don't mix

The Lisp remote-triple-store client could signal an error after evaluating a SPARQL UPDATE statement. The UPDATE would occur but the result would be mis-interpreted and signal an error. This has been corrected.

Bug20851 - SPARQL FILTERs could be too strict

Certain combinations of triple patterns in a BGP and FILTERs could cause AllegroGraph to misapply the FILTERs. For example, depending on the underlying triple-store, this query

SELECT ?first_film ?actor ?second_film {  
  ?first_film dbpp:starring dbpr:Jack_Black , ?actor .  
  ?second_film dbpp:starring dbpr:Joan_Cusack , ?actor .  
  FILTER ( ?actor != dbpr:Joan_Cusack )  
  FILTER ( ?second_film != ?first_film )  
  FILTER ( ?actor != dbpr:Jack_Black )  
}

could have misapplied the ?actor filter and produced fewer results than expected.

This has been corrected.

Bug20838 - SPARQL CONSTRUCT queries could fail for large result sets

Due to limitations in the internal mechanisms used to keep track of intermediate results, SPARQL CONSTRUCT queries that returned more than 100,000 rows would cause a segmentation fault. This has been corrected.

Bug20830 - Some inequalities could use the wrong comparisons

Some SPARQL 1.1 queries could select the wrong comparison code when comparing two variable clauses with either >= or <=. This could lead to incorrect results. This has been fixed.

Bug20821 - EXISTS or NOT EXISTS and additional filters did not mix

The new SPARQL 1.1 EXISTS and NOT EXISTS filters failed to parse correctly when combined with other FILTER clauses. This has been fixed.

Bug20805 - encodeforuri could produce incorrect results

The SPARQL 1.1 encode_for_uri could produce incorrect results for some UNICODE characters. This has been corrected.

AGWebView

Rfe10885 - Setting duplicate suppression from AGWebView

Added "Suppress duplicate statements" feature to the repository overview page. This sets the duplicate-suppression-strategy switch, which can be set to prevent new duplicate triples (both duplicates already in the store, and duplicates within the same commit) from being committed to the store.

Bug20996 - Webview: js error creating a new repo

This fixes a javascript error when creating a repository from the Catalog page.

Bug20993 - Webview: Internet Explorer (IE) compatibility mode

AGWebView now includes an html meta header for IE9 so that IE9 will use its documentMode instead of IE7 compatibility. Also tested with IE8.

Also using html5 doctype to get standards mode in all browsers.

Bug20963 - Fix session url in AGWebView

Urls for sessions are changed in Bug20814 (described under AllegroGraph Server: General Changes above). This fixes AGWebView and changes the url for its use of sessions to include the uuid.

Bug20894 - AGWebView not working with ie9 on intranet

When connecting to AGWebView on an intranet site IE may automatically use Compatibility Mode. A syntax error causes it to only display the "Loading" message and not continue. Work-around is to use a full url, or otherwise not use compatibility mode. The syntax error is now fixed so an intranet does not cause the error.

Bug20843 - AGWebView sometimes clears the editor

The execute button would clear the query editor (and the results area). This happened when the same query had been executed sometime before in the same browser followed by logout/login. The browser history mechanism had not been fully cleared. Recent queries are cleared after logout. This is now fixed.

Changes to the Lisp API

Bug20884: improve Lisp client remote-triple-store SPARQL interface

The remote-triple-store implementation of the Lisp client was unable to execute SPARQL queries with non-trivial filter clauses. This has been fixed. (Repeated in the SPARQL section above.)

Bug20883: improve the Lisp client's remote-triple-store caching interface

The Lisp client's remote-triple-store implementation did not support the spogi-cache-enabled-p or spogi-cache-size methods. This has been corrected.

Bug20185 - Remote lisp client used wrong value for restriction reasoning

For a remote reasoning store, this method used the wrong string internally, leading to an error when a query was made:

(setf (restriction-reasoning-enabled-p *db*) t)

For a remote reasoning store, this method enabled restriction reasoning if the key :restriction-reasoning-enabled was present regardless of the key's value. Now respects t or nil.

(apply-rdfs++-reasoner :restriction-reasoning-enabled nil)

Documentation

No significant changes.

Python Client

Rfe11313 - Remove export*, use response capture

RepositoryConnection.export and exportStatements have been deprecated. They used client side serialization of statements that was buggy and not up to spec. The functionality has been replaced with a way to capture server responses of any supported mime type.

A new context manager function on RepositoryConnection is called saveResponse, and can be used as follows:

with open('out', 'w') as response:  
    with conn.saveResponse(response, 'application/rdf+xml'):  
        # The server response is written to response  
        conn.getStatements(None, None, None)

You can reference the function testsaveresponse in src2/franz/openrdf/tests/tests.py for examples.

Java Client

Rfe11493 - Support writing N-Quads from an AGModel

With this change, the Jena adapter now supports writing to N-Quads via AGModel.write(out, "N-QUADS").

Rfe11438 - Provide N-Quads parser/writer META-INF/services in agraph.jar

With this change, the agraph.jar now provides N-Quads implementations for org.openrdf.rio.RDFParserFactory and org.openrdf.rio.RDFWriterFactory in META-INF/services (Sesame doesn't yet provide N-Quads implementations).

Rfe11436 - Add Sesame 2.6.2 src jar

With this change, the openrdf-sesame-2.6.2-onejar-src.jar has been added to eclipse project classpaths (repo and distribution) for convenience.

Rfe11397 - Support RepositoryFactory and Repository configuration

With this change, the Java client now supports Sesame's RepositoryFactory api and Repository configuration from RDF data. See javadoc for the new com.franz.agraph.repository.config package for details.

Rfe11240 - Support user/role management

With this change, AllegroGraph superusers can now dynamically manage users and roles. See the AGServer javadoc for details.

Rfe11230 - Support masquerading as another user

With this change, an AllegroGraph superuser can now masquerade as another user when issuing requests. See the javadoc for AGRepositoryConnection#setMasqueradeAsUser for details.

Bug20946 - IE javascript error in Mold: innerHTML is null

Work-around: the error dialog can be dismissed and the search results are still displayed. Fixed.

Bug20811 - Allow prolog queries from javascript

To run javascript the user must have the "execute arbitrary code" permission which should be enough to run prolog queries too. The bug that was fixed prevented the user from running prolog queries from javascript.

AllegroGraph 4.5

Server Changes

Rfe11338 - CJK bigram tokenizer

A new tokenizer :simple-cjk for Chinese/Japanese/Korean (CJK) text has been added. It uses bigrams for consecutive CJK characters as words. To use this tokenizer, specify :simple-cjk as the tokenizer option when creating or modifying a freetext index.

The bigram tokenizer can be used for CJK mixed text, but its simplicity may result in false positives. It also tends to index a much larger number of words compared to other tokenizers. See Free-text Indices and create-freetext-index.

Rfe11298 - Spread access to the same triple store across shared backends

Previously, when using shared backends, operations for a particular triple store tended to be routed to a single shared backend, even if it was busy. Now concurrent operations for a particular triple store are routed to a shared backend that is not already processing a request for that same store. Additional backends (up to the value of the "Backends" configuration parameter) will be started if necessary to achieve this goal. This is expected to increase concurrency in such situations.

Dedicated sessions are unaffected by this change.

Since this change is most beneficial when there are many shared backends available, the default setting of the "Backends" configuration parameter has been changed from 3 to 10.

Rfe11257 - Backup/restore user namespace settings

agraph-backup is enhanced in two ways:

The user namespace settings are now handled by the backup-all/backup-settings/restore-all subcommands.

The backup-settings subcommand can take a path to the executable of an older AllegroGraph's version of agraph-backup, and call it to archive the store data. With this feature, archiving an entire repository from an older AllegroGraph repository becomes a single command.

See Backup and Restore.

Rfe11099 - Add Apache Solr support to AllegroGraph

Now you can connect an external Solr text search engine from AllegroGraph server. With new prolog functions you can issue a query to the Solr engine to retrieve associated document IDs. See Solr Text Indices.

Bug20750 - Unused store closer holds lock too long

The thread which closes idle triple stores in backends held an important lock while trying to acquire locks on individual stores. If a lock on a store was being held by another thread (e.g., one running a long query), this could cause the whole backend to become unresponsive to other unrelated requests. This has been corrected.

Bug20736 - Federated-triple-store could mistreat xsd:strings in queries

In some cases a federated-triple-store could fail to account for the equality of plain literals and literals with a datatype of xsd:string. This has been corrected.

Bug20717 - Fix the UPI not in string table query issue

Depending on the result format and the client, queries that generated new strings (e.g., using SPARQL 1.1's BIND clause) could lead to a UPI not in string table error. This has been corrected.

Bug20712 - Implement xs:date and xs:time XPath constructor functions

The constructor functions for xs:date and xs:time are now implemented.

Bug20697 - Zero length property path with future parts

Zero length property path patterns now correctly match future parts that are not yet in the db:

?x :p{0} :a .

Note that the same pattern with a literal worked even without this fix:

?x :p{0} "o" .

Bug20690 - Property path results cardinality

In the presence of cycles, property path expressions did not always return results with the right cardinality. This has been fixed.

Bug20616 - Fix race condition in triple store termination

AllegroGraph now waits for a triple-store's subprocesses to exit when the last connection is closed instead of killing them while they could potentially be executing a clean shutdown.

Avoid Logging a backtrace during unclean shutdown

Under certain circumstances (such as when 'killall agraph' was used), there was a race condition during process termination that could result in an error and backtrace being logged to agraph.log. This has been corrected.

HTTP Client

Rfe11308 - Support UPDATE via POST /repository/ID/statements

The Sesame 2 protocol allows statements to be added to a repository by sending a POST request to /repository/ID/statements. AllegroGraph previously only supported update via POST requests to /repository/ID.

Now SPARQL update statements may be directed to either location. You can also POST an RDF document or data in the form of an application/x-rdftransaction to /repository/ID/statements.

Rfe11213 - Allow superusers to run queries with another user's security filters

AllegroGraph sessions belonging to superusers now allow their owners to use the X-Masquerade-As-User header in HTTP requests to activate another user's security filters.

Bug20671 - List the Hub process in the HTTP API

AllegroGraph now lists the Hub process in the HTTP API's process list and in WebView. The Hub process can not be killed or asked to perform a backtrace.

SPARQL

Rfe11320 - Trying to use SERVICE should signal an error

AllegroGraph's SPARQL engine does not support federated query (http://www.w3.org/TR/sparql11-federated-query/). Previously queries using the SERVICE keyword would fail silently. Now these queries will signal an error.

Rfe11302 - Support clauses with no variables

The SPARQL 1.1 engine now supports basic graph patterns where one or more of the clauses have no free variables. Previously, a query like

select * {  
   :a :p1 :c .  
      :p2 ?value .  
}

could produce incorrect results. This has been corrected.

Rfe11262 - Improve SPARQL 1.1 filter transformation

AllegroGraph's SPARQL 1.1 engine is able to move some filters out of the query processor and into the storage layer. For example, a query like

select * {  
  ?a rdf:type ?class .  
  filter( str(?class) > 'z' )  
}

will move the inequality into a filter involving only the first clause. In some cases, however, the query processor could leave behind a stub filter which would unnecessarily add to total query time (without altering the results). This is now corrected.

Several other minor efficiency improvements have also been added to the query engine.

Rfe11254 - Improve SPARQL 1.1 sub-query implemenation

A problem in the sub-query implementation led to two failures in the DAWG test-suite. These have been corrected and additional tests have been added to the regression tests

Rfe11245 - Optimize Queries where the right-hand side of a join is a BGP

The SPARQL 1.1 engine now handles queries where the right-hand side of the join is a BGP more efficiently. Previously, it would compute the RHS separately and then join the two sets. Now, it merges the RHS BGP into the existing LHS set directly. This can provide some speed and memory improvements for some queries.

Rfe11135 - Implement SPARQL 1.1 DISTINCT aggregation

AllegroGraph's SPARQL 1.1 engine now implements DISTINCT in its aggregates. This means that a query like

select ?gender (count(distinct ?profession) as ?pCount) {  
  ?person k:profession ?profession .  
  ?person k:sex ?gender .  
} group by ?gender order by ?pCount

will now correctly return a count of the distinct professions for each gender rather than a count of all professions.

Rfe11098 - Integrate Apache Solr with SPARQL 1.1 query engine

Assuming that you have setup Apache Solr with an AllegroGraph triple-store, you can now query it using the SPARQL 1.1 query engine. See the description of using Solr in Solr Text Index. When used with SPARQL 1.1, there are two storage strategies: addinng a triple that associates the text with a Solr document ID and telling Solr to associate the text with the triple-id directly. Both are described in the referenced document.

Rfe11070 - Improve performance of variable predicate queries

AllegroGraph's SPARQL engine now recognizes queries that have variables in the predicate position and provides custom methods to optimize them. An example of such a query is:

SELECT ... {  
   ... clauses ...  
   ?s ?p ?o .

Bug20778 - Several fixes for various SPARQL 1.1 issues

There were numerous issues in the initial release of the SPARQL 1.1 query engine. Some of the problems were caught by customers and others by internal review. This release of AllegroGraph fixes:

issues with several functions including: encodeforuri, some new mathematical functions, isLiteral, etc.
fixes several code paths that could lead to the UPI is not in string dictionary error,
fixed some edge cases in the SPARQL parser,
made some conversion functions more stringent so that, for examples, 10 dollars would raise a SPARQL type error rather than being treated as the number 10.

There remain some known issues with SPARQL; please see the SPARQL release notes for details.

Bug20760 - Several improvements in SPARQL aggregation.

Do not include unbound variables in COUNT aggregation. Previously, AllegroGraph would include these in the count.
Remove bindings for SUM, AVG, etc. if any of the bindings in the group are unbound. Previously, AllegroGraph would simply ignore the unbound values are return the SUM of the remaining elements.
Allow non-numeric bindings for MIN and MAX. Previously, AllegroGraph assumed that MIN and MAX only applied to numeric bindings.

Bug20756 - Multiple OPTIONAL clauses for the same variable could fail

The SPARQL 1.1 query engine could fail to find some bindings when multiple OPTIONAL clauses referred to the same variable. For example, a query like

SELECT ?first ?last {  
  ?s ab:lastName ?last .  
  OPTIONAL { ?s ab:nick ?first . }  
  OPTIONAL { ?s ab:firstName ?first . }  
}

could fail to find bindings for ?first that come from the second OPTIONAL. This is now corrected.

Bug20752 - Queries with both filters and property paths could fail

If a SPARQL 1.1 property path query used FILTERs and these FILTERs were moved into the triple patterns, then it was possible for the filters to be ignored. This has been corrected.

Bug20732 - SPARQL 1.1 property path improvements

In some cases, SPARQL 1.1 property path queries could do too much work and return too many results. This has been corrected.

Bug20723 - Improve handling of SPARQL filter predicates

Some of the SPARQL 1.1 filter predicates were failing to function properly when used in some contexts because their return value was not being correctly transformed from an internal representation to an external one. This has been corrected.

Bug20722 - SPARQL 1.1 parser treated whitespace differently

The SPARQL 1.1 parser required whitespace after a period and before an additional SPARQL keyword. This meant that a query like

select * { ?a ?b ?c.filter( ?c > 3) }

would signal a parse error. This has been corrected.

Bug20721 - Improve handling of certain OPTIONAL edge cases

Depending on the data in the store, the SPARQL 1.1 engine could sometimes fail to process queries like:

select ?a ?b ?c ?d {  
  optional { ?a a ?b . }  
  { ?c <urn:a:b> ?d . }  
}

where the OPTIONAL could lead to an empty group of bindings on the left-hand-side during query processing. This has been corrected.

Bug20711 - Improve some date and dateTime comparison issues

The SPARQL 1.0 and 1.1 engines behaved inconsistently with respect to one another for certain comparisons between xsd:dates and xsd:dateTimes. This could lead to differences in query behavior. These issues have been reconciled.

Bug20695 - Problem when computing range-queries on dateTimes

Both the SPARQL 1.0 and the SPARQL 1.1 engine could fail to correctly compare dateTime datatypes with fractional seconds. This has been corrected.

Bug20689 - Parsing of SPARQL property path expressions where a group is followed by a + as in

?x (ex:p)+ ?y

no longer results in a error.

AGWebView

Bug20751 - Agwebview could submit queries twice in some browsers

On some browsers AGWebView could submit queries twice. This led to unnecessary work for SPARQL or Prolog SELECT queries and to duplicate data for SPARQL UPDATE queries. This has been corrected.

Bug20738 - AGWebView could not start remote sessions

A recent change in AllegroGraph broke AGWebView's ability to start sessions on remote stores. This is now corrected.

Changes to the Lisp API

No significant changes.

Documentation

Rfe11236 - Fix command line documentation of the agdebug tool

The agdebug tool now correctly indicates a non-optional argument.

Python Client

Rfe11229 - Add support for user masquerading to python client

Added support for user masquerading to Python client by adding RepositoryConnection.runAsUser(username).

Java Client

Rfe11329: AGConnPool, remove unnecessary dependency on Jena

If using AGConnPool with Sesame API, but not Jena, the Jena libraries are no longer required.

Rfe11310: AGConnPool should rollback when borrowing

Pool will now always rollback when borrowing a connection. This makes the transaction catch up to the most recent.

The 'testOnBorrow' property for the pool is now redundant, so generally not recommended.

Rfe11242: Upgrade to Sesame 2.6.2 jars

With this change, the Java client has been updated to use the Sesame 2.6.2 jars, including support for AGUpdate and AGRepositoryConnection#prepareUpdate methods. Java Sesame Tutorial example13 has also been updated to reflect use of prepareUpdate.

Bug20795: AGConnPool does not close connections

Closing the AGConnPool now closes sessions properly.

AllegroGraph 4.4.0.1

Server Changes

Bug20667 - XSD float exponent fix

Due to rounding issues when encoding, exponents of floats in the :sparql-xml result format were - very seldom - off by one. This has been fixed.

Bug20615 - TrxLogRetain replication now works on secondaries

The catalog directive: TransactionLogRetain replication now archives properly for replication secondaries.

HTTP Client

No significant changes.

SPARQL

Rfe11219 - Add query option support for all SPARQL constructs

AllegroGraph's SPARQL parser supports specifying query options as part of the query string. These take the form

prefix franzOption_OPTIONNAME: <franz:OPTIONVALUE>

In particular, you can tell AllegroGraph which query engine to use with the queryEngine parameter. For example, to use the SPARQL 1.0 engine in a query, you would add the prefix

prefix franzOption_queryEngine: <franz:sparql-1.0>

This extension is now supported for all SPARQL constructs: ASK, CONSTRUCT, DESCRIBE, SELECT and UPDATE.

Rfe11218 - Improve maximum solution size handling

Because the SPARQL 1.1 engine operates on the entire query solution simultaneously it can be more memory intensive than the SPARQL 1.0 engine for some queries. There was an internal check to warn if query solution sets were becoming too large. This check was causing some queries to fail rather than simply signaling a warning. This has been changed so that these queries will write a warning to the log rather than failing.

Rfe11202: SPARQL Update example with batch update

With this change, Java tutorial example 13 has been augmented to show how to evaluate SPARQL Update queries, including one that batches updates in a single request. [This item is repeated in the Java Client section.]

Bug20768 - SPARQL 1.1 property paths could return incorrect results

AllegroGraph was failing to filter non-matching rows for property path queries like

select ?a {  
  ?a owl:sameAs{1} ?a .  
}

meaning that it would emit results when it should not. This has been fixed.

Bug20692 - Sparql 1.1 CONTAINS function can fail

Corrected a problem with some instances of the SPARQL 1.1 CONTAINS function.

Bug20676 - Geospatial datatype mappings could cause query errors

Changes to some query optimization code in the AllegroGraph SPARQL engines (both sparql-1.0 and sparql-1.1) were incompatible with geospatial datatype mappings (geospatial predicate mappings were not effected). This could lead to parse errors during query processing. This has been fixed.

Bug20665 - SPARQL-1.1 UNION doesn't see with-variable bindings on the RHS

The SPARQL 1.1 engine was losing track of the variable bindings on the right-hand side of a UNION. This could cause queries like

select ?s1 ?s2 {  
  { ?s1 a ?p . }  
  union  
  { ?s2 a ?p . }  
}

(with a variable binding for ?p) to produce incorrect results. This has been corrected.

Bug20664 - SPARQL-1.1 CONCAT function signaling errors with typed literals

The SPARQL-1.1 CONCAT function could signal an error instead of generating the correct result when one or both of the arguments with a typed literal. This has been corrected.

AGWebView

Rfe11198 - Improve WebView's bulk loading UI

WebView no longer conflates bulk load mode with agload usage. It's now possible to use agload's multi-core optimized loading mechanism and to optionally turn bulk load mode on. Some aspects of the file loading UI in WebView have also been optimized: The format selection element is now a combo box, and is located closer to the file input field.

Bug20662 - Reload triple count in WebView if load fails

WebView will now display the correct triple count in a store after an import got stopped by an error (e.g. the triple license limit).

Bug20641 - Improve restoring backups from WebView

When restoring from a backup in the WebView catalog overview, AllegroGraph will now automatically upgrade the newly-created store to the newest version. Additionally, there is now an option to set the store up for use as a replication secondary.

Bug20637 - Fix problem where SPARQL Update did not work with AGWebView

Changes to SPARQL UPDATE caused some UPDATE queries to fail in AGWebView. This has been corrected.

Changes to the Lisp API

Bug20640 - Fix printing of encoded-ids in remote lisp client

AllegroGraph's remote lisp client now correctly translates encoded IDs' UPIs back into strings. It can also correctly translate variable-width encoded IDs into UPIs now.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe11202: SPARQL Update example with batch update

AllegroGraph 4.4

Server Changes

Rfe11082: Transactional duplicate deletion

INCOMPATIBLE CHANGE: AllegroGraph no longer supports duplicate deletion at index optimization time. The database creation parameters indicating duplicate deletion strategy are no longer accepted.

Instead, AllegroGraph now allows deletion of duplicates using the delete-duplicate-triples API function, which is also accessible via the HTTP API call DELETE /statements/duplicates.

Rfe11039 - Include AllegroGraph version in ps output

AllegroGraph subprocesses modify their command line so that they can supply identifying information to programs like 'ps' and 'top'. Now, in addition to the name of the subprogram, the version of AllegroGraph is included. Also, for the AllegroGraph Service Daemon process, the port number that it is listening on is included.

Rfe11017 - Add an agdebug command line utility

AllegroGraph now ships with a command line utility called agdebug that will output a summary the contents of the transaction log files in a store. This is primarily useful for diagnosing problems with transactions and the database's storage system.

Rfe10984 - Fix serialization issues with encoded IDs

Both the turtle and the RDF/XML serializers failed to correctly handle encoded IDs. This has been corrected.

Rfe10981 - Improve upgrading/migration process

The agraph-backup tool has two new commands, backup-all and restore-all, to help migrating existing databases to the new version of AllegroGraph or copying all catalogs and repositories to a new location. See Backup and Restore and [Repository Upgrading[upgrade-guide].

Rfe10948 - Log secondary index creation

Adds log entries to the AllegroGraph log when a secondary index (also called a UPI map) is created.

Rfe10853 - Support fti charset in Python client

Keyword arguments innerChars, borderChars, and tokenizer have been added to createFreeTextIndex and modifyFreeTextIndex.

getFreeTextIndexConfiguration also returns these new configuration properties and values. See put-freetext-index.

Rfe10783 - Add per triple-store control of secondary indices

Modified the Lisp and HTTP APIs so that each triple-store can specify whether or not it should use existing secondary indices and/or create new secondardy indices during query. Since these indices must be maintained and updated when modifying the triple-store, automatic creation should be turned off until the store is quiescent. The default is to use secondary indices when they exist but not to create new ones. Use [create-secondary-indices-p[]/(setf create-secondary-indices-p), and use-secondary-indices-p/(setf use-secondary-indicies-p)* to control the behavior.

Rfe10568, Rfe11172 - Set the default query engine via agraph.cfg

AllegroGraph supports two SPARQL query engines. The algebra engine has been available for several years and is well tested, stable and robust. The newer, set-based, engine, named sparql-1.1 can be significantly faster than the algebra engine but may contain bugs that the algebra engine does not.

Out of the box, AllegroGraph uses the sparql-1.1 engine. This can be changed by using the QueryEngine parameter in the configuration file. If you include

QueryEngine algebra

Then AllegroGraph will use the algebra engine. If you leave the setting off or use

QueryEngine sparql-1.1

Then AllegroGraph will continue to use the set based query engine.

The value of this configuration is read each time a triple-store is created or opened.

Rfe10179, Rfe10540 - SNA optimizations

Improved the efficiency of SNA operations in several ways. All SNA generator functions are now slightly faster. SNA generators that use select now save the select plan and re-use it. This can make these generators hundreds of times faster than before. The SNA system will also use secondary indices (UPI-maps) when available.

Rfe10398 - Improved speed of low-level AllegroGraph operations

Optimized the code used to copy UPIs and parts of triples. This produces a query speedup in SPARQL queries and most Prolog select queries. Some count-only Prolog select queries can run slightly slower.

Rfe10090, Rfe10541 - optimizations to AllegroGraph secondary indices

Improves both the creation of secondary indices (UPI-maps) and their efficiency in use. Queries that make heavy use of secondary indices will operate more quickly.

Rfe9786 - Make shm server global

A new configuration directive GlobalShmServerDirectory is added to specify a directory where the global shared-memory server creates its socket. When omitted, the value of SettingsDirectory is used.

Bug20619 Optionally not inherit trx log arch params during restore

The configuration file catalog directive TransactionLogArchive is no longer inherited from a backup database during a restore. In addition, when the --replica flag is set during agraph-backup restore, TransactionLogRetain and TransactionLogReplicationJobname are no longer inherited from the backup and must be set manually in the config file catalog section prior to restore. This is so that a replication primary and replication secondary can have independent transaction log archiving behavior. The flag --replica also ensures --nocommit is set and --newuuid is not set, and should be used as a replacement for --nocommit when restoring backups for replication client purposes.

Bug20460 - Encoded IDs and reasoning triple-stores mix not

If reasoning was used during a query, then encoded IDs were not correctly recognized which led to incorrect results. This has been corrected.

Bug20443 - Encoded IDs did not work with the TriX serializer

The TriX serializer was incorrectly emitting encoded IDs as if they were plain literals instead of resources. This has been corrected.

Bug20355 - Durably save transactions on warm-standby client

The warm-standby secondary machine now saves transaction log entries in a way that ensures they've been flushed to disk durably as soon as the entire commit has been read.

Bug20332 - Trix parser can get confused and try to encode plain literals

Under some circumstances, the TriX parser could attempt to encode plain literals as if they were typed literals which could cause a data import to fail. This has been corrected.

Bug20230 - Fix deletion slowdown under replication

The replication server used a connection to the replicated triple store that ensured it would never expunge deleted triples, slowing down operations if many deletes had occurred. Now, the replication server correctly expunges these triples and does not slow down.

Bug19957 - Turtle parser should support type-mapping

The turtle parser was not respecting a triple-store's type-mapping settings. This meant that triples added via turtle would not have their objects converted into encoded UPIs. This has been corrected.

Spr38044 - Restrict possible values for TransactionLogRetain

AllegroGraph will now signal an error when invalid values for TransactionLogRetain are encountered in the config file.

HTTP Client

No significant changes.

SPARQL

Rfe10985 - Spin functions and magic-properties

SPIN functions and magic-properties have been implemented. The SPIN implementation is described in Spin document.

Rfe10737 - SPARQL 1.1 BIND support

Add support for SPARQL 1.1's BIND language construct. In particular, queries like

select ?title ?costPlusTax {  
?book <http://purl.org/dc/elements/1.1/title> ?title .  
?book <http://example.org/ns#price> ?cost .  
?book <http://example.org/ns#tax> ?tax .  
bind( ?cost * (1 + ?tax) as ?costPlusTax )  
}

are now supported.

Rfe10736, Rfe10743 - SPARQL 1.1. exists, not exists, and minus

AllegroGraph's SPARQL engine now handles the MINUS operator of SPARQL 1.1. It also handles the new EXISTS and NOT EXISTS filter operators.

Rfe10700 - SPARQL 1.1 assignment

SPARQL 1.1 assignment is now implemented so that queries like

select ?title ?cost ?discount (?cost * ?discount as ?totalCost)  
where {  
?book :title ?title .  
?book :price ?cost .  
?book :discount ?discount .  
}

are now supported.

Rfe10699 - SPARQL 1.1 UPDATE support

The SPARQL parser and implementation now support the UPDATE specification at http://www.w3.org/TR/2011/WD-sparql11-update-20110512/.

Note: This change is incompatible with some existing uses of SPARQL UPDATE.

AGWebView

Rfe11131: Allow triggering deletion in WebView

It is now possible to invoke duplicate deletion from WebView.

Rfe10631 - Allow loading files via agload in WebView

WebView now has a "Bulk Load" option to speed up loading large files. Since this option uses the agload utility internally, it also allows specifying an error handling strategy other than "Cancel".

Bug20492 - Improve code editor in WebView

The text editor in WebView (used for e.g., queries) has been improved for browser compatibility (mainly with Internet Explorer) and functionality. It now expands automatically to fit the text contained, displays line numbers and automatically indents code.

Changes to the Lisp API

Rfe11082: Transactional duplicate deletion (repeated from the General section)

INCOMPATIBLE CHANGE: AllegroGraph no longer supports duplicate deletion at index optimization time. The database creation parameters indicating duplicate deletion strategy are no longer accepted.

Instead, AllegroGraph now allows deletion of duplicates using the delete-duplicate-triples API function, which is also accessible via the HTTP API call DELETE /statements/duplicates.

Documentation

No significant changes other than new features documented.

Python Client

Rfe11122 - Add "delete duplicates" API function to python client

RepositoryConnection.deleteDuplicates:

Delete all duplicates in the store. Must commit.

mode - "spog" or "spo"

Java Client

Rfe11121 - Support transactional duplicate deletion

With this change, the Java client now provides the method AGRepositoryConnection#deleteDuplicates(comparisonMode) for deleting spog and spo duplicates in a transactional manner. See the javadoc for more details.

See Change History for user-visible changes to earlier releases.

Rfe11117: java api for SPIN (adds to rfe10988)

New methods added to AGRepositoryConnection: listSpinFunctions() listSpinMagicProperties()

New classes: AGSpinFunction, AGSpinMagicProperty

Rfe10622 - Http response handlers now have their own package

With this change, the Java client's HTTP response handlers have been refactored and moved into a com.franz.agraph.http.handler package.

AllegroGraph 4.3.3

Server Changes

Rfe10984 - Fix serialization issues with encoded IDs

Both the turtle and the RDF/XML serializers failed to correctly handle encoded IDs. This has been corrected.

Rfe10967 - Nicer CSV output option

AllegroGraph now supports another CSV output format (MIME type application/processed-csv) that uses the UTF-8 encoding, prints literals with numeric types as numbers, and doesn't wrap string literals in quote marks.

Bug20460 - Encoded IDs and reasoning triple-stores failure

If reasoning was used during a query, then encoded IDs were not correctly recognized which led to incorrect results. This has been corrected.

Bug20451 - Fix encoded ID API on remote lisp client

The remote lisp client was missing a few functions in the encoded ID API: The functions that were missing are next-encoded-upi-for-prefix, encoded-id-prefix-registered-p and encoded-id-prefix-in-use-p. These are now implemented and available on the remote lisp client.

Bug20443 - Encoded IDs did not work with the TriX serializer

The TriX serializer was incorrectly emitting encoded IDs as if they were plain literals instead of resources. This has been corrected.

Bug20416 - Use proper n-triples syntax for text/csv

The CSV generator (when requesting MIME type text/csv) for query results will now always generate valid CSV such that the entries imported into a spreadsheet or a database are in correct n-triples part syntax.

HTTP Client

No significant changes.

SPARQL

No significant changes.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe10983 - Query execution with initial bindings

With this change, the Java client's Jena adapter now supports creating query executions with initial bindings using the method: AGQueryExecutionFactory#create(query,model,initialBinding);

AllegroGraph 4.3.2

Server Changes

No significant changes.

HTTP Client

Bug20424 - Disable 5-minute HTTP request timeout

AllegroGraph now allows HTTP API requests to run (and generate results) for longer than 5 minutes.

SPARQL

No significant changes.

AGWebView

No significant changes.

Changes to the Lisp API

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Bug20272 - NPE when no response stream is available

Previously, the Java client's HTTP response handler could throw an opaque NullPointerExeption if no response stream is available (this might occur due to a server crash). With this change, the response handler checks for a null stream and throws an exception advising the user that no response stream is available and to inspect the server log file for possible explanations.

AllegroGraph 4.3.1

Server Changes

Rfe10889 - Provide guaranteed duplicate removal behavior

As long as no triples are concurrently being added, completion of a level 2 database optimization will now ensure that all duplicate triples (as defined by the current duplicate triple removal mode) are deleted.

Rfe10850 - Ag3 plugins for TBC 3.5

TBC 3.5 no longer provides built-in support for AG3 databases. With this change, AG3 plugins are made available to work with TBC 3.5. The plugins are not compatible with previous versions of TBC due to conflicts with the built-in support for AG3.

AG3 plugins for TBC 3.5 will be available from this Eclipse update site:

http://franz.com/ftp/pub/agraph/eclipse/tbc/3.5

Bug20408 - Implement encoded-ids for lisp remote-triple-store

It is now possible to register, unregister and list encoded-id prefixes when using the remote lisp client.

Bug20397 - Load triples from files faster on remote lisp client

The remote lisp client now can load large files into the server, and will no longer run out of memory when doing so.

Bug20378 - Improve duplicate reduction in UPI-map based cursors

AllegroGraph reduces the number of triples examined during SPARQL queries by skipping subsequent duplicates when iterating over a cursor. This duplicate reduction was not occuring for cursors based on secondary indices (UPI-maps) which could lead to decreased query performance. This has been corrected.

Bug20372 - Fix part printing for literals and resources with same name

AllegroGraph would incorrectly print literals that had the same name as a resource. This is now fixed.

Bug20355 - Durably save transactions on warm-standby client

The warm-standby secondary machine now saves transaction log entries in a way that ensures they've been flushed to disk durably as soon as the entire commit has been read.

Bug20336 - Fix inconsistency on warm standby secondaries

Replication secondary instances for AllegroGraph would sometimes query and attempt to display inconsistent data while replication was copying transactions from the primary instance. This has now been fixed: AllegroGraph ensures that only data from transactions that have been fully been replayed is used.

Bug20230 - Fix deletion slowdown under replication

HTTP Client

Rfe10893 - Allow "unlimited" concurrent HttpClient connections

Previously, the Java client's AGServer class would default to using an HttpClient that limited the number of concurrent connections per host to 20 (up from the HttpClient's default of 2, per RFC 2616 sec 8.1.4), which could still result in unnecessary confusion and socket churn when interacting with application level connection pool management. With this change, the defaultMaxConnectionsPerHost and maxTotalConnections are both increased to Integer.MAX_VALUE to avoid conflict with application level connection pool management.

SPARQL

Bug20382 - Sparql queries that returned `*` could confuse remote-triple-stores

If a SPARQL query returned all of its variables (e.g.. SELECT * ...), then the remote-triple-store could fail to determine the actual variables returned by the query. This could in turn lead to incorrect results from clients that use this information (e.g., Gruff). This is now corrected.

AGWebView

Rfe10924 - Enable download of server log file from WebView

WebView now allows viewing and downloading the agraph.log server file for diagnostic purposes. To open the log, select the logfile entry in the Admin menu.

Changes to the Lisp API

No significant changes.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Bug19798 - Illegal resource in Jena reified statements

Previously, the Jena adapter's createReifiedStatement method could create reified statements containing illegal blank node ids that were based on generated UUIDs, manifesting in invalid part errors from the server. With this change, these reified statements are now created using legal resources. AGGraphMaker instances can now be created with a specified ReificationStyle (the default style is ReificationStyle.Minimal).

AllegroGraph 4.3

Server Changes

Rfe10790 - improve logic for finding the sort-upi-map binary

There were cases when the Lisp client could fail to locate the path to the sort-upi-map external program. This has been corrected.

Rfe10665 - AG4 plugins for TBC 3.5.0

With this change, the AG4 plugins have been modified to work with TBC 3.5.0. The plugins are not backwards-compatible with previous versions of TBC due to changes in its API.

AG4 plugins for TBC 3.5 will be available from the update site: http://franz.com/ftp/pub/agraph/eclipse/tbc/3.5

Rfe10636, Bug20160 - signal an error if a blank node ID is too big, support for more blank nodes per store

AllegroGraph previously supported only 2^32 blank-nodes per triple-store. This is now increased to 2^56. Furthermore, if AllegroGraph tries to create a blank-node with an ID larger than the maximum supported, it will now signal an error rather than silently reusing existing nodes.

Rfe10632 - Enable per-user security-filtering of stores.

AllegroGraph now has a new class, security-filtered-triple-store, which allows specifying criteria that triples must fulfill in order to be accessible in that store.

AllegroGraph also automatically filters stores for users of a remote-triple-store if these users (or any of their roles) have security-filters defined.

Rfe10090, Rfe10541 - Optimizations to AllegroGraph secondary indices

Improves both the creation of secondary indices (UPI-maps) and their efficiency in use. Queries that make heavy use of secondary indices will operate more quickly.

Rfe10508 - Japanese full text search

An alternative text indexer for Japanese is provided. From the Lisp client, it is available by supplying :japanese as the :tokenizer keyword argument to create-freetext-index. When you use the Japanese tokenizer, we recommend you set :minimum-word-size to 2 or even 1 (the default is 3, but there are many Japanese words that consist of 2 characters).

The following is an example configuration for indexing Japanese text in literals of the object part of triples whose predicate is !<<description>.

(create-text-index "ja" :predicates '(!<description>)  
                        :stop-words nil  
                        :index-fields '(:object)  
                        :tokenizer :japanese  
                        :minimum-word-size 2)

Rfe10020 - add support for owl:SymmetricProperty

AllegroGraph's RDFS++ reasoner now supports owl:SymmetricProperty so that it will make inferences like:

P rdf:type owl:SymmetricProperty  
a P b  
===> b P a

In the process of implementing this addition, several other small efficiency improvements were made so that the reasoner is now usually faster even though it is doing more work. For example, the time to run all 14 LUBM-50 queries went from 10.04 seconds down to 9.44 seconds.

Rfe8278 - Add ability to customize the tokenizer used by freetext indexing

The Lisp API create-freetext-index takes two additional keyword arguments, border-chars and inner-chars, to fine-tune the word tokenization behavior. Each argument takes a freetext-charset object, specifying a set of characters that can appear at the borders of a word, or inside a word. A freetext-charset object can be created by a new API freetext-charset.

The following example creates a freetext index in which only alphanumeric characters and a hyphen #\- consists a word; that is, #. and #_ breaks words.

(create-freetext-index "custom" :predicates '(!<description>)  
                       :inner-chars (freetext-charset '(:alphanumeric #\-)))

Bug20289 - Do not allow stored procedures to enter the debugger

If a stored-procedure written in lisp calls the function error with a non-error condition, the AllegroGraph backend process running the stored procedure would enter the debugger. This resulted in an unresponsive backend.

Now, the backend will catch invocations of the debugger, report the failure, and continue running.

Bug20273 - Suboptimal index queries broken

Under certain circumstances, suboptimal index queries could return triples that did not match the query pattern. This has been fixed.

Bug20160, Rfe10636 - signal an error if a blank node ID is too big, support for more blank nodes per store

Bug20133 - Select queries should create filtered triple-stores only as needed

Prolog select queries were sometimes creating temporary graph-filtered triple-stores unnecessarily which wasted resources and time. This is now corrected: graph-filtered triple-stores are created only when necessary.

Bug19834 - Resource manager connection management bugs

Previously, global variables were used to maintain connection information between Lisp clients and one of the AllegroGraph subprocesses. This could cause problems if a single Lisp client made connections to two separate AllegroGraph servers or if an AllegroGraph server was stopped and restarted. This has been fixed.

Bug20037 - allow trailing slash on Server URL in AG4 dialogs

Previously, putting a trailing slash on the Server URL in AG4 dialogs would result in an error message that was hard to debug. With this change, dialogs are now robust to trailing slashes.

HTTP Client

Rfe10839 - Freetext index charsets and tokenizer in HTTP API

The HTTP API for freetext index management now supports the optional arguments "tokenizer" as well as "borderChars" and "innerChars", which specify the tokenizer and character sets for word boundaries.

Bug20250 - Fix federated remote stores in HTTP API

The /session resource didn't correctly create sessions when specified using the <http://example.com/repositories/test> syntax. Now, POSTing to /session correctly opens a session on a remote triple store.

SPARQL

Bug20305 - Encoded IDs were incompatible with SPARQL CONSTRUCT queries

SPARQL CONSTRUCT queries will not produce triples whose subject or predicate are literals. The check for this was incorrectly excluding triples with encoded IDs in the subject (or predicate) because these IDs have a different UPI type-code that the one used for resources. This has been corrected.

Bug20288 - Encoded IDs don't work with SPARQL/XML and SPARQL/JSON output

The SPARQL/XML results format would crash when used with encoded IDs; The SPARQL/JSON results format generated output but emitted encoded IDs as typed literals rather than as resources. Both of these problems are corrected.

AGWebView

Rfe10728 - Improve WebView store overview user interface

The AllegroGraph WebView store overview page now is better structured and allows easier access to commonly-used functions.

Bug20304 - Fix codemirror for Internet Explorer 9

Internet Explorer 9 users can now use WebView's text editor with syntax highlighting.

Bug20302 - Fix loading more results in Internet Explorer

WebView can now load more than 100 results (using the [more] link) in Internet Explorer.

Bug20296 - Fix query result downloads in Firefox 3.

Fix a JavaScript incompatibility that caused downloading query results from WebView to fail in Firefox version 3.

Bug20203 - Fix WebView for newer Internet Explorer versions.

Internet Explorer 8 and 9 support a new history tracking mechanism, and deprecated features that WebView used to keep track of history before. Now, WebView makes use of this new mechanism when it's available. This allows Internet Explorer to correctly run AllegroGraph WebView again.

Bug20069 - Uploading file into a context from WebView is broken

The error " is not a valid part" would always appear. This is now fixed.

Changes to the Lisp API

Bug20146 - Export UPI hash-table functions from db.agraph.user

In the transition from AllegroGraph 3 to AllegroGraph 4, the UPI hash-table functions ceased being exported from db.agraph.user. This is now corrected.

Bug20145 - (Lisp API only) loading AllegroGraph should `provide` module :agraph4.

AllegroGraph previously only provided the module :agraph which meant that using

(require :agraph4)

could load agraph4.fasl unnecessarily. This is fixed. AllegroGraph 5.0 note: the module to require is :agraph5 and the file is agraph5.fasl starting in version 5.0.

Bug20144 - Fix interference from remote-triple-store ping

To ensure that a remote-triple-store remains opened, AllegroGraph's lisp API sends a ping message in the background in regular intervals. Previously, it would use the same session object to record responses from the server as a user's requests. This would cause errors when a ping message interrupts a long-running request.

Now, AllegroGraph uses a separate session object on the client's side to send these ping messages.

Documentation

Bug20149 - Correct documentation of datatype and predicate mappings (Lisp)

The Lisp reference manual had several typos in its discussion of datatype and predicate mappings. These have been fixed.

Python Client

No significant changes.

Java Client

Rfe10135 - Improve javadocs

Added overview pages, more docs for classes and methods, and links to AllegroGraph documentation for introduction, catalog, federation, etc.

Rfe10745 - Add Jena text indexing code example

With this change, a Jena text indexing code example is now available in the distribution in JenaTutorialExamples.java as example 12; this will eventually also be written up in the Jena tutorial document, but the focus here is on making the code example available as it is useful on its own.

Rfe10624 - AGQuery limit and offset modifiers

With this change, the Sesame and Jena AGQuery classes now allow limit and offset to be specified by the query object, outside of any limit or offset specified in the query string, making it easier for apps to override these modifiers.

Rfe10623 - Jena AGQueryExecution count methods

With this change, the Jena api now supports counts queries via the AGQueryExecution#countSelect() and countConstruct() methods. Solutions are counted server-side and only the count is returned to the client.

Rfe10492 and Rfe10493 - javadoc for AGRepositoryConnection

Add javadoc that points to SessionPorts for relevant methods. Add javadoc for type mappings, linking to http and lisp docs.

Rfe10452 - support N-Quads responses in the Java client

With this change, the Java client now supports N-Quads responses from the server. The property -Dcom.franz.agraph.http.defaultRDFFormat=NQUADS can now be used to configure the client to request N-Quads format by default for RDF statement responses; when this property is not present, the default format is TriX. The method AGHttpRepoClient#setPreferredRDFFormat() can also be used to override the default RDFFormat on a per connection basis.

Rfe10450 - Jena Prolog and transaction code examples

With this change, Jena Prolog and transaction examples are now available in the distribution in JenaTutorialExamples.java as examples 17, 18, and 22; they will eventually also be written up in the Jena tutorial document, but the focus here is on making the code examples available as they are useful on their own.

Rfe10447 - add AGGraphQuery#count() method

With this change, the java client now supports counting for graph queries (e.g CONSTRUCT and DESCRIBE) with the AGGraphQuery#count() method. Also added javadoc for the AGTupleQuery#count() method.

Rfe10446 - add count() examples to the Java tutorial

With this change, the Java tutorial shows how to use the AGTupleQuery#count() and AGGraphQuery#count() methods to to obtain query result counts without returning results (counting is done server-side, only the count is returned).

Rfe10445 - add example/test getting triple ids via prolog

With this change, example17 in TutorialExamples.java shows how to get triple ids via Prolog. TripleIdTests has also been augmented to get triple ids via Prolog.

Rfe10411 - Streaming large query results

When AGTupleQuery.evaluate() is used for sparql and prolog queries, the results are returned by the server in sparql/xml format. This is parsed by the Sesame library using a SAX XML parser and the results are collected in an ArrayList before the TupleQueryResult is returned. For very large result sets, this will cause an OutOfMemoryError.

Alternatively, AGTupleQuery.evaluate(handler) can be used where the handler methods are called as results are parsed. This can handle any length results without a memory error, but the API is less convenient. A new example has been added to the tutorial to demonstrate this: TutorialExamples.example3a()

A new class AGStreamTupleQuery can be used to wrap an AGTupleQuery so the TupleQueryResult will be streamed back. This provides the familiar iteration API rather than a handler, but the http response is only parsed as the iterator is advanced. The usage of this API is explained further in the javadocs: AGTupleQuery query = conn.prepareTupleQuery(QueryLanguage.SPARQL, "SELECT ..."); query = new AGStreamTupleQuery(query); TupleQueryResult results = query.evaluate();

There is also a new method in AGRepositoryConnection that will cause the connection to automatically wrap all tuple queries to use streaming: AGRepositoryConnection.setStreamResults(boolean streamResults)

Instead of SAX, the AGStreamTupleQuery uses an XMLStreamReader for parsing. Initial performance results suggest that in addition to being resistant to OutOfMemoryError this uses less overall memory than the SAX parser for small and medium result sets (10-500).

This approach of streaming results has not yet been applied to getStatements().

Rfe9445 - Upgrade slf4j, funnel through jcl, and add source jars

With this change, the java client now uses slf4j-api-1.6.1, and uses slf4j-jcl-1.6.1 to funnel sesame and jena logging through commons-logging-1.1.1 (used by httpclient-3.1). Source jars are also included for javadoc and to faciliate debugging.

AllegroGraph 4.2.1c

Server Changes

Clarify what retain for replication means

If the TransactionLogRetain is set to replication then all transaction logs will be saved until replication starts at which point those log files no longer needed will be archived. This behavior may be further modified in a future release.

Bug20257 - fix convertdb for some rare cases

The convertdb utility failed to upgrade some stores that had transaction logs with a certain size. Now it can correctly upgrade these stores.

Bug20236 - Fix startup when HTTPProxy is specified in config

AllegroGraph now starts up correctly when a HTTP proxy specification is present in the configuration file.

Bug20223 - Fix agload for Turtle files

A change in agload broke loading Turtle-formatted files with agload. Now, agload can correctly load the Turtle format again.

HTTP Client

No user-visible changes.

SPARQL

No user-visible changes.

AGWebView

No user-visible changes.

Bug20200 - Clarify format support in import forms in WebView.

AllegroGraph now supports loading .gz-compressed files (and autodetecting their file format) from the server's local file system. The file import dialogues in WebView also have a note about the file name format for autodetection.

Changes to the Lisp API

Rfe10790 - improve logic for finding the sort-upi-map binary

There were cases when the Lisp client could fail to locate the path to the sort-upi-map external program. This has been corrected.

Documentation

No user-visible changes.

Python Client

No user-visible changes.

Java Client

No user-visible changes.

AllegroGraph 4.2.1a

Server Changes

Rfe10689 - Increase unique string limit to ~4B

The limit on the number of unique strings in the string table was previously 1,431,655,765. This limit has been increased to 4,294,967,295 entries.

Rfe10562 - Change encoded-id marker to @@ (from @#)

The marker used to indicate that a URI represents an encoded-ID has been changed from @# to @@.

Rfe10532 - Return encoded ids in ascending order

The .../encodedIds?prefix=""&amount=# http request was returning the ids in reverse order. This has been fixed to return them in ascending sequential order.

Rfe10521 - Deleted hash table management based on generation

Deleted hash tables are now cleaned up as soon as they are no longer needed, preventing the store from accumulating unnecessary files.

Rfe10200 - Regulate speed of mergers

Triple index mergers now regulate their speed based on the number of triples to be merged and the rate at which triples are added after a merge has started.

AllegroGraph 4.2.1.

Server Changes

Rfe10483, Rfe10528, Bug20012 - Inadequate transaction log durability

When using the default TlogSyncMethod of "odirect", the durability of transaction log files was not correctly ensured. This release fixes that issue by preallocating transaction log files and fsyncing them before using them. Additional transaction log files are created as needed. In addition, transaction log files that are no longer needed are now recycled instead of deleted.

This change will have an impact on database creation time due to the initial preallocation. To compensate for this, the default TransactionLogSize parameter has been changed from 1 gigabyte to 256 megabytes.

There is a new configuration parameter DesiredTlogFiles which specifies the number of transaction log files which should be preallocated at database creation time. The default value is 2. Specifying a larger value helps lower the probability of additional transaction log files being created during commits. This parameter is documented in Server Configuration.

Rfe10475 - Improve I/O behavior under memory pressure

When under memory pressure, Linux can sometimes generate far more I/O than expected when minor page faults occur during accesses to memory mapped files. This can result in very poor performance, especially when the I/O device is already congested. This change instructs the operating system not to perform these extra I/Os.

Rfe10439 - RDSF++ reasoner now does not create duplicate inferred triples

In some cases, the RDFS++ reasoner could follow multiple owl:sameAs paths and generate many duplicate inferred triples. This increased AllegroGraph's memory footprint and decreased RdFS++ reasoning performance.

The reasoner now generates only a single inferred triple for each owl:sameAs path when searching for sameAs statements with a fixed subject and predicate or for ones with a fixed predicated and object.

Rfe10434 - Rename Lisp API function to duplicate-deletion-strategy

The Lisp API function duplicate-deletion-strategy alters the way AllegroGraph manages duplicate triples during index optimization operations. It can be one of:

t - the same as :spog (see below).
nil - do nothing to duplicates.
:spo - ensure that no more than one triple with a given Subject, Predicate and Object exists after index optimization operations are complete. A triple may be deleted if it differs from another only in its graph.
:spog - ensure that no more than one triple with a given Subject, Predicate, Object and Graph exists after index optimization operations are complete.

Previously, this function was called delete-duplicates-during-merge-p.

This setting is also available via the HTTP interface.

Rfe10413 - Add check for 64-bit Linux

Added a check to install-agraph and configure-agraph scripts to ensure that they're running on a 64-bit Linux. This should reduce confusion in the case where AllegroGraph Server was accidentally installed on a 32-bit Linux host.

Rfe10400 - Minor cursor implementation changes

Some changes were made to enhance AllegroGraph's cursor mechanisms. These do not alter existing cursor behavior but help pave the way for future improvements.

Rfe10399 - Verify lisp client and server version match

As the lisp client is very sensitive to changes in the backend API exposed by the AllegroGraph server, it now checks that it is connecting to the correct version of the server.

Rfe10363 - Improved error message when a file cannot be found

The error message generated when a database file is not found has been improved to make it more clear what is causing the problem.

Rfe10354 - Redesigned --quiesce feature of agtestclient

The quiesce functionality in agtestclient was unnecessarily complicated and inefficient. The feature has been redesigned to require running on the host of the secondary server in question and also require the expected count and a timeout. This is sufficient for the purposes for which the quiesce function is used. Expected count must either be queried from the primary or the triple-count must be known beforehand.

A bug was also fixed with the order of the arguments in the delete function. (They had been reversed.) Also, help documentation was created, which can be accessed by running agtestclient with no arguments. Finally, the --add argument was renamed to the more appropriate --add-random.

Rfe10345 - Don't log warnings when opening NFS triple-stores.

Opening a triple-store on NFS caused unnecessary warnings to be issued. These warnings are no longer logged.

Rfe10342 - Command line wrapper for the lubm-generator Lisp module

Note: lubmgen can easily contaminate or erase a production database if used without clear understanding of its purpose.

Rfe10337 - Improve cursor resource management

Improve the management of the cursors AllegroGraph uses internally for triple counting and triple deletion.

Rfe10335 - Improve logging of (adjusted) store parameters

AllegroGraph will now log human-readable configuration values when opening a store if a human-readable presentation is available. (The system determines whether a value has a human-readable presentation or not. That is not under user control. Values which do not have a human-readable presentation are also logged but not in a human-readable way.)

Rfe10308 - AllegroGraph can use UPI-maps to run queries in more cases

Increased the number of times when AllegroGraph can use UPI-maps (secondary indices) to run queries. This improves the enhancements made in Rfe10141 and Rfe10142.

Rfe10243 - opening database should honor existing merge mode as default

It may have been possible that the duplicate removal mode is rolled back accidentally by reopening the database. The problem is fixed.

Rfe10184 - Improve custom hash table efficiency

Improve the way AllegroGraph handles some internal hash table computations. This will provide some very small performance improvements to general AllegroGraph operations. Note that none of these operations occur in inner loops so the performance gains will not be markedly noticeable.

Rfe10114 - Change default for TransactionLogRetain to "recovery"

In prior releases, the default value for the TransactionLogRetain setting was "all", which meant that, unless configured otherwise, all transaction log files since the the database was created would be saved. This release changes the default to "recovery", which means that only the transaction log files required for instance crash recovery are retained. See Transaction Log Archiving.

Rfe9338 - Improved error message due to session port connection failure

When using methods that start a session (setAutoCommit, addRules, and registerSNAGenerator), if there is a connection error, the error message points to the documentation about SessionPorts.

Bug20122 - Handle disk full on transaction log writing

Previously, encountering a full filesystem while writing to a transaction log would result in a crash. Now, AllegroGraph will (if possible) log a message to the logfile and to stderr noting the problem and will periodically retry the write operation until it succeeds.

Bug20118 - graph type-mapping could lead to incorrect data

If a triple was added with a graph field that was mapped to an encoded UPI (for example, a geospatial point or a typed literal) and the object field was not mapped (for example, a plain literal or a resource), then AllegroGraph would fail to correctly store the triple's object. This has been fixed.

Bug20089 - encoded-ids not working in non-object fields for clients

AllegroGraph was failing to translate encoded-IDs in fields other than the object field for any non-direct Lisp client. This has been fixed.

Bug20050 - Improve speed of bulk load mode in agload

agload was incorrectly handling the --bulk parameter which caused it to perform significantly more slowly when bulk mode was turned on. With this change, bulk mode is 6 to 7 times faster than it was before.

Bug20031 - agraph-backup backs up the same file twice

Previously, agraph-backup was backing up one of the database files twice. Though this did not result in a corrupt backup, it was wasting time and space. This problem has been fixed.

Bug19996 - Repair durability guarantees for NFS stores.

The change to implement rfe10216 in AllegroGraph 4.2 accidentally broke durability guarantees on NFS stores: Transaction logs on NFS were opened in an asynchronous write mode, which could lead to commits getting lost in the case of a crash.

Bug19969 - Correct problem printing times with fractional seconds

If a fractional second had a value less than 0.001, then AllegroGraph would incorrectly print the time (or dateTime) using scientific notation. It now prints these values correctly. E.g., it now prints 16:46:40.000133 rather than 16:46:40.1.33e-4.

Bug19954 - Commit/rollback could sometimes result in crash

Fixed a programming error which could potentially cause a commit or rollback to result in a program crash.

Bug19950 - Handling of tranaction log files corrected

In this release the 'recovery' and 'archive' values for the TransactionLogRetain catalog parameter function correctly. Previously these two values were treated as if 'all' were specified.

Bug19933 - Made agload recognize "nq" file format (nquads).

Previously, agload would reject files with extension .nq. Now they are correctly identified as nquad files and loaded.

Bug19916 - Improve :terse output

Modified the heuristic for the :terse output so that it will not emit empty strings. Previously, a URI like 'http://www.foo.com/bar/' would have a terse representation of ''; now it will use 'bar/'.

Bug19836 - Date and datetime with negative year

XSD requires BCE dates, for example, these are now accepted:

(value->upi "-0002-04-12" :date)  
(value->upi "-0002-04-12T12:23:00Z" :date-time)

Bug19682 - Federated-triple-stores error for type-mappings

Type mappings can now be used with federated stores where all component stores have the same type mapping, for example in methods such as collect-all-type-mappings and get-triples. Function collect-all-type-mappings returns the intersection of mappings for all component stores.

Mappings can still only be set on component stores, not a federated store.

Bug19116 - Datatype-mappings are not applied to graph

When adding a triple, if the graph is a literal with a datatype that can be mapped like object, it will be. Subjects and predicates are not mapped at all.

At most 3 triples can be added, which is the same as before:

with object and graph mapped by datatype
with object mapped by predicate-mapping and graph by datatype if this results in a different triple than #1
with no mappings if :preserve-strings is t

HTTP Client

Rfe10360 - Better HTTP error messages for encoded-id errors.

Stack traces are no longer logged for too-wide encoded-id formats or bad re-registration of such ids.

Bug20124 - Make number of initial HTTP workers configurable.

The number of initial HTTP workers started by the AllegroGraph server used to be exactly the number of backends. However, that doesn't take into account any frontend sessions like WebView. This would cause long-running requests on startup (like opening a triple store) to cause shorter-lived requests to wait until the longer-running request was finished.

AllegroGraph now has a new top-level configuration parameter HTTPWorkers (defaulting to 50), which lets administrators specify how many HTTP worker threads to launch on startup.

Bug20022 - Add location-rewriting to the AG reverse-proxy.

The HTTP reverse-proxy that dispatched requests to server back-ends did not handle redirect responses with Location headers in any special ways. This could sometimes lead to clients being redirected to a backend URL, which did not work. The proxy now properly rewrites the Location header to point at the front-end URL.

Bug20013 - Requests to back-ends with Expect: headers fail

Due to a bug in the HTTP system, sending a request with a body an Expect: header (which CURL does by default) would cause a deadlock in the intra-server communication, and thus a timeout. This has been fixed.

Bug19948 - Allow full HTTP delete-duplicates control

The HTTP API used to expose deleteDuplicates as a boolean value, but there are actually three states: off, delete by SPO, and delete by SPOG. The strings "spo" and "spog" (as well as the existing booleans) can now be used in the HTTP API when inspecting or setting duplicate deletion.

SPARQL

Rfe10486 - SPARQL query planning should not require the PSOGI index

When a query involves filtering on potentially encoded triples (e.g., numeric or date ranges), then the SPARQL algebra query engine checks to see if certain ranges of triples exist during its planning phase. The checks that SPARQL makes requires the POSGI index and are inordinately slow if that index is not available. Previously, AllegroGraph would make the checks regardless. Now, it will now skip these checks if the index is not there.

Rfe10374 - Improve efficiency of some SPARQL queries

SPARQL's notion of equality requires AllegroGraph to examine several variants of some IRIs. For example, the literal "4" should be equivalent to the literal "4"^^xsd:string.

This change improves the efficiency of AllegroGraph's SPARQL equality computations. The performance improvement will depend on the query.

Rfe10371 - Improve SPARQL's handling of DISTINCT LIMIT queries

Improve SPARQL's performance for queries with a LIMIT whose solutions must be DISTINCT.

Rfe10348 - Iri= for subject and predicates (generalize SPARQL constraints)

Extended IRI= constraints for subjects and predicates. This allows queries like:

select * where { ?s ?p ?o . filter( ?s = ex:foo ) }

to operate as if they were written:

select * where { ex:foo ?p ?o . }

If multiple constraints are simultaneously operative, then only one will be used for a given pattern. Currently, these will be selected in the order: subject, object, predicate.

Queries for which this optimization is relevant will see greatly increased speed.

Rfe10344 - Improve SPARQL statistical BGP reordering logic

The statistical BGP reordering function could produce sub-optimal results if a SPARQL query had multiple levels of nested UNIONs or OPTIONALs. This could cause queries to take much longer than necessary.

Rfe10312 - Optimize IRI= for object fields in SPARQL queries

This optimization converts IRI= constraints for object fields in SPARQL queries into the equivalent storage level get-triples calls. Previously, a query like

SELECT ?article  
WHERE { ?article rdf:type ?type .  
        ?article swrc:pages ?value  
        FILTER (?type = bench:Article ) }

would fetch all of the rdf:type triples and then filter for those whose object field was bench:Article. Now the query will be executed as if it had been written as

SELECT ?article  
WHERE { ?article rdf:type bench:Article .  
        ?article swrc:pages ?value  
 }

Rfe10118 - Added interface to SPARQL query analysis to Python client

After calling RepositoryConnection.prepareTupleQuery() on a SPARQL select query, one can call analyze() on the resulting query object to get a string returned with information about which indices would be used to satisfy the query.

Currently, the query object's method is:

def analyze(self, analysisTechnique=None, analysisTimeout=None):

Analysis is only available for SPARQL queries.  
 
analysisTechnique defaults to "executed", which executes  
the query to perform dynamic analysis. "static" analysis is  
the other option.  
 
For analysisTimeout, pass a float for the maximum number of  
seconds to run the query if executed.

Rfe9507 - Add initial support for SPARQL subqueries

SPARQL subquery support includes ORDER BY, LIMIT, and DISTINCT. An example query would be:

PREFIX fn:    <http://www.w3.org/2005/xpath-functions#>  
PREFIX boat:  <http://example/boats/vocab/>  
PREFIX boats: <http://example/boats/>  
SELECT ?boat ?boatName  
FROM boats:boats1  
FROM boats:boats2  
WHERE {  
  # Select the 2 boats with the shortest names.  
  { SELECT ?boat WHERE {  
    ?boat boat:placed ?position ;  
          boat:name ?name .  
    } ORDER BY asc(fn:string-length(?name)) LIMIT 3 }  
 
  # Select their crew and the boat's name.  
  ?boat boat:name ?boatName .  
}  
ORDER BY desc(fn:string-length(?boatName)) desc(?boatName)

Note that subqueries are a pre-release SPARQL 1.1 feature; documentation and conformance tests are not yet available from the W3C, semantics are subject to change, and the implementation might be incomplete in a manner that is visible to user code. Please contact Franz support if you suspect that you have found a discrepancy.

Bug20070 - fix problem with geospatial encoded UPIs and some output formats

If a SPARQL query returned geospatial encoded UPIs and used the SPARQL-XML result format, then an error would occur. The same error could occur when serializing geospatial UPIs in the turtle format.

There was also a problem correctly serializing geospatial output using the SPARQL-JSON format: incorrect latitude and longitude values were sometimes generated.

All of these problems are corrected.

Bug20047 - Can't download query results with missing values as CSV

Taking a SPARQL query that produced unbound variables (say, with OPTIONAL) and asking for a comma-separated value return format would return an error. This has been fixed.

Bug19989 - Sparql CONSTRUCT could generate invalid blank nodes

SPARQL CONSTRUCT using the N-Triples format could generate invalid blank nodes.

Bug19832 - Correct bug in low level SPARQL equality code

Correct a bug in some rarely executed SPARQL equality code. The bug would have caused some queries in some datasets to fail to run.

AGWebView

Rfe10370 - Use only relative URLs in WebView

This makes it possible to set up a reverse-proxy that proxies, for example, host:80/allegrograph/ to host:10035/, without breaking WebView.

Bug20051 - Downloading query results from WebView was broken

After recent patch (bug20022), trying to download query results from WebView always resulted in a 404 response. This fixes that.

Bug20016 - Fix horizontal scrolling in WebView for Firefox

A bad CSS rule was preventing Firefox from ever displaying a horizontal scrollbar for WebView. This made it hard to inspect, for example, query result tables that were wider than the window. This is now fixed.

Bug20009 - 4.2 WebView can't handle some data saved by 4.1

Saved queries saved in 4.1 would cause an error when used in 4.2, because of a change in the way the client-side code is handling JSON data. This change fixes that, by falling back to the old parsing method if the new one fails.

Bug19972 - Webview breaks when it doesn't find JSON support

On older browsers without a JavaScript JSON object, WebView's detection of that object was failing, causing the whole script to abort. It now properly detects, the functionality, replacing it when not present.

Changes to the Lisp API

Rfe10420 - Add lisp client function optimize-indices

AllegroGraph's Lisp client now has a function optimize-indices that will cause the server to rearrange the indices for optimal query performance. This optimization pass can take a long time and cause a lot of I/O operations to be performed on large stores.

Rfe10066 - Add option to use rapper for load-turtle

The lisp API load-turtle now has a :use-rapper-p keyword parameter. See load-rdf/xml documentation for argument usage details.

Rfe9715 - Lisp client package now includes Lisp tutorial examples

The subdirectory tutorial now holds the Lisp and data example files discussed in the documentation.

Bug19672 - Triple-store-exists-p leaks server information

The Lisp client's successful return value from triple-store-exists-p sometimes provided the store's path on the server's filesystem.

The API has been fixed to return only the values nil or t.

Bug19133 - get-triple now applies geospatial mappings

get-triple now correctly applies geospatial mappings to the triple returned. Also, the get-triple now returns a fresh triple (rather than one which might be reused by internal mechanisms).

Documentation

New JavaScript document for programming the server

See JavaScript.

New Security document describing data security

Rfe10341 - Improve documentation search

Added basic documentation searching to the AllegroGraph documentation.

Python Client

Rfe10505 - use session to get new blankNodeIds

AGValueFactory was using the main repository url to get blank node ids. Use connection session instead if available. Using the session can reduce load on the primary server.

If the ValueFactory was created by Repository instead of RepositoryConnection, the old behavior remains.

Rfe10418 - Call indices/optimize from Python

Added an optimizeIndices(level, wait) method to RepositoryConnection for exposing the http indices/optimize POST request to Python client.

Rfe10326 - Support deleteDuplicates setting from Python client

The Catalog.createRepository function now has a deleteDuplicates keyword argument for setting the store's deleteDuplicates behavior on creation. If unspecified, the behavior is the server's default setting.

The store's deleteDuplicates setting is also available from a Repository object instance either as a property variable (delete_duplicates) or as setDeleteDuplicates and getDeleteDuplicates methods. This is a per-store setting.

See the HTTP Protocol - SPARQL Endpoint documention for setting values and behaviors (text search for deleteDuplicates).

Rfe10325 - Added evalJavaScript to Python client

Added an evalJavaScript function to the RepositoryConnection class for evaluating server-side JavaScript code.

See the server's JavaScript documentation for details on programming the server in JavaScript.

Java Client

Rfe10389 - Upgrade to sesame 2.3.2

With this change, the java client distribution has been upgraded to use openrdf sesame 2.3.2, a minor upgrade from 2.3.1.

Rfe10373 - Support deleteDuplicates mode in java

With this change, the Java Client supports setting a repository's policy for deleting duplicates using methods AGRepository#setDeleteDuplicatesMode and AGRepository#getDeleteDuplicatesMode.

Rfe10372 - Support periodic commit during add/load

With this change, the java client now supports the /statements commit parameter for periodic commits during add/load via the AGRepositoryConnection's setUploadCommitPeriod and getUploadCommitPeriod methods.

Rfe10257 - Support script loading in sessions

Added method to AGRepositoryConnection: addSessionLoadScript(String scriptName)

When a dedicated session is created, the named scripts on the AllegroGraph Server will be loaded.

Rfe10256 - Support loadInitFile switch in java

Added method to AGRepositoryConnection: setSessionLoadInitFile(boolean loadInitFile)

When a dedicated session is created, the initFile on the AllegroGraph Server will be loaded if this param is true.

Javadocs on AGRepositoryConnection reorganized to describe sessions.

Rfe10201 - Support adding N-Quads from java

With this change, the java client now supports adding N-Quads data from Sesame and Jena using the RDFFormat AGRDFFormat.NQUADS and the language "NQUADS", respectively.

Rfe10177 - Support getting statements by id

With this change, the java client now supports getting statements by triple id using the AGRepositoryConnection getStatements(String... ids) method. Note that there is currently no method for obtaining a statement's id in the java client, so this api is for experimental use by developers and subject to change.

Rfe10174 - Support adding Turtle from Sesame and Jena

With this change, Turtle formatted data can now be added to a repository using the RDF4J and Jena APIs.

Rfe10059 - SPOGI triple cache management

With this change, the SPOGI-cache can now be managed from the java client using AGRepositoryConnection methods enableTripleCache(size), getTripleCacheSize(), and disableTripleCache().

Rfe9436 - Session lifetime management

With this change the lifetime (maximum idle time) of a session can be set using the AGRepositoryConnection#setSessionLifetime method. The default session lifetime (in effect when not set with the method above) can be set with the AGHttpRepoClient.setDefaultSessionLifetime method. When none of these are set, the default session lifetime is 1 hour.

Rfe8362 - Allow geospatial queries against graph/context field

A boolean useContext parameter can be given to /geo/box, /geo/circle, /geo/haversine, and /geo/polygon, to make them query the context field, rather than the object field of the triples.

Bug19906 - Writing a JavaScript NaN value as JSON loops forever

When running, for example, the JavaScript program 'NaN', and asking for a JSON response, the server would never respond, because the float-printing code didn't handle 'special' float values. We now convert NaN and Infinity to strings in JSON output.

Bug19342 - Java LUBM Prolog and SPARQL queries

With this change, Java LUBM Prolog and SPARQL queries are updated to use distinct in appropriate places and located in the test.lubm package as AGLubmProlog and AGLubmSparql; ant targets lubm-prolog and lubm-sparql have been added to facilitate running the queries from the command line.

AllegroGraph 4.2

The new (in 4.2) document Release Notes has information about the 4.2 release. The document you are reading simply lists, for the current and for previous releases, user-visible changes (new features, bug fixes, and other modifications) in an abbreviated format. All the 4.2 changes listed here are also listed in the Release Notes but that document does not list changes in earlier releases. We suggest that users start with that document and refer to this one only if they need information about earlier releases.

Welcome to AllegroGraph 4.2. See Release Notes for general release information. The following, organized by category, are user-visible changes in AllegroGraph 4.2.

Server Changes

Rfe10293 - Warn against using ReiserFS.

As ReiserFS causes long delays on checkpoints, AllegroGraph now warns (in the server log file) if a store resides on ReiserFS. To ease debugging, it also logs the file system types of every store directory now.

Rfe10267 - Do not request duplicate removal if there are no duplicates detected

Internal handling of duplicate removal process is slightly improved. This fix also eliminates excessive "Dupes deleted: 0" messages from the agraph.log file.

Rfe10255 - Allow minimum text-index word size of 1

It is now possible to create freetext indices with a minimum word size of 1 character (the minimum used to be 2).

Rfe10216 - Support alternative tlog synchronization methods.

AllegroGraph now supports a new per-catalog configuration option, TlogSyncMethod, which allows users to specify one of three synchronized writing methods for transaction logs: ODIRECT, SYNC, and fsync.

As before, ODIRECT is the default and the recommended choice on ext3 file systems. For catalogs residing on non-ext3 file systems, the other choices may yield performance benefits.

Rfe10191 - Have install-agraph pass its arguments to configure-agraph

You can now do './install-agraph /dir/ --non-interactive ...' to install and configure agraph in a single command.

Rfe10182 - Improve efficiency when waiting for an in-memory chunk

Previously, when waiting for an in-memory chunk to become available, a commit operation would poll once per second. The polling has been replaced with a wait on a semaphore which will result in immediate notification when an in-memory chunk becomes available. This results in lower average times for commits that must wait for in-memory chunks.

Rfe10173 - Upgrade plugins to support TBC 3.4.0-Beta1

Previously the AllegroGraph plugins were only supported on TBC 3.3.x; with this change, the plugins can also be used with TBC 3.4.0-Beta1.

Rfe10167 - Log euid and runas information at daemon startup.

AllegroGraph now logs which user started the AllegroGraph service daemon, which user it was meant to switch to once running, and the currently effective user ID.

Rfe10162 - Perform a clean shutdown on agraph-control stop

Previously, running the agraph script with the stop argument would not verify that open databases were shut down correctly, leading to a forced recovery from transaction logs upon next startup.

Now, running the agraph script with the stop argument will wait for up to 60 seconds for open triple-stores to shut down cleanly.

To get the quicker shutdown behavior, use force-stop argument or kill the Service Daemon process with the QUIT signal.

Rfe10156, Rfe10157 - Improve lock acquisition and store creation on NFS.

Previously, creating a triple-store left open a very narrow window in which another process could clobber the store parameters. Now, the triple-store is locked immediately on creation. Additionally, newly-created triple-stores can now be cleaned up correctly when created on NFS.

Rfe10149 - Support setting a graph for inferred triples in HTTP sessions

Using the syntax <store>[rdfs++#] it is now possible to specify which graph should be used for the inferred statements in a reasoning store created as an HTTP session.

Rfe10147 - Export 'custom-service' macro from the db.agraph package

This macro was crudely inserted directly into db.agraph.user before. It is now imported there from db.agraph, like all other AllegroGraph functionality.

Rfe10143 - support bulkMode in connections

With this change, connections can now be put into bulkMode. When in bulkMode, data can be added/loaded more quickly, but there is no guarantee of durability in the event of a crash. Added method AGRepositoryConnection#setBulkMode to set the connection's bulkMode (defaults to false), and added method AGRepositoryConnection#isBulkMode to check the mode.

Rfe10141, Rfe10142 - Use UPI-maps more often inProlog queries

Improved the Prolog query optimizer to use UPI-maps in more cases. Also added a new function, qm, that will use a UPI-map whenever possible. The optimizer changes will help existing queries run more quickly; the new functor will allow more hand-tuned query optimizations.

Rfe10117 - implement run-time (dynamic) query index analysis

Query index analysis now comes in two flavors: the existing :static flavor (from rfe10075) and the new :executed flavor. The later provides an accurate account of the index usage because it actually runs the query while monitoring which indices are touched. Because the query is run, this flavor of analysis can be much more time consuming.

Note that the default analysis technique is to use the new :executed flavor. You can use a different technique by using the queryAnalysisTechnique query parameter. You can also specify a timeout using the queryAnalysisTimeout parameter. See Query Analysis.

Rfe10069 - AllegroGraph v4.0.6 introduced a dependency on a patch to update message digest functions. Now, loading the AllegroGraph client will fail early if this patch is missing, and will report an actionable error.

In addition to the check for the SHA1 patch, AG will now report missing patch names to the user, and offer to install these patches specifically.

Rfe10021 - Support Soundex freetext indices

This adds a supported freetext wordfilter 'soundex', which will reduce words to their Soundex equivalent when indexing or searching. Soundex is described in this Wikipedia article.

Rfe9877 - Lazier collection of FTI results

The triple-ids matching a free-text query will no longer all be accumulated at the start of the query, but (as far as possible) lazily, while triples are being fetched. This should improve performance of limited queries with lots of results.

Rfe9861 - Improve method of sending an x-rdftransaction

Previously, sending an application/x-rdftransaction request required escaping to the client's http layer. With this change, the method AGRepositoryConnection#sendRDFTransaction can now be used; this method is useful for bundling add/remove operations into a single request and minimizing round trips to the server. Changes are committed iff the connection is in autoCommit mode. For increased throughput when sending multiple rdftransaction requests, consider using autoCommit false and committing less frequently.

Rfe9746 - Various improvements to agload

Added the --rapper command line option to tell agload to use the external rapper program when parsing RDF/XML.
Added :trix to the list of supported input formats
Added :save as an error strategy to write errors to the file agload.log and continue.
Added --recursive to process source directories recursively.

Rfe9657 - Unattended execution of configuration script

The configure-agraph script now has --non-interactive mode, where all its parameters are specified on the command line, so that it can be called from a script without human intervention.

Rfe9631, Rfe10028 - db-get-triples improvements, UPI map usage

Improved the performance of AllegroGraph's internal get-triples functors and extended them to use secondary indices (UPI maps) when they are available.

This greatly improves the performance of many standard queries for both SPARQL and Prolog select. For example, the SP2 benchmark for 10,000 authors goes from a total time of 112.58 seconds to 37.91 seconds.

Rfe9622 - support getting the server's version information

With this change, the client can now get the server's version, build date, and revision information using the AGServer methods getVersion(), getBuildDate(), and getRevision(), respectively.

Rfe9517 - Unify AServe and AGraph messages in the AGraph log

Our HTTP server used to output log messages in its own format. We now route these through the AGraph logging system, causing them to follow a unified format.

Rfe9515 - Drop deleted triples from text index during merge.

The freetext index used to never un-index anything it had indexed. It now leaves out triples that have been deleted when merging index chunks, which will prevent unbounded index growth in stores where lots of triples are being deleted.

Rfe9470 - Reduce madvise calls during compressed index lookup

Small performance improvement during load and query.

Rfe9296 - Transactionally safe duplicate triple elimination

There are improvements in the functionality to remove duplicate triples. Removal is done more eagerly and more robustly. When opening a database, you can pass one of the following values as the :delete-duplicates-during-merge argument:

nil - do not remove duplicates (default)
:spog - duplicate triples with the same s, p, o and g are deleted.
:spo - duplicate triples with the same s, p, o are deleted
t - same as :spog, for backward compatibility.

Rfe8414 - Add encoded-ids to AllegroGraph

Adds support to register and query encoded-ids and to use these to generate sequences of UPIs in a triple-store.

Bug19910: agraph-recover now returns 0 when successful, 1 otherwise

Agraph-recover now correctly returns 0 to shell when successful. (see Point-in-Time Recovery).

Bug19897 - SHM hangs if socket creation fails

Previously, if a socket creation failure occured in the SHM server (which is a critical subprocesses associated with every database instance), database startup would hang. Now the error is immediately reported, allowing AllegroGraph server to otherwise continue operating.

Bug19891 - Fix log level test for log-backtrace and when-debugging.

Error logging no longer ignores the preferred log level for certain messages.

Bug19888 - Prevent logging gratuitous errors on forced shutdown.

AllegroGraph no longer leaves error messages in the server log file when force-terminated with the killall command. (Trying to write such log messages could result in the service daemon trying to communicate with dead processes.

Bug19864 - Some cursors could fail to correctly free resources

There were situations in SPARQL querying where a cursor could fail to correctly release its resources even when it was correctly discarded. This could lead to resource starvation. This is now fixed.

Bug19848 -- Improve behavior of `qm` when UPI-maps are not available

The qm functor now gracefully drops back to the behavior of the q functor when UPI-maps are not available. Previously, it would signal an error.

Bug19842 - Error message server-pid-and-client-pid-must-match-error

Fix error message when server is running as agraph and client is running as root.

Bug19813 - Fix store parameter merging for configured values.

Opening a triple-store by passing configuration parameters to open-triple-store now correctly overrides those values that are already present in the store's parameters configuration file.

Bug19816 - FTI Merger sometimes produces corrupt chunks

In a database with deleted triples, it was possible for the text-index merger to output a chunk that could not be read, which would bring down the whole FTI subsystem.

Bug19800 - support empty enumerations in owl:oneOf

AllegroGraph's RDFS++ reasoner signaled an error when an owl:oneOf claimed that some class X was one of the empty set. Now it works correctly for this case too.

Bug19791 - Fix checkdb's --port argument.

Previously, the checkdb utility could check triple-stores only if they were served on the default port, 10035. Now, it uses the value from its --port argument to determine the port to connect to.

Bug19786 - Checkdb now exits with 0 on success.

The checkdb utility now follows the unix convention of returning 0 if no problems were detected.

Bug19785 - Convertdb now exits with 0 on success.

The convertdb program now follows the unix convention of returning the status code 0 if there were no errors.

Bug19777 - Replication to multiple secondaries improved

The speed of replication will no longer be determined by the slowest secondary. Each replication job will now proceed as fast as each secondary database can process the data

Bug19766 - Sub-optimal index selection

Previously, in certain circumstances (such as when a range query was being used and one or more of the standard index flavors were not available) the index flavor selection code would exclude some index flavors that it could have used to satisfy a query more efficiently. This has been fixed.

Bug19773 - Dropping an index is not crash recovery proof

Fixed a bug where dropping an index followed by uncleanly shutting down a database would result in an error during restart recovery.

Bug19765 - Detect more cases of string table corruption.

The checkdb program can now detect many more inconsistencies in string tables.

Bug19764 - N-Quad output was incorrect

The output of AllegroGraph's N-Quad serializer when printing single triples was incorrect.

This is now fixed.

Bug19697 - improve performance of part printing

Increased the speed and memory efficiency of part printing.

Bug19657 - Fix race-condition in text-index merger

The merger could sometimes pick the chunks it was to merge from an outdated set of chunks, resulting in chunks being merged multiple times. This would cause duplicate triples to show up in FTI search results.

Bug19600 - direct-upi wire format does not handle nil values

The wire format used to communicate query results to a remote-triple-store broke when given a nil value, though the result of SPARQL queries with an OPTIONAL part may return them. This is now fixed.

Bug19587 - Suboptimal encoding of some characters in shm-queue.

Some characters were being encoded with an extra unnecessary byte. This problem has been corrected.

Bug19578 - Clearnamespaces XMLTransaction tag is broken

There was a bug in the XMLTransaction parser that causes requests containing a <clearNamespaces/> tag to fail. This has been rectified.

Bug19550 - Fix reification in rdf/xml parser

Fixed a bug in which certain statements imported from RDF/XML files would not be reified correctly.

Bug19549 - RDF/XML parser loses the trailing /

When importing from RDF/XML sources, AllegroGraph used to rewrite URIs that came without a path, but with a trailing slash and import them without the trailing slash (e.g., "http://franz.com/" would be imported as "http://franz.com"). Now, it imports all URIs verbatim.

Bug19530 - Fix off-by-one error in text-index code.

Adding a 255-character (the maximum) word to a text index would cause searching and merging of index chunks to break, because a too-small buffer was allocated when traversing the index. This is now fixed.

Bug19522 - Prevent spurious warning in some Prolog select queries

If both :limit and :count-only were specified in a Prolog select query, AllegroGraph would signal a spurious warning. This is now corrected.

Bug19517 - improve Lisp API for remote-triple-stores

Previously, if a :url parameter was used, AllegroGraph would ignore other parameters to create-triple-store like :user and :port that may be inconsistent with :url value. AllegroGraph now ensures that the :url and any other parameters agree and merges parameters into the :url if necessary. This means that you can now use constructs like:

(create-triple-store "foo"  
                     :triple-store-class 'remote-triple-store  
                     :url "http://blade4.franz.com"  
                     :catalog "java-tutorial"  
                     :user "test" :password "xyzzy"  
                     :port 11101)

Bug19515 - Correct precision loss in printing of RDF time and dateTimes

Certain RDF time and dateTime values with fractional seconds could fail to print all of the digits stored for the value. This patch corrects this problem.

Bug19508 - Prevent text-index from leaking files on crashes

When processes crashed 'hard' (hang, or kill -9), the text index would sometimes leave files around that it never cleaned up. It now takes care to (at some point) delete every file it creates.

HTTP Client

Rfe10278 - Alphabetize store and catalog lists in HTTP responses

The lists of repositories and catalogs returned through the HTTP API were ordered more or less randomly. This made finding a repository in a long list in WebView cumbersome. They are now sorted by name.

Rfe10234 - Inefficient limited Prolog queries over HTTP

The HTTP server used to fetch all results, then throw away the extra ones, when executing a Prolog query with a limit argument. It now only fetches needed results.

Rfe10230 - Accept application/x-turtle as a synonym for text/turtle

Some systems are still using the pre-standard MIME type for Turtle documents. The AllegroGraph HTTP server now accepts that as a content-type when uploading triples.

Rfe10220 - Provide an HTTP interface for controlling bulk mode

It used to be only possible to turn on bulk-load mode per load request. This was slow when using many small requests. In a session, it is now possible to persistently turn bulk mode on and off through requests to /session/bulkMode.

Rfe10210 - Don't backtrace on unsupported-db-version error

The HTTP server no longer logs a backtrace and returns a server-error code when trying to open a store with the wrong version. Instead, it returns a PRECONDITION FAILED error.

Rfe10165 - Expose turtle loading over HTTP.

The /statements HTTP service now accepts request bodies in Turtle format, using a Content-Type header of text/turtle. Such files can now also be loaded from WebView.

Rfe10078 - Make HTTP protocol no longer hide inaccessible repositories.

It used to be that HTTP services /repositories and /catalogs hid all repositories and catalogs not accessible by the current user. This was unnecessarily confusing. Now all present objects are returned, with readable and writable flags indicating whether the current user has access to them.

Rfe10009 - Implement Turtle loading

Loading of files in the Turtle (Terse RDF Triple Language) format is now supported.

Rfe9986 - Support Sesame 2.2-style XML transactions

The Sesame project, which is the origin of the XML transaction format our HTTP server supports, have slightly changed the format to be simpler and more flexible. We now support this update, while remaining compatible with the old-style transactions.

Rfe9912 - Remove SharedMemSize parameter from HTTP protocol and HTTP docs

The SharedMemSize parameter was still mentioned in the HTTP protocol docs, and understood by some HTTP services, even though it is no longer used.

Rfe9910 - Reuse sockets for backend-frontend communication

Use HTTP keep-alive to reuse the sockets that the front-end uses to talk to the back-end.

Rfe9838 - Use HTTP keep-alive in the remote Lisp client.

Lisp remote-triple-stores now re-use their sockets when making HTTP requests.

Bug19909 - Inter-server HTTP communication should use local addresses

When the server tried to address itself over HTTP, it would often use the URL provided in the HTTP request that trigerred this communication. If that came from outside the local network, the address in this URL might not be useable. It now uses a local address.

Bug19871 - HTTP server does not unescape \u in resources

In resources passed to the HTTP server either through HTTP parameters, or through JSON data, N-Triples backslash escape sequences were left in the literals, rather than being unescaped. The server now properly handles these.

Bug19846 - Ensure the HTTP client explicitly frees SPARQL cursors

SPARQL queries from the HTTP client did not explicitly discard the cursor they created. This could cause resource exhaustion in situations where many, many queries were operating simultaneously. This is now fixed.

Bug19801 - HTTP character encoding should not depend on system locale

The system's default character encoding was accidentally being used when writing HTTP responses. This worked fine on systems with a UTF-8 locale, but caused the server to return inconsistently encoded responses on other locales.

Bug19661 - HTTP JSON output for query results truncates row on first unbound result.

The code that wrote out SELECT query results in application/json format would stop writing a row at the first null it encountered, leaving that null and any values after it out of the result.

Bug19616 - Allow multiple contexts to be specified in HTTP /size

Contrary to the Sesame docs, we only accepted a single context argument to our /size HTTP service. This has now been fixed -- you can pass multiple context parameters, and the returned value will be the sum of their sizes.

SPARQL

Rfe10207 - Efficiency improvement for some SPARQL graph queries

Improve the way the SPARQL engine handles simple GRAPH clauses with a variable graph argument that does not appear within the rest of the clause. For example:

select ?someGraph ?o  
where {  
 ex:a ex:inGraph ?someGraph .  
 graph ?g { ex:a a ?o . }  
}

SPARQL semantics dictate that the query act as if the graph clause is checked for every graph in the dataset. In this case, however, it is clear that the value of ?g cannot alter the behavior of the inner query and that the intent is to determine the graph slots of any of the selected triples.

Note that this optimization is currently very conservative and only occurs when the graph clause is a simple BGP. Future work will extend the behavior to more complex queries.

Rfe10061, Bug19663 - Improve SPARQL negation-as-failure parsing

Nested instances of the SPARQL negation-as-failure idiom were not being parsed correctly leading to incorrect query results. This fix corrects the problem.

Rfe10030 - Share statistical planner between Prolog and SPARQL

Use the foundations of the Prolog select statistical planner in SPARQL queries to reorder BGPs. This improves the speed of many SPARQL queries. Note that this planner requires the POSGI index. If this index is not available, then the coverage planner is used instead.

Because the planner is statistical in nature, it will not always choose the best plan. In general, performance is improved (for example, the 10,000-triple SP2 benchmark queries almost double in speed (from 56-seconds down to 33-seconds) but some queries go slightly slower using the new planner. We will continue to enhance the statistics used and the planner's capabilities in future releases.

If your queries are running more slowly than expected, you can tell SPARQL to use the older coverage-based heuristic planner or tne identity planner (i.e., the planner that does no clause rearranging. Planner selection varies by client. See the client SPARQL documentation for more details.

Bug19809 - fti:matchExpression is sometimes treated like fti:match

There was a bug in the SPARQL code that sometimes caused S-espression-style text queries (as per fti:matchExpression) to be parsed as textual queries (fti:match).

Bug19681 - fix SPARQL problem in queries using both FROM and FROM NAMED

A SPARQL query that used FROM and FROM NAMED together to create the dataset could incorrectly omit some of the the graphs mentioned in the FROM clause. This could cause the query to produce the wrong results.

This problem has been corrected.

Rfe10061, Bug19663 - Improve SPARQL negation-as-failure parsing (repeated from Rfe10061 listing)

Nested instances of the SPARQL negation-as-failure idiom were not being parsed correctly leading to incorrect query results. This fix corrects the problem.

AGWebView

Rfe10300 - Allow optimizing of stores from WebView

There is now an optimize store control in the store overview in WebView. This can be used to optimize a store for querying after a bunch of data has been added.

Rfe10280 - Expose scripting in WebView

There is now a 'Scripts' link in the navigation bar, where one can run JavaScript and (with the right permissions) Lisp scripts interactively, as well as store and edit server-side user script files.

Rfe10277 - Improve help text for WebView part input

The examples given now demonstrate that both backslash-escapes (\n, \uHHHH) and unicode characters are allowed.

Rfe10244 - Add JavaScript server-scripting support

It is now possible to send an /eval request with content-type "text/javascript", and have the body be evaluated as JavaScript. Custom services can also be written in JavaScript.

Rfe10236 - Disable query planner selector in WebView for Prolog queries

To make it clear that the query planner dropdown doesn't apply to Prolog queries, it is now greyed out when selecting a non-SPARQL language.

Rfe10185 - Allow the user to specify a limit for queries in WebView

For heavy queries, the default limit (200) imposed by WebView might be too big. There is now a dropdown above the query input that allows one to choose the amount of results that should be retrieved.

Rfe10181 - Leave WebView widgets in the default browser style

This removes the custom CSS styles for WebView inputboxes and buttons, making the the application look a little more native in modern browsers.

Rfe10155 - Allow a query planner to be selected in WebView

Adds a drop-down menu to the WebView query input page where one can select which planner to use.

Rfe10128 - Rework WebView namespace management

Both the store's default namespaces and the user's namespaces are now managed in the same place. This to reduce confusion---people assumed they could use the namespaces they defined for the store in their queries, without first selecting them.

Rfe10126 - Show PIDs in WebView's process list

Under #processes, the Unix PID of each of the processes is now listed after the process name. The /processes HTTP service now returns tuples instead of just names.

Rfe10116 - Show running sessions in WebView

Once you navigated away, the browser history used to be the only way to get back to a session through WebView. There is now a list of active sessions (if any) in the catalog overview page.

Rfe10039 - Support backup and restore through WebView.

There is now (for superusers) the option to create stores from backups, and to back up existing stores, through the WebView interface. (This is not recommended for large stores.)

Bug19880: Resizing the query input field in WebView is broken

You could drag the resize corner widget, but when you released it nothing happened. This is now fixed.

Bug19868 - Webview tries to write to the installation dir

The server would try to cache gzipped JavaScript files in its installation directory, which fails when it is running under a user that doesn't have write access to that directory. It now uses the settings directory (called the Directory to store data and settings by the configuration program, with default name .../data/) for the cache.

Bug19858 - AGWebView accidentally shows Turtle as export format

We don't currently support exporting to Turtle, so this shouldn't be selectable.

Bug19762 - Fix problem in process page in WebView

A bug had snuck into the WebView code that prevented the #processes page from coming up when in a catalog view.

Bug19741 - Add example to context input field help in AGWebView import

It used to say only that it accepted 'N-Triples' syntax.

Bug19523 - Fix several bugs in WebView's query result display.

Prolog queries returning the default graph now display properly. Prolog queries returning non-flat-lists (such as (?x (?y ?z))) now produce an error message rather than no feedback at all. Literals ending in a backslash are now parsed properly, rather than treated as invalid.

Changes to the Lisp API

Rfe10296 - Check minimum build id on client startup.

The AllegroGraph lisp client will now check if the lisp has all the required runtime patches installed, and will present an error and resolution options if that is not the case.

Rfe7148 - Optimize away keyword scanning in get-triples

db.agraph:get-triples now uses a compiler macro to optimize the calls so no runtime CPU time is spent keyword scanning.

In Lisp if you want to trace get-triples calls, you should declare the function notinline. Alternatively, you can trace db.agraph::get-triples-positional, the non-exported internal function, which the compiler macro uses.

Bug19890 - load-rdf/xml-from-string (Lisp API) could cause memory error

If called with a very long string and no explicit base-uri, load-rdf/xml-from-string tried to create a base-uri by querying the file system with the string. In some situations, this could cause a buffer overflow which would lead to abnormal client termination. This commit fixes the base-uri resolution algorithm so that it handles strings without querying the file system. The underlying buffer problem has also been corrected.

Bug19678 - serialize-rdf-n3 fails with encoded UPIs

The Lisp API function serialize-rdf-n3 would fail if it was asked to serialize triples with any encoded UPIs. This is now corrected.

Bug19069 - improve error handling of N-Triples parser

The continue-on-errorp function to load-ntriples can now specify how to behave when an error occurs more precisely. It can return one of

:continue (the default): to ignore an error and continue
:stop-parsing: to ignore the erorr and stop parsing the rest of the current source
:error: to signal the error in any case

Documentation

Rfe9657 - server-installation.html document changed to describe non-interactive running of configure-allegrograph.

Python Client

Rfe10221 - BulkMode support added to Python client

Bulk Mode is a repository-wide setting that affects all connected clients.

Use the bulk_mode property on a repository object to set and get the bulk_mode state. Here are some examples.

repository.bulk_mode = True print "The bulk mode state is ", repository.bulk_mode

If you only have a connection object, you can access the property with conn.repository.bulk_mode.

Any commits while repository.bulk_mode is true are done so using BulkMode on the server.

Java Client

Rfe10303 - Support revised bulk-load mode interface

With this change, the Java client now supports the revised bulk-load mode interface. Bulk mode is now persistent (sticks even after closing and reopening the store), and is controlled with the AGRepository#setBulkMode and AGRepository#isBulkMode methods, rather than per session.

Bulk-load mode can now only be controlled through a store-wide toggle (/bulkMode over HTTP), since it is a store-wide setting, and turning it on and off a lot is very slow.

Rfe10197 - Support for encodable-ids and generated URIs

With this change, a namespace (viewed as a set of URIs) can now be registered as encodable. URIs in the namespace have local parts that follow a declared format, allowing for efficient encoding and in some cases generation of unique URIs in the namespace. Once registered, a conformant URI in the namespace is auto-encoded when added to the store, resulting in a smaller string table and faster query speed. (This is a Java client version of the encoded-ids feature, see Encoded IDs for more information.)

Rfe10189 - Java Interface for Stored Procedures

The stored proc feature and API are experimental, and subject to change in a future release. For this reason, the new methods are marked as deprecated.

The AllegroGraph server defines a new API for defining Stored Procedures and they are installed like Custom Services.

The primary API is:

AGRepositoryConnection.callStoredProc(functionName, moduleName, args)

A low-level API is also exposed:

AGHttpRepoClient.callStoredProcEncoded()  
AGSerializer.serializeAndEncode()  
AGDeserializer.decodeAndDeserialize()

Rfe10119 - Add analyze() method to provide query analysis

Sesame AGQuery subclasses can now call the analyze() method to obtain a query analysis for the query rather than evaluating it.

Rfe9971 - JAVA API for HTTP protocol checkVariables

Related to bug19157: SPARQL error if ORDER variable isn't in query

Add method setCheckVariables(boolean) to both query classes: com.franz.agraph.jena.AGQuery com.franz.agraph.repository.AGQuery

Rfe9858 - Support ResultSet nextSolution() method

Previously, the legacy ResultSet API method nextSolution() would throw an UnsupportedOperationException; with this change the method is now supported.

Rfe9846 - JENA graph unions and graph-scoped reasoning

With this change, the method AGGraphMaker#createUnion can be used to create a union of named graphs. AGInfModels can now be created over models backed by a union graph, and reasoning will be properly scoped to just the union graph; this change allows sound reasoning over a proper subset of data in the store.

Rfe9534 - Add source jar to the client download

With this change, an agraph-[version]-src.jar file is now included in agraph-[version]-client-java.tar.gz downloads. The download's Eclipse project's .classpath is modified to attach the source jar to the agraph-[version].jar to facilitate browsing and debugging.

AllegroGraph 4.1.1

Welcome to AllegroGraph 4.1.1. This release of AllegroGraph provides overall product stability and efficiency improvements, with special attention to improvements in AGWebView and documentation.

Server Changes

Rfe9796 - Make tlog-write-checkpoint-indicator use :allocate-temporary

The checkpoint indicator file, which records the location of the last checkpoint record that has been written to the transaction log file, is now written using the new internal API for creating temporary files. This ensures atomic updates on disk.

Rfe8994 - Change behavior when a filesystem is full

A change has been made to the AllegroGraph server that makes the database server block if one of the file systems is found to be full during a write operation. Operations automatically continue once space has been made available by the system administrator.

Rfe9319 - Freetext index configuration and search support

With this change a free text index can be created by specifying its configuration, and free text performed can be performed w.r.t particular freetext indices.

Rfe9534 - Add sources for jena, arq, and json jars

Previously the source code for these 3rd party libs had to be obtained separately; this change includes them as a debugging convenience in the git repository, and sets up the .classpath to reference them in Eclipse.

Rfe9969 - Update clojure api

AGraph Clojure API updated for agraph v4, clojure 1.2. Build support for leiningen 1.3.1, as well as ant.

Clojure and Java source are in the Git repository: http://github.com/franzinc/agraph-java-client Added README.md for use and development instructions.

Clojure and Java jars are available in the Clojars Maven repository: http://clojars.org/groups/com.franz agraph-java-client and openrdf-sesame-onejar When AGraph v4.1 is released, these will be updated and agraph-clj-4.1.jar added.

Changed license to EPL, same as the java code.

Removed agclj.sh in preference for 'lein repl' or 'lein swank'.

Rfe9999 - Auto-adjust MinMergeChunks

If MergeThreshold is configured to a value lower than MinMergeChunks, the value of latter is now automatically adjusted to the value of the former. This ensures that a merge will actually be done when the threshold is reached.

Rfe10119 - Add analyze() method to provide query analysis

Sesame AGQuery subclasses can now call the analyze() method to obtain a query analysis for the query rather than evaluating it.

Bug19658 - Agload needs bin directory symlink

The symbolic link for the agload program was not being created in the 'bin' directory by install-agraph and the RPM installation. This has been fixed.

Bug19674 - After-the-fact index creation should mind the amount of files it opens

Fixed a bug that caused "too many open files" error and terminating the server when large number of chunks had to be merged.

Bug19698 - Changes to prepareGraphQuery and prepareBooleanQuery

prepareGraphQuery and prepareBooleanQuery now also accept the string "SPARQL" or "PROLOG" as a query language parameter. Previously, only prepareTupleQuery allowed the string values as a convenience in place of the QueryLanguage.SPARQL or QueryLanguage.PROLOG objects.

Bug19720 - Fix "don't attach as root" check

The check which ensures that you don't attach to a triple store as "root" was checking the real uid, not the effective uid, preventing it from working properly under su/sudo. This has been fixed.

Bug19739 - Compressed index files not unmapped when closed

Triple index files which were mapped during queries were not being properly unmapped when they became obsolete. This problem has been fixed.

HTTP Client

Rfe9756 - Return 400, not 500, when trying to delete an open store over HTTP.

The HTTP server was returning (and logging) an internal server error when it tried to delete a store that was still being held open (through a DELETE or PUT request to the store's URL). It now returns a 400 (bad request) response.

Rfe9971 - Java api for http protocol checkVariables

Related to bug19157: SPARQL error if ORDER variable isn't in query

Added method setCheckVariables(boolean) to both query classes: com.franz.agraph.jena.AGQuery and com.franz.agraph.repository.AGQuery.

Rfe10120 - Make sure custom-service exists in agraph4.fasl

This adds the HTTP backend code to the agraph fasl, so that it is possible to compile custom services in an image that has this fasl loaded. AllegroGraph 5.0 note: the file is agraph5.fasl starting in version 5.0.

SPARQL

Bug19157 - SPARQL error if ORDER variable isn't in query

When the order by clause has a var not known from the query, ignore it (the default), or identify it in an error message (when sparql-check-variable-errors-p or checkVariables in the http protocol is true).

Bug19618 - Properly escape strings in SPARQL/JSON output.

Newlines and other special characters are now escaped according to JSON rules in SPARQL/JSON results returned by the SPARQL engine.

Bug19619 - Fix typed literal writing in SPARQL/JSON output.

The SPARQL engine was not writing out typed literals in SPARQL/JSON documents properly. Those in UPI form got a wrong type property ("literal" rather than "literal-typed"), those in future-part form were written out as XML.

AGWebView

Rfe9492 - Add export to AGWebview

Adds the possibility to download repositories and query result sets through AGWebView. Several output formats are supported, depending on whether triple data (n-triples, n-quads, rdf/xml, trix) or select results (sparql+xml, csv) is downloaded.

Rfe9587 - Show progress indication in WebView uploads

Rather than just seeing a spinner animation, users doing an upload in WebView now see how much data has been uploaded already.

Rfe9898 - Add drop-down menus to WebView's navigation bar.

To make it possible to link more functionality from the navigation bar without running out of space, groups of links are now put into drop-down menus.

Rfe9900 - Optimize loading of WebView scripts and style-sheets.

Serves the JavaScript and CSS files used by WebView in concatenated form, allowing gzipped responses. This causes the base webview page to go from 25 files totalling over a quarter megabyte to 2 files and 68 kilobyte.

Rfe9950 - Provide a way to start exploring a store in WebView

This adds a paragraph to a repository overview containing links to some simple queries that can help the user explore the store.

Rfe9951 - In WebView user management, show drop-downs for stores.

When granting permissions on a store, the user no longer has to type the names, but can select them from a drop-down menu.

Rfe9964 - Ask confirmation before letting user un-superuser himself in WebView

Once your user loses its superuser flag, it can no longer manage users, so one wrong click here could lock yourself out. There is now a confirmation dialog to ensure you intended to do this.

Rfe9990 - Do not display \uHHHH character escapes in WebView

For non-ASCII characters, WebView showed the N-triples escape sequence instead of the character itself, even when 'short' part display was turned on. This is now fixed.

Rfe10025 - More robust recent-query saving in WebView

On modern browsers, you'll now be able to list your recent queries even after navigating away from WebView and back. On top of that, the list of recent queries is no longer tied to a specific repository.

Rfe10026 - WebView now shows a spinning animation when loading a file

Before, there was no proper feedback indicating that something was happening during a dataset upload.

Rfe10027 - In WebView's graph view, fix nodes after dragging

Nodes can be rearranged by dragging in the graph view. A node now stays fixed after having been dragged, making it much easier to lay out the graph.

Rfe10050 - Make specifying a graph when adding from WebView easier.

The graph input field in the add-triple dialog is now always visible. WebView will try to retrieve a list of graphs that are used in this store, and allow you to choose them from a drop-down.

Rfe10051 - Allow a graph/context to be specified when importing in WebView.

This extends the import file and import URL dialogs with a context field, which is used to specify a graph to import into.

Rfe10052 - Add a general delete-triples UI to WebView

This adds a 'delete statments' control to the repository overview page, from which you can delete triples, by wildcard. Also includes the existing-graph select control from rfe10050, to make it easy to drop a graph.

Rfe10127 - Allow init files to be loaded in sessions created from WebView

There is now a checkbox 'load initfile' in the widget that creates new sessions from a catalog view in WebView.

Bug19581 - Newly registered users in WebView don't get enough access

When user-registration is turned on, new users were supposed to get the same access permissions as the anonymous user, but due to a bug they did not get any access. This is now fixed.

Bug19536 - In WebView's graph view, clicking on a node was broken.

Clicking on a node in the graph did not actually take you to that node's overview page, but raised a script error instead. This is now fixed.

Bug19552 - Can't start a session from catalog page in WebView

There was a bug that would prevent users from starting a session from a (non-root) catalog page in WebView. This has been fixed.

Documentation

Rfe9574 - Lisp client documentation

The AllegroGraph Lisp Client Quick-Start guide has been expanded to include explicit Lisp Client set-up instructions and an explanation of local vs. remote triple-store connections.

Rfe9750 - Improve document navigation

The navigation panel (to the right side of most documentation pages) and the document map page have been revised and extended to provide more organized and thorough access to the many documents and web pages that describe AllegroGraph.

Rfe9883 - Javadoc included in the distribution

Javadocs for com.franz.agraph are in the javadoc folder in the distribution file, and they link to external javadocs for java, jena, and sesame.

Rfe9929 - User manual for 4.1.1 AGWebView

The AllegroGraph documentation has been extended to include a User's Guide for WebView. This document presents task-oriented guidance on using the WebView interface for managing users and repositories.

Rfe9936 - Document TopBraid Composer plugin

In this release the TopBraid Composer (TBC) plugin installation notes got an upgrade with numerous screen-shot illustrations and further notes on how to integrate TBC with the AllegroGraph Virtual Machine.

Rfe10046 - AllegroGraph Text Indices

There is a new, high-level document describing AllegroGraph's free-text indexing features.

Rfe10077 - Add user-selected triple indices to tutorials

The Python, Java, and Jena tutorials have an augmented example1() showing how to add and remove triple indices for greater efficiency.

Rfe10108 - AllegroGraph Triple Indices

There is a new, high-level document describing AllegroGraph's system of triple indices and how to manage them.

Rfe10139 - Emphasize "agraph" user vs. AG superuser

The server-installation document now emphasizes the differences between the "agraph" operating system account and the AllegroGraph superuser account. Users sometimes confused the two accounts, which caused connection issues.

Rfe10159 - Rapper documentation

There is a new document describing the use of the Rapper command-line utility for pre-processing incoming data files to make them load more quickly in AllegroGraph.

Bug19620 - Fix typos in documentation of create-freetext-index

The description of the index-fields argument was a jumble of a sentence.

AllegroGraph 4.1

Welcome to AllegroGraph 4.1. This release of AllegroGraph provides overall product stability and efficiency improvements, with special attention to backup/restore, point-in-time recovery, and warm standby features.

Rfe9535 - Use managed files throughout the storage layer

The storage layer has been changed so that all files are allocated through the central resource manager mechanism. This makes all files eligible for placement through FilePlacementRule parameters in the database configuration. Previously, some files could not be placed using such rules.

Rfe9772 - More advanced server scripting

This introduces server-side script files, which can be loaded into sessions, and a convenient way to define custom HTTP services from such files. See the HTTP protocol documentation for details.

Rfe9784 - Keep track of which pages of deleted bitmap have been touched

In order to speed up backup operations, the AllegroGraph server now keeps track of which pages in the deleted bitmap file are used.

Rfe9785 - Revise backup and restore

AllegroGraph now uses a new, single-file archive format for storing backups. The new backup and restore functionality is described in the server documentation.

Rfe9789 - Support callback parameter when writing JSON output

Requests returning JSON data can now specify a callback=[x] HTTP parameter, which will cause the response to be wrapped in [x]([response]), so that the resulting document can be loaded as a JavaScript program (which is useful for working around cross-domain restrictions in browsers).

Rfe9798 - Allow relative directories in config files

In the AllegroGraph server configuration file, relative pathnames can now be specified for file and directory parameters. Such pathnames are considered relative to the directory where the server configuration file is located.

Rfe9805 - Point in time recovery

The point in time recovery facilty allows the a database backup to be restored to its state at a particular point in time.

Rfe9835 - Use stream API to allocate blocks in sparse deleted bitmap

The database server code has been updated to explicitly allocate blocks in the deleted bitmap file using file system calls. That way, full filesystem conditions will be handled in a graceful manner.

Rfe9840 - Improve "disk full" behavior in agraph-backup

When agraph-backup detects a "disk full" condition, it now performs an interactive recovery if it was started from a terminal. When started non-interatively (i.e. from cron), no recovery is performed and an error message is generated.

The exit code of agraph-backup is now always 0 if the operation succeeded, 1 if it failed.

Progress reporting in agraph-backup has been improved.

Rfe9842 - Agraph-backup: merge parameters.dat while restoring

When restoring a database from a backup, the parameters established in the parameters.dat file stored in the backup database are copied to the restored database. File placement related parameters are skipped in this process.

Rfe9888 - Do not log backtrace for HTTP proxy pipe errors.

When a back-end crashed, this often caused two back-traces to be logged, one for the back-end and one for the front-end. The second one was just noise, so it is now suppressed.

Rfe9889 - Allow resource manager to forget a file

A new internal API function has been added to the resource manager library to support the archival of transaction log files.

Rfe10055 - Properly expose the no-commit flag

The no-commit mode, used primarily by the warm standby replication process, is now something that can be explicitly turned on and off.

Rfe10056 - Require authentication for warm-standby connections.

The socket on which a warm-standby server listens used to be accessible by anyone. It now requires a valid username and password when initiating a connection.

Bug19391 - Agraph-install no longer outputs paths with double slashes

The helper script agraph-install would output paths with double slashes when given a path with a trailing slash as argument. This was mostly harmless, and is now fixed.

Bug19420 - Chunk writers leak disk space when db crashes

A problem has been corrected that could lead to stray files being left on disk when a database was brought down uncleanly.

Bug19446 - Catalog parsing problem - confusing error message

The error message that the configuration file parser generates when overlapping directories are specified in the configuration file has been improved.

Bug19487 - Solve ':void external-format' errors in logs.

The AG log files occasionally showed unexpected errors of 'Attempt to translate characters using a :void external-format'. These were harmless, but they are now handled properly, and no longer logged.

Bug19526 - Back up tlog file only up to backup checkpoint

The backup code has been changed so that transactions which executed while the backup was running are not included in the backup.

Bug19684 - Include the triple-id index in the standard indices

If specific indices are not requested when a store is created, then it will have the standard set of seven: :ospgi, :posgi, :spogi, :gospi, :gposi, :gspoi, and :i.

Prior to this change, the :i index was not included in this standard set.

Lisp Client

Bug19442 - Make sure remote-triple-store accepts :server argument.

The :server argument to the remote-triple-store class was being incorrectly rejected, making it impossible to open stores on other machines.

SPARQL

Bug19474 - Improve SPARQL describe queries

excess variables are no longer included in the output of DESCRIBE
CBD (Concise Bounded Description) logic is now implemented so that we correctly handle blank nodes
Many general efficiency improvements

Bug19499 - Fix SPARQL XML and JSON output of geospatial values

Retrieving SPARQL SELECT results containing geospatial values as SPARQL+XML or SPARQL+JSON was broken because an internals change hadn't been propagated to the SPARQL engine. This is now fixed.

AGWebView

Rfe9815 - Fix accept-encoding issue for WebView in Internet Explorer 7

This fixes a problem where Internet Explorer 7 would display "Internet Explorer cannot display the webpage" when opening WebView (or any other page served by AG). Internet Explorer fails to handle deflated content, so we now serve it gzipped responses instead.

Rfe9899 - Fix typo and wrong word choice in WebView.

When adding a notable node, 'Desription' was corrected to 'Description'. When looking at a text index that indexes all predicates, it used to say 'None' under predicates, now it says 'All'.

Bug19500 - Fix problem with displaying geospatial results in WebView

When a query result contained geospatial data, a bug in WebView was triggered that caused the result to fail to display. This bug has been fixed.

Bug19525 - Fix failure when opening a federated session in WebView.

Trying to create a federated session in WebView resulted in a PRECONDITION FAILED error. This was caused by a bug in the way session data was being returned by the server, causing WebView to believe it was not on a federated store. The problem has been fixed.

Documentation

Spr37148 - Question about connecting using java

Two lines of code were inconsistent between the Java tutorial HTML file and the tutorial examples code. The tutorial has been edited to correct this issue.

Rfe9783 - AllegroGraph's isolation model

The AllegroGraph Overview document has been augmented with a new section on AllegroGraph's ACID-compliant isolation model.

Rfe9806 - Enhance SPARQL examples in tutorials

In the Python, Java and Jena tutorials, example13() now uses the SPARQL "contruct" to create has-grandchild triples, and then "describes" a grandparent and a grandchild.

Rfe9821 - Make SPARQL examples more robust

Example10() of the Python/Java tutorials has been extended to show examples of SPARQL's FROM, FROM DEFAULT, and FROM NAMED operators, and an example of the UNION operator.

Rfe9831 - Python/Java tutorials, illustrate use of conn.size() with contexts.

Example10() of the Python and Java tutorials has been extended to show examples of the conn.size() method returning the number of triples found in specific subgraphs.

Rfe9915 - Java and Python tutorials demonstrate SPARQL and RDFS++

Example19() of the Java and Python tutorials has been augmented to demonstrate how to use a SPARQL query with RDFS++ reasoning.

Rfe10104 - Point-in-time recovery document

There is a new document describing AllegroGraph's point-in-time recovery feature.

Rfe10105 - Replication and warm-standby document

A new document on triple store replication and warm stand-by has been added to the document set.

Rfe10106 - Update backup/restore document

The backup/restore document has been updated to reflect the new features available in AllegroGraph 4.1.

Rfe10107 - Transaction Log Archiving document

Transaction logs increase without limit unless archived. A new document describes the transaction log archiving feature of AllegroGraph.

Bug19454 - Python tutorial explains how "int" becomes "long"

In example15() of the Python tutorial we show registerDatatypeMapping() requesting a nativetype "int" and AllegroGraph responding with XMLSchema.LONG. This is correct, and the text now explains why.

Document MinimumCheckpointInterval and MaxRecoveryTime parameters

The meaning of these parameters have been documented in the daemon configuration documentation.

AllegroGraph 4.0.6a

Welcome to AllegroGraph 4.0.6a. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9837 - Avoid log warning when opening an existing repo

Added method AGCatalog.hasRepository(id) which tests for existance. Added AGCatalog.openRepository(id) which will open but not create. Added AGCatalog.createRepository(id,strict) where the strict option will throw an exception if the repository already exists.

Rfe9980 - getCatalog now works with the root catalog

Added AGCatalog.isRootId(id) to identify id's for the root catalog. Enhanced AGServer.getCatalog(id) to accept root catalog identifiers. Also added AGServer.getCatalog() to return the root catalog.

Rfe10041 - Implement owl:oneOf

AllegroGraph's RDFS++ reasoner now includes the following inference:

if the class X is owl:oneOf A, B and C  
then each of A, B and C are of type X.

Rfe10075 - Implement query index use analyzer, version 1

If analyzeIndicesUsed=true is added to a SPARQL HTTP query string, then AllegroGraph will analyze the query to determine which indices it will use. The results are returned as plain text and written to the AllegroGraph log. They will look like

(desired spogi optimal 1 actual spogi optimal 1)  
(desired posgi optimal 6 actual ospgi suboptimal 4)

Where each line refers to one of the indices required by the query. The line indicates which index flavor the query really wants and which index the query used based on what is available to the store. The additional information indicates whether or not the index choice was optimal, suboptimal or required a full scan. The numbers indicate whether or not additional filtering was required and can be ignored for now.

Bug19650 - Fix problem federating repositories from the root catalog

Previously an exception was thrown when attempting to federate repos from the root catalog; this change addresses that.

AllegroGraph 4.0.6

Welcome to AllegroGraph 4.0.6. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9535 - Use managed files throughout the storage layer

Rfe9620 - Increase limit on number of index chunks

Previously, the maximum number of index chunks was based on the limit on the number of file descriptors (RLIMITNOFILE) which, in a default Linux configuration, resulted in a limit of 100 chunks. Now the maximum is based on the limit on the number of memory mappings (determined by reading /proc/sys/vm/maxmap_count), which usually results in a much higher limit.

If the limit on the number of index chunks is reached, further commit activity will be blocked. Therefore this higher chunk limit will reduce the likelihood of blocked commits.

Rfe9731 - Change transaction log aligment calculation

An internal change was made to the way the alignment for transaction log records is calculated so that opening transaction log files can be defered until records are written.

Rfe9762 - Add someValuesFrom and allValuesFrom restriction reasoning

AllegroGraph's hasValue reasoning has been renamed restriction-reasoning and now covers owl:hasValue, owl:someValuesFrom and owl:allValuesFrom. In particular, AllegroGraph will now infer:

C owl:onProperty P  
C owl:hasValue v  
a rdf:type C  
==> a P v  
 
C owl:onProperty P  
C owl:hasValue c  
a P c  
==> a rdf:type C  
 
C owl:onProperty P  
C owl:someValuesFrom D  
a P b  
a P x  
b rdf:type D  
(and will not infer x rdf:type D)  
==> a rdf:type C  
 
C owl:onProperty P  
C owl:allValuesFrom D  
a type C  
b P x  
==> x type D

Because the reasoner now implements more than owl:hasValue, it has been renamed the restriction-reasoner. The accessor function has-value-reasoning-enabled has been deprecated and the new accessor is restriction-reasoning-enabled.

The HTTP protocol changes to use restriction instead of hasvalue as the name of the reasoner that supports hasValue reasoning (and the other restrictions).

Rfe9769 - Log merge statistics

In addition to logging how long a merge took, the following additional information is logged: Number of triples read, number of triples written, number of deleted triples expunged, and number of duplicate triples dropped.

Rfe9772 - More advanced server scripting

This introduces server-side script files, which can be loaded into sessions, and a convenient way to define custom HTTP services from such files. See the HTTP protocol documentation for details.

Rfe9776 - Standardize blank node serialization

Use the same blank node syntax in all of AllegroGraph's serializers (N-Triples, RDF/XML, TriX, etc.).

Rfe9789 - Support callback parameter when writing JSON output

Rfe9803 - Bulk loading support

AllegroGraph now supports a bulk loading mode that allows changing the triple-store without updating the transaction log. Both the HTTP statements service (used to load data into a store) and the agload bulk loader have been enhanced to use this new feature.

Use the bulkMode parameter to specify that you want bulk loading on in the HTTP service

Use the --bulk option to specify that you want bulk loading on when using agload.

Depending on hardware and the actual data, loading with the transaction log turned off is between 10 and 33% faster than loading with it on.

Rfe9812 - Add client API for triple-store index manipulation

Add the following new functions to the Lisp API:

add-index - add a new index to a store
drop-index - remove an index from a store
triple-store-indices - return a list of current indices for a store
valid-index-flavors - return a list of all valid index flavors

Note that changes made by 'add-index' and 'drop-index' are not made until 'commit-triple-store' is called.

Rfe9889 - Allow resource manager to forget a file

A new internal API function has been added to the resource manager library to support the archival of transaction log files.

Rfe9907 - Add with-indices support to agload

agload now supports custom index selection using the --with-indices command-line option.

Rfe9913 - Improve single-float printing

AllegroGraph could lose precision when printing single float values. For example, 0.0018333651 would print as 1.833365E-3 rather that 1.8333651E-3. This has been corrected.

Rfe9930 - Support adding, dropping, listing repository indices

Added methods addIndex, dropIndex, listIndices, and listValidIndices to AGRepositoryConnection to support managing repository indices.

Rfe9953 - Share transaction log buffer across attachees

Each process (backend) that attached to a store used to allocate its own in-memory buffer for writing to the transaction log.

A single buffer is now allocated in shared memory and used across all backends, saving valuable RAM.

Rfe9966 - Front-end server-socket backlog parameter is too low

The kernel will only save a certain amount of pending connection attempts. If the server doesn't accept them quickly enough, they get discarded. The default backlog size was too small. It is now set to use the maximum the operating system allows, making the chance of discarded connections much, much lower.

Rfe9970 - Upgrade path for configurable indices

Code has been added to convert pre-v4.0.6 databases to be useable with v4.0.6 in terms of configurable indices.

Rfe9972 - Updates to database upgrade code

The internal API for database upgrades has been improved to make more information about the upgraded database available to the code that performs (part of) the upgrade.

Rfe9992 - Backup/restore improvements

agraph-backup now accepts a --catalog command line parameter. Also, agraph-backup now only backs up the used portions of the string table files, resulting in faster backup and restore times.

Rfe10012 - Consistency checker for sstab

A utility has been added to the AllegroGraph distribution that validates the contents of the on-disk string table data structures.

To validate the integrity of a database, the new "checkdb" utility should be used.

Rfe10019 - Setup automatic datatype-mapping for triple-ids

There is now an automatic datatype mapping between triple-id UPIs and the datatype <http://franz.com/ns/allegrograph/4.0/triple-id>. This means that these UPIs will be be read and written in this format. For example, the triple

ex:test rdf:refersTo "18"^^<http://franz.com/ns/allegrograph/4.0/triple-id> .

will be imported such that its object is an encoded UPI that refers to triple ID 18 using the AllegroGraph specific triple-ID reification.

Bug19171 - Importing RDF files with long text literals is much faster.

AllegroGraph now uses a string output stream to collect characters in text literals instead of string+'ing them together. This speeds up importing long literals with many XML escapes. Files that previously took 4 hours to import now take ~2 seconds.

Bug19181 - Correct newline parsing for n-triple imports

Previously, the AllegroGraph n-triple format parser would count newline and carriage return characters as white space, allowing incorrectly-formatted triples to be imported. Now AllegroGraph allows a triple to be specified on one line only.

Due to this change, the AllegroGraph n-triple parser now reports the correct line number when a parse error occurs. This line number can now be used in a text editor to jump to the location of the problem.

Bug19253 - Accept resources for :base-uri in load-rdf/xml.

Previously, the RDF parser disregarded resources used as :base-uri in the RDF import. Now, it accepts all resources and other future-parts and uses their value to determine the base URI for the import.

Bug19266 - Correct cursor re-use problem

AllegroGraph was aggressively re-using storage layer cursors which could lead clients to point at cursors that were no longer valid. This could lead to incorrect query results.

Bug19391 - Agraph-install no longer outputs paths with double slashes

The helper script agraph-install would output paths with double slashes when given a path with a trailing slash as argument. This was mostly harmless, and is now fixed.

Bug19407 - Allow the direct Lisp client to log to the server log file

The direct client (which is used internally in the HTTP server) can now write to the server log file directly.

Bug19420 - Chunk writers leak disk space when db crashes

A problem has been corrected that could lead to stray files being left on disk when a database was brought down uncleanly.

Bug19442 - Make sure remote-triple-store accepts :server argument.

The :server argument to the remote-triple-store class was being incorrectly rejected, making it impossible to open stores on other machines.

Bug19446 - Catalog parsing problem - confusing error message

The error message that the configuration file parser generates when overlapping directories are specified in the configuration file has been improved.

Bug19460 - Issue installing agraph 4.0.5 over 4.0.4

A bug has been fixed that resulted in errors when trying to open a database that was created with AllegroGraph v4.0.4. This problem has been resolved.

Bug19569 - Java client now helps delete orphaned saved queries

Preparing a Sesame AGQuery currently has the side effect of saving the prepared query on the server in memory for the duration of the session. A client app that prepares many queries during a session thus uses up server memory. If an AGQuery instance goes out of scope, the saved query is no longer needed on the server and is currently only deleted when the session expires. With this change, the client library no longer uses the saved query service during a prepare query, pending further cost-benefit analysis of the saved query service. Client code can explicitly request that a query be saved using an experimental AGQuery.setSaveName() method. The client library also tries to help delete orphaned saved queries sooner by tracking when AGQuery instances are garbage collected on the client and issuing delete requests for the orphaned saved queries in subsequent query requests.

Bug19474 - Improve SPARQL describe queries

Excess variables are no longer included in the output of DESCRIBE
CBD (Concise Bounded Description) logic is now implemented so that we correctly handle blank nodes
Many general efficiency improvements

Bug19487 - Solve ':void external-format' errors in logs.

Bug19489 - Directory where pidfile resides is chown'd to agraph user

In prior releases, when running as root, the agraph-control script would chown the directory where the pidfile resides to the agraph user. This was okay when the recommended pidfile path of /var/run/agraph/agraph.pid was used, but if another path that was shared with non-agraph files was used, the chown was undesirable. This problem has been fixed.

Bug19497 - Fix convertdb

As AllegroGraph v4.0.6 changes the on-disk database format, existing databases need to be converted. The conversion process is performed by the "convertdb" utility, which has been brought up to date with respect to recent storage layer changes.

Bug19499 - Fix SPARQL XML and JSON output of geospatial values

Retrieving SPARQL SELECT results containing geospatial values as SPARQL+XML or SPARQL+JSON was broken because an internals change hadn't been propagated to the SPARQL engine. This is now fixed.

Bug19509 - Agload could load inconsistently on virtual machines

Some virtual machines could reveal a fault in agload that caused inconsistent triple loading. This could result in agload reporting a negative triples-per-second load rate and in different amounts of triples being loaded on different runs. This patch corrects the problem.

Bug19539 - Support dynamic catalogs

Support for dynamic catalogs as documented in the AllegroGraph manual has been restored.

Bug19548 - Correctly import default namespaces in rdf/xml literals.

Previously, when importing XML literals that had a default namespace set, the RDF/XML import in AllegroGraph would silently drop the default namespace. Now, the correct xmlns clause is set for these xml literals.

Bug19551 - Fix handling of blank nodes in RDF/XML parser.

Previously, the RDF/XML parser failed to import empty xml elements like <foo x="y"/> by creating references to empty string literals instead of blank nodes like the RDF/XML syntax specification mandates. Now, the RDF/XML parser correctly creates blank nodes.

Bug19560 - Provide sstab upsizing upgrade path

v4.0.5b increased the sizes of the string table files without changing the database version number. Databases created with v4.0.5a or earlier must be upgraded using the convertdb utility to have this problem corrected.

Bug19562 - Improve error message when AllegroGraph loaded on an unsupported platform

AllegroGraph is currently only supported on linux-64 but the error message produced when trying to load agraph4.fasl on a non-supported platform was nonsensical. This patch improves the error message to make it clear what has gone wrong and what remedies to take. AllegroGraph 5.0 note: the module to require is :agraph5 and the file is agraph5.fasl starting in version 5.0.

Bug19565 - Crash can occur when file descriptor limit is large

When running as a user with a large file descriptor limit (larger than 1024), it was possible for AllegroGraph to crash under certain circumstances. This problem has been corrected.

Bug19567 - Log handles remain open after SIGHUP

Due to a mistake in the way the daemon detached from the terminal, the initial log file would never be closed, even when a HUP signal was sent to the daemon. This is now fixed.

Bug19595 - Repair createGraph() for anonymous graphs

Previously createGraph() in the no-arg case was trying to use an illegal blank node id based on a Jena-internal AnonID; this fixes that problem.

Bug19602 - String table fails to recover short strings from transaction log

An error has been corrected that could have resulted in the loss of string table data if a database was uncleanly shut down.

Bug19614 - Bus error during prolog query

Some queries that returned AllegroGraph specific UPI types (e.g., for reified triple-ids or default-graphs) could cause a bus error rather than signaling the correct message that these types cannot be serialized using RDF/XML. Corrected.

Bug19632 - N-Quads serializer was emitting the default-graph incorrectly

The N-Quads serializer was needlessly including the graph even for triples in the default-graph of the triple-store. Furthermore, it was emitting these graphs using invalid syntax. Both of these issues are fixed.

HTTP Interface

Rfe9885 - Fix HTTP connection keep-alive for 204 responses.

HTTP requests that caused a 204 (no content) response incorrectly had their connection closed by the server, even when they requested keep-alive. This meant the client had to create a new socket for their next request, causing unnecessary overhead.

Rfe9888 - Do not log backtrace for HTTP proxy pipe errors.

When a back-end crashed, this often caused two back-traces to be logged, one for the back-end and one for the front-end. The second one was just noise, so it is now suppressed.

Rfe9906 - Support index inspection and manipulation over HTTP.

Under /indices, a triple store's indices can now be inspected, created, and dropped over HTTP. Adds an index argument to store-creation requests, to set the indices at creation time.

Bug19506 - Make HTTP server more cautious about closing stores

It was possible for the HTTP server to close a triple store that it was still reading from. This would cause the reading request to fail. The server now waits until pending requests have finished when closing one of its stores.

Documentation

Rfe9936 - User document for Top Braid Composer and AG

Added an installation page for the AllegroGraph TopBraid Composer plugin.

Lisp Client

Bug19452 - Race condition in lisp client startup

While connecting to a database, the Lisp client could sometimes crash with the following error message:

The value of db.agraph.spawn::hub is nil,  
which is not of type db.agraph.spawn::hub

A fix has been developed to prevent that problem from occuring.

Python Client

Rfe9906 - Add support for index changes to the Python client

A connection object in the Python client now has listIndices, addIndex, and dropIndex methods for inspecting and manipulating the indices on a triple store.

Rfe9965 - Python miniclient now supports retry on ECONNRESET

The python miniclient now retries on connection failures with errno ECONNRESET. Previously it would just fail the request.

Java Client

Rfe9845 - Support owl:Restriction reasoning in the Java client

The Java client now supports hasValue, someValuesFrom, and allValuesFrom reasoning. In the Sesame AGQuery API one can now specify which entailment regime to use when including inferences in query results; the new Restriction regime is now available as an option, while RDFS++ remains the default. Similarly, using the Jena AGReasoner API, one can set up AGInfModels with either a Restriction reasoner or the default RDFS++ reasoner.

SPARQL

Bug19474 - Improve SPARQL describe queries

Excess variables are no longer included in the output of DESCRIBE
CBD (Concise Bounded Description) logic is now implemented so that we correctly handle blank nodes
Many general efficiency improvements

Bug19499 - Fix SPARQL XML and JSON output of geospatial values

Retrieving SPARQL SELECT results containing geospatial values as SPARQL+XML or SPARQL+JSON was broken because an internals change hadn't been propagated to the SPARQL engine. This is now fixed.

WebView

Rfe9815 - Fix accept-encoding issue for WebView in Internet Explorer 7

Bug19500 - Fix problem with displaying geospatial results in WebView

When a query result contained geospatial data, a bug in WebView was triggered that caused the result to fail to display. This bug has been fixed.

Bug19525 - Fix failure when opening a federated session in WebView.

AllegroGraph 4.0.5d

Welcome to AllegroGraph 4.0.5d. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9966 - Front-end server-socket backlog parameter is too low

AllegroGraph 4.0.5c

Welcome to AllegroGraph 4.0.5c. This release of AllegroGraph v4.0 provides efficiency improvements in the area of reasoning.

Server Changes

Rfe9887 - Exclude inferences from Sesame queries by default

Sesame queries (Tuple, Graph, and Boolean) will now be set to exclude inferences by default (this applies when the setIncludeInferred method has not been called on the query instance).

Bug19484 - Jena always uses reasoning

Fixed so reasoning is off by default and is enabled when InfModel is used.

Bug19485 - setAutoCommit(true) causes exception

True is the default, but calling this would raise an exception, so this has been fixed.

Bug19491 - Jena reasoning not on full graph

In the Jena API, AGInfModel, reasoning was previously done on graphs (default or named), and is now done on the full repository. Reasoning on scoped graphs will be included in a future release.

AllegroGraph 4.0.5b

Welcome to AllegroGraph 4.0.5b. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9866 - Increase limit on string file sizes

Previously, the files that held string data were limited to 32GB each. This release expands that limit to 512GB each.

Bug19503 - Reclaim cursor resources more aggressively

Certain SPARQL operations could acquire cursor resources and not release them immediately. Depending on garbage collection and server load, this could eventaully hang the server.

This patch ensures that SPARQL releases cursors immediately during triple-existence checks which prevents the resource issues from arising.

AllegroGraph 4.0.5

Welcome to AllegroGraph 4.0.5. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9334 - License support for agraph with triple-limit and expiration

AG now requires a license code which includes a limit on the number of triples and an expiration date, both of which may be unlimited.

If the license code is missing, invalid, or expired, it defaults to Free version, 5 million triples with no expiration.

Rfe9348 - Added offset keyword argument to getStatements/evalFreeTextSearch

The getStatements and evalFreeTextSearch methods on RepositoryConnection objects now support an optional offset keyword argument for skipping a number of results before returning values from the result set.

Rfe9471 - Limit maximum checkpoint frequency by configuration variable

A new configuration variable, MinimumCheckpointInterval, has been added that can be set to modify the frequency at which checkpoints are written by AllegroGraph. It defaults to the the CheckpointInterval period or five minutes if interval based checkpointing is not used.

The SkipPostMergeCheckpoint configuration variable has been removed.

Rfe9564 - Alternate mode for checkpoint triggering

In addition to triggering checkpoints in regular time interval, AllegroGraph now offers an alternate mode for triggering checkpoints based on the expected recovery time. It is enabled by the MaxRecoveryTime catalog/database parameters and needs to be set to the maximum number of milliseconds that AllegroGraph should spend recovering transactions from the transaction logfile during recovery. When AllegroGraph detects that the established maximum recovery time would be exceeded, a checkpoint is written. Note that the maximum recovery time period does not include the flat startup fee for a database that is independent of the number of transaction to recover from the transaction log file.

Rfe9604 - Suppress unnecessary checkpoints

In previous releases, AllegroGraph automatically created checkpoints when a database was started, stopped or a certain time interval had elapsed. This has been changed so that checkpoints are only written in these situations if there had been any commits since the last checkpoint has been created.

Rfe9608 - Have HTTP /eval service handle multiple returned values

Extra values returned by evaluated forms used to be discarded. The server now attempts to return them as a list of values, if the Accept format supports such a thing.

Rfe9691 - Remove race condition in internal API

In order to support a new use case, a race condition has been removed from the internal resource manager API.

Rfe9709 - Improved automated test for database recovery

Automated test coverage for database recovery has been improved.

Rfe9713 - Document `load-nquads`

Added the documentation for load-nquads and load-nquads-from-string to the Lisp Reference Guide.

Rfe9726 - Minor agload improvements

improve agload output so that it is easier to understand,
correct agload failure to load the TriX format,
expose support for N-Quad format, and
other minor internal improvements

Rfe9729: Make it possible to disable interval based checkpointing

Interval based checkpointing can now be disabled by setting the CheckpointInterval database parameter to zero.

Rfe9732 - Disable interval based checkpointing when recovery time based checkpointing is enabled

When recovery time based checkpointing is enabled, AG now switches off interval based checkpointing by default.

Rfe9766 - Ensure that the AllegroGraph server and the Lisp client use the same UID

Because the Lisp client must have full read/write access to a database's files and shared memory data structures, it must run under the same UID as the AllegroGraph server. This change enforces this requirement to avoid potentially confusing errors.

Bug19101 - with-variables->bindings-array is too strict

AllegroGraph's SPARQL engine raises an error when imposed bindings (the :with-variables keyword argument to run-sparql) are not a subset of the variables mentioned in the body of the query.

This error checking failed to include variables mentioned in a CONSTRUCT template or DESCRIBE variable list, which meant that query invocations which imposed bindings for variables mentioned in the CONSTRUCT template alone would raise an error.

This is now corrected.

Bug19168, Bug19332 - Ensure that user functions are compiled correctly

This patch improves the way that functions generated at run-time are compiled. This includes SNA generators and select query plans. These should all perform slightly better with this change.

Bug19209 - Fix bug with reasoner and rollback

In certain situations, the RDFS reasoner could become confused and fail to update its data caches after the triple-store was rolled-back. This is corrected.

Bug19339 - Update UPI serialization

There was a printer escape problem in the N-Triple serializer such that only one backslash was output when two should have been. I.e., \a instead of \a. This prevented serialized triples from being re-imported into AllegroGraph. This problem has been corrected.

Bug19429 - Problems with catalog option FilePlacementRule in agraph.cfg

A defect has been fixed that resulted in an error when a database created from a catalog that specified a FilePlacementRule was opened.

Bug19431 - Wrong parameters logged during database startup

A defect has been fixed that made the AllegroGraph server potentially log wrong parameters to the server log file during database startup.

Changes to the Lisp API

Rfe9643 - Remove synchronize-automatically from the Lisp API

In AllegroGraph 3, synchronize-automatically controlled whether or not a triple-store was immediately synchronized after the addition of new triples. Because this level of control is unnecessary in AllegroGraph 4, the variable is being removed.

Bug19188 - [Lisp API] create-triple-store :if-exists :error problem

The Lisp API function create-triple-store behaved incorrectly if the :if-exists keyword argument was :error. An error would occur whether or not the triple-store already existed. Fixed.

Documentation

Rfe9475 - Document catalog parameters vs database parameters

The AllegroGraph server configuration documentation has been updated to include information on how existing databases can be reconfigured.

Rfe9721 - Restore Lisp freetext tutorial

Adds a tutorial that describes the way freetext indexing works in version 4 of AllegroGraph.

Bug19411 - Configuration and performance documentation updated

The server documentation has been updated to reflect the removal of the 'Aux' catalog directive and the addition of the StringTableDir and FilePlacementRule directives.

Java Client

Rfe9743 - Upgrade to Sesame 2.3.1

Previously the Java client was using Sesame 2.2.4; this upgrades the client to the latest version of Sesame 2.

Jena Client

Rfe9744 - Upgrade to Jena 2.6.2

Previously the Java client was using Jena 2.6.0; this change upgrades the client to the latest version of Jena.

AllegroGraph 4.0.4

Welcome to AllegroGraph 4.0.4. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe8792 - Clean up partially created triple stores

When creating a triple store fails, the AllegroGraph server now deletes any files and directories that it may have created up to the point where the error occured. This makes it easier to retry creating the triple store once the condition that made creation fail has been removed.

Rfe9123 - Extend test functions to start and stop db server

The AllegroGraph server can now be stopped by providing the --stop-server switch to the agraph executable. The --config or --pid-file parameters must be present on the command line.

Rfe9292 - AllegroGraph Loader

Added the AllegroGraph Loader tool for parallel loading of large files.

Rfe9367 - AllegroGraph log should include the config file

A change has been made to AllegroGraph so that configuration file directives are written to the server log file at "info" log level during configuration file parsing.

Rfe9451 - Pause text-index merges during checkpoints

We make sure the text-index merger is paused when a checkpoint is taking place, to prevent resource contention from making the checkpoint take long, which would make other subsystems wait.

Rfe9497 - Ensure that select queries release temporary indices

Previously, select queries that used temporary indices (UPI maps) would not free them when the query completed. This could result in poor memory management and eventual poor system responsiveness. This is now corrected because queries release their temporary indices when they complete.

Rfe9508 - Add configuration options for manual control of file placement

Two new configuration options, StringTableDir and FilePlacementRule, have been added to the catalog and database configuration. They can be used to control the placement of individual database files.

Rfe9518 - SHM server handles client failures

Increased robustness of the background shared memory server so that it won't crash when a client shuts down prematurely while communicating with the server.

Rfe9579 - Include catalog name in conditions signaled by the storage layer

Error messages generated by the storage layer now include the catalog name in addition to the name of the database that was being acted upon when the error occured.

Rfe9580 - Use less strict permissions on UPI map and transaction log files

In prior releases, UPI map and transaction log files were created with mode 0600, which prevented them from being read by other users. This made is difficult to share databases. Now, UPI map and transaction log files are created with mode 0666 and the umask is used to determine the final file permissions. This keeps the permissions on these two types of files in line with the other database files.

Rfe9602 - Optimize various UPI coercion functions

Optimized some of the code paths that AllegroGraph uses to convert strings and future-parts into UPIs. This will provide a small performance benefit (1-2%) in some aspects of triple-store loading and querying.

Rfe9642 - Transaction logging adjustments

The recovery code has been adjusted so that automatic recovery after a machine crash is possible if the crash happened while a record was written to the transaction log.

Rfe9642 - Transaction logging adjustments for warm standby

The transaction log now includes marker records for the beginning of a commit that is used by the warm standby mechanism. Existing databases must be upgraded to accomodate for the new transaction log format.

Rfe9654 - Remove Aux configuration parameter

The Aux configuration parameter that was used in previous versions of Allegrograph to make a database use multiple disks for its files has been removed. In order to make use of multiple disks, the TransactionLogDir, StringTableDir and FilePlacementRule should now be used.

Rfe9710 - Make warning about uploading big files through WebView clearer.

Since browsers aren't very good at uploading multi-gigabyte files, WebView makes it clear that there are better alternatives for doing that.

Bug19199 - Lock database directories

When an Allegrograph server opens a database, it will now lock the database directory so that if a second server is configured to use the same directory, it cannot open the database. This protects the database from corruption which would be the result of two servers opening it at the same time.

Bug19203 - Restore the :use-planner option of the `select` query planner

The :use-planner option of the select query planner was being ignored. It now functions correctly.

Bug19221 - Durability issue related to file deletion

A problem has been identified that could result in database files to be wrongly deleted during checkpoint processing, resulting in database corruption. The problem has been fixed.

Bug19251 - Spurious warning message about nonexistent file ctl0

During database creation, the server logged a spurious warning message about a file "ctl0" that could not be deleted. This has been fixed.

Bug19256 - Database recovery fails with partially written checkpoint

A problem was identified that could cause database recovery to fail when the transaction log contained an incomplete checkpoint record. This problem has been corrected.

Bug19263 - Server log contains cut-off lines

Due to a programming error, it could happen that lines in the server log file were garbled or cut off. This program has been corrected.

Bug19264 - Opening databases is slower than it needs to be

A bug has been found that caused opening databases to be much slower than needed. The reason was that the "df" command was invoked by the database server for every file instead of once per directory. This has been corrected.

Bug19279 - Deleted triple data structure maintenance only happens once

After each merge (but only if 15 minutes has elapsed), a task runs which performs maintenance on the data structures which track deleted triples. Due to a logic bug, this operation was only happening once, even if subsequent merges occurred. This has been corrected.

Bug19282 - Improve parsing of XSD single and double floats

AllegroGraph was parsing XSD doubles using a single float parser which led to some values being incorrectly converted to infinity. AllegroGraph was also treating numbers outside the range of IEEE floating point representations inconsistently. It now always converts numbers smaller than the minimum representable value for a float to zero and numbers larger than the representable value for a float to positive or negative infinity (as appropriate).

Bug19303 - Improve algorithm used to find sort-upi-map program

In some cases, AllegroGraph was unable to find an external program needed for UPI map creation. This patch fixes the problem.

Bug19308 - Make namespaces available to code evaluated server-side.

Code run through the eval-in-server service is now given access to the namespaces that the user defined. It used to run in an environment without namespaces.

Bug19328 - Make remote stores raise an error when created while open

It is now an error to call create-triple-store on a store that is still being held open as a remote-triple-store (in Lisp).

Changes to the Lisp API

Bug19298 - Improve the printing of future-parts

(Lisp only) Future-parts were not printing properly when print-circle was bound to true. In addition, future-parts like !<foo> were not printing correctly (the angle brackets were left off). This patch corrects both of these problems.

Documentation

Rfe9339 - Improve 3-to-4 code conversion document

The Java and Jena sections of the 3.x to 4.x code conversion document have been greatly expanded.

Rfe9549 - Many improvements in the Lisp documentation

Improved and clarified many minor issues in the Lisp reference guide.

Rfe9690 - Documentation for AllegroGraph Loader

A documentation page for the AllegroGraph Loader has been added to AllegroGraph 4.0.4.

Rfe9711 - More detail in Java client installation instructions

The instructions for importing the AllegroGraph Java client into Eclipse have been augmented with more detail to make the process easier for new users.

WWW495 - Add LUBM Benchmarks to documentation navigation list

A link to the web page on AllegroGraph's performance on the Lehigh University benchmark (LUBM) has been added to the navigation bar of the Franz documentation pages.

Text Indexing

Rfe8932 - Allow sorting of freetext-searches by relevance.

freetext-get-triples and the HTTP /freetext service now support an argument that will cause the results to come back in sorted order of relevance, using a simple weighting scheme that takes into account word frequencies in the document and commonness of words across documents.

Rfe9356 - Add fuzzy text-index searching.

Freetext search expressions can now contain fuzzy terms, written as (fuzzy "term" [max-distance]) in s-expression queries, and term~ or term~2 in textual queries, which will match anything with a Levenshtein distance less than max-distance (or 1/5 word length when no explicit distance is given).

WebView

Rfe9504 - Improve handling of WebView file uploads to use less memory.

File uploads through WebView went through a few too many memory buffers, causing them to fail on systems without a lot of free memory. They are now directly streamed, and never stored in memory in their entirety.

AllegroGraph 4.0.3

Welcome to AllegroGraph 4.0.3. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe8999 - Use full blank-node IDs in RDF/XML output.

The nodeID properties found in RDF/XML documents created by AllegroGraph can now be sent back to the server to identify the blank node they refer to. They used to be shorter IDs, which were meaningless outside of the document.

Rfe9363 - Revise use of fsync and fdatasync and move to checkpoint time

In previous versions, AllegroGraph used the fsync() system call in various places to synchronize the on-disk state of database files. This could result in unexpected I/O activity when databases resided on ext3 file systems, as fsync() on ext3 always synchronizes all files, not just the file that has been requested to be synchronized. To improve this, AllegroGraph now syncs files only at checkpoint time.

Rfe9490 - increase application's initial heap size

Increase AllegroGraph's initial heap size to reduce garbage collection frequency and thereby improve application performance.

Rfe9505 - Improve efficiency of string dictionary lookup

Increased the efficiency of string lookup by roughly 20%.

Rfe9541 - Server installation script avoids writing double-slashes to agraph.cfg

Previously in configure-agraph, if the user gave a directory name with a trailing slash, the cfg file would be written with double-slashes in various dir names.

Now, the trailing slash is removed to avoid double-slashes. A double-slash is legal in Unix, but is visually unappealing.

Rfe9560 - Truncated cursors free resources more aggressively

Truncated cursors now clean up their environment more quickly which improves general AllegroGraph resource management.

Rfe9567 - Don't print a backtrace on socket timeout errors.

The server no longer logs a backtrace when a socket timeout is raised.

Rfe9586 - Prevent unneeded file reads in metadata-rollback

This provides an increase in efficiency when rolling back a triple store. The metadata system was re-reading its status file on rollback, even when it didn't have to.

Rfe9593 - Optimize string-dictionary

Optimized and stream-lined AllegroGraph's string-dictionary code. This provides a 5 to 6-percent speed increase in many string related operations.

Rfe9598 - Improve efficiency of string table intern operations.

Previously when strings were interned, more work than necessary was being performed if an entry already existed for a string. This extra work is now avoided.

Rfe9665 - Adjusted merge invocation algorithm

Changes have been made to the algorithm which determines when index merges should occur. These changes should make merge behavior more predictable and efficient.

Rfe9673 - Make sort-upi-map operation more verbose

In order to aid in tracking down a hard-to-reproduce bug, the sort-upi-map program now logs information about which files it is manipulating.

Bug19041 - Delete-triple-store only deletes files from the main directory (not the aux)

Previously, if a database used Main and Aux directories and was deleted using delete-triple-store, the files in the Aux directories were not deleted. This has been corrected.

Bug19085 - Fixed race condition in delete-list.dat handling

A file named "delete-list.dat" is used to maintain a list of which files should deleted at the next checkpoint. There was a race condition in the handling of this file which could result in it being deleted at an inopportune time. This problem has been corrected.

Bug19178 - Fix race condition in metadata system

In some circumstances, opening a store or rolling back could cause the system to try to read a non-existent file. This is now solved.

Bug19182 - Performance regression with concurrent writes.

A performance problem has been identified in the mechanism that Allegrograph uses to translate from internally-used logical file names to real on-disk path names. Previously, such lookups were made by a centralized process using a remote procedure call. Now, lookups are solely made within the requesting process without requiring an inter-process exchange.

Bug19192 - Merger may not notice that the database is shutting down

Previously, under certain circumstances, the merger may fail to shut down in an orderly manner during an otherwise normal database shutdown. This problem has been fixed.

Bug19189 - Prevent HTTP back-ends from holding their lock too long

This bug was most visible when using cursors in remote-triple-stores -- the client had to read out the whole cursor before it could make other requests. This is now fixed.

Bug19201 - Catch and log errors raised in the process-services thread.

Errors raised within the per-process thread that managed, for example, back-trace requests were not properly logged, but caused the debugger to be entered. This is now fixed.

Bug19206 - Unrecoverable database due to checkpoint file id reuse

A problem has been identified that could result in an unrecoverable database in situations where there have been unclean shutdowns during recovery. This problem has been fixed.

Bug19222 - Do not fsync temporary files during checkpoint processing

Previously, AllegroGraph called fsync for all files, including temporary files, that make up a database. This has been changed so that temporary files are not fsynced.

Bug19223 - Recovery hangs

A problem has been identified that could cause AllegroGraph to hang during database recovery afer an unclean shutdown. This problem has been fixed by starting the merger processes earlier so that they are available during the recovery process.

Bug19236, Bug19237 - improve geospatial subtype mapping support

The geospatial sub-type mapping code was occasionally being called incorrectly which could lead to spurious errors during query. In addition, the error message reported when a geospaital subtype was not found was being displayed incorrectly. Both of these problems are now corrected.

Bug19244 - Delete old deleted-shmem-files

It has been discovered that the deleted-shmem-files, which are used by the triple deletion functionality, have never been deleted when they became obsolete. This wasted disk space with stale data. The problem has been fixed.

Bug19257 - Deleted hash file deleted at the wrong point in time

A problem has been identified that could lead to the deletion of a deleted hash table file while it was still being referenced. This has been fixed.

Bug19276 - Possible unnecessary post-merge full scan of indices

After an index merge, a task sometimes runs to perform maintenance of the data structures used to track deleted triples. This operation requires a full scan of indices. Previously, this full scan was performed even if it was unnecessary. This has been fixed.

Changes to the Lisp API

Rfe9595 - Reduce unnecessary type-checking from Lisp API functions

The exported Lisp API functions (like get-triples or add-triple) were performing unnecessary type checks. This commit removes these checks and thereby increases the speed of code that uses this API.

This change also means that the Lisp API no longer supports using triple-store names to designate stores. That is, functions now require a triple-store instance and will no longer accept a triple-store name.

Bug19173 - Corrected bug in Lisp API for remote-triple-stores

Previously, cursors for remote-triple-stores could produce an error when a query returned too many unique strings. This error will no longer occur.

Bug19177 - Fix broken interpretation of namespaces in Sesame transaction documents

The 'name' and 'prefix' attributes of the addNamespace and removeNamespace tags were interpreted the wrong way. ('name' should hold the URL, and 'prefix' the short name).

Documentation

Rfe9339 - 3.x to 4.x code conversion document added

At the request of users, we have provided a document that lists the changes in the AllegroGraph Lisp, Python, and Java APIs in the upgrade from AllegroGraph 3.x to AllegroGraph 4.x versions.

Rfe9432 - Updates to Python API document

There were several updates to the Python API documnent to better delineate the distinction between public and private methods in 4.0.3.

Rfe9504 - Document TempDir configuration directive

The location where the server creates temporary files can now be controlled by specifying a TempDir option in a configuration file. This option is now described in the Daemon Configuration web page.

Rfe9540 - Audited the Python API doc for currency

The Python API document has been updated to reflect recent changes in the Python API.

Rfe9599 - Update Java section of 3-to-4 conversion document.

The v3-to-v4 code conversion document, convert3to4.html, has been updated to enlarge on the information about the new AllegroGraph Java Client. [5.0 note: the convert3to4.html document is no longer part of the documentation set.]

Bug19320 - Include links to upgrading information in documentation

Added links from the navigation section of each document to the "converting from AllegroGraph 3 to 4" documentation.

SPARQL

Rfe9561 - Improvement in SPARQL recycling of storage-layer cursors.

The SPARQL query engine now more aggressively discards and recycles cursors over on-disk indices. This improves the behavior of the system under load.

Gruff

New overview/scroller panes

The View menu has three new commands for using an overview/scroller pane.

New submenu File | Export Displayed Data As ...

A new "Export" child menu on the File menu allows exporting the currently-displayed triples or nodes to a file in one of several standard formats.

Child menu names are now included in the documentation.

Until now, a menu command that's on a cascading child menu, such as "Drawing Options | Derived Node Labels | Capitalize First Word", appeared in the docs as simply "Drawing Options | Capitalize First Word", which could make it hard to find on the menus. The docs now include the full path.

New freetext command "Add | Edit the Set of Registered Predicates".

There are new commands for registering some or all predicates for freetext queries at one time, and for unregistering some in agraph4.

Rfe9090 - Initial port to the agraph4 freetext API

The freetext commands now work on agraph 4.

Export commands failed when bridge lines were present

The new File | Export commands failed when bridge lines are present. This is now fixed.

AllegroGraph 4.0.2

Welcome to AllegroGraph 4.0.2. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9084 - Return Prolog parse errors as bad requests, not internal server errors.

When issuing a Prolog with a syntax error, the response no longer indicates an internal server error, but is returned as a bad request error, tagged "MALFORMED QUERY".

Rfe9313, Bug18592 - Improved clause reordering in `select` queries

Modified the clause size estimation code to make it work better with AllegroGraph 4.0. The change greatly increases the speed of some Prolog-based select queries.

Rfe9345 - Ensure that distribution does not contain extra debug scaffolding

We now ensure and test that certain developer tools and debugging aids are not part of the distributed AllegoGraph. Removing these from the released code will have a small performance gain in some parts of the system.

Rfe9351 - Don't write deleted hash table if no triples have been deleted

Previously, the shared memory hash table that is used to track deleted triples was written to a file at every checkpoint. With this change, the table is only written when deletion of triple actually has happened. This speeds up trivial checkpoints as well as database open and close operations by about 35%.

Rfe9404 - Improve error message when bulk loading N-Triple data

The error signaled when loading invalid N-Triple data has been made clearer.

Rfe9435 - Support offset argument to geospatial query services

For consistency, all services that support a limit argument now also have an offset argument, which allows results to be skipped.

Rfe9440 - Improve cursor code efficiency in some situations

Improved some low-level cursor implementation details that provide a small but measurable speed and space boost for some queries.

Rfe9447 - Improve detection of SPARQL Negation-As-Failure idiom.

This change allows for SPARQL's idiom for Negation-As-Failure to be detected more reliably, and thus executed more optimally.

In particular, this allows for the OPTIONAL component to contain FILTERs, so long as they refer only to variables bound within the OPTIONAL itself.

One such query might be:

PREFIX foo: <http://www.example.com/foo/foo_model#>  
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>  
SELECT ?foo ?type  
WHERE {  
   ?foo a foo:Bar .  
   ?foo a ?type .  
   OPTIONAL {  
    ?subType rdfs:subClassOf ?type .  
    FILTER(?subType != ?type)  
   }  
   FILTER(!BOUND(?subType))  
}

Rfe9448 - Add upi< and upi<= functors to prolog select

Added additional functors to the Prolog-based select language to make some operations easier to use.

Rfe9462 - Improved database shutdown process

The shutdown process that AllegroGraph uses when a database is closed has been improved to be more orderly and reliable.

Rfe9464 - Support adding and removing of namespaces from Sesame XML transactions

A client program that uses the <setNamespace> and <removeNamespace> tags has been shown to exist (TopBraid), so they have been implemented.

Rfe9493 - Added mixed workload to events test

A -m parameter was added to the events test to run a Phase 6 mixed workload after the normal phases complete. This runs a mixture of queries, adds, and deletes until it is interrupted.

Bug19070 - /etc/init.d/agraph 'status' doesn't work

Previously, running "/etc/init.d/agraph" resulted in an error message instead of reporting the status on the agraph service. This has been fixed.

Bug19078 - Improve error message for invalid use of create or open-triple-store

The error message signaled when creating (or opening) a triple-store with incorrect parameters has been improved. The documentation of both functions is also clearer.

Bug19089 - Make sure daemon doesn't hang when it encounters an error

Turns off a debugging switch that was, unintentially, switched on by default, causing the service daemon to enter the debugger when encountering an error.

Bug19090 - Improve handling of graph-filtered-stores

Graph filtered triple-stores are used in various places in AllegroGraph's query engine. Prior to this fix, they did not implement some of the necessary API and could generate an error.

Bug19091 - Prevent server shutdown from locking up when a session is open.

The server would lock up when sent a sigterm if an HTTP session was still open. This is now fixed.

Bug19101 - Error detection for with-variables and CONSTRUCT queries.

AllegroGraph's SPARQL engine raises an error when imposed bindings (the :with-variables keyword argument to run-sparql) are not a subset of the variables mentioned in the body of the query.

The resolution for this bug corrects the issue.

Bug19121 - Fix bug in namespace caching for HTTP back-ends.

Occasionally (depending on the system clock rolling over to the next second at the wrong moment) a newly defined (or deleted) namespace wasn't being seen by a back-end, causing 'unknown namespace' errors. This is now fixed.

Bug19123 - Allow reasoning-triple-stores to create new-blank-nodes

Reasoning-triple-stores (and other encapsulated-triple-store classes) were unable to mint new blank IDs. This prevented them from loading any RDF data that contained blank nodes. This fixes the problem.

Bug19145 - sendfd library does not handle "out of file descriptors" condition gracefully

When a client process experienced file descriptor shortage while connected to an Allegrograph store, memory corruption could occur. This has been corrected.

Bug19151 - Refcounts on open indices not updated aggressively

Previously, if regular query operations were performed against a database that was continually growing, file handle exhaustion could occur due to incorrect reference counts on indices. This problem has been fixed.

Bug19155 - Correct printing of invalid-characters-in-name condition

This fixes a syntax error in the formatting of the condition used to signal an invalid triple-store name or catalog. Previously, the error message produced was difficult to interpret; now it is clearer.

AllegroGraph Server: HTTP Interface

Rfe9227: Allow an HTTP proxy to be configured for the server

The HTTPProxy configuration directive can now be used when the server has to use an HTTP proxy to connect to the internet (for example, to load data from a URL).

Rfe9318 - Remove persistent federations from the HTTP protocol

Accessing federations of stores is now done by opening a session for such a federation, not by creating a persistent federated store and opening that.

Rfe9450 - Allow committing and rolling back through HTTP headers

As a performance optimization, it is now possible to have commit or rollback commands piggyback on other HTTP requests, using the x-commit and x-rollback HTTP headers.

Rfe9458 - Expose a full-featured textual free-text query language over HTTP

The 'pattern' argument to the /freetext service now understands strings like 'foo | bar' (foo OR bar), 'foo*' (everything starting with foo), '"foo bar"' (matching the exact string "foo bar") and even 'quux (foo | bar)' (quux AND (foo OR bar)).

Rfe9458 - Tweak free-text query expression format

When searching from the Lisp API, a phrase search is now done with '(phrase "foo bar") rather than "\"foo bar\"", and a wildcard match is done with '(match "wild") rather than just "wild".

Rfe9483: Add continueOnError parameter to HTTP statement loading

For N-Triples and N-Quads data, this argument can be used to make the load continue when malformed data is encountered, skipping the broken triples.

Rfe9496 - Support clearNamespaces tag in Sesame XML transaction documents

This tag was overlooked in the original implementation. It is now supported.

Rfe9499 - Support prepared queries in the HTTP protocol

It is now possible to 'prepare' SPARQL queries, either as a side-effect of executing them, or as a separate action. The requests for executing the prepared query will be more lightweight, since they don't have to pass in the query itself and most of its arguments, and skip the query-parsing stage.

Bug19088 - Allow a default namespace to be defined over HTTP

Due to the server stripping off trailing slashes from URLs and redirecting, the PUT request to ../namespaces/, which is required to create a namespace without a name, was redirected to ../namespaces, and didn't work. The redirecting behaviour is now turned off for this URL.

Changes to the Lisp API

close-triple-store has new 'commit' keyword

close-triple-store has a new 'commit' keyword argument.If true, then the commit-triple-store will be called before the store is closed.

user-reasoner keyword removed

The deprecated use-reasoner keyword argument has been removed from make-tutorial-store; use apply-reasoner-p instead.

sync-triple-store has been removed

The deprecated sync-triple-store has been removed; use commit-triple-store instead.

Rfe9576 - :expected-size argument to open-triple-store and create-triple-store no longer supported

To specify the expected size of a new database, edit the catalog configuration file before creating the database. To change the expected size of an existing database, change the ExpectedStoreSize parameter in the parameters.dat file of the database.

Bug19046 - Fix error generated by future-part printer

If a namespace mapping did not exist, then the future-part printer could signal an error instead of correcty printing the part. [Lisp API]

Bug19081 - Change return type of ego-group-layers from vector to list

The ego-group-layers function was incorrectly returning a vector of results instead of a list. Its behavior now matches the documentation: it returns a list of lists where each sublist contains nodes discovered at that level. [Lisp API]

Documentation

Improved Lisp reference manual documentation

Made several minor formatting changes and added documentation for the upi<= function and functor.

Document new free-text index HTTP API.

The creation and configuration of multiple text indices over HTTP is now described in the protocol document.

Rfe9169 - Add links to Javadocs from AllegroGraph documentation

Added links from the navigation bar of each AllegroGraph documentation webpage to the Javadocs.

Rfe9314 - Update Lisp API docs to reflect changes in freetext-indexing API

Indexing is no longer managed by keeping a list of predicates, but by creating one or more indices, each with multiple configurable parameters.

Rfe9318 - Update the HTTP protocol documentation for changed federation API

The services for creating and deleting persistent federations are gone, a more flexible way to open sessions, which also allows federation of stores, has been added.

Rfe9377 - Update free-text indexing example in Python tutorial

The 12th example in the Python tutorial now shows how to create multiple indices and search for both text literals and URIs. The example introduces these new methods:

createFreeTextIndex  
getFreeTextIndexConfiguration()  
evalFreeTextSearch()

Rfe9378 - Update Python documentation to reflect new federation API

The Python tutorial has been updated to reflect recent changes in the Python API for creating federated sessions.

Rfe9383 - Document new-style federation in the java client

The 16th examples in the Java tutorial has been updated to reflect the new API for creating a federated repository.

Rfe9500 - Update free-text-index entries in Python API html doc.

The connection object methods for free-text indexing have been updated to reflect recent changes in the API.

Java Client

Rfe9318 - Support session-based (rather than persistent) federation

The methods in AGServer for creating and deleting federations have been replaced by a single method, federate, which takes any number of repository objects, and returns a 'virtual' repository that federates all of them.

Rfe9392 - Use accept-encoding gzip to get back compressed results

Get and Post requests now use the Accept-Encoding: gzip header to request that responses be compressed, to improve performance when results are large.

Modified both tutorial examples files to call closeAll prior to deleteRepository in exampele1, as not doing so can result in orphaned sessions on the toClose list.

Rfe9436 - Make session lifetime configurable

Session lifetimes can now be configured for each connection, and should be set prior to making a method call that initiates a dedicated session. The default session lifetime is 1 hour. Tutorial example 22 sets the session lifetime to 120 seconds to demonstrate.

Rfe9442 - Add basic information to the jar manifest

The Java client jar manifest now includes information about the current implementation, including name, version, vendor name, vender URL, time of creation, etc.

Rfe9499 - Support query preparation

Calls to prepareTupleQuery, prepareGraphQuery, and prepareBooleanQuery now return query objects that will, during the next call to evaluate(), request that the server save the query for future reuse (in dedicated sessions only), eliminating some parsing and other overhead in subsequent calls to evaluate(). Tutorial example 14 has been modified to use non-autocommit mode so that it demonstrates query preparation in a dedicated session and multiple calls to evaluate() with different variable bindings.

Bug19086 - JenaTutorialExamples now uses Graph#getName()

The tutorial examples now use Graph#getName() to get the graph's name, rather than using Graph#toString(), which now prints out additional human readable information about the graph.

Bug19092 - Jena ResultSet and PrefixMapping compliance

AGPrefixMapping now supports methods qnameFor, removeNsPrefix, samePrefixMappingAs, setNsPrefixes, shortForm, expandPrefix, lock, and withDefaultMappings. The toString method now returns human readable information about the prefix mapping.

AGResultSet now supports methods getResourceModel, getResultVars, and remove.

AGQuerySolution now supports methods contains, get, getLiteral, getResource, and varNames.

Python Client

Rfe8610 - Help finding pairs of UPIs

The 23rd example in the Python tutorial has been extended to demonstrate how to use inequalities to reduce the number of permutation matches in SPARQL and Prolog queries.

Rfe9311 - Expose revised FTI API in the Python client

Text-indexing is no longer configured by adding predicates, but by creating (and updating or deleting) multiple separate indices.

Rfe9318 - Move to session-based federation API.

To federate stores, one now calls AllegroGraphServer.openFederated, which opens a session that federates the given stores, rather than creating a persistent federated store and opening that.

Rfe9357 - Support wordFilter option to text-indices in Python client

The createFreeTextIndex and modifyFreeTextIndex methods of a connection now allow a wordFilters argument to be specified, which allows accent removal and word stemming to be configured.

Text Indexing

Rfe8775 - Support multiple text-indices per repository, add configuration parameters.

Multiple free-text indices can now be active in a repository, and there is an API to create, query, modify, and delete them. New parameters supported are: stop words, minimum word length, whether to index resources, which literals to index, indexing non-object parts, and the possibility to index all triples, regardless of predicate.

Rfe8775 - Add triples to free-text index as soon as they are added.

A commit used to be necessary to be able to find newly added triples through the free-text index. They can now be queried (by the current transaction) as soon as they are added.

Rfe9357 - Add word-filters to text-index configuration (accent removal, stemming)

It is now possible (both in the Lisp API and over HTTP) to specify a list of word filters when defining a freetext index. These are operations that are performed on every word that is indexed, and every word that is queried, allowing various 'normalisations' of words. Currently allows two filters, "drop-accents" to remove accents from letters, and "stem.english" to apply basic, English-language word stemming.

Changes to WebView

Rfe8775 - Allow free-text reconfiguration through WebView.

WebView now allows the creation, modification, and deletion of free-text indices, exposing a UI for all the new configuration parameters.

Rfe9408 - Expose free-form sessions in WebView

There is now, for users with the "session" permission flag, a control on the main WebView page for starting and entering a session over a federated, reasoning, filtered, and/or remote store.

Rfe9438 - Show server version in WebView

WebView now shows the exact version of the server in its header.

Rfe9453 - Support bulk-adding of namespaces in WebView

Clicking 'bulk input' when adding a namespace now pops up a new input dialog, where a set of namespaces can be typed or pasted all at once.

Bug19105 - Restore Safari-compatibility for WebView

WebView was broken on recent versions of the Safari browser. This has now been remedied.

AllegroGraph 4.0.1

Welcome to AllegroGraph 4.0.1. This version of AllegroGraph contains updates to the AllegroGraph 4.0 release, including:

Cursor optimizations for increased performance
Improved caching of RDFS++ reasoning results
Improved SPARQL range-query efficiency
Heuristic optimization of negation as failure in SPARQL queries
SPARQL query performance tuning documentation added
Hardware Profiles added to Performance-Tuning Guide

This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.

Server Changes

Rfe9137: Heuristic optimization for negation as failure in SPARQL queries.

SPARQL queries which use the common OPTIONAL/FILTER (!bound(?foo)) idiom to express negation as failure are now handled more efficiently, computing fewer unnecessary intermediate results.

This optimization can be disabled by setting the non-exported variable sparql.algebra::rewrite-naf-p to nil.

Rfe9320: Improved caching of RDFS++ reasoning results

Improved RDFS++ reasoner efficiency by selectively caching intermediate results. This makes many queries involving the reasoner between 2 and 10 times faster.

Rfe9352, Bug19067, Bug19068: Improve internal cursor operations

Optimizing how AllegroGraph treats empty cursors (i.e., cursors known to return no results) so that operations with them are faster.

Rfe9374: Improved SPARQL range-query efficiency in some cases

Improved the detection of range queries that will return no results which allows SPARQL to optimize other cursor operations.

Bug18978: Optimizing cursor creation and filtering

Improved the way AllegroGraph internally manages cursor creation and triple-filtering to increase performance.

Documentation

Hardware Profiles added to Performance-Tuning Guide.

The performance-tuning guide has a new section describing the kind of hardware Franz recommends for users who will be deploying applications on the scale of 3-4 billion triples.

Rfe9374: SPARQL query performance tuning documentation added.

The performance tuning guide has been extended to describe possible approaches to handling typed literals to improve SPARQL query performance for numeric and date FILTERs.

AllegroGraph 4.0

Welcome to the new AllegroGraph v4.0. This latest version of AllegroGraph is designed take full advantage of symmetric multi-processing hardware and brings enterprise database features to the world of Semantics and RDF. AllegroGraph, now incorporates:

Full ACID database compliance.
Full Recoverability.
Online backup and restore.
Automatic Indexing.
HTTP-based client/server libraries supporting: Java, Sesame, Jena and Python.

This release of AllegroGraph v4.0 is focused on enhancements to database backup and restore, Java Jena client library compliance, and overall product stability improvements.

Server Changes

Add a Lisp function to list the catalogs on a server.

The function #'catalogs, exported from the db.agraph package, can now be used to find out which catalogs a server provides.

Added ego-group-layers function

ego-group-layers returns a nodes ego-group organized by the depth at which each node was found. [Lisp API]

Documentation: ego-group-layers (node depth generator)

Return a list of lists of the nodes in nodes ego group.

Each element of the result is a list of the nodes discovered at that list's depth.

These are the nodes in the graph that can be reached by following paths of at most length depth starting at node. The paths are determined using the generator to select neighbors of each node in the ego-group. See the description of SNA generators for additional details.

Added support for `auto-commit` to the bulk load functions

The commit parameter to load-ntriples, load-rdf/xml and load-trix can be one of

nil - do not commit
t - commit at the end of the load
a number N - commit every N-triples

Changed behavior of breadth-first-path-search when maximum-depth is given.

Previously, when the breadth-first-path-search functions were given a maximum depth, AllegroGraph would return the shortest path or paths found that were no longer than that depth. Now, AllegroGraph returns all paths found that are no longer than maximum depth.

This can be very expensive for graphs with high connectivity. To return only the shortest path, use a maximum depth of nil. [Lisp API].

Since the Prolog SNA functors use the underlying SNA functions, their behavior has also changed.

Change the keyword value used to identify typed literals

Previously, AllegroGraph used :typed-literal to specify a typed literal. To make things more consistent with :literal :literal-language, this has been changed to :literal-typed. [Lisp API]

Define an error type to signal failures to connect to the AllegroGraph server.

Any operation (opening, creating, deleting, probing) that has to talk to the server will now raise a condition of type db.agraph:could-not-connect-to-server-error if it fails to find a server on the specified port.

Fix mistake in instance-subprocess communication protocol.

This would cause the instance to go into an infinite loop and become unresponsive when one of its children died.

Make delete-duplicates-on-merge setting persistent.

The flag was not properly restored when opening a triple store.

Rfe7381 - Add a db keyword argument to several API commands Added the db argument to the following commands

part->string
part->value
part->ntriples
part->long
part->concise
part->terse
print-triple

[Lisp API]

Rfe8765 - Improved Prolog query performance

Many Prolog queries execute more than 30% more quickly.

Rfe8860 - Geospatial functions now accept strings

The geospatial functions in AllegroGraph previously required a subtype object. They now accept strings which name a subtype object. This makes them easier to use. [Lisp API]

Rfe9146 - Allow 'merging' of store metadata on concurrent change.

Before, two sessions both adding a type-mapping and then committing would cause a conflict error. The metadata system is now clever enough to merge such changes.

Rfe9209 - Don't log process termination messages before cleanup.

Both instance processes and their children used to output '... terminated' before they cleaned up, which in some cases led to them hanging or crashing after they claimed to have terminated.

Rfe9212 - Periodically clean up deleted triple data structures

Previously, the data structures which track deleted triples could grow without bounds. Now, they are periodically examined and obsolete portions are purged.

Rfe9291 - Use XSD instead of internal type-codes for /mappings

The /mapping protocol now returns lists of

mapping-kind datatype/predicate XSD-type

rather than

mapping-kind datatype/predicate numeric-type-code

This makes both debugging and general use of the mappings much simpler for clients.

Rfe9294 - Added SNA path functors to bind only the first matching path

Added breadth-first-search-path and bidirectional-search-path functors to correspond with the existing depth-first-search-path functor. These all run SNA path finding algorithms and stop after finding the first matching path.

Rfe9315 - Do early probe to see if file system supports >4GB file offsets

Allegrograph requires that the underlying file systems support offsets longer than 32 bits. Some file systems (i.e. NFSv2) do not support long offsets, which caused file system related failures in varying situations. Allegrograph now probes each of the file systems specified in a configuration to make sure that long file offsets are supported.

Rfe9322 - Changes in encoded date and time UPIs.

Corrected a problem where the fractional seconds stored in an encoded UPI could be lost. The update fixes the stored representation and invalidates existing triple-stores that have any time, date or date-time encoded UPIs.

Rfe9329 - Improved loading of a required shared library.

Previously, AllegroGraph required loading of a particular shared library (librt) that did not always exist by the same name on all Linux distributions. This problem has been corrected.

Rfe9330: Set up selinux security context on shared library during installation

Previously, on systems with SELinux enabled, the agraph server would not start up properly unless the operator issued the following command first:

chcon -t textrel_shlib_t <agraph lib dir>/libacl*.so

Now, this command is automatically performed during RPM package install or when using the install-agraph script that comes with the tarball package.

Rfe9340 - Disallow overlapping catalog directories.

Having, for example, one catalog in /tmp/foo, and another in /tmp/foo/bar, or two catalogs in the exact same directory, is no longer allowed, since this is bound to cause stores to be corrupted.

Rfe9362: Output a log message when deleting a triple-store.

Deleting a store wasn't visible in the log before. It now notes which store was deleted.

Spr36556 - Reduce number of duplicate rdf:type inferences

AllegroGraph's type inference was returning a triple for each possible inference path rather than filtering out the duplicates. I.e., if a rdf:type B could be inferred two ways, then the reasoner would return two triples. This no longer happens.

Spr 36565 - Standardize the printing of encoded types

AllegroGraph would sometimes print typed literals non-syntactically. This no longer occurs.

Bug18679 - Made part->value function more consistent

The part->value function now returns type-codes for both UPIs and future-parts. Previously, it returned a type-code for UPIs and a keyword (like :literal or :anon) for future-parts. [Lisp API].

Bug18679 - Removed the deprecated part-value

The deprecated part-value function has been removed. Use part->value in its place. [Lisp API]

Bug18738 - Make query-planner work on remote-triple-stores

Previously, select queries that needed to reorder their clauses would instead signal an error on remote-triple-stores. This patch corrects this problem and lets remote-triple-stores work correctly with the query-planner. [Lisp API]

Bug18840, Bug19003 - Inconsistent state after duplicate purging

Total triple counts were wrong after duplicate purging. By default, AllegroGraph purges duplicate triples from the index during merge operations. Previously, when duplicate triples were purged, the total count of triples in the store was not updated. Also, in most cases, one of the seven triple indices would not be purged of duplicates, resulting in wasted storage and potentially inconsistent query results. These problems have been corrected.

Bug18879 - Corrected add-triple so that it could accept large strings

Unlike the bulk loaders, the add-triple function was unable to accept strings longer than 8092 characters and would signal an error. Now add-triple can accept strings of arbitrary size.

Bug18800 - Unbound prolog variables could pollute `select` query results

Queries like

(select (?s ?t)  
 (q ?s !<http://franz.com/sna#marriage>  
!<http://franz.com/sna#peruzzi>)  
 (optional (q ?s !rdf:type ?t)))

could return unbound expessions like `{Unbound 14a170c9}` instead of `nil`. Now these queries correctly return `nil`.

Bug18880 - Corrected serialization of RDF/XML

The serializer sometimes added extra angle brackets around typed literals. Now it correctly outputs them as, e.g.,

"42"^^<http://www.w3.org/2001/XMLSchema#int>

Bug18947 - Make service daemon responsible for deleting stores

delete-triple-store used to only check whether the store was open in the current image, and thus deleted stores even if another process had them open. This moves the responsibility for deleting stores to the service-daemon, which actually knows whether anyone has the store open.

Bug18949 - Fix geospatial type mapping in federated stores

AllegoGraph was not handling geospatial type-mappings in federated-triple-stores and could fail to return all of the triples from every leaf store. This is corrected.

Bug18972 - Correct printing of times with fractional seconds

When a time or dateTime with fractional seconds had leading zeros in the fractional seconds part, AllegroGraph could fail to print it. I.e., a time like 10:31:55.028192 might print as 10:31:55.281900. This printing problem has been corrected.

Bug19010 - Fix 'store in use' bug when accessing store over HTTP.

Fixes a bug where a store accessed over HTTP could not be deleted or replaced through the direct Lisp API until its handle had timed out.

Bug19018 - Make sure instance processes are cleaned up when breaking

Fixes a few cases where unexpected errors would cause the daemon to get confused about the stores it had open (instance crash, client dying while opening or creating).

Bug19024 - Show catalog name in store-status error messages

Errors like 'store exists' and 'store not found' used to only contain the store name, now they also have the catalog. Trying to perform an operation on a non-existent catalog now raises an error that clearly indicates the catalog doesn't exist.

Bug19045 - Use . as the fractional seconds separator in times and dateTimes

Previously AllegroGraph used the comma (,) as the separator in its printed representation of XSD times and dateTimes. This is acceptable ISO 8601 notation but not acceptable XSD notation. This bug fixes the problem so that fractional seconds are separated from whole seconds by a period (.).

Bug19055: Fix potential hang when committing

Under certain circumstances, commits could hang indefinitely while waiting for chunk merging to complete, but chunk merging would never commence due to a counter having an invalid value. This problem has been corrected.

Bug19061: upi->String table race condition fix

Previously, under high load conditions, errors could occur during string table operations. This problem has been fixed.

AllegroGraph Server: HTTP Interface

Allow nested lists in JSON-shaped Prolog-query output.

application/json results for Prolog queries can now contain nested arrays of results, to make it easier to use some of our SNA functionality.

Allow the creation of sessions with federated, graph-filtered, or reasoning stores.

It is now possible to spawn HTTP sessions on a store built up by applying an arbitrary combination reasoning, federation, and graph-filtering to concrete triples.

Cause output of --http-trace switch to actually get flushed to the file.

Before, it was buffered, and would only be visible after a large amount was written, or the server was shut down.

Clean up namespaces when a repository is deleted.

Before, deleting and re-creating a repository would leave the old namespaces intact.

Enable streaming of XML bodies when loading statements.

A problem in our SAX parser made this difficult before, but that has since been fixed. This makes it possible to load large files with less overhead.

Fix a bug that prevented session permissions to show up in WebView user-contol UI.

(No further text.)

Fix a bug with namespace caching.

Sometimes, back-ends wouldn't notice a user's namespaces had changed when they were updated twice within the same second.

Fix bug in remote geospatial queries.

There was a problem that broke remote-triple-stores when more than one geospatial subtype occurred in a query response. This is now fixed.

Fix user-access bug in HTTP protocol.

Fix a bug that prevented users with access to a store in the root catalog, but not the whole catalog, from accessing this store over HTTP.

Make user-access-configuration interface in WebView more obvious

It was easy to mistake the widget for adding access for an indication that access had already been granted.

Support changing user passwords through the HTTP interface

There is now /users/[NAME]/password, to which one can POST a new password.

Support resetting namespaces to the defaults.

There is now a reset argument to the namespace clearing interface, which, when true, will cause the default namespaces to be restored.

Rfe9131 - Add defaultGraphName parameter to SPARQL query interface

This is used to provide a way to refer to the default graph inside a SPARQL query using a URI.

Rfe8999 - Use a unified format for serializing blank nodes.

Before, each component had its own format. This moves the HTTP server to the unified format.

Rfe9183 - Silence AServe's socket-reset warnings

The log is no longer polluted with spurious connection-reset messages.

Rfe9185 - Support delete-duplicates-during-merge remotely

Remote-triple-stores now support the :delete-duplicates-during-merge initarg, and the db-delete-duplicates-during-merge accessor.

Rfe9217 - Add more help text to the WebView interface.

Potentially confusing input fields in WebView (such as those where resources or literals are entered) now have a question-mark icon that pops up an explanation.

Rfe9236 - Support SPARQL/Update over HTTP.

POST requests to a repository can now issue a SPARQL/Update query, if the user has write access to the store.

Rfe9247 - Expose server processes in HTTP interface

It is now possible to list, inspect, and kill the server's processes through the HTTP interface. Additionally, one can start up telnet servers in the processes, in order to directly debug them.

Rfe9247(2) - Allow process control from WebView

Superusers can now view the server processes, and open telnet servers in them, through the WebView UI. (See the 'Processes' link in the navigation bar at the top-level page.)

Rfe9295 - Add a commit argument to PUT/POST /statements.

This supports auto-committing every X triples, to make uploading huge files viable.

Rfe9310 - The PUT /repository/[name] now overwrites by default.

The PUT /repository/[name] service in the HTTP protocol now overwrites existing repositories by that name unless override=false is passed.

Bug19002 - Fix bug in HTTP protocol namespace caching.

Fix a bug that caused problems with namespaces after deleting and re-creating (over HTTP) a store that had namespaces defined.

Bug19005 - Corrected problem in client lookups of individual datatype mappings.

Previously, a request for an individual datatype mapping would return nil regardless of how the mapping was defined. Now it returns the correct value.

Documentation

Server Installation Page Updated to be Ubuntu-Friendly

The server-installation instructions have been updated to make them more friendly to people who don't know how to install Python, python-pycurl, and python-cjson packages. This included some specific suggestions for Ubuntu users for RPM installation and Python client installation.

Rfe9322 - Added client tutorials to the documentation

The documentation now contains tutorials on how to use the Python and Java clients. Please see the links off of the main web pages.

Java Client

Expanded Coverage of Literal Values in Java Tutorial.

The Java tutorial has been revised and expanded to document the behavior of literal typed values (in example5()). This includes the default behavior of untyped literals, as well as their use in getStatements queries as well as filtered and direct SPARQL queries. The section covers strings, floats, ints, untyped values, dates and times.

Java Tutorial Describes Managing User Accounts

The Java tutorial has a new section on "Creating AllegroGraph Users with WebView." It describes how to create a manage new AllegroGraph user accounts.

Rfe8812 - Jena compliance improvements

Added compliance tests for Jena GraphMaker and Model interfaces and corrected several non-compliances.

Properly handle the case of getting a namespace uri for a prefix that does not exist, and add support for expanding a QName.

Add GraphMaker support for creating anonymous graphs, listing graphs, checking existence of a graph, and returning existing graph objects rather than creating new ones when possible.

Throw AlreadyExistsException when attempting to create an existing graph in strict mode, and throw a DoesNotExist exception when attempting to remove or open a non-existent graph in strict mode.

Rfe8812 - Jena Graph compliance improvements

Added compliance tests for the Jena Graph interface and corrected several non-compliances.

AGBulkUpdateHandler's removeAll method properly clears a graph.

Added a custom AGCapabilities class declaring handlesLiteralTyping to be false, as AGGraph does not currently support D-entailment.

Add support for the executeInTransaction method in AGTransactionHandler, though using begin/commit/rollback explicitly is preferred.

AGTripleIterators are now tied to AGGraphs, calling remove on the iterator will delete from the graph.

Rfe9359 - Jena Prolog support.

Added support for Prolog select queries over a Jena model. Queries must be written to select for variables that are bound to RDF parts, rather than to arbitrary Lisp objects.

Python Client

Removed Common Logic

Removed the experimental and undocumented client side Common Logic query language. Please contact [email protected] for information if you were using this language.

Removed JDBC result sets

Removed the ability to get JDBC result sets. Please contact [email protected] if you need ways to get the result data without instantiating the corresponding Literal, URI, or BNode objects.

Namespace implementation is now server-side

Removed the client-side namespace implementation in favor of a server-side namespace management.

"Duplicate Triples" added to Python Tutorial.

The Python Tutorial has a new section on "Duplicate Triples", based on example23(). It describes the sources of duplicate triples and repetitive query results, and prescribes techniques for bringing them under control.

Python Tutorial Describes Managing User Accounts.

The Python tutorial has a new section on "Creating AllegroGraph Users with WebView." It describes how to create a manage new AllegroGraph user accounts.

Expanded Coverage of Literal Values in Python Tutorial.

The Python tutorial has been revised and expanded to document the behavior of literal typed values (in example5()). This includes the default behavior of untyped literals, as well as their use in getStatements queries as well as filtered and direct SPARQL queries. The section covers strings, floats, ints, untyped values, dates and times.

Python Examples from Command Line

The Python tutorial example file (tutorialexamples40.py) can now be run from the command line.

$ python tutorial_examples_40.py  runs all tests.  
$ python tutorial_examples_40.py all  runs all tests.  
$ python tutorial_examples_40.py 1 5 22   runs tests 1, 5, and 22

Rfe9295 - Changes addStatements and addFile

addStatements and addFile can now auto-commit triples based on a count for large loads of triples. Set the add commit size to a positive integer by calling setAddCommitSize or setting the addcommitsize property on the RepositoryConnection for size-based commits, or set to None or 0 for regular commit semantic on the add methods.

Using this feature can decrease the server's memory usage during loading.

Rfe9316 - RepositoryConnection.setRuleLanguage removed

RepositoryConnection.setRuleLanguage has been removed. Both addRules and loadRules already take an language parameter which defaults to QueryLanguage.PROLOG as the only language currently supported for rules.

Bug19001 - Changes to RepositoryConnection.clearNamespaces API

RepositoryConnect.clearNamespaces now supports a reset parameter (which defaults to True). clearNamespaces deletes all namespaces in the repository for the current user. If the reset argument is True, the user's namespaces are reset to the default set of namespaces, otherwise all namespaces are cleared.

Bug19004 - Make sure Python clients ping their session.

Previously, the thread to keep a dedicated session alive was created but not started, making it possible for Python clients to have their session time out on them.

AllegroGraph 4.0m2

General

The M2a release includes the following major functionality:

Full database backups and restores.
Text Indexer Enhancement for databases with more than 1 Billion text elements and strings with international characters.
Infiniband testing and performance document.
Deleted Triples Data structure improvement that removes the previous hard-coded limit on the number of triples that can be deleted per transaction.
Cluster Manager Design Document.
Jena functionality that was outlined as being highest priority.
Federation of AllegroGraph 4 databases
Support for Social Network Analysis
Support for two dimensional geospatial analysis.
Support for 48-bit triple IDs.
User definable placement of the transaction log.

There were many minor enhancements and bug fixes.

Note that AllegroGraph databases created with release m1 or m2 will have to be converted to a new format before use with version m2a. See Repository Upgrading.

Performance and Robustness

Improve logging by making some messages more concise and adding additional messages to improve diagnostics.
Improve and make string tables more robust
- better consistency checks
- compress large strings (>= 256 chars) in the string table
Added conversion tools for moving m1 triple-stores to m2
Support SIGUSR1-based reload in agraph-control script
After merging UPI tables, schedule a checkpoint to delete files which are no longer needed.
Safer file deletion for UPI tables to prevent problems with untunded databases.
Optimize UPI table merging, do not merge table portions that can't have common entries.
Improve client handling of instance process death
Improve upgrade checkpoint handling
Improved database versioning framework
Reference count indices so that they do not disappear out from under cursors
Rmprove transaction log upgrade handling
Writes to transaction log files are now aligned to the filesystem block size to allow for optimal performance when used with certain storage subsystems.
Control placement of transaction log (with TransactionLogDir)
The maximum number of triples that can be added to a triple store is now 2^48 (formerly 2^32).
Added TransactionLogDir config file parameter. This specifies where transaction log files should go. If not specified, defaults to the value of Main.
The agraph server process now responds to SIGUSR1 by reloading the server configuration file. The agraph-control script and /etc/init.d/agraph scripts now accept the 'reload' command and will send SIGUSR1 to the agraph server process.
Enhanced in-the-field debugging support. When requested by Franz tech support, you can send SIGUSR2 an AllegroGraph subprocess to get a backtrace for all threads.

Other Enhancements

configure-agraph now turns off terminal echo before prompting for the super user password. Also, since the password is no longer echoed, configure-agraph prompts for the password twice to make sure it has been entered properly.
support fractional seconds in time and dateTime
improve support for duplicate deletion during merge operations
improve and unify the printing of blank nodes
make the opening and re-opening of federated-triple-stores more robust
document group centrality
add API to auto-commit (on a per store basis)
add depth-first-search-path functor
add :plain encoding
improve handling of polygons in the geospatial sub-system
cleanup part printing
Remove line breaks in server log files to make automated processing and filtering easier.
Crash instance process if one of its helper processes crashes.
Detect death of instance process in client. Generate clear error message instead of hanging the client.
Add database versioning framework. Implement stepwise upgrade paths for databases that have been created with different software versions.

Fixed bugs and other corrections

Fix for bug18831: Agraph.cfg "overwrite" option appends to previous file instead of overwriting it as it should.
Fixed bug18774: Running "agraph-control start" twice would overwrite a valid pid file. This has been fixed.

SPARQL

Significant SPARQL performance improvements throughout, particularly in Left Joins (OPTIONAL), DISTINCT and REDUCED, and certain numeric FILTERs.
Massive performance increase for queries with both ORDER BY and a small LIMIT.
Imposed bindings (:with-variables) now apply to the output portion of SPARQL CONSTRUCT and DESCRIBE queries.
Some queries that use the SPOGI cache as an optimization could return duplicates. Fixed.
Similarly, the free-text index now returns distinct results in all cases, which avoids some unnecessary duplicate results.
Add the ability to use a graph URI, on both input and output, as a signifier for the default graph UPI of the current store.
Complete xsd:string/simple literal entailment. This allows you to seamlessly switch between using xsd:strings and plain literals without query changes. Plain literals are more efficient for the storage layer, and encouraged in all circumstances.
Improvements (in both efficiency and correctness) for SPARQL comparisons and XQuery functions when handling encoded values.
RDF/XML serializer can now handle on-the-fly namespace abbreviation, at the cost of more verbose output.
SPARQL extensions (e.g., geospatial forms) are now permitted by default.
URIs can now be used to specify geospatial subtypes, as well as the more verbose and less portable UUID form.

Java Client

Added Social Network Analysis capability.
Updated the Java Tutorial and tutorial examples to include Social Network Analysis.
Added Geospatial analysis capability.
Updated the Java Tutorial and tutorial examples to include Geospatial Analysis.
Established a Jena semantic framework API.
Created the Jean Tutorial, and a file of Jena tutorial examples.

Python Client

Added back support for SPOGI cache enabling/disabling
Enabled returning SPARQL-results+json or SPARQL-results+xml result from a query rather than a binding set
Introduced a boolean argument to openSession and session API for whether or not to load the initfile in the process handling the session
Federation support.
Introduced several options and better command-line option parsing for the load stress script
Minor updates to Python Tutorial and examples.

Changes to earlier releases

Significant changes in AllegroGraph 4.0 compared to earlier releases makes changes to earlier releases irrelevant to release 4.0 and later. Contact [email protected] if you have questions about releases prior to 4.0.

AllegroGraph 8.1.1 Change History

Version 8.1.1 user visible changes

AllegroGraph Server: General Changes

HTTP Client

SPARQL

AG-643 - incorrect results when using complex FILTER [NOT] EXIST clauses

AG-737 - SPARQL parser fails on space-separated + property path modifiers

AG-770 - inconsistent results of joins with UNION on SBQE

AGWebView

AG-354 - Fix losing content of query tabs in New WebView after reload

AG-367 - Make the query language option local to a query tab instead of being global for a user

Changes to the Lisp API

Prolog

Documentation

Distributed AllegroGraph

Version 8.1.0 user visible changes

AllegroGraph Server: General Changes

HTTP Client

SPARQL

AGWebView

Option to stay logged in in New WebView

New WebView page now lists regular repos and vector repos

Links in New WebView rearranged

Large Language Model changes.

Vector database are now backed up when their associated repo is backed up

Lineparse changes

Carriage returns are ignored in Lineparse files

Changes to the Lisp API

Prolog

Documentation

New LLM Split document

Distributed AllegroGraph

Version 8.0.0/8.0.1 user visible changes

AllegroGraph Server: General Changes

Some changes to install-agraph

Users will be logged out after 30 minutes of idle time

Two new AuthPolicy values external-token and token-external

Further allowed abbreviations of server specs

Some changes when configuring AllegroGraph

Exiting shared backend processes

SHACL now allows specifying all graphs

Text starting with a # is a comment in agraph.cfg

Names of Fedshard shards now end in .fshard

Backing up settings now backs up attributes data

Rfe17027: Add delete-duplicate-triples and get-dupliate-triples support to agtool

Bug26968: Occasional segmentation faults when using user attributes

Bug26926 - Bogus results produces by joins on optional variables

Bug26397 - login failures caused by AGWebView tabs of different versions

HTTP Client

Rfe17140 - support prefixed names in HTTP API endpoints accepting parts

Rfe17040 Add RedirectHTTPOnHTTPS AG directive

SPARQL and GRAPHQL

Single quotes work like double quotes to designate strings in GraphQL

New configuration and query options for query caching

Rfe17214 - Opt-in unification of owl:sameAs duplicates in SPARQL queries

Rfe17183 - support for owl:SymmetricProperty in materializer

Rfe16975 - Use the same default base URI for all backends/sessions

Rfe16328: Warn about unknown constants in SPARQL DESCRIBE clauses

Bug27048 - query parsing failures with aggregates and outer VALUES

Bug27090 - SPARQL SERVICE calls do not follow HTTP redirects

Bug26880 - Saved queries HTTP API ignores uri/baseURI arguments

Bug26772: Obscure errors in patterns with repeated variables on MJQE

Bug26675: Possible incorrect bindings order in SPARQL solutions

Bug25697 - Wrong results with OPTIONAL clauses sharing a variable

Changes to the Lisp API

Prolog

Documentation

Distributed AllegroGraph

Version 7.3.1 user visible changes

AllegroGraph Server: General Changes

Bug26839 - Fix SSLCipherSuite directive

Bug26806 - Turtle parser fails on prefixes starting with base/prefix

Bug26775 - Turtle parser fails on boolean literals followed by dots

bug26765 - Problem with Admin/Audit log page

Bug26754 - Load-transform fails on floating point numbers in JSON

Bug26747 - Allow using transform templates with JSON list elements

bug26135 - change in stored queries format in 7.3.0

Bug26755 - Errors when using numeric values in load-transform templates

HTTP Client

SPARQL

AG-643 - incorrect results when using complex `FILTER [NOT] EXIST` clauses

AG-737 - SPARQL parser fails on space-separated `+` property path modifiers

AG-770 - inconsistent results of joins with `UNION` on SBQE

Text starting with a `#` is a comment in agraph.cfg

Rfe17214 - Opt-in unification of `owl:sameAs` duplicates in SPARQL queries

Rfe17183 - support for `owl:SymmetricProperty` in materializer

Rfe16328: Warn about unknown constants in SPARQL `DESCRIBE` clauses

Bug27048 - query parsing failures with aggregates and outer `VALUES`

Bug27090 - SPARQL `SERVICE` calls do not follow HTTP redirects

Bug26880 - Saved queries HTTP API ignores `uri`/`baseURI` arguments

Bug25697 - Wrong results with `OPTIONAL` clauses sharing a variable

Bug26806 - Turtle parser fails on prefixes starting with `base`/`prefix`

Bug26872 - "Not CaaT state" errors with large `OFFSET` and `DISTINCT`

agtool `--output` option replaced by `--output-format`.

Rfe16785: Deprecate `--results-format` in favor of `--output-format`

Rfe16175: Unify `run-sparql`'s `rdf-format`/`results-format` arguments

Rfe16711 - implement `query-results-limit` user access flag

Rfe16639 - Rename agtool command `token` to `tokens` for consistency

Rfe16562: Support limiting `agtool query` output to results only

Bug26515 - Restore `reorderDuringExecution` query option

Bug26460: Empty RDF graph was returned by SPARQL's `DESCRIBE * WHERE ...`

Bug26452 - MJQE may omit bindings introduced by SPARQL `VALUES` clause