Release 5.0.2

This document lists all changes for the most recent major release (releases with number X.Y) and subsequent minor releases (with numbers X.Y.Z and perhaps further identifiers), including major and important changes and a list of all user-visible modifications. Change History repeats the list of user-visible modifications for this release and includes similar lists for earlier releases.

The release described in the documentation set is 5.0.2.

Major Changes in release 5.0.2

Version 5.0.2 is a maintenance release which fixes several bugs, only two of which (Bug22800 and Bug23060) are user-visible. Bug22800 fixed a problem downloading query results or audit logs from AGWebView (when using https). Bug23060 fixed a potential race condition when doing multiple replications. Neither bug affected the data itself, just viewing or replicating data.

Major Changes in release 5.0.1

Version 5.0.1 is a maintenance release which fixes a bug that manifests itself when replication is combined with deleting duplicates (Bug23016). Doing so could result in an incorrect triple count and garbage triples being added to the database. We recommend that users using replication and duplicate triple deletion recreate their replicas from a fresh backup.

Major Changes in release 5.0

Version 5.0 is a major release with various important new features, in particular:

• New 3D and multi-dimensional geospatial functionality (see N-dimensional Geospatial Overview).
• Update to new Sesame 2.7.x transactional semantics (see Javadocs (Sesame and Jena)).
• Update to new Apache Jena v2.6.x (see Javadocs (Sesame and Jena)).
• The server supports new optional specific transaction "begin".
• Improved support for encrypted client connections.
• WebView query page improvements: display of execution time and abort options (see WebView).
• Query Engine Improvements.
• Support for Top Braid Composer 4.5. See Top Braid Composer Plugin.

Release 5.0 programmer notes

• The fasl (compiled Lisp) file which must be loaded into Allegro CL to use AllegroGraph is now agraph5.fasl. Previously it was agraph4.fasl. The require form is now (require :agraph5).

• A bug in text indexing was uncovered and fixed. Text indices created since version 4.13 contain some bogus entries. We recommend that you drop and recreate any existing text indexes. See the Bug22837 entry.

Major changes in release 4.14.2

This version, released at the same time as 5.0.1, contains the same replication/delete duplicates fix for Bug23016 as 5.0.1, but updating release 4.14.1.

Major Changes in release 4.14.1

Version 4.14.1 is primarily a maintenance release but there are some important enhancements as well:

• A new feature allows getting a list of duplicate triples which will be deleted when duplicates are deleted. Thus you can tell which triples will be deleted (and, using the list which is in Nquads format, restore the deleted duplicates if desired). See Deleting Duplicate Triples.

• Enhanced audit email notifications: you can now specify multiple recipients, each receiving notifications of a customized list of audit events. See the Auditing document.

Major Changes in release 4.14

Version 4.14 has fixes and enhancements in the following areas:

• New AllegroGraph data export facility: see Data Export.

• Enhanced and documented HTTP interface to the audit utility: see Auditing, which describes the interface and has links to other relevant documentation.

• Enhanced password control for users along with other security measures. Configuration options now can be set to effect the following:

  • Allow users to set their own passwords.

  • Ensure password quality (so minimum number of characters and minimum number of characters of particular types can be specified).

  • Make passwords expire (so passwords must be changed regularly).

  • Suspend user accounts after a number of failed login attempts or failure to use an account for a specified length of time.

  • Have email notification sent to a specified email address when certain actions are taken.

Also, superusers may not have all access and permissions by default (depending on the value of the new SuperUserCanAccessAllData configuration option, whose default is `yes'.) If they are restricted, they can add access and permissions to themselves, but the fact they did so is recorded if auditing is enabled. See Server Configuration and Control where the new options implementing these features are described. See also Auditing, Security Implementation, and WebView.

• Redesigned Query page in AGWebView. The page has been redesigned and now includes a Cancel button allowing users to cancel uncompleted queries. See the documentation in the WebView documentation.

Major Changes in release 4.13.2

Version 4.13.2 is a maintenance release which supplies an updated OpenSSL library containing a fix for the OpenSSL Heartbleed bug. Other than the availability of Top Braid Composer version 4.4 and 4.4.1 (noted just below), there are no other changes from version 4.13.1. Because AllegroGraph uses OpenSSL, the Heartbleed OpenSSL bug can introduce a security leak. See this Allegro CL Tech Corner article at http://franz.com/support/tech_corner/heartbleed040914.lhtml for links to information about the bug. The end of that article describes how to update earlier versions of AllegroGraph to fix the Heartbleed bug. If you have version 4.13.2 installed, the update described in that article is not necessary as the relevant libraries are already updated in version 4.13.2.

• Support for Top Braid Composer 4.4 and 4.4.1. See Top Braid Composer Plugin. The new TBC version is now available for download (no change to AllegroGraph is required).

Changes in this and some earlier releases

This document lists changes in the current release and several previous releases. See the Change History for a list of all changes in earlier releases.

AllegroGraph 5.0.2 user-visible changes

AllegroGraph Server: General Changes

Bug23060 - Fix race when starting multiple replication jobs

Fixed a race when spawning processes in the primary to handle replication. When triggered, this bug would hang worker threads in the primary's Service Daemon, and leave the secondary trying to start replication hung in an unrecoverable state.

AGWebView

Bug22800: Fix two AGWebView links when using https

The Download button when viewing query results in AGWebView and viewing Audit Logs would fail if requested via https. This bug is now corrected.

AllegroGraph 5.0.1 user-visible changes

AllegroGraph Server: General Changes

Bug23016 - Incorrect triple-count and garbage triples

Under certain circumstances (particularly when delete-duplicate-triples is used), crash recovery or replication can result in an incorrect triple count and garbage triples being added to the database. This has been corrected. We recommend that users using replication and duplicate triple deletion recreate their replicas from a fresh backup. Replication is described in the Replication document.

HTTP Client

Bug23040 - Performing the same ASK query more than once under REST could error

Under certain circumstances, if an ASK query that used joins was evaluated more than once under the HTTP REST API, then AllegroGraph would signal an error. This has been corrected.

AllegroGraph 5.0 user-visible changes

AllegroGraph Server: General Changes

Rfe13506 - Minor query optimization for triple pattern filters

AllegroGraph now performs triple pattern filtering slightly more efficiently when the current set of solutions has only a single member.

Rfe13405 - Warn if transparent hugepages are enabled

Due to the potential for negative performace impact, AllegroGraph now logs a warning into agraph.log on startup if Transparent Hugepages are enabled. See this section of Performance Tuning for information on the transparent hugepage issue and its effect on performance.

Rfe13376 - Improve hashing performance

AllegroGraph's internal hashing functions have been further optimized leading to small general performance improvements.

Rfe13346 - configure-agraph should allow for specification of port

The configure-agraph script now prompts for the Port configuration parameter and accepts the --port command line argument. See Running the configuration script in Server Installation.

Rfe13265 - Update N-Triple and N-Quad parsers

AllegroGraph's N-Triple and N-Quad parsers now support the full W3C specifiation and pass all tests. The performance is also slightly improved. Note that this change means that AllegroGraph will no longer load invalid N-Triple and N-Quad source data unless the relax-syntax argument is used. See Data Loading, the Lisp API functions load-ntriples and load-nquads, and [PUT/POST /repositories/name]/statements in REST/HTTP interface.

Rfe13172: Allow setting size of strings hash table at database creation time

A new database configuration parameter StringTableSize is now available. It can be used to configure the number of hash table slots used for the string storage hash table and setting it properly can enhance performance. See Catalog Parameters that Improve Performance in Performance Tuning for more information on performance-enhancing options for details.

Rfe13146 - Improve suboptimal index scan performance

The efficiency of low-level cursor operations has been improved when an optimal index for the cursor is not available.

Rfe12872 - Remove ag3-export / ag4-import

The obsolete ag3-export and ag4-import utilities have been removed. Serializing data to nquads in AllegroGraph 3 and loading it into AllegroGraph 4 is the recommended alternative.

Rfe11580: Added --no-daemonize flag to agraph binary

The agraph binary is the actual program which starts AllegroGraph. It is not typically invoked directly, but rather using the agraph-control script, but it can be called directly. When it is and --no-daemonize is supplied, the service daemon will remain in the foreground. See agcontrol.

Bug22837 - Text index incorrectly indexing the string "nil"

Since version 4.13 the text indexer was incorrectly indexing the word "nil" for triples that were ineligible for text indexing. This resulted in unnecessary work being performed and space being consumed. This has been fixed. We recommend that you drop and recreate any existing text indexes.

Bug22835 - Fix numa info parsing failure

At startup, information about numa (non-uniform memory access) topology is collected and logged. Previously, parsing failed with an error if a cpumap file in /sys/ contained a comma separated list of numbers instead of just a single number. This bug has been fixed.

Bug22767 - Use agraph5 for the Lisp client fasl name

AllegroGraph 5.0 now uses agraph5.fasl instead of agraph4.fasl for the name of the Lisp fasl file.

Bug22754 - Concurrent duplicate triple deletion can result in inaccurate triple count

Under some circumstances duplicate triple deletion transactions which occur concurrently with other deletion transactions could result in an inaccurate triple count. This has been corrected.

Notes:

• Though the triple count could become inaccurate, the integrity of the triples in the database remained intact.

• The triple count on already-affected databases will remain inaccurate until they are recreated.

Bug22566 - Encoded ID validation was too weak

In some cases, a URI representing an invalid encoded ID could be accepted by AllegroGraph instead of signaling an error. This has been corrected.

Bug22564 - Data import could fail to determine the correct base URI

When importing a compressed RDF/XML file, AllegroGraph could fail to determine the correct base URI. This has been corrected.

Bug22559 - RDF/XML parser should validate the base URI if supplied

Previously, the RDF/XML parser failed to validate the base URI it was given which could lead to invalid data. This has been corrected.

Bug21867 - Ensure text index queries are called using strings

AllegroGraph sometimes called lower level text index query functions with invalid arguments. This has been corrected.

Bug21460 - Polygons data cannot be exported and imported

AllegroGraph now converts resources of the form

<http://franz.com/ns/allegrograph/4.14/index#_N> 

(where N is replaced by a whole number) into encoded UPIs rather than hashed ones. These URIs are used to serialize AllegroGraph polygon data which means that polygons can be efficiently exported and imported. Note that the Lisp API upi->value will return the numeric representation for the UPI rather than the string representation. Use part->string if the string representation is needed.

Bug20497, Rfe11047 - Fix fti too many open files error

Previously, when constructing a new text index for a database with existing triples, there was potential for the operation to fail with a "Too many open files" error. This has been fixed.

HTTP Client

Rfe13304 - Support relaxed load syntax for remote triple-stores

The HTTP REST API for loading statements now includes the relaxSyntax argument. When true, N-Triple and N-Quad data are not validated as stringently. For example, blank node names may include underscores. See PUT/POST statements in REST/HTTP interface.

Rfe13249 - Augment optimize-indices API to allow for single index optimization

The HTTP API's indices/optimize service now takes an additional index query parameter which can be specified multiple times. If omitted, then all of a store's indices will be optimized. If supplied, each parameter value should be the name of an index whose optimization is being requested. See [POST /repositories/name]/indices/optimize in REST/HTTP interface.

Rfe11547 - Support Sesame query timeout parameter in HTTP

AllegroGraph's SPARQL HTTP REST interface now supports an optional timeout parameter. If specified, it must be a number greater than zero which denotes the number of seconds to limit SPARQL query execution time. If unspecified, then there is no limit on the length of a time a query may run. See [GET/POST /repositories/name in REST/HTTP interface.

Rfe11221 - General type-mapping improvements

The interface for setting up predicate- and datatype-mappings has been simplified. The same HTTP REST API calls can be used to setup regular XSD datatype mappings and geospatial ones (including cartesian, spherical and n-Dimensional). In each case, the datatype must be specified using its URL. E.g., the 10x10 cartesian mapping with resolution 1 would use the URL:

<http://franz.com/ns/allegrograph/3.0/geospatial/cartesian/0.0/10.0/0.0/10.0/1.0> 

See Type mappings in REST/HTTP interface.

Bug22576 - POST|PUT statements service should not require eval permission

AllegroGraph would not allow client-side files to be imported into a store unless the user had eval permission. This has been corrected. Of course, eval permission is still required to import server-side files.

SPARQL

Rfe13510 - Improve performance of the SPARQL query REST API

AllegroGraph now automatically caches query information across REST API calls to improve overall query performance.

Rfe13323 - Provide API to retrieve only the ID part of an encoded ID

There is now a SPARQL function, , that retrieves the non-prefix portion of an encoded ID. For example, if the encoded ID has IRI , then the function will return "3489a". In a query, this would look like:

select * {  
  _:b ex:identifier ?id .  
  bind( <http://franz.com/ns/allegrograph/fn#encodedIdId>(?id) as ?value )  
} 

Rfe13305 - Deprecate old-style SPARQL GEO syntax

Because AllegroGraph's magic properties support a richer range of geospatial features than the old-style SPARQL GEO syntax, the old syntax is being deprecated and will now signal a warning when it is used. See N-dimensional Geospatial Overview.

Rfe13297 - Reduce memory consumption for distinct aggregation in SPARQL queries

AllegroGraph now better manages memory when evaluating distinct aggregation queries like:

select ?dept (count(distinct ?student) as ?sc) where {  
  ?dept a :Department .  
  ?student :memberOf ?dept .  
} group by ?dept  

Rfe13289 - When logging, use query supplied namespaces for abbreviations

Previously, AllegroGraph would ignore SPARQL query supplied namespaces when displaying resources during query logging. It now uses these namespaces to improve the readability of the display.

Rfe13171 - Improve query checks for unknown variables

AllegroGraph now warns more consistently when a valid SPARQL expression uses variables in unreasonable ways. For example, this query is perfectly valid but the ORDER BY variables are not bound by the query so it is unlikely to be what was intended:

select ?a ?year ?label {  
  ?a a :Car .  
  ?a :modelYear ?year .  
  ?a rdfs:label ?label .  
} order by ?autoYear 

Rfe12470 - Apply multiple constraints to the same triple pattern

Previously, AllegroGraph could only apply a single logical FILTER expression to any given triple pattern. This meant that queries like:

select * {  
  ?s ?p ?o .  
  filter( ?p = ex:pred1 && ?o > 10 )  
}  
would optimize the query plan with either the ?p constraint or the ?o  
constraint but not both. This has now been improved so that queries  
like the above will use both constraints simultaneously and thereby  
run much more efficiently. 

Bug22803 - Fix bug in authorizationBasic query option

The authorizationBasic query option was not being parsed correctly which meant that queries that tried to use it would fail. This has been corrected.

AllegroGraph 4.14.2 user-visible changes

This release contained the same replication/delete duplicates fix for Bug23016 as release 5.0.1.

AllegroGraph 4.14.1 user-visible changes

Bug22802 - Support userinfo in SPARQL service request URIs

URIs used in SPARQL service requests did not allow the inclusion of userinfo information so that queries like this would fail:

select * {  
  service <http://user:[email protected]:7001/repositories/sp2b-1e4> {  
    ?s ?p ?o  
  }  
} limit 5 

This has been corrected.

Bug22750 - Correct an error in SPARQL property path queries

In some circumstances, a SPARQL property path query could signal an error when trying to find paths between fixed endpoints. This has been corrected.

Bug22735 - SPARQL federated query could fail to recognize geospatial data

SPARQL federated query (via the SERVICE) clause could fail to transform returned results using AllegroGraph geospatial encoding. This would lead to query errors and has been corrected.

Bug22735 - SPARQL federated query could fail to recognize geospatial data

SPARQL federated query (via the SERVICE) clause could fail to transform returned results using AllegroGraph geospatial encoding. This would lead to query errors and has been corrected.

Bug22734 - Query options should be passed along to the HOST of any SERVICE clauses

Previously, any custom query options set using the PREFIX notation were not passed along to the remote server when using SPARQL Federated query. Now, these options are passed to the remote server which provides more control when that server is running AllegroGraph.

Bug22732 - Correct errors for SERVICE clauses that used blank node labels

SERVICE clauses that used blank node labels (or property paths that might be rewritten using blank node labels) could lead to an error when AllegroGraph tried to merge these bindings with the containing query. This has been corrected.

Bug22731 - Error when using property paths in a SERVICE query with distinct results

If a query asked for DISTINCT results and had a property path query inside of a SERVICE clause, then AllegroGraph would signal an error during query planning. This has been corrected.

Bug22730 - Some uses of UNION and VALUES could lead to an error

Some uses of UNION clauses and SPARQL in-line data via the VALUES clause could lead to a run-time query error. This has been corrected.

Bug22727 - EXISTS filters sometimes performed more computation than necessary

SPARQL EXISTS (and NOT EXISTS) filters were performing query computation even when the main result set was empty (and hence had nothing to filter). This has been corrected.

Bug22660 - Temporal magic properties introduce new variables

Temporal magic properties like were introducing new variables into queries so that a query like

prefix m: <http://di.uoa.gr/~charnik/music_db#>  
prefix time: <<http://franz.com/ns/allegrograph/3.0/temporal/>  
select * {  
  m:p_1939-08-19 time:pointBefore m:p_1980-12-04 .  
} 

was returning bindings instead of empty rows. This has been corrected.

Bug22588 - Further improve handling of multiple range filters on the same variable

AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like

select * {  
  ?s ex:test ?age .  
  filter( ( ?age >= 10 && ?age <= 20 ) ||  
          ( ?age >= 60 && ?age <= 70 ) ||  
          ( ?age >= 80 && ?age <= 90 ))  
} 

will now execute as the combination of two range queries rather than a single scan. This change is similar to the change for bug22535 but more general.

Bug22569 - Improve canceling of SPARQL SNA generators

Previously, neither the administrator jobs list nor the query canceling API functioned properly if a query used an SNA generator that was implemented via a separate SPARQL query. This has been corrected. The SNA generator SPARQL queries will no longer appear in the administrator jobs list and they will now inherit any timeout from the query that they are running in service of. This means that canceling the parent query will cancel these child queries.

Bug22541 - Improve handling of invalid typed literals

Previously, the SPARQL parser would generate a general error and log a backtrace if a query included illegal XSD datatypes like "2014-Mar-13"^^xsd:date. These are now treated as SPARQL parse errors.

Bug22541 - Improve clause reordering for some SPARQL GRAPH queries

AllegroGraph's statistical BGP reordering could lead to sub-optimal query plans for certain queries that involved GRAPH queries and FILTERs on ranges. This has been corrected.

AGWebView

Rfe13530 - Rearrange some of AGWebView's settings

The "show long parts" and "show contexts" settings are global to the browser session but they were only available on the query viewer. These settings are now also accessible via the new Utilities | Settings menu. See the AGWebView document.

Rfe13504: Add a "large operations warning" to AGWebView

AGWebView's site settings now include a place to enter text that will appear before it executes an operation that potentially exposes a large amount of repository data. Note as well that the site settings are now available via the new Utilities | Settings menu.

Rfe13183 - Agwebview uses Unicode rather than \u escaping to display data

AGWebView now presents non-ASCII characters using Unicode instead of their \uXXXX escape sequences.

Rfe12523 - Improve data import dialogs in AGWebView

The data import dialogs in AGWebView now support AllegroGraph's relaxed syntax and continue on error options. Relaxed syntax allows the RDF importers to accept data that it not strictly correct but is still easily understandable. For example, a blank node label is not allowed to include the - (dash) character but many source files use it regardless. Normally, AllegroGraph will not allow these files to be imported but if relax syntax is turned on, it will. Continue on error allows AllegroGraph to continue loading data even if it encounters a bad triple or quad. Note, this option only affects line based formats like N-Triples or N-Quads. As ignoring errors is an all or nothing affair, this option should be used with caution: it will not be possible to determine how many triples were skipped or even if any errors were encountered. These new options correspond to the --error-strategy and the --relax-syntax options to agload.

Rfe11466 - Use double-clicks to navigate the file picker in AGWebView

AGWebView's file picker for selecting files on the server side used single-clicks rather than double clicks to change directories. This has been corrected.

Bug22800 - Fix regression in v4.14 that breaks proxying through frontend

There was a regression in v4.14 that partially broke the use of AGWebView over https. This is fixed. Additionally support is added for proxying requests to https sessions through an http frontend, or to http sessions through an https frontend.

Changes to the Lisp API

Rfe13249 - Augment optimize-indices API to allow for single index optimization

The Lisp API's optimize-indices function now takes an additional indices argument. This can be nil (the default) or :all to optimize all of the triple-store's indices, a single index name (like :psogi) to optimize one index or a list of index names to optimize each of them.

Rfe13184 - Check for uninterned UPIs when adding triples

Previously, AllegroGraph did not ensure that the UPIs in a triple being added were already interned in the triple-store's string-table. This led to the possiblity of adding triples that could not be re-serialized. The default is now to check UPIs unless the code doing the adding is wrapped in the new while-not-validating-that-upis-are-in-string-table macro.

Rfe6878 - open-triple-store can take a triple-store instance

The open-triple-store function can now be called using an existing triple-store instance. In this case, it will return a new instance of the same underlying store.

Bug22811 - Fix use of direct Lisp client when AllowHTTP is false.

Previously, the direct Lisp client would be unable to access an Allegrograph server if it was started with the config directive AllowHTTP set to false. This bug is now fixed.

Bug22772 - Include inferred-triple-p in the remote Lisp client

The inferred-triple-p function was inadvertently left out of the remote Lisp client. This has been corrected.

Bug22703 - Add relax-syntax argument to load-nquads|ntriples-from-string

The funtions load-nquads-from-string and load-ntriples-from-string were missing the relax-syntax argument. This has been corrected.

Prolog

No significant changes.

N-dimensional geospatial

Rfe13364 - Add automate-nd-datatype-mappings switch to agload

agload now supports AllegroGraph's automatic N-Dimensional geospatial type mappings. If the new --automate-nd-datatype-mappings option is specified on the command-line, then any geospatial datatypes in a source file will be automatically recognized and added to the triple-store. See Data Loading.

Rfe13350 - Add additional standard namespace abbreviations

The following namespace abbreviations have been added to AllegroGraph:

• nd -
• ndfn -
• keyword -

These are helpful in using the new N-Dimensional geospatial functionality.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe12921 - Upgrade to sesame-2.7.11 and jena-2.11.1 jars

With this change, the Java Client has been upgraded to use Sesame 2.7.11 and Jena 2.11.1 jars. Previously, the Java Client used Sesame 2.6.8 and Jena 2.7.2 jars.

Rfe12920 - Tutorial for connection pooling using Sesame

Example24 has been added to the Sesame tutorial examples to create and manage connection pools using Sesame. Java Sesame tutorials in general are discussed in Java Sesame Tutorial. Example24 does not have a writeup and is in the file TutorialExamples.java discussed in Java Sesame Tutorial.

Bug22854 - TutorialExamples.java signals warnings in java client

With this change, Tutorial examples run with out any runtime warnings. Also some warnings are fixed in callimachus tests

Bug21911 - Implementation of Jena QueryExecution methods

Previously, the AGQueryExecution class had no implementation for following methods:

• execConstructTriples()
• execDescribeTriples()
• getQuery()
• setTimeout(long timeout),
• setTimeout(long arg0, TimeUnit arg1)
• setTimeout(long arg0, long arg1)
• setTimeout(long timeout1, TimeUnit timeUnit1, long timeout2, TimeUnit timeUnit2)

This has been corrected. An example (example24) for a Jena Graph Union has been added to the Jena examples. Jena tutorials in general are discussed in Java Jena Tutorial. Example24 does not have a writeup and is in the file JenaTutorialExamples.java.

AllegroGraph 4.14.1 user-visible changes

AllegroGraph Server: General Changes

Rfe13193 - Parameterize email notifications

This change builds on rfe13104 and makes it possible to specify for which audit events emails should be sent and to what address. The body of the email contains the triples added for the audit event in pretty printed Turtle format.

The AdminEmail and AdminEmailSMTPHost configuration directive were removed in favor of the new AuditEventsToEmail directive. (Note this is a backward-incompatible change. See the Auditing document.)

The following rule asks for emails to be sent to [email protected] via the gmail smtp host (defined with SMTPHost directive) when any of the expirePassword, addUser or deleteUser audit events trigger:

AuditEventsToEmail to="[email protected]", smtphost="gmail", \  
  events="expirePassword,addUser,deleteUser" 

If there is only one smtp host defined then it can be omitted:

AuditEventsToEmail to="[email protected]", \  
  events="expirePassword,addUser,deleteUser" 

Note that auditing must be enabled for this to work. See the Audit documentation for more information.

Rfe13201 - Restrict the range of backend ports

This change adds the BackendPorts global configuration directive to restrict the range of ports on the loopback interface to be used by shared back-ends. Note that the maximum number of shared back-ends is limited by the number of ports in the specified range even if the Backends directive is higher. Configuration options are described in Server Configuration and Control.

Bug22675 - Simple queries may not show up in the jobs list

This is a continuation of the work done for bug22536 and increases the number of queries that can be controlled in the jobs list.

Bug22666 - run-fti-merge doesn't fail fast

Previously, if the text index merger process encountered an error during merging, it would log the error and a backtrace to agraph.log and then continue operation. If the error condition persisted, this logging could continue indefinitely, resulting in agraph.log growing to excessive size. This has been fixed.

Bug22641 - Listing and cancelling queries

Previously, job management (available via /jobs REST service and also admin/job in AGWebView) had a bug which caused concurrent requests to fail with:

the assertion (endp (intersection  
(db.agraph.spawn::receiver-for db.agraph.spawn::r)  
(db.agraph.spawn::receiver-for db.agraph.spawn::receiver))) failed 

This bug has been fixed.

Bug22641 - Listing and cancelling queries

Previously, job management (available via /jobs REST service and also admin/job in AGWebView) had a bug which caused concurrent requests to fail with: the assertion (endp (intersection (db.agraph.spawn::receiver-for db.agraph.spawn::r) (db.agraph.spawn::receiver-for db.agraph.spawn::receiver))) failed This bug has been fixed.

Bug22636 - Accumulation of datafile mappings during unshuffling

For very large databases it was possible for index optimizers to exceed address space limitations and throw errors during large optimization operations. This has been fixed.

Bug22624 - Level 2 optimization should stop after single round

Previously, if a level 2 optimization was requested and, while it was running, additional triples were loaded into the database, the level 2 optimization would run again, redoing most of the work that it had done before. This has been fixed.

HTTP Client

Rfe12546 - Implement get-duplicate-triples.

Add support for retrieving the set of duplicate triples in a repository via the following REST/HTTP service call:

GET /repositories/[name]/statements/duplicates

See Deleting Duplicate Triples.

SPARQL

Bug22692 - Improve error logging for SPARQL update commands

AllegroGraph now prints additional information to the query log when errors are encountered during SPARQL update processing.

Bug22601 - SPARQL INSERT WHERE could fail to encode typed literals

When adding triples using the SPARQL DELETE/INSERT/WHERE update command, it was possible for typed literals to be saved as unencoded UPIs (stored in the string table) rather than as the more efficient encoded UPIs (stored directly with the triples). Depending on other settings, this could lead both to inefficent queries and missing results. The problem has been corrected.

Bug22588 - Further improve handling of multiple range filters on the same variable

AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like

select * {  
  ?s ex:test ?age .  
  filter( ( ?age >= 10 && ?age <= 20 ) ||  
          ( ?age >= 60 && ?age <= 70 ) ||  
          ( ?age >= 80 && ?age <= 90 ))  
} 

will now execute as the combination of two range queries rather than a single scan. This change is similar to the change for bug22535 but more general.

Bug22540 - Handle failures of WHERE parts of update queries

Previously, if the query in the WHERE part of an update query failed (it timed out, ran out of memory, or was cancelled), the update failed with a bus error. This bug has been fixed.

Bug22334 - SERVICE doesn't deal with encoded IDs

SPARQL Federated query was not working correctly with AllegroGraph encoded IDs. This has been corrected.

AGWebView

Rfe12546 - Add Export Duplicate Statements link in AG Webview

On the Repository Overview page there is now an option to have the browser download the set of duplicate triples in the repository. Use the adjacent dropdown box to choose the method by which duplicates are determined (SPO or SPOG). See Deleting Duplicate Triples.

Changes to the Lisp API

Rfe12546 - Implement get-duplicate-triples.

Add support for retrieving the set of duplicate triples in a repository with the following Lisp API get-duplicate-triples, which returns a cursor of all duplicate triples visible inside the current transaction. See also Deleting Duplicate Triples.

Prolog

No significant changes.

Documentation

New document Deleting Duplicate Triples discusses the issue of duplicate triples in a store.

Python Client

Rfe12546: Add getDuplicateStatements to AGRepositoryConnection

Adds a new method to the RepositoryConnection class called getDuplicateStatements, that returns the set of duplicate triples in the repository. The method signature is:

def getDuplicateStatements(mode) 

Duplicate triples are determined by the mode argument, which can be specified as either "spo" (compare as ntriples) or "spog" (compare as nquads). See also Deleting Duplicate Triples.

Java Client

Rfe12546 - Add getDuplicateStatements to AGRepositoryConnection

Adds a new method to the RepositoryConnection class called getDuplicateStatements, that returns the set of duplicate triples in the repository. The method signature is:

def getDuplicateStatements(mode) 

Duplicate triples are determined by the mode argument, which can be specified as either "spo" (compare as ntriples) or "spog" (compare as nquads). See also Deleting Duplicate Triples.

AllegroGraph 4.14 user-visible changes

AllegroGraph Server: General Changes

Rfe13188 - Validate that strings that look like encoded IDs are encoded IDs

AllegroGraph will now signal an error if a triple-store has any registered encoded IDs and a resource is added that looks like an encoded ID but whose prefix is not registered. E.g., it is now an error to try to add a triple which contains a part like:

<http://www.unregistered.encodedID/test@@0023a> 

(Unless, of course, is registered!). Previously, AllegroGraph would have accepted the triple and converted the URI into a hashed UPI in the string dictionary.

Rfe13144 - Add bulk mode switch to agmaterialize command line tool

agmaterialize now supports a --bulk command-line option. If specified, then materialization will be run in bulk mode. This mode provides approximately 30% faster materialization. Note that bulk mode is global to the store and means that transactions will not be durable (in particular, will not be logged). Therefore, the switch should generally be used only when the store is not being used for other operations. See the Materializer document for information on agmaterialize.

Rfe13116 - Audit events for AGWebView login, logout, timeout

With this change, if auditing is enabled, then logIn, logOut, logOutWithTimeout events are added to the audit log whenever a user logs in, logs out, or when a login session times out. See Auditing.

Rfe13108 - Allow restricting superuser access to triples data

This change adds a new global configuration directive: SuperUserCanAccessAllData. If it is enabled (the default), then users with the Superuser permission will have read/write access to all repositories (just like in previous AllegroGraph versions). If it is turned off, then the superuser needs to be granted access to repositories. This is most useful when auditing is enabled and any change to user permissions is logged, as superusers will be able to grant themselves any permissions they want and auditing must be enabled if the fact that a superuser granted him/herself permission is to be recorded.

Rfe13104 - Notify admin by email of user account status changes

This change adds three new global configuration directives: SMTPHost, AdminEmail, and AdminEmailSMTPHost. With SMTPHost one can associate login, password, port, etc with a server:

SMTPHost gmail \  
  server="smtp.gmail.com", ssl=true, starttls=false,\  
  from="[email protected]", login="[email protected]", \  
  password="somepassword" 

If AdminEmail is defined, then emails for user account status changes events are sent to it:

AdminEmail [email protected] 

If there are multiple SMTPHost definitions, then one can explicitly select one of them:

AdminEmailSMTPHost gmail 

See the documentation in Security Implementation for more information.

Rfe13094 - Add EvalAllowed configuration directive

This change introduces the EvalAllowed global configuration directive. If it is yes (the default), then the "Evaluate arbitrary code" permission bits are effective. If it is no, then arbitrary code evaluation is disabled for all users (including superusers), regardless of the value of a user's actual "Evaluate arbitrary code" permission. Configuration directives are described in Server Configuration and Control.

Rfe13091 - Password and account expiration, suspending, disabling

This change introduces a number of parameters to control passwords and accounts. All are described in Server Configuration and Control. The new options are:

PasswordExpiry: The time since the last password change after  
which the password will be expired. One cannot login with an  
expired password, it can only be used to change the password.  
 
PasswordExpiryGrace: The time since password expiry after which  
the account is disabled. Only the administrator can reenable  
accounts. This option does not affect users with superuser  
permission.  
 
MaxFailedLogins: The number of failed logins in a row after which  
the account is suspended.  
 
AccountUnsuspendTimeout: The time after which suspended accounts  
are unsuspended.  
 
AccountExpiry: The time since the last authenticated activity of a  
user after which the account is deleted. This option does not  
affect users with superuser permission. 

Rfe13058 - Control password complexity

This change adds four new global configuration directives: PasswordMinLength, PasswordMinUppercaseChars, PasswordMinDigitChars, PasswordMinSpecialChars to allow the administrator to specify password complexity thresholds. All are described in Server Configuration and Control.

Rfe12941 - Allow use of more than 64GiB of query memory

Previously, the start address of the heap holding the query data structures has been such that when the heap grew beyond 64GiB it was likely to collide with shared libraries mapped there which made allocations fail. This change moves the heap so that collision is less likely and much larger heaps are possible.

Rfe12929 - Add changePassword audit event

With this change, when a user's password is changed and auditing is enabled, an audit event of type <http://franz.com/allegrograph/4.11/audit-log#changePassword> is logged.

Rfe12915 - Improve reporting of shared memory allocation errors

Previously, if allocation of shared memory used by processes serving a database instance failed, the error message was an opaque 'end-of-file'. With this change, the error message expicitly mentions the allocation failure.

Rfe12884 - Include failed login attempts in the audit log

With this change, when auditing is enabled, a failedLogin event is written to the audit log on authentication failures.

Rfe12848 - Removal of query temporary files

Sometimes queries write intermediate results to disk when they will not fit in memory, for instance with an ORDER BY query. If a query failed, some of its temporary files were not removed in a timely manner. With this change, temporary files are always deleted whether the query finishes or fails.

Rfe12740 - Include store name in optimize log message

The initial and final log messages for triple store optimizations now include the store name.

Bug22527 - Sorting UPI maps

Previously, on certain filesystems sorting of UPI maps could fail, which broke the repository instance. The only indication of this bug was the log message "Sort command failed". This bug has been fixed.

Bug22507 - Include id of deleted user in audit report

Previously, the audit report did not include the user id of the affected user associated with deleteUser events. This bug has been fixed.

Bug22466 - Setting geospatial type mappings did not register the subtype

Previously, setting a geospatial datatype or predicate mapping did not also add the geospatial subtype to the triple-store. This has been corrected.

Bug22465 - In the turtle format, QNames are a superset of XML QNames

AllegroGraph's turtle parser required QNames (like rdf:type) to be XML QNames but the turtle standard approved in February 2014 allows additional flexibility. In particular, the first character of the local part of a QName may start with a digit. Previously, a QName like rdfs:0 would have signaled a parse error. Now, it will be accepted.

Bug22450 - RDF/XML serializer could re-use node abbreviations

For efficiency, the RDF/XML and RDF-N3 serializers maintain a mapping from URI to prefix abbreviation (e.g., from to rdf:type). It was possible for the map to become corrupted during serialization leading to different nodes being printed as if they were the same. This has been corrected.

Bug22422 - Federated triple-stores cannot use geospatial magic properties

The check for whether or not a triple-store could use a geospatial magic property was too stringent. This has been corrected so that federated triple-stores can be queried with geospatial magic properties. Note that every store in the federation must agree on any geospatial predicate type mappings used in the query. If they do not, then an error will be signaled during query planning.

Bug22371 - Fix race condition when using free-text indices.

Previously, a race condition existed where a client A would fetch the list of free-text indices on a repository, but have the set of indices modified by a client B, thus rendering client A's list out of sync. This could result in unexpected errors in certain situations, such as when opening repositories. Now, attaching to free-text indices is synchronized so that this race condition is avoided.

Bug22363 - Fix log rotation

A bug was introduced in v4.13 where a SIGHUP sent to the Service Daemon would not cause the log file to be closed and reopened. This is now fixed.

Bug22345 - Use customer namespaces in remote clients

Similar to rfe12679 (under SPARQL below), AllegroGraph now uses customer namespaces when serializing output in RDF/XML for all clients. Previously, temporary namespaces were used.

Bug22319 - Materializer not inferring rdfs:subClassOf reflexivity

The materializer was not always correctly inferring that classes were rdfs:subClasses of themselves. This has been corrected. This change also improves the performance of AllegroGraph's OWL 2 RL materializer significantly.

Bug22300 - Fix error when system time is in the past

Previously, if the system time in the environment of deployment was before the build time of the AllegroGraph, then queries would fail with an error saying:

Attempt to do an array operation on nil which is not an array. 

This bug has been fixed.

Bug22297: ensure-not-lingering should latch

Previously, if a request to delinger a store could not be honored because it was still in use by another process, the request would be ignored. Now, the request is deferred until the store is no longer in use.

Bug22293 - Make agraph-backup support dynamic catalogs.

Previously, backup and restore operations would fail when operating on repositories in dynamic catalogs. Backup operations were unable to locate the main directory of the repository. Restore operations were unable to locate the main directory of dynamic catalogs.

Now, backup and restore operations support repositories created in dynamic catalogs. They follow the same rules as other repositories, most notably that when performing a restore operation, all dynamic catalogs must already have been created, or the restore will fail.

See Catalog definitions in Server Configuration for a description of dynamic catalogs.

Bug22259 - Ensure proper transaction log stream alignment on restart

Previously, under certain unusual circumstances, the transaction log stream could be left in a state that prevented a database from reopening properly. This has been fixed.

Bug22243 - Less fragmentation in the shared memory

With this change, the shared memory allocator is more aggressive about coalescing freed regions which reduces the tendency of shared memory to fragment.

Bug22229 - Geospatial type mappings could cause duplicate triples

If a triple-store registered a predicate and a datatype geospatial type mapping that applied to the same triple, then that triple would be added to the store twice. This has been corrected.

Bug22228 - Rare error when opening a triple store

Under extremely rare circumstances, opening a triple store could result in the following error:

The assertion (integerp db.agraph.storage::chunk-size) failed. 

This has been fixed.

Bug22226 - More efficient memory usage

With this fix, the query engine does a more thorough job of giving freed memory back to the operating system which enables more efficient operation under high concurrency workloads.

Bug22211 - Trix parser did not treat Return (ASCII 13) as whitespace

The TriX parser was not treating Return characters (ASCII 13) as whitespace which sometimes led to parse errors. This has been corrected.

Bug22208 - Concurrent queries in shared backends

When the system is under load, a shared backend may execute several queries concurently. Under rare circumstances, the queries involved could change each other's query options and produce incorrect results. This bug has been fixed.

Bug22198 - Listing freetext indices should not error for federated triple-stores

Previously, federate-triple-stores would signal an error when asked to list their freetext indices. They now return an empty list.

Bug22141 - Attempt to close instance during agraph-backup --supersede restore operations.

Previously, using agraph-backup to restore into an open triple-stores would fail. Now, when the --supersede option is specified, such restores can succeed. agraph-backup is described in Backup and Restore.

HTTP Client

Rfe12963 - SERVICE URIs with query strings

Previously, AllegroGraph dropped the query string in a SERVICE URI when making a request. In this example:

SERVICE <http://sparql.bioontology.org/ontologies/sparql?apikey=XXX> 

the 'apikey' was not sent to the remote web server. This has been fixed.

Rfe12952 - Add global configuration directive UseMainPortForSessions

Normally, HTTP requests to a dedicated session go directly to the process serving that particular session. This has low overhead, but it can make firewall management more difficult because more ports have to be open. With this change, the new UseMainPortForSessions directive may be enabled in the configuration file which makes all communication be proxied through the front-end. Previously, similar functionality existed only in the Java client controlled by the UseMainPortForSessions property.

Rfe12798: Implement repo-specific scripts

The existing scripting HTTP API (sitescripts) manages scripts that can be used with any repository. An HTTP API has been added for managing scripts that are local to a given repository, called "reposcripts". Reposcripts can only be loaded and run against the repository under which the script was loaded. See Scripting the server in HTTP Protocol for more information.

Bug22524 - HTTP protocol should not send partial responses

Previously, queries that used Chunk-at-a-Time (CaaT) processing over the HTTP REST interface would begin streaming results back as soon as the first chunk was processed. If a query timeout was in effect, this meant that AllegroGraph could issue a 200 (OK) response and then only send back only a partial response body. To prevent this, AllegroGraph now computes the entire query result before sending any HTTP response when there is a time out active.

Bug22476 - Unauthorized response for javascript eval

Previously, if the user had no eval permission, attempts to execute javascript failed with HTTP status 400 and a "MALFORMED PROGRAM" message. With this change, the coreect HTTP status 401 'unauthorized' is returned.

Bug22445 - Don't percent-decode path component of HTTP requests twice.

Previously, path components of HTTP requests sent to AllegroGraph were sometimes being percent-decoded twice. For example, the URI

http://localhost:10035/repositories/test%252frepo/statements 

could be seen by AG as either

http://localhost:10035/repositories/test%2frepo/statements 

or

http://localhost:10035/repositories/test/repo/statements 

depending on the URI being requested. This bug isn't likely to have been encountered unless you were attempting to include special characters in repository or catalog names. It is now fixed. Users with applications that make direct HTTP requests should check their request URIs to ensure that they do not double-percent-encode any path components, as these may stop working or have unexpected behavior. Percent-decoding of URI query arguments is not affected.

Bug22369 - Fix server-side javascript http-request property values.

Previously, the property values of javascript http-request objects (method, url, body, etc.) could raise errors due to returning invalid objects. This bug has been fixed, and return values should match what is described in the documentation.

Bug22329 - Fix handling of HTTP 'Expect: 100-Continue'

Previously, AllegroGraph did not send the '100 Continue' response to requests with 'Expect: 100-Continue' headers when the request was forwarded internally to another component. This happened, for instance, when a request to a shared backend or a dedicated session was forwarded via the frontend and also when a requested made directly to a dedicated session was forwarded to the frontend. With this change, '100 Continue' is sent as appropriate.

SPARQL

Rfe13029 - Use POST instead of GET for remote SPARQL queries

AllegroGraph now uses HTTP POST requests instead of HTTP GET requests when talking to remote SPARQL endpoints. This allows for larger query strings to be sent.

Rfe13028: Improve use of partial results when making SERVICE queries

AllegroGraph now makes more use of partially computed results when making federated SPARQL queries using the SERVICE clause. This results in more efficient queries against SPARQL endpoints.

Rfe12911 - Improve estimates for magic geospatial properties

AllegroGraph's size estimates for geospatial queries that use SPARQL magic properties are now more accurate. This improves query clause reordering and can greatly improve query performance.

Rfe12891 - Improve efficiency of some SPARQL geospatial queries

Some queries that use SPARQL magic properties to use AllegroGraph's geospatial engine will now operate more efficiently.

Rfe12822 - Add temporaryFileDiskSpace query option

Sometimes queries write intermediate results to disk when they will not fit in memory, for instance with an ORDER BY query. With a huge query it is possible for a such temporary files to fill the filesystem. In order to prevent this, the temporaryFileDiskSpace query option may be set. For example:

PREFIX franzOption_temporaryFileDiskSpace: <franz:10G> 

As with all query options, it is also possible to change the default value globally in the AllegroGraph configuration file. For example:

QueryOption temporaryFileDiskSpace=10G 

The default is to have no limit. Query options are described here in the SPARQL Reference; the AllegroGraph configuration file is described in Server Configuration.

Rfe12788 - Raise error for invalid use of SPARQL geospatial functions

AllegroGraph now signals an error at parse time when a geospatial function is used with a predicate that is not associated with a geospatial type-mapping.

Rfe12750: SPARQL Property Path cache improvements

SPARQL Property Path computations use a cache to improve the efficiency of transitive closure calculations. The data structures for this cache have been rewritten to be more memory and time efficient. Secondly, the heuristic that the query engine uses to determine whether or not the cache should be aggressively or lazily filled have been improved.

Rfe12470 - Apply multiple constraints to the same triple pattern

Previously, AllegroGraph could only apply a single logical FILTER expression to any given triple pattern. This meant that queries like:

select * {  
  ?s ?p ?o .  
  filter( ?p = ex:pred1 && ?o > 10 )  
} 

would optimize the query plan with either the ?p constraint or the ?o constraint but not both. This has now been improved so that queries like the above will use both constraints simultaneously and thereby run much more efficiently.

Rfe12284 - Signal an error if a SPARQL GEO pattern uses a constant object or graph place

The SPARQL GEO Clause uses its first pattern to link with AllegroGraph's geospatial system. The object place of this pattern must be left unbound. Previously, AllegroGraph would silently return no results when the place was bound. Now, it will signal an error. For example, the following query would have previously run but returned incorrect results but now it will signal an error.

prefix s: <http://franz.com/ns/allegrograph/3.0/geospatial/spherical/km/-180.0/180.0/-90.0/90.0/>  
select distinct ?event ?coord ?date ?platform ?EventType {  
  GEO SUBTYPE s:1.0  
  BOUNDINGBOX( POINT(-084,34), POINT(-082,36)) {  
    # this is an error because the first clause is the one  
    # that connects the query with AllegroGraph's geospatial  
    # sub-system  
    ?event a <http://UCore-SL-Taxonomy-OWL-DL#ObservationEvent> .  
    ?event mo:hasGeospatialCoordinate ?coord .  
  }  
  WHERE { }  
}  

Rfe12240 - Improve bound variable analysis when using magic properties

The SPARQL 1.1 query planner now recognizes that variables set via assignment using the BIND clause can be treated as bound when determining whether or not a magic property can be used. This means that queries like

select ?event ?coord {  
  bind(44.30 as ?xMin) .  
  bind(33.40 as ?yMax) .  
  (?event ?coord) franzGeo:inBoundingBoxXY (swp:startLatLong ?xMin 44.50 33.30 ?yMax)  
} 

can now be used.

Rfe11880 - Improve handling of some SPARQL GRAPH clauses

AllegroGraph's analysis of when and how it can execute SPARQL GRAPH clauses has been improved. Depending the query, this enhancement can speed execution by a factor of 10 or more.

Rfe10132 - Improve performance of find unique values for a column queries

SPARQL queries that ask for the DISTINCT values of a particular column are now sometimes significantly faster. Previously, they performed linearly in the size of the triple-store; now, they perform linearly in the size of the number of values returned. In particular, for stores with a small number of predicates or graphs (relative to the size of the store), queries like these will be much, much faster:

SELECT DISTINCT ?p { ?s ?p ?o }  
SELECT DISTINCT ?g { graph ?g { ?s ?p ?o } }  
SELECT DISTINCT ?class { ?s a ?class } 

Note that aggregate queries still require a full-scan of the store, so a query like:

SELECT ?p (count(1) as ?pCount) {  
  ?s ?p ?o .  
} 

will still be expensive.

Bug22658 - Using BIND and alternation in property paths could lose solutions

A query like

bind(ex:foo as ?head)  
?head ex:p1 | (ex:p2/ex:p3) ?tail 

could fail to find all solutions. This has been corrected.

Bug22529 - Improve use of intermediate information during query

AllegroGraph's set based SPARQL 1.1 query engine sometimes used an intermediate result set when it would have been more efficient to compute the solutions directly. This has been corrected.

Bug22535 - Improve handling of multiple range filters on the same variable

AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like

select * {  
  ?s ex:test ?age .  
  filter( ( ?age >= 10 && ?age <= 20 ) ||  
          ( ?age >= 60 && ?age <= 70 ))  
} 

will now execute as the combination of two range queries rather than a single scan.

Bug22485 - Improve pattern size estimation logic

AllegroGraph's estimation of the size of triple-patterns in a BGP has been improved. This in turn improves the automatic BGP re-ordering and leads to better query plans when using the statistical clauseReorderer.

Bug22430 - Cross products with unbound variables could lose results

Depending on the query algebra and the order in which clauses were evaluated, it was possible for AllegroGraph to lose some solutions with unbound variables when computing a cross product. This has been corrected.

Bug22391 - Make SPARQL rand() results more independent

Due to an unintended sharing of random number generator state, SPARQL rand() calls in queries running on the same backend can be produce the same stream of pseudo random numbers.

Bug22389 - The chunkProcessingMemory query option could not be specified in a query

The chunkProcessingMemory could only be set using the AllegroGraph config file rather than being able to also specify it in using the PREFIX notation in a SPARQL query. This has been corrected.

Bug22480 - Range queries could cause other filters to be ignored

If a query had multiple FILTERs that applied to a single pattern and one of these FILTERs created a range query, then it was possible for the other FILTERs to be ignored. This has been corrected.

Bug22372 - SPARQL JSON results format was using literal-typed instead of typed-literal

The SPARQL JSON results serializer was emitting literals that had an associated datatype as

"name" : {"type":"literal-typed", "datatype":" D ", "value":" S "} 

rather than

"name" : {"type":"typed-literal", "datatype":" D ", "value":" S "} 

This has been corrected.

Bug22348: EXISTS and NOT EXISTS FILTERs could fail

If an EXISTS or NOT EXISTS FILTER used variables from the outer query in a FILTER (rather than a BGP), then too many solutions could be returned. E.g., a query like:

PREFIX : <http://example.com/>  
SELECT ?x ?n WHERE {  
  ?x :p ?n  
  FILTER NOT EXISTS {  
    ?x :q ?m .  
    FILTER(?n = ?m)  
  }  
} 

uses the bindings for ?n in the NOT EXISTS FILTER and ?n is not one of the variables that the NOT EXISTS FILTER binds itself. Previously, AllegroGraph will leave ?n unbound and therefore return too many results. This has been corrected.

Bug22336 - Allow the use of magic properties in more situations

The SPARQL 1.1 query planner ensures that a magic property can only be used when all of its required variables are bound. Previously, however, the test for bound variables was overly strict and did not, for example, take BIND or VALUES into account. This has been corrected. For example, this query would previously have been disallowed:

prefix sna: <http://franz.com/ns/allegrograph/4.11/sna/>  
prefix pred: <http://www.testData.com/predicate/>  
prefix : <http://ex#>  
select * {  
  values ?start { <http://www.testData.com/graph/10444> }  
  ?graph sna:nodalNeighbors (:graphSubClass ?start) .  
} 

because AllegroGraph would not have seen that ?start is bound by the VALUES clause. Now this query will function as expected.

Bug22347 - Query options were sometimes ignored in SPARQL update commands

Previously, AllegroGraph PREFIX options were ignored during the WHERE clause of a SPARQL DELETE/INSERT/WHERE command. This has been corrected.

Bug22321 - Use other sets more in many more cases

The set-based SPARQL query engine now makes better use of partially computed result-sets during query execution. This can greatly accelerate many queries.

Bug22272 - Correct unbound variable handling in MINUS

In rare circumstances variables that were unbound in the right-hand side of a SPARQL MINUS clause could be incorrectly matched with bound variables on the left hand side. This has been corrected.

Bug22270 - Correct BGP estimation for predicates not in the store

If a query clause had either a fixed subject or object and a predicate that was not in the triple-store, then AllegroGraph's statistical estimator could return a positive number rather than zero. This has been corrected.

Bug22253 - Combining reorderDuringExecution and Property Path queries could lose filters

Under rare circumstances, the query engine could lose track of FILTERS in queries that used SPARQL Property Paths and the AllegroGraph reorderDuringExecution option. This has been corrected.

Bug22246 - Catch argument count mistakes in query functions

SPARQL queries that contained functions being called with an incorrect number of arguments would silently fail. These errors are now detected and reported at query time. For example, a query like:

prefix gf: <http://franz.com/ns/allegrograph/3.0/geospatial/fn/>  
select * {  
  bind( gf:cartesianDistance(1, 2, 3, 4) as ?b )  
}  
would have previously returned a single unbound result because the  
`cartesianDistance` function should only be given two arguments. Now,  
AllegroGraph will signal an error. 

Bug21941 - Magic properties could lose track of FILTERs

In rare circumstances a query that used magic properties and FILTER on some of the variables in the properties could incorrectly filter out too many solutions. This has been corrected.

AGWebView

Rfe13159 - Enhance AGWebView query GUI

AGWebView's SPARQL query user interface now supports query log viewing, query plan inspection and reports additional details about query execution such as query time and warnings. The HTTP REST API for query has a new parameter, returnQueryMetadata. If supplied and true, then the HTTP response will include a JSON payload containing information about the query in an HTTP header named x-query-info. The revised query page is described here.

Rfe13103 - Reposcripts added to AGWebView

Access to the new Reposcripts API has been added to AGWebView (see the Rfe12798 release notes entry, which has links to other documentation).

Rfe13068 - Login timeout for AGWebView

The new global configuration directive LoginTimeout controls the expiry of inactive AGWebView login sessions. See the description of that option in Server Configuration and Control for the details.

Rfe12925 - Only run AGWebView queries when the Execute button is pushed

Previously, AGWebView executed recent and saved queries as soon as they were selected. Now AGWebView only executes queries when the execute button is clicked.

Rfe12579 - Allow users to change their own passwords

This change introduces the PasswordChangeAllowed global configuration directive. If it is yes (the default), then users can change their own passwords via AGWebView. This is a change in default behavior (there was no facility to allow users to change their own passwords in earlier releases). If you set the value of the option to no, the old behavior will be restored. Configuration directives are described in Server Configuration and Control. User passwords are described in Security.

Rfe12275 - Add query cancellation interface to AGWebView's query pane

In addition to the existing job canceling screen, AGWebView now provides query cancellation directly from the SPARQL query interface. The revised query page is described here.

Bug22536 - Simple queries may not show up in the jobs list

Previously, simple queries like

select  { ?s ?p ?o . filter( fn(?o) ) } 

could fail to appear on the query jobs list in AGWebView. AllegroGraph now checks for query timeout more often which means that even these queries will appear and be cancelable.

Bug22260 - Fix AGWebView audit log date range

Previously, when the start date or the end date was specified in the audit log query (at /#audit), then the query failed with a type error. This bug has been fixed.

Bug20820 - AGWebView's multi-core data import could not specify a graph

When using multi-core loading, AGWebView was incorrectly formatting the graph it sent to AllegroGraph's parallel data import tool. This meant that it was impossible to specify a graph. This has been corrected.

Changes to the Lisp API

Rfe12862 - Extend Lisp geospatial functions with subject, object and graph arguments

Add s, o and g arguments to the Lisp API geospatial functions:

• get-triples-haversine-miles
• get-triples-haversine-km
• get-triples-geospatial-bounding-box
• get-triples-geospatial-radius

The new arguments act as additional filters on the triples returned by the functions. g can only be used when the use-g argument is nil and o can only be used when the use-g argument is non-nil.

Rfe12830 - Warn when a query has constants that are not in the store

In the Lisp API, the fourth return value of run-sparql now contains information about any warning that occurred during query planning and execution. This includes a warning when a query contains a constant that is not in the store. For example:

select * { ?s <ex://#notHere> ?o } 

would generate a warning regarding <ex://#notHere>. Additional warnings will be added in the future.

Rfe12546 - Implement get-duplicate-triples.

Add support for retrieving the set of duplicate triples in a repository via the following API calls

Lisp client:

get-duplicate-triples (&key (mode :spog) (db db) Returns a cursor of all duplicate triples visibile in db inside the current transaction.

• :db - defaults to the value of \db\
• :mode - specifies how triple equality is determined. It can be :spog or :spo and defaults to :spog.

REST/HTTP service

GET /repositories/[name]/statements/duplicates

Bug22322 - remote-triple-store does not honor default-prefixes argument to run-sparql

The default-prefixes argument to the Lisp run-sparql function was being ignored for remote-triple-stores. This has been corrected.

Bug22225 - Memory corruption in the Lisp direct client

Previously, if an unclosed database object became garbage, memory corruption could occur which lead to unpredictable errors such as segmentation faults under extremely rare circumstances.

Bug22167 - Fix flaws in delete-duplicate-triples.

delete-duplicate-triples is documented as preserving the triple with the lowest possible triple-id from a group of duplicate triples. Flaws in the algorithm used to delete duplicate triples made it possible that the triple preserved would not necessarily have the lowest possible triple-id, and could depend on the order in which the duplicates were sorted. This change corrects that problem.

Prolog

Bug22179 - Allow safe Prolog query requests in shared backends

Previously, AllegroGraph only allowed Prolog queries when a user had eval privileges for the store. Now, eval privileges are required only for Prolog queries that make use of functors that escape to Lisp.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe12952 - Update Java client for handling of useMainPortForSessions

When the server configuration option UseMainPortForSessions is enabled, all requests to sessions are multiplexed through the frontend port. In this case, the Java system property useMainPortForSession is redundant and its value is not consulted. This means that one can turn on this feature on the client if it's not enabled in the server config, but cannot turn it off.

Bug22392 - Hang with useMainPortForSessions

Previously, when the Java client's useMainPortForSessions feature was enabled, a request returning a large response could hang. This bug has been fixed.

Bug22323 - Fix UseMainPortForSessions when using SSL

Previously, when the com.franz.agraph.http.useMainPortForSessions property was set to true, the Java client arranged for all communication with a session be funneled through the front-end. This fix makes this feature work with SSL connections. Also, with this change it is required that SSLPort is set in the configuration file, otherwise using SSL connections to dedicated sessions with fail.

AllegroGraph 4.13.2 user-visible changes

The only change was to replace the SSL library with one that avoided the heartbleed bug.

AllegroGraph 4.13.1 user-visible changes

AllegroGraph Server: General Changes

Bug22255 - Concurrent full index optimization requests

Previously, if index optimization was requested with a full index optimization already in progress, the database instance would log an error and hang. This bug has been fixed.

AllegroGraph 4.13 user-visible changes

AllegroGraph Server: General Changes

Rfe12645 - Add a command line interface to AllegroGraph's materializer

AllegroGraph now includes a command line tool to materialize the inferred triples in a triple-store. The tool is named agmaterialize. It can currently be run only on the same machine where the AllegroGraph server is running. The command line interface is described in the Materializer documentation.

Rfe12642: Improve handling of datafile free space information

Previously, datafile free space information was maintained in a fixed sized data structure which could run out of room after a certain level of datafile fragmentation was reached. This has been corrected.

Rfe12590 - Shared backend scheduling improvements

Previously, when all shared backends were busy, further requests were held until a backend became available. Such pending requests were scheduled for execution in basically random order with latency up to 1s.

With this change, pending requests are scheduled rather than essentially random. The scheduling is basically FIFO (first in, first out, meaning the oldest request is processed first), but FIFO may be violated for efficiency reasons. (Strict FIFO behavior is undesirable, because it greatly reduces AllegroGraph's ability to schedule requests optimally under stress.)

Also with this change, scheduling latency is now minuscule and shared backends are now allowed to execute slightly more queries in parallel if all backends are busy, which allows greater efficiency in most cases.

Rfe9523 - Logging improvements

This change adds a centralized logging facility to the AllegroGraph daemon. The new logger guarantees no interleaving of messages and is able to deal with full filesystem conditions gracefully. Lisp direct clients have now the option of sending log messages to an AllegroGraph server with the with-log-routed-to-server macro. See the documentation for more. Previously, log messages were sent to the AllegroGraph server belonging to the first direct connection opened in the client.

Bug22220 - Race condition while determining last written transaction log id

Previously, a rare error could be encountered when starting replication or while switching roles, indicating that replication failed due to a missing transaction log that had not yet been written to disk. This problem has been fixed. Now, only transaction logs on disk are used to start replication.

Bug22184 - Instance shutdown and replication

In rare circumstances, when the instance belonging to a replicated repository was shut down, writing the transaction log could fail. This bug has been fixed.

Bug22156 - date-string-to-upi doesn't handle zero time zones consistently

The date-string-to-upi functor was not treating a Zulu time zone specified as "Z" consistently with the rest of AllegroGraph's date and time parsers. This has been corrected.

Bug22111 - Database names may not contain special characters

Because it is impossible to reliably use databases (also called repositories) in session specifications if their names contained characters that are used in those specifications, the list of characters which must not be used in names has been expanded. The following characters are now disallowed in database (repository) names:

\ / ~ : Space $ { } < > * + [ ] | 

Previously, only the following four characters were disallowed in database (repository) names:

\ ~ / : 

Note that this change only effects new databases (repositories); existing databases (repositories) with any of the newly invalid characters in their names will continue to be accessible. It may, however, result in difficulties when attempting to use them in more complex session specifications.

Bug22100 - More strict, faster parsing of some xsd datetimes

The parsers for xsd:date, xsd:dateTime and xsd:time used to be lenient and accept things like:

20131118         # missing separator  
2013-11-18T12:27 # missing seconds 

and many other convenient but non-standard formats. The new parser only accepts what the XML Schema specification mandates which also allows it to be faster. See XSD Date and Time datatypes for more information on these datatypes.

Bug22086 - Materializer deletes triples with the same subject, predicate and object

The initial version of AllegroGraph's materializer deleted all triples with the same subject, predicate and object. This limitation has now been lifted: the materializer now only deletes triples with matching subject, predicate, object and graph.

Bug22032 - Control whether or not external references are followed during RDF/XML import

Previously, AllegroGraph ignored external references in RDF/XML source files. The Lisp API now includes a new keyword argument to load-rdf/xml:

• :resolve-external-references : if true, the external references will be followed. The default continues to be nil.

agload also has a new command-line switch. Supplying either of -x or --external-references will cause agload to follow external references. See Data Loading.

Bug21953 - Allow import of triples to reasoning-triple-store

A bug was introduced in v4.12 that prevented import of triples via load-* routines (e.g. load-rdf/xml) on reasoning-triple-stores, when the source argument was passed a URI. This problem has been fixed. This change also provides a better error message when attempting to import statements to a federated-triple-store, which is not allowed.

Bug21656 - Cleaner daemon shutdown

When the AllegroGraph daemon was stopped, despite the claim of doing a 'clean shutdown' in agraph.log, the shutdown of the instances was not clean and the corresponding repositories would potentially require crash recovery the next time they were opened. Also, such a shutdown lost any index optimization work performed since the most recent checkpoint.

With this change, instance processes perform a checkpoint if necessary which makes shutdown slower, but startup faster and it makes sure the index optimization results persist.

Bug20773 - Support federations of RDFS++ stores

Previously, queries against a federation of RDFS++ stores would cause an error during query planning. This has been corrected.

HTTP Client

Rfe12763 - Update HTTP API for the materializer

The REST API for materializeEntailed now includes an inferredGraph argument that can be used to specify which graph newly entailed triples are added to (or deleted from). The materializer is described in Materializer.

SPARQL

Rfe12767 - Improved the efficiency of single group aggregate queries

The code generated for single group aggregate queries is now slightly more efficient.

Rfe12762 - Optimizing level 2 index optimization

With this change, level 2 index optimization (the default) no longer tries to optimize access for triples added since optimization was started. This allows index optimization to finish even if triples are added continuously at a high rate. The new code is faster on average, has the same worst case disk usage (about two times the original size), but it has a higher average disk usage.

Rfe12760 - Improve efficiency of some SPARQL Property Path queries

AllegroGraph now builds some property path data structures as needed rather than computing all of them up front. Depending on the data, this can greatly enhance the efficiency of some queries.

Rfe12757 - Improve efficiency of some property path queries

In some circumstances, AllegroGraph could choose the less efficient of two strategies when evaluating a property path query. AllegroGraph now ensures that it makes the better choice.

Rfe12717 - Handle RDFterm-equality of strings consistently

Previously, AllegroGraph implemented sameTerm() using a strict interpretation of RDF literal equality when comparing plain literals and typed literal strings but implemented = using a more relaxed interpretation. These tests are now done consistently using the relaxed interpretation. In particular,

'hello' is RDFterm-equal to 'hello'^^xsd:string 

This is in accordance with both common usage and the RDF 1.1 specification.

Rfe12699 - Improve efficiency of SPARQL equality tests on resources

The efficiency of equivalency tests in SPARQL filter expressions has been improved.

Rfe12681 - Optimize interprocess communication

The performance of message passing between different AllegroGraph server processes has been slightly improved.

Rfe12679 - Use query local namespace abbreviations in result serialization

AllegroGraph now uses the namespaces defined by a SPARQL query's prefixes when serializing output in RDF/XML. Previously, temporary namespaces were used instead.

Rfe12670 - Allow /SPARQL access to AllegroGraph as a SPARQL endpoint

Previously, access to an AllegroGraph triple-store as a SPARQL endpoint was via sending a query to

SERVER:PORT/respositories/NAME?query=... 

This REST API has been augmented so that sending a query to the triple-store's URL with a trailing /sparql also works. E.g.,

SERVER:PORT/respositories/NAME/sparql?query=... 

This is not required by the SPARQL standard but is in conformance with many other existing endpoints.

Rfe12657 - More efficient handling for the VALUES clause

AllegrGraph now rearranges queries so that VALUES clauses are evaluated first when possible. This improves query efficiency as VALUES clauses are generally small.

Rfe12656 - Improve execution of some nested joins

AllegroGraph now evaluates nested joins more efficiently.

Rfe12498 - Improve full-scan warning heuristics

AllegroGraph warns when a SPARQL query executes a full-scan. Heretofore, this meant that a query like

select * { ?s ?p ?o } limit 100 

could have produced a full-scan warning even though it is benign. AllegroGraph now saves its warnings for when they may actually indicate a problem with the query.

Rfe11981 - New query option chunkProcessingMemory

Previously when chunk-at-a-time processing was enabled (see query option chunkProcessingAllowed), the query option chunkProcessingSize controlled memory consumption by limiting the number of intermediate results stored in memory. The total memory consumption depended on the number of variables and the complexity of the query, so it was rather difficult to know how much a query will actually use. With this change, chunkProcessingSize is deprecated and chunkProcessingMemory can be specified instead. The unit of chunkProcessingMemory is a byte but the usual abbreviations are also accepted ("8G" (the default), "2M").

Rfe11515 - Fully support the SPARQL 1.1 HTTP Protocol

AllegroGraph now fully supports the five different HTTP request types specified in the SPARQL 1.1 HTTP Protocol. These are:

• Query
  • via GET with query parameters
  • via POST with urlencoded parameters
  • via POST with a plain query string (using the content-type of application/sparql-query).
• Update
  • via POST with urlencoded parameters
  • via POST with a plain update request string (using the content-type of application/sparql-update).

In addition, AllegroGraph continues to support SPARQL update using POST where the update command is encoded in the query portion of the HTTP request.

Bug22350 - Distinct queries could produce duplicate results in rare cases

Under rare circumstances it was possible for the query engine to produce duplicate results when using chunk-at-a-time (CaaT) processing. This has been corrected.

Bug22214 - Encoded IDs failed to compare properly in SPARQL queries

Inequality comparisons between encoded IDs were always returning true. This has been corrected.

Bug22204 - decimal to decimal cast loses precision

When in a SPARQL query an xsd:decimal was cast to xsd:decimal it lost precision as it was converted to a single float internally. This bug has been fixed.

Bug22203 - Fix decimal parsing in SPARQL queries

In SPARQL query strings, decimals in the (-1,0) interval were parsed with the wrong sign. For example, -0.9 was parsed as 0.9. This bug has been fixed.

Bug22153 - Some wild card property path queries could generate incorrect paths

In rare circumstances, the data structures used to cache property path information could be invalidated leading to too many path results. This has been corrected.

Bug22147 - Add support for ObjectId's in the _id field.

The SPARQL MongoDB magic predicate now works with documents containing ObjectId's in the _id field.

Bug22142 - Expressions on aggregates fail for empty group

If a query wrapped an aggregate with an expression like

select (xsd:float(avg(?age)) as ?avgAge) ... 

and the result set contained an empty group (i.e., a group with no members), then AllegroGraph would signal error. This has been corrected.

Bug22134 - SPARQL aggregates on expressions with errors could compute incorrect results

Other than COUNT, a SPARQL aggregate on an expression should return unbound if that expression evaluates to a type error during computation. AllegroGraph was incorrectly returning the aggregate of any bindings produced after the last type error. This has been corrected.

As an example, if AllegroGraph needed to compute the MIN( ?p * 2 ) and ?p was bound to 4, 7, 15, "hello", 6, and 1, then AllegroGraph would previously have returned 2 (the minimum of 12 and 2). Now, it will return no binding for that expressions.

Bug22123: Empty construct template will cause an error

If a SPARQL CONSTRUCT had an empty template like

CONSTRUCT {} WHERE { ?s a ?o } 

then AllegroGraph would signal an error. This has been corrected.

Bug22094 - Correct sparql< and sparql> for date and dateTime comparisons

Because there is no defined operator mapping for inequality comparisons between xsd:date and xsd:dateTime, comparing them should throw a SPARQL type-error. AllegroGraph was incorrectly returning nil instead. This has been corrected. See XSD Date and Time datatypes for more information on these datatypes.

Bug22074 - DELETE templates with unknown variables caused an error

AllegroGraph would signal an error if a SPARQL DELETE/WHERE command's DELETE clause contained variables that were not bound by the WHERE clause. This has been corrected. An example of such a command would be:

DELETE { graph ?unknownGraph { ?s ?p ?o } }  
WHERE { ?s ?p ?o }  
 

Bug21561 - Remote-triple-stores do not support the table results format

Remote-triple-stores did not support returning tables from SPARQL queries. This has been corrected.

AGWebView

Bug22161 - Adding security filters in Firefox

Previously, with Firefox, clicking the 'add' button in the security filters table of the user management page (Admin/users in the menu) of AGWebView did not submit the form. This bug has been fixed.

Changes to the Lisp API

Rfe12632: Improvements in Lisp API to AllegroGraph materializer

Materialization previously occurred only for triples in the default-graph of the triple-store. It now occurs for all of the graphs in the store. Materialization on subsets of the graphs can be accomplished by creating a graph-filtered triple-store and calling materialize-entailed-triples on it.

Materialization previously placed inferred triples into a special graph that could not be utilized by the rest of AllegroGraph (the special graph behaved similarly to but not exactly like the usual AllegroGraph default-graph). It now places inferred triples into a graph that can be specified in the call to materialize-triples using the materialized-graph argument. It defaults to .

Two new arguments were added to delete-materialized-triples. These are:

• db - which controls the triple-store that is modified.
• materialized-graph - which controls the triples that are deleted. It defaults to .

Materialization now logs its start and end to the AllegroGraph log file. The overall efficiently of the materializer has also been improved.

Bug22132 - calling run-sparql with the :with-variable argument with a timezoned upi

When the Lisp function run-sparql was used in conjunction with a :with-variable argument that bound a variable to an encoded xsd:dateTime or xsd:time, then in some cases comparisons involving that variable failed to match equivalent representations of the same xsd:dateTime or xsd:time with a different timezone. This bug has been fixed.

Prolog

Bug22171: Select0 should return UPIs when it can

Previously, the Prolog select0 query macro could return future-parts rather than UPIs. For example, this query would return a future-part.

(select0 (?a) (= ?a !rdf:type)) 

This could cause problems in the unlikely event that namespace abbreviations changed when running the same query multiple times. select0 now returns UPIs so that the above behavior is not possible. Note that if a future-part represents a triple that is not currently in the triple-store, then a query can return UPIs that cannot be printed (because the UPI will not be in the string dictionary).

Bug22157 - Temporal functors could signal an error when used with all bound values

Previously, AllegroGraph used real numbers to represent times, dates and dateTimes. Now, the internal representation includes time-zone information which means that comparing two values can no longer use simple inequalities. The Prolog temporal functors like point-before-datetime had not been updated to reflect these changes. This oversight has been corrected. This change means that the Prolog functors will use the same XSD logic used by the SPARQL date and time functions.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

No significant changes.

See Change History for user-visible changes to earlier releases.