Release Notes
HarperDB 2.3.1, Penny Release (Current Release)
A bug in HarperDB 2.3.0 was identified related to manually calling the create_attribute
operation. This bug caused secondary indexes to be overwritten by the most recently inserted or updated value for the index, thereby causing a search operation filtered with that index to only return the most recently inserted/updated row. Note, this issue does not affect attributes that are reflexively/automatically created. It only affects attributes created using create_attribute
.
To resolve this issue in 2.3.0 or earlier, drop and recreate your table using reflexive attribute creation. In 2.3.1, drop and recreate your table and use either reflexive attribute creation or create_attribute
.
To resolve this issue in 2.3.0 or earlier, drop and recreate your table using reflexive attribute creation. In 2.3.1, drop and recreate your table and use either reflexive attribute creation or create_attribute
.
HarperDB 2.3.0, Penny Release
Features/Updates
- CORE-1191, CORE-1190, CORE-1125, CORE-1157, CORE-1126, CORE-1140, CORE-1134, CORE-1123, CORE-1124, CORE-1122
Added JWT Authentication option (See documentation for more information) - CORE-1128, CORE-1143, CORE-1140, CORE-1129
Addedupsert
operation - CORE-1187 Added
get_configuration
operation which allows admins to view their configuration settings. - CORE-1175 Added new internal LMDB function to copy an environment for use in future features.
- CORE-1166 Updated packages to address security vulnerabilities.
Bug Fixes
- CORE-1195 Modified
drop_attribute
to drop after data cleanse completes. - CORE-1149 Fix SQL bug regarding self joins and updates alasql to 0.6.5 release.
- CORE-1168 Fix inconsistent invalid schema/table errors.
- CORE-1162 Fix bug which caused
delete_files_before
to cause tables to grow in size due to an open cursor issue.
HarperDB 2.2.2, Penny Release
10/27/2020
- CORE-1154 Allowed transaction logging to be disabled even if clustering is enabled.
- CORE-1153 Fixed issue where
delete_files_before
was writing to transaction log. - CORE-1152 Fixed issue where no more than 4 HarperDB forks would be created.
- CORE-1112 Adds handling for system timestamp attributes in permissions.
- CORE-1131 Adds better handling for checking perms on operations with action value in JSON.
- CORE-1113 Fixes validation bug checking for super user/cluster user permissions and other permissions.
- CORE-1135 Adds validation for valid keys in role API operations.
- CORE-1073 Adds new
import_from_s3
operation to API.
HarperDB 2.2.0, Penny Release
08/24/2020
Features/Updates
- CORE-997 Updated the data format for CSV data loads being sync'd across a cluster to take up less resources
- CORE-1018 Adds SQL functionality for
BETWEEN
statements - CORE-1032 Updates permissions to allow regular users (i.e. non-super users) to call the
get_job
operation - CORE-1036 On create/drop table we auto create/drop the related transactions environments for the schema.table
- CORE-1042 Built raw functions to write to a tables transaction log for insert/update/delete operations
- CORE-1057 Implemented write transaction into lmdb create/update/delete functions
- CORE-1048 Adds
SEARCH
wildcard handling for role permissions standards - CORE-1059 Added config setting to disable transaction logging for an instance
- CORE-1076 Adds permissions filter to describe operations
- CORE-1043 Change clustering catchup to use the new transaction log
- CORE-1052 Removed word "master" from source
- CORE-1061 Added new operation called
delete_transactions_before
this will tail a transaction log for a specific schema / table - CORE-1040 On HarperDB startup make sure all tables have a transaction environment
- CORE-1055 Added 2 new setting to change the server headersTimeout & keepAliveTimeout from the config file
- CORE-1044 Created new operation
read_transaction_log
which will allow a user to get transactions for a table by timestamp, username, orhash_value
- CORE-1043 Change clustering catchup to use the new transaction log
- CORE-1089 Added new attribute to
system_information
for table/transaction log data size in bytes & transaction log record count - CORE-1101 Fix to store empty strings rather than considering them null & fix to be able to search on empty strings in SQL/NoSQL.
- CORE-1054 Updates permissions object to remove delete attribute permission and update table attribute permission key to
attribute_permissions
- CORE-1092 Do not allow the
__createdtime__
to be updated - CORE-1085 Updates create schema/table & drop schema/table/attribute operations permissions to require super user role and adds integration tests to validate
- CORE-1071 Updates response messages and status codes from
describe_schema
anddescribe_table
operations to provide standard language/status code when a schema item is not found - CORE-1049 Updates response message for SQL update op with no matching rows
- CORE-1096 Added tracking of the origin in the transaction log. This origin object stores the node name, timestamp of the transaction from the originating node & the user.
Bug Fixes
- CORE-1028 Fixes bug for simple
SQL SELECT
queries not returning aliases and incorrectly returning hash values when not requested in query - CORE-1037 Fixed an issue where numbers with leading zero i.e. 00123 are converted to numbers rather than being honored as strings.
- CORE-1063 Updates permission error response shape to consolidate issues into individual objects per schema/table combo
- CORE-1098 Fixed an issue where transaction environments were remaining in the global cache after being dropped.
- CORE-1086 Fixed issue where responses from insert/update were incorrect with skipped records.
- CORE-1079 Fixes SQL bugs around invalid schema/table and special characters in
WHERE
clause
HarperDB 2.1.1, Penny Release
05/22/2020
Highlights:
- CORE-1007 Added the ability to perform
SQL INSERT
&UPDATE
with function calls & expressions on values. - CORE-1023 Fixed minor bug in final SQL step incorrectly trying to translate ordinals to alias in
ORDER BY
statement. - CORE-1020 Fixed bug allowing 'null' and 'undefined' string values to be passed in as valid hash values.
- CORE-1006 Added SQL functionality that enables
JOIN
statements across different schemas. - CORE-1005 Implemented JSONata library to handle our JSON document search functionality in SQL, creating the
SEARCH_JSON
function. - CORE-1009 Updated schema validation to allow all printable ASCII characters to be used in schema/table/attribute names, except, forward slashes and backticks. Same rules apply now for hash attribute values.
- CORE-1003 Fixed handling of ORDER BY statements with function aliases.
- CORE-1004 Fixed bug related to
SELECT*
onJOIN
queries with table columns with the same name. - CORE-996 Fixed an issue where the
transact_to_cluster
flag is lost for CSV URL loads, fixed an issue where new attributes created in CSV bulk load do not sync to the cluster. - CORE-994 Added new operation "
system_information
". This operation returns info & metrics for the OS, time, memory, cpu, disk, network. - CORE-993 Added new custom date functions for AlaSQL & UTC updates.
- CORE-991 Changed jobs to spawn a new process which will run the intended job without impacting a main HarperDB process.
- CORE-992 HTTPS enabled by default.
- CORE-990 Updated
describe_table
to add the record count for the table for LMDB data storage. - CORE-989 Killed the socket cluster processes prior to HarperDB processes to eliminate a false uptime.
- CORE-975 Updated time values set by SQL Date Functions to be in epoch format.
- CORE-974 Added date functions to
SQL SELECT
column alias functionality.
HarperDB 1.3.1, Alby Release
2/26/2019
Features
- Clustering connection direction appointment
- Foundations for threading/multi processing
- UUID autogen for hash attributes that were not provided
- Added cluster status operation
Bug Fixes and Enhancements
- More logging
- Clustering communication enhancements
- Clustering queue ordering by timestamps
- Cluster re connection enhancements
- Number of system core(s) detection
- Node LTS (10.15) compatibility
- Update/Alter users enhancements
- General performance enhancements
- Warning is logged if different versions of harperdb are connected via clustering
- Fixed need to restart after user creation/alteration
- Fixed SQL error that occurred on selecting from an empty table
HarperDB 1.3.0, Alby Release
11/2/2018
Features
- Upgrade: Upgrade to newest version via command line.
- SQL Support: Added
IS NULL
for SQL parser. - Added attribute validation to search operations.
Fixes
- Fixed
SELECT
calculations, i.e.SELECT
2+2. - Fixed select OR not returning expected results.
- No longer allowing reserved words(https://harperdbhelp.zendesk.com/hc/en-us/articles/115004292473-HarperDB-Reserved-words)for schema and table names.
- Corrected process interruptions from improper SQL statements.
- Improved message handling between spawned processes that replace killed processes.
- Enhanced error handling for updates to tables that do not exist.
- Fixed error handling for NoSQL responses when
get_attributes
is provided with invalid attributes. - Fixed issue with new columns not being updated properly in update statements.
- Now validating roles, tables and attributes when creating or updating roles.
- Fixed an issue where in some cases“undefined” was being returned after dropping a role
HarperDB 1.2.0, Alby Release
7/10/2018
Features
- Time to Live: Conserve the resources of your edge device by setting data on devices to live for a specific period of time.
- Geo: HarperDB has implemented turf.js into its SQL parser to enable geo based analytics.
- Jobs: CSV Data loads, Exports & Time to Live now all run as back ground jobs.
- Exports: Perform queries that export into JSON or CSV and save to disk or S3.
Fixes
- Fixed issue where CSV data loads incorrectly report number of records loaded.
- Added validation to stop
BETWEEN
operations in SQL. - Updated logging to not include internal variables in the logs.
- Cleaned up
add_role
response to not include internal variables. - Removed old and unused dependencies.
- Build out further unit tests and integration tests.
- Fixed https to handle certificates properly.
- Improved stability of clustering & replication.
- Corrected issue where Objects and Arrays were not casting properly in
SQL SELECT
response. - Fixed issue where Blob text was not being returned from
SQL SELECT
s. - Fixed error being returned when querying on table with no data, now correctly returns empty array.
- Improved performance in SQL when searching on exact values.
- Fixed error when ./harperdb stop is called.
- Fixed logging issue causing instability in installer.
- Fixed
read_log
operation to accept date time. - Added permissions checking to
export_to_s3
. - Added ability to run SQL on
SELECT
without aFROM
. - Fixed issue where updating a user’s password was not encrypting properly.
- Fixed
user_guide.html
to point to readme on git repo. - Created option to have HarperDB run as a foreground process.
- Updated
user_info
to return the correct role for a user. - Fixed issue where HarperDB would not stop if the database root was deleted.
- Corrected error message on insert if an invalid schema is provided.
- Added permissions checks for user & role operations.
HarperDB 1.1.0, Alby Release
4/18/2018
Features
- Users & Roles:
- Limit/Assign access to all HarperDB operations
- Limit/Assign access to schemas, tables & attributes
- Limit/Assign access to specific SQL operations (
INSERT
,UPDATE
,DELETE
,SELECT
)
- Enhanced SQL parser
- Added extensive ANSI SQL Support.
- Added Array function, which allows for converting relational data into Object/Hierarchical data: http://www.harperdb.io/blog/sql-queries-complex-objects-array-function
Distinct_Array
Function: allows for removing duplicates in the Array function.- Enhanced SQL Validation: Improved validation around structure of SQL, validating the schema, etc..
- 10x performance improvement on SQL statements.
- Export Function: can now call a NoSQL/SQL search and have it export to CSV or JSON.
- Added upgrade function to CLI
- Added ability to perform bulk update from CSV
- Created landing page for harperdb.
- Added CORS support to HarperDB : https://harperdbhelp.zendesk.com/hc/en-us/articles/115002214294-HTTP-access-control-CORS-
Fixes
- Fixed memory leak in CSV bulk loads
- Corrected error when attempting to perform a
SQL DELETE
- Added further validation to NoSQL
UPDATE
to validate schema & table exist - Fixed install issue occurring when part of the install path does not exist, the install would silently fail.
- Fixed issues with replicated data when one of the replicas is down
- Removed logging of initial user’s credentials during install
- Can now use reserved words as aliases in SQL
- Removed user(s) password in results when calling
list_users
- Corrected forwarding of operations to other nodes in a cluster
- Corrected lag in schema meta-data passing to other nodes in a cluster
- Drop table & schema now move the table & schema or table to the trash folder under the Database folder for later permanent deletion.
- Bulk inserts no longer halt the entire operation if n records already exist, instead the return includes the hashes of records that have been skipped.
- Added ability to accept EULA from command line
- Corrected
search_by_value
not searching on the correct attribute - Added ability to increase the timeout of a request by adding
SERVER_TIMEOUT_MS
to config/settings.js - Add error handling resulting from SQL calculations.
- Standardized error responses as JSON.
- Corrected internal process generation to not allow more processes than machine has cores.