Idyl E3 Release Notes

This page provides a changelog of Idyl E3 versions. New versions are typically available in cloud marketplaces within a week or two following the release date.

Version 3.0.0 (December 3, 2017)

This version converts Idyl E3 to a microservice that only performs entity extraction. The functionality of sentence extraction, tokenization, and language detection are now provided by Prose Sentence Extraction Engine, Sonnet Tokenization Engine, and Renku Language Detection Engine, respectively. As a result, the Idyl E3 API has changed.

Version 2.6.0 (November 6, 2017)

  • New: GPU capable edition of Idyl E3 that uses GPUs to create and use entity models.
  • New: Models that fail to load are now blacklisted and reload attempts will not happen until Idyl E3 is restarted.
  • New: Added scripts to perform model validation to avoid having to run the jar files directly.
  • New: $IDYL_E3_HOME environment variable must be set. On AWS Marketplace images this is automatically set to /opt/idyl-e3.
  • New: HMAC API authentication now includes the hash of the text being submitted.  (See the documentation.)
  • New: Entity extraction requests can now cover multiple entity types by setting the type parameter to a comma-separated list of types, e.g. person,place.
  • New: Deep learning models now publish metrics to the console and/or to Graphite.
  • Fix: Fixed timestamp checking when using HMAC API authentication.
  • Fix: Text in the Token Model Validator incorrectly said Sentence Model Validator.
  • Fix: Model validation tools omitted the number of folds in the logged message.
  • Fix: Stats reporter could cause an error when updating confidence values.
  • Tweak: Generated data files are stored in $IDYL_E3_HOME/data.
  • Tweak: Improved thread-safety to ensure accurate responses.
  • Tweak: Removed deprecated /ingest API endpoint.
  • Tweak: Tightened permissions on Idyl E3 files on AWS Marketplace images.
  • Tweak: Update checks are now disabled by default – setting can be changed in idyl-e3.properties.

Version 2.5.1 (October 5, 2017)

Version 2.5.0 (August 28, 2017)

  • New: English-language person entity models can now be trained using data in the CoNLL-2003 format. (Note this only applies to English-language person entity models.)
  • New: Added support for training and using deep learning neural network entity models. (See blog post.)
  • New: Changed to 3-letter ISO 3166-1 alpha-3 language codes. 2-letter codes are still supported by use of 3-letter codes are recommended.
  • New: Added support for the use of dictionary entity models.
  • New: Added support for custom annotation types when training entity models.
  • New: Added a setting to control how duplicate entities (from a single extraction request) are handled.
  • New: A TCP endpoint accepts streaming text. This endpoint deprecates the /ingest API endpoint.
  • Fix: Entities extracted without a probability are now correctly assigned a probability of 1.0 instead of 100.0.
  • Fix: Lemma and parts-of-speech model manifests could be incorrectly generated as token model manifests.
  • Tweak: Restructured the idyl-e3 application directory into bin/ and lib/ directories.
  • Tweak: Added shell scripts for performing model training.
  • Tweak: Changed the ASCII banner displayed when starting Idyl E3.
  • Tweak: Removed auto as a valid parameter for language. Language detection is no longer supported.

Version 2.4.0 (May 2, 2017)

  • New: Adds support for the Idyl Annotation Format (IAF) that lets the annotations for entity model training text be stored external to the training text. (See blog post.)
  • New: Allows the user to configure how duplicate entities are handled. (See blog post.)
  • Tweak: Changed the console starting banner to “Idyl E3.”
  • Tweak: Removed the deprecated API v1 (/api/v1).

Version 2.3.0 (March 9, 2017)

  • Important: Version 1 of the API has been deprecated and will be removed in a future release. Please migrate to the version 2 API as soon as possible.
  • New: HMACSHA512 API authorization option. (See the API documentation.)
  • New: Parts-of-speech-based feature generator for custom models. Model training and evaluation can now consider each token’s part-of-speech.
  • New: Tools to create parts-of-speech and lemmatization models.
  • New: Added special character feature generator. (See blog post.)
  • New: Added word normalization feature generator. (See blog post.)
  • New: Added token part of speech feature generator. (See blog post.)
  • New: Added model version verification to make sure each model is compatible with the version of Idyl E3.
  • New: Added sort parameter to V2 /extract endpoint to control how extracted entities are sorted before returned. Entities can be sorted by confidence, entity text, and by occurrence in the input text. (See the API documentation.)
  • New: Added new plugin for using Google Cloud Natural Language API to perform supplementary entity extraction. (See blog post.)
  • New: Added new API endpoint to sanitize input text. The identified entities are “blacked out” from the input text. (See the API documentation.)
  • New: There is now an Idyl E3 SDK for Go.
  • New: The API endpoints now respond to HEAD requests with HTTP 405 Method Not Allowed. This change is to support smoother integration with Apache NiFi’s PostHTTP processor. (See blog post.)
  • Fix: Fixed problem where entity model validation via separate data can return an incorrect F-measure of -1.0.
  • Tweak: Added Idyl E3 version number to the README.txt file.
  • Other minor changes and fixes.
  • User Documentation

Version 2.2.0 (February 1, 2017)

  • New: Added heuristic confidence filtering. (See blog post.)
  • New: Added support for any UTF-8 language.
  • New: Added support to the V2 API for providing metadata for extracted entities.
  • New: Statistics reporting to Graphite and the console tracks entity counts and requests. (See blog post for scripts to create a Graphite AMI.)
  • New: A separate timing metric for annotations is now available to be reported.
  • New: Models can now have names that are used to identify the model. A model’s name is assigned by the user when the model is generated.
  • New: Extract, ingest, and annotate endpoints now accept an optional type parameter that restricts the entity extraction to a specific entity type.
  • Tweak: Removed the confidence and documentId parameters from the /annotate endpoint as they are not used.
  • Tweak: An empty encryption key is now invalid. A key must be at least one character.
  • Tweak: If API authentication is enabled but no API key is specified a warning will be logged in Idyl E3’s log file.
  • Fix: The confidence threshold defaults to 0 when not provided during an API /upload request.
  • Fix: API V2 /extract now correctly returns 400 instead of 500 when an entity extraction request does not contain any text.
  • Fix: Specifying “all” for the language to an entity extraction request could fail to load the appropriate tokenizer or sentence detector for each available entity model.
  • Other minor changes and fixes.
  • User Documentation

Version 2.1.0 (Released January 6, 2017)

  • New: Added V2 REST API endpoints for /extract and /ingest accept input text in the request body. (The Idyl E3 client SDKs on GitHub have been updated to use the V2 APIs.)
  • New: Added LEXS document processing plugin.
  • New: Built on OpenNLP 1.7.0.
  • Tweak: A model encryption key is no longer optional when creating a custom model.
  • Tweak: All API endpoints return UTF-8.
  • Fix: Fixed an error that can happen when an entity extraction request’s language contains no entity models.
  • Other minor changes and fixes.
  • User Documentation

Version 2.0 (Released December 20, 2016)

This is a major revision that focused primarily on performance improvements.

  • New: Can now now use custom sentence and token models.
  • New: Plugin system allows for extending Idyl E3’s functionality to be able to consume text from other sources, parse binary files, perform pattern-based extraction, and publish entities to external services.
  • New: Simplified configuration properties.
  • New: Includes tools to create entity, sentence, and token models.
  • New: Supports integration with UIMA through the Idyl E3 UIMA connector.
  • New: Removed web dashboard.
  • Tweak: API endpoints for health, extraction time, and status were combined into a single /api/status endpoint.
  • Other minor changes and fixes.
  • User Documentation

Version 1.12.0 (Released August 17, 2016)

  • New: Added optional integration with EntityDB. EntityDB replaces Idyl E3’s entity store functionality with a new product devoted solely to storing and querying entities.
  • Tweak: Minor improvements and fixes.

Version 1.11.0 (Released on June 24, 2016)

  • Fix: Entity persistence when using an RDBMS-backed entity store. (Noted as a problem in the 1.10.0 Release Notes below.)
  • Fix: The dashboard’s Entity Models page now accurately displays the entity models and each’s status.
  • New: Added support for multilingual documents. (Specify “all” as language parameter to the API. See the API documentation for more information.)
  • New: Added support for extracting dates from English text.
  • New: Added support for using a date filter in an EQL-based rule.
  • New: Added support for using entity models for the same entity type concurrently. (For example, you can now use an English person entity model and a German person entity model at the same time.)
  • New: Added ability to select which tokenizer (whitespace or character-based) to use during extraction.
  • New: Image files ingested through the file system will be processed by OCR.
  • New: Text from binary files (DOCX, PDF) is extracted when ingesting from the file system.
  • Tweak: Performance improvements for extracting email addresses, Twitter usernames, and hashtags.
  • Tweak: No longer required to set environment variables prior to running Idyl E3.

Version 1.10.0

  • Tweak: Regions for AWS services are now provided by endpoint URL.
  • Known Problem: There is a known issue concerning the persistence of entities to a RDBMS database-backed entity store. This problem will be fixed in version 1.11.0.

Version 1.9.0

  • New: Can now use an EQL statement as a condition in a rule.
  • Tweak: All statistical time units are now in milliseconds.
  • Tweak: Removed the “skip integrations” option from the dashboard extraction form.

Version 1.8.0

  • New: Added rules engine to process extracted entities.
  • New: Added ability to consume entity extraction requests from an Amazon Web Services Simple Queue Service (SQS) queue.

Version 1.7.1

  • Fix: Fixed model loading issue.

Version 1.7.0

  • Minor UI and performance improvements.
  • Entity models are now configurable.

Version 1.6.0

  • Minor UI and performance improvements.
  • Restarts are no longer required after changing settings.
  • Now supports optional API authentication.
  • Entity Filters can be enabled via the UI.
  • Added AWS SQS visibility timeout setting.
  • Added AWS SNS message subject setting.
  • Added AWS CloudWatch metric name setting.
  • Updated cURL example API calls.
  • Fixed API upload failing if parameters not provided.
  • Settings are stored encrypted.

Version 1.5.3

  • Adding support for MongoDB as an entity store.
  • Adding EQL as a method for querying the entity store.
  • API changes updated in the Idyl E3 client SDKs.
  • Added /api/eql API endpoint.
  • Added /api/query API endpoint.
  • Added support for extracting Twitter usernames.
  • Added support for extracting email addresses.
  • Added support for extracting phone numbers.
  • Fixed issue handling uploaded documents.

Version 1.5.2

  • Simplifying statistics.
  • Performance improvements.
  • Adding /api/ingest API endpoint.
  • Removed text limit on API POST requests.

Version 1.5.1

  • Fix problem where average extraction time could be incorrect.
  • Fix problem where extracted text could be returned empty.

Version 1.5.0

  • Added support for DynamoDB as an entity store.
  • Added ability to query the entity store by entity enrichments.
  • Added ability for Idyl E3 to restart itself when necessary after settings are changed.
  • Added documentId to the query API.
  • Added support for integrating with Idyl Cloud using a Mashape key.

Version 1.4.0

  • The Idyl E3 log is now located at /var/log/idyl-e3/idyl-e3.log.
  • The service name is now idyl-e3.
  • The Idyl E3 properties file is now stored at /etc/idyl-e3/idyl.properties.
  • Added entity store feature.
  • Added Idyl Cloud integration for entity disambiguation and enrichment.
  • Added entity querying API.
  • Added additional entity extraction API parameters for document ID, refTag, and context.
  • Released Idyl E3 SDK 1.4.0 for java and .NET.

Version 1.3.1

  • Minor changes.
  • Released Idyl E3 SDK 1.3.1 for java and .NET.

Version 1.3.0

  • Redesigned dashboard.
  • Supports both person and place entities.
  • Released Idyl E3 SDK 1.3.0 for java and .NET.