Idyl E3 Release Notes

This page provides a changelog of Idyl E3 versions. New versions are typically available in cloud marketplaces within a week or two following the release date.

Version 2.6.0 (Not yet released)

  • New: GPU capable edition of Idyl E3 that uses GPUs to create and use entity models.
  • New: Models that fail to load are now blacklisted and reload attempts will not happen until Idyl E3 is restarted.
  • New: Added scripts to perform model validation to avoid having to run the jar files directly.
  • New: $IDYL_E3_HOME environment variable must be set. On AWS Marketplace images this is automatically set to /opt/idyl-e3.
  • New: HMAC API authentication now includes the hash of the text being submitted.  (See the documentation.)
  • New: Entity extraction requests can now cover multiple entity types by setting the type parameter to a comma-separated list of types, e.g. person,place.
  • New: Deep learning models now publish metrics to the console and/or to Graphite.
  • Fix: Fixed timestamp checking when using HMAC API authentication.
  • Fix: Text in the Token Model Validator incorrectly said Sentence Model Validator.
  • Fix: Model validation tools omitted the number of folds in the logged message.
  • Fix: Stats reporter could cause an error when updating confidence values.
  • Tweak: Generated data files are stored in $IDYL_E3_HOME/data.
  • Tweak: Improved thread-safety to ensure accurate responses.
  • Tweak: Removed deprecated /ingest API endpoint.
  • Tweak: Tightened permissions on Idyl E3 files on AWS Marketplace images.

Version 2.5.0 (August 28, 2017)

  • New: English-language person entity models can now be trained using data in the CoNLL-2003 format. (Note this only applies to English-language person entity models.)
  • New: Added support for training and using deep learning neural network entity models. (See blog post.)
  • New: Changed to 3-letter ISO 3166-1 alpha-3 language codes. 2-letter codes are still supported by use of 3-letter codes are recommended.
  • New: Added support for the use of dictionary entity models.
  • New: Added support for custom annotation types when training entity models.
  • New: Added a setting to control how duplicate entities (from a single extraction request) are handled.
  • New: A TCP endpoint accepts streaming text. This endpoint deprecates the /ingest API endpoint.
  • Fix: Entities extracted without a probability are now correctly assigned a probability of 1.0 instead of 100.0.
  • Fix: Lemma and parts-of-speech model manifests could be incorrectly generated as token model manifests.
  • Tweak: Restructured the idyl-e3 application directory into bin/ and lib/ directories.
  • Tweak: Added shell scripts for performing model training.
  • Tweak: Changed the ASCII banner displayed when starting Idyl E3.
  • Tweak: Removed auto as a valid parameter for language. Language detection is no longer supported.

Version 2.4.0 (May 2, 2017)

  • New: Adds support for the Idyl Annotation Format (IAF) that lets the annotations for entity model training text be stored external to the training text. (See blog post.)
  • New: Allows the user to configure how duplicate entities are handled. (See blog post.)
  • Tweak: Changed the console starting banner to “Idyl E3.”
  • Tweak: Removed the deprecated API v1 (/api/v1).

Version 2.3.0 (March 9, 2017)

  • Important: Version 1 of the API has been deprecated and will be removed in a future release. Please migrate to the version 2 API as soon as possible.
  • New: HMACSHA512 API authorization option. (See the API documentation.)
  • New: Parts-of-speech-based feature generator for custom models. Model training and evaluation can now consider each token’s part-of-speech.
  • New: Tools to create parts-of-speech and lemmatization models.
  • New: Added special character feature generator. (See blog post.)
  • New: Added word normalization feature generator. (See blog post.)
  • New: Added token part of speech feature generator. (See blog post.)
  • New: Added model version verification to make sure each model is compatible with the version of Idyl E3.
  • New: Added sort parameter to V2 /extract endpoint to control how extracted entities are sorted before returned. Entities can be sorted by confidence, entity text, and by occurrence in the input text. (See the API documentation.)
  • New: Added new plugin for using Google Cloud Natural Language API to perform supplementary entity extraction. (See blog post.)
  • New: Added new API endpoint to sanitize input text. The identified entities are “blacked out” from the input text. (See the API documentation.)
  • New: There is now an Idyl E3 SDK for Go.
  • New: The API endpoints now respond to HEAD requests with HTTP 405 Method Not Allowed. This change is to support smoother integration with Apache NiFi’s PostHTTP processor. (See blog post.)
  • Fix: Fixed problem where entity model validation via separate data can return an incorrect F-measure of -1.0.
  • Tweak: Added Idyl E3 version number to the README.txt file.
  • Other minor changes and fixes.
  • User Documentation

Version 2.2.0 (February 1, 2017)

  • New: Added heuristic confidence filtering. (See blog post.)
  • New: Added support for any UTF-8 language.
  • New: Added support to the V2 API for providing metadata for extracted entities.
  • New: Statistics reporting to Graphite and the console tracks entity counts and requests. (See blog post for scripts to create a Graphite AMI.)
  • New: A separate timing metric for annotations is now available to be reported.
  • New: Models can now have names that are used to identify the model. A model’s name is assigned by the user when the model is generated.
  • New: Extract, ingest, and annotate endpoints now accept an optional type parameter that restricts the entity extraction to a specific entity type.
  • Tweak: Removed the confidence and documentId parameters from the /annotate endpoint as they are not used.
  • Tweak: An empty encryption key is now invalid. A key must be at least one character.
  • Tweak: If API authentication is enabled but no API key is specified a warning will be logged in Idyl E3’s log file.
  • Fix: The confidence threshold defaults to 0 when not provided during an API /upload request.
  • Fix: API V2 /extract now correctly returns 400 instead of 500 when an entity extraction request does not contain any text.
  • Fix: Specifying “all” for the language to an entity extraction request could fail to load the appropriate tokenizer or sentence detector for each available entity model.
  • Other minor changes and fixes.
  • User Documentation

Version 2.1.0 (Released January 6, 2017)

  • New: Added V2 REST API endpoints for /extract and /ingest accept input text in the request body. (The Idyl E3 client SDKs on GitHub have been updated to use the V2 APIs.)
  • New: Added LEXS document processing plugin.
  • New: Built on OpenNLP 1.7.0.
  • Tweak: A model encryption key is no longer optional when creating a custom model.
  • Tweak: All API endpoints return UTF-8.
  • Fix: Fixed an error that can happen when an entity extraction request’s language contains no entity models.
  • Other minor changes and fixes.
  • User Documentation

Version 2.0 (Released December 20, 2016)

This is a major revision that focused primarily on performance improvements.

  • New: Can now now use custom sentence and token models.
  • New: Plugin system allows for extending Idyl E3’s functionality to be able to consume text from other sources, parse binary files, perform pattern-based extraction, and publish entities to external services.
  • New: Simplified configuration properties.
  • New: Includes tools to create entity, sentence, and token models.
  • New: Supports integration with UIMA through the Idyl E3 UIMA connector.
  • New: Removed web dashboard.
  • Tweak: API endpoints for health, extraction time, and status were combined into a single /api/status endpoint.
  • Other minor changes and fixes.
  • User Documentation

Version 1.12.0 (Released August 17, 2016)

  • New: Added optional integration with EntityDB. EntityDB replaces Idyl E3’s entity store functionality with a new product devoted solely to storing and querying entities.
  • Tweak: Minor improvements and fixes.

Version 1.11.0 (Released on June 24, 2016)

  • Fix: Entity persistence when using an RDBMS-backed entity store. (Noted as a problem in the 1.10.0 Release Notes below.)
  • Fix: The dashboard’s Entity Models page now accurately displays the entity models and each’s status.
  • New: Added support for multilingual documents. (Specify “all” as language parameter to the API. See the API documentation for more information.)
  • New: Added support for extracting dates from English text.
  • New: Added support for using a date filter in an EQL-based rule.
  • New: Added support for using entity models for the same entity type concurrently. (For example, you can now use an English person entity model and a German person entity model at the same time.)
  • New: Added ability to select which tokenizer (whitespace or character-based) to use during extraction.
  • New: Image files ingested through the file system will be processed by OCR.
  • New: Text from binary files (DOCX, PDF) is extracted when ingesting from the file system.
  • Tweak: Performance improvements for extracting email addresses, Twitter usernames, and hashtags.
  • Tweak: No longer required to set environment variables prior to running Idyl E3.

Version 1.10.0

  • Tweak: Regions for AWS services are now provided by endpoint URL.
  • Known Problem: There is a known issue concerning the persistence of entities to a RDBMS database-backed entity store. This problem will be fixed in version 1.11.0.

Version 1.9.0

  • New: Can now use an EQL statement as a condition in a rule.
  • Tweak: All statistical time units are now in milliseconds.
  • Tweak: Removed the “skip integrations” option from the dashboard extraction form.

Version 1.8.0

  • New: Added rules engine to process extracted entities.
  • New: Added ability to consume entity extraction requests from an Amazon Web Services Simple Queue Service (SQS) queue.

Version 1.7.1

  • Fix: Fixed model loading issue.

Version 1.7.0

  • Minor UI and performance improvements.
  • Entity models are now configurable.

Version 1.6.0

  • Minor UI and performance improvements.
  • Restarts are no longer required after changing settings.
  • Now supports optional API authentication.
  • Entity Filters can be enabled via the UI.
  • Added AWS SQS visibility timeout setting.
  • Added AWS SNS message subject setting.
  • Added AWS CloudWatch metric name setting.
  • Updated cURL example API calls.
  • Fixed API upload failing if parameters not provided.
  • Settings are stored encrypted.

Version 1.5.3

  • Adding support for MongoDB as an entity store.
  • Adding EQL as a method for querying the entity store.
  • API changes updated in the Idyl E3 client SDKs.
  • Added /api/eql API endpoint.
  • Added /api/query API endpoint.
  • Added support for extracting Twitter usernames.
  • Added support for extracting email addresses.
  • Added support for extracting phone numbers.
  • Fixed issue handling uploaded documents.

Version 1.5.2

  • Simplifying statistics.
  • Performance improvements.
  • Adding /api/ingest API endpoint.
  • Removed text limit on API POST requests.

Version 1.5.1

  • Fix problem where average extraction time could be incorrect.
  • Fix problem where extracted text could be returned empty.

Version 1.5.0

  • Added support for DynamoDB as an entity store.
  • Added ability to query the entity store by entity enrichments.
  • Added ability for Idyl E3 to restart itself when necessary after settings are changed.
  • Added documentId to the query API.
  • Added support for integrating with Idyl Cloud using a Mashape key.

Version 1.4.0

  • The Idyl E3 log is now located at /var/log/idyl-e3/idyl-e3.log.
  • The service name is now idyl-e3.
  • The Idyl E3 properties file is now stored at /etc/idyl-e3/
  • Added entity store feature.
  • Added Idyl Cloud integration for entity disambiguation and enrichment.
  • Added entity querying API.
  • Added additional entity extraction API parameters for document ID, refTag, and context.
  • Released Idyl E3 SDK 1.4.0 for java and .NET.

Version 1.3.1

  • Minor changes.
  • Released Idyl E3 SDK 1.3.1 for java and .NET.

Version 1.3.0

  • Redesigned dashboard.
  • Supports both person and place entities.
  • Released Idyl E3 SDK 1.3.0 for java and .NET.