Privacy Policy Changes

We want to make you aware of a recent change to our Privacy Policy. We added a paragraph to the “Non-personal identification information” section about product update checks. The new paragraph describes the information that is transmitted when our products perform an updated version check. Remember that update checks can always be enabled or disabled -- please check the product's documentation for instructions or contact us.


Idyl E3 2.0

Update: Idyl E3 2.0 is now available on the AWS Marketplace: https://aws.amazon.com/marketplace/pp/B01BSQUR2K

Today we are announcing Idyl E3 2.0. It has been over a year since version 1.0 was introduced and we'd like to thank our users for helping us to reach this milestone. The main goals of version 2.0 were to make Idyl E3 extensible and increase performance. We would like to thank our users for helping us get to this milestone release. We could not have done it without your feedback and comments.

Idyl E3 is available for download from our website. Look for Idyl E3 2.0 to be available on the AWS Marketplace and other channels shortly thereafter.

Three Editions

Idyl E3 2.0 will be available in three editions:

Idyl E3 Free Edition

This edition of Idyl E3 is free. It includes an English-persons entity model and no plugins. This edition can be customized with plugins and models to meet your requirements.

Idyl E3 Standard Edition

The Standard Edition includes everything in the free edition plus model evaluation tools and priority email technical support.

Idly E3 Analyst Edition

The Analyst Edition includes everything in the standard edition plus all plugins and supports unlimited custom models.

Plugins

In Idyl E3 1.x, things like email addresses and phone numbers were extracted through built-in functionality called extraction modules. In version 2.0 we are introducing plugins. There are two types of plugins - a plugin type that perform an entity extraction and a plugin type that publishes the extracted entities. Plugins can be downloaded from our website and installed in your Idyl E3. The following plugins are currently available or will soon be:

Text Consumption Plugins

  • Consume input text from Kafka topic
  • Consume input text from Kinesis stream

Entity Extraction Plugins

  • Phone numbers extraction plugin
  • Email addresses extraction plugin
  • Hashtags extraction plugin
  • User mentions extraction plugin

Document Processing Plugins

  • Parse text from PDF files

Entity Publisher Plugins

  • AWS Kinesis Firehose publisher plugin
  • EntityDB publisher plugin

Internal changes were made to improve Idyl E3's performance to lower the time to extract entities. One change was the removal of the web-based dashboard. Configuration is now done directly through the properties file.

Custom Sentence and Token Models

Also new in version 2.0 to increase performance is the ability to generate and use custom sentence and token models. In versions 1.x, internal models were used for sentence detection and sentence tokenizing. These models were not always representative of the input text so their performance was degraded. In version 2.0 you have the option to generate sentence and token models from your data or use the legacy internal models just as versions 1.x did. You can still create your own entity models.

UIMA

Idyl E3 2.0 supports integration with UIMA through the Idyl E3 UIMA connector.


EntityDB and AWS CloudWatch Metrics

We added the ability for EntityDB to report metrics to AWS CloudWatch. The metrics reported include the numbers of entities stored and indexed. The screen capture of an AWS CloudWatch graph is shown below. The system that generated the metrics illustrated by the chart was composed of 5 EntityDB t2.micro instances in auto-scaling group behind an elastic load balancer. An SQS queue was used for the entity queue and entities were persisted to a MongoDB database also running on a t2.micro instance. (This architecture was created using the CloudFormation templates in the GitHub repository.)

EntityDB CloudWatch metrics

As the metrics show, the entities are being stored at a rate much faster than the entities are being indexed. We will be working to make the index rate (orange line) more closely follow the stored rate (blue line).


Open Source Updates

In the past week we made the following updates to our open source projects.

Entity Model - Updated to include a new Span class on entity. The Span class identifies the location of the entity in the source text by token and by character indexes. This update was made to the entity-model and entity-model-net projects. Version 1.0.8 of entity-model was published to Maven Central and version 2.0.0 of entity-model-net was published to NuGet.

Idyl E3 UIMA Annotator - An update was made that annotates the entities based on the character index instead of the token index so the entities are properly annotated in UIMA. (The Idyl E3 UIMA Annotator requires Idyl E3 1.13.0 which is not quite ready but look for it soon.)

Idyl E3 Client SDKs - The Idyl E3 client SDK for Java was updated to use entity-model 1.0.8. The Idyl E3 client SDK for .NET was updated to use the new MountainFog.EntityModel 2.0.0 package from NuGet.

Anthology - Anthology was updated to include the ability to load balance Idyl E3 entity extraction requests. You can now specify multiple Idyl E3 endpoints per entity type when defining the routes.

EntityDB - EntityDB was updated to use entity-model 1.0.8.

 


Idyl E3 UIMA Annotator

We have published a new project to our GitHub. The new project is a UIMA annotator that uses Idyl E3 for named entity recognition. When added to a UIMA pipeline, the annotator will send the text that is the subject of analysis to Idyl E3. The project is licensed under the Apache Software License, version 2.0.

The Idyl E3 UIMA annotator requires Idyl E3 1.13.0 which will be available very soon.


Idyl Talk - New Open Source Project

We have pushed a new open source project to our GitHub called Idyl Talk. The goal of Idyl Talk is to replace traditional interface-defined software communication with natural language text.

When software communicates with other software, either internally or with external software, the communication is defined by interfaces. These interfaces tell each side how to communicate. Interfaces are an essential piece of good design. But what happens when two components have to communicate, and for whatever reasons, it is difficult (or impossible) to define the interface? Idyl Talk addresses this problem by letting software components communicate using natural language English text.

Imagine your refrigerator talking to your smartphone app to update your shopping list. The communication might look a bit like this:

{
    inventory: {
        "milk": "low",
        "eggs": 12
    }
}

Your smartphone receives the message and an app notifies you that you need milk. For this to be possible the developers of the refrigerator and the smartphone app have to agree on some interface that dictates the communication between the devices. This requires collaboration, and of course, time and money.

Now, imagine that when you are running low on milk your refrigerator sends the following message to your smartphone app:

You are low on milk.

The agreed-to interface here is the English language. With Idyl Talk can now create devices that are enabled to communicate even if they do not exist yet! The app processes the received message and alerts you that you are low on milk.

Sound interesting? We think so! We welcome your contributions to the project as it matures and grows. Check out Idyl Talk on GitHub.

See a listing of all our open source projects.


EntityDB is Open Source

EntityDB is now open source on GitHub. It is licensed under the AGPLv3. The goal of EntityDB is to provide an integration solution for storing, managing, and querying entities (persons, places, and things). Everyone is welcome to contribute to its development and future as we work toward a first release.

EntityDB provides a choice of underlying databases. MySQL, MongoDB, Cassandra, and DynamoDB are currently supported. The Entity Query Language (EQL) is also included in the open sourced code. EQL provides an abstraction layer for querying the entities regardless of the underlying database.

Proprietary licenses are available for situations where the AGPLv3 is not suitable. Please contact us for more information.


Idyl E3 1.12.0

Idyl E3Look for Idyl E3 1.12.0 to be available on various cloud marketplaces this week. Version 1.12.0 starts the separation from the entity stores we announced in our last post. It also contains some minor fixes and improvements. (See the Idyl E3 Release Notes.)

There will be multiple versions of Idyl E3 1.12.0 available. The versions will differ based on what entity models are included in the version. One version will not have any entity models making it ideal for scenarios when you want to use your own generated entity models. As a reminder, you can create entity models from your own data for use with Idyl E3. Using your own data to generate models will result in models that perform better than our models for your type of data.


Idyl E3's entity store and EntityDB

Along with the ability to extract entities from text, Idyl E3's entity store feature allows you to save the extracted entities to a database of your choice. Supported databases include a relational database like MySQL and the NoSQL databases MongoDB and DynamoDB. In addition to save the entities to a database you can also query the entities using a special language called Entity Query Language (EQL). EQL has a SQL-like syntax letting you select entities based on conditions in the query. Your EQL query is translated into a native query for your selected database. A single EQL query can be executed against MySQL, MongoDB, and DynamoDB.

The entity store feature of Idyl E3 is being separated from Idyl E3 into its own product called EntityDB. This separation will allow Idyl E3 to focus on entity extraction. Idyl E3 will integrate with EntityDB's public API to still provide entity storage services.

EntityDB will continue to support the same databases as well as a new database - Apache Cassandra. Cassandra is ideally suited for storing entities and will allow for large-scale querying and analysis. The Cassandra-based entity store will support EQL queries but you will also have the ability to query it using other tools like SparkSQL.

Look for the first version of EntityDB to be available in the near future. We have a large roadmap for EntityDB and plan to add features incrementally over a series of releases.


Mountain Fog, Inc. Listed in AWS Marketplace for the U.S. Intelligence Community

Mountain Fog, Inc. Listed in AWS Marketplace for the U.S. Intelligence Community

Idyl E3 Entity Extraction Engine now available to 17 US Intelligence Agencies in a Cloud Marketplace

June 20 – Morgantown, WV – Mountain Fog, Inc., a leading provider of natural language processing software for commercial and law enforcement users, today announced it is among the first group of technology vendors to be listed in Amazon Web Services (AWS) Marketplace for the U.S. Intelligence Community (IC). AWS Marketplace for the U.S. IC is designed exclusively for the 17 intelligence agencies to evaluate, purchase, and deploy in minutes via 1-Click® a broad array of common software infrastructure, developer tools, and business software products, with the categories of products and vendors growing over time.

Mountain Fog’s product, Idyl E3 Entity Extraction Engine, analyzes multilingual natural language text and identifies persons, places, and things within the text. Its integrated rules engine and entity persistence capabilities provide a complete solution for processing unstructured text. “Idyl E3 can help government agencies manage unstructured text and turn it into usable information. We are pleased to offer Idyl E3 on the AWS Marketplace for the U.S. IC in order to give more agencies immediate access to its capabilities,” said Mountain Fog president Jeff Zemerick.

AWS Marketplace for the U.S. IC provides the same purchasing convenience, open and transparent license terms and conditions, and variety of pricing models, including hourly usage and annual subscription, as the commercial AWS Marketplace. It also supports Bring-Your-Own-License (BYOL) so that agencies can more easily migrate existing software licenses and applications. AWS Marketplace for the U.S. IC is part of the Commercial Cloud Services (C2S) program, under the Director of National Intelligence (DNI) Intelligence Community (IC) Information Technology Enterprise (IC ITE). For more information on AWS Marketplace for the U.S. IC, contact icmp@amazon.com.

###

About Mountain Fog, Inc.

Mountain Fog was founded in 2011 to develop innovative language processing solutions. Our team of developers and engineers specialize in big-data analysis, natural language processing, and cloud systems. Our philosophy is simple – provide the best products we can and back them up with unparalleled customer support. (412) 206-1079 | sales@mtnfog.com

 

 

 


User-created entity models

Idyl E3 offers can extract many types of entities such as building, cities, and more. In instances where we do not offer the type of entity you need we will soon be offering a tool to create your own entity models. You will be able to create your own entity models from your own text giving you entity models customized for your own use-cases. This ability is expected to be available in an upcoming release so stay tuned for more details.


Idyl E3 on the VMware Solutions Marketplace

Idyl E3 is now available through the VMware Solutions Marketplace. Check it out at https://solutionexchange.vmware.com/store/products/idyl-e3!


Idyl E3 1.7.1 on the AWS Marketplace

Idyl E3 1.7.1 is now available on the AWS Marketplace. Launch it for free now!


Idyl E3 1.7.1 Debian Package Available

Ubuntu LinuxThe debian package for Idyl E3 1.7.1 is available for free download in your account. If you do not have an account you can create one (also free!).

See the Idyl E3 on Ubuntu Server page for more details.


Idyl E3 on Docker Hub

Idyl E3 is now available as a docker container through Docker Hub. Be sure to star the Idyl E3 repository on Docker Hub and to send us your feedback!  Learn more about the Idyl E3 docker containers.


Idyl E3 is Now Free


Idyl E3
is now available for free. And to make it even better, Idyl E3 will soon also be available for Windows Azure, VMware ESXi, and as a standalone download. Idyl E3 will continue to be available through the AWS Marketplace. Deploy Idyl E3 to the platform of your choice.

And to make it even better, the entity models used by Idyl E3 for entity extraction are now configurable and additional models are available for download. Idyl E3 comes with a base model for extracting person entities that is fully functional. Models capable of extracting more entities with higher confidence are available.

We will still provide support for Idyl E3. Priority support is available as well as development and integration support to help you get Idyl E3 integrated into your systems.

We are very excited about the expanded availability of Idyl E3. If you need help getting started with Idyl E3 or have any questions please contact us.


New Website

In case you haven't noticed we have rolled out our website update. The goal of this update was to improve usability. We felt that our previous website was at times hard to navigate with too much text. In the new website we are going for simplicity, especially on the product pages.

The new website also features a new and improved My Account. Soon you will be able to access your downloads and purchase history from your account. The new website also features improved single sign-on with our .

We have also migrated our blog to the new website. No longer do you have to leave our main site to checkout our newest blog posts.

So, please bear with us over the next few days as we iron out any issues and missed 404 errors.


Idyl E3 1.6.0 Available on the AWS Marketplace

Idyl E3 1.6.0 is now available on the AWS Marketplace. This release brings some good fixes and exciting new features.

Here's some of what's new in 1.6.0:

  • Access to the API can now require authentication.
  • Restarts are no longer required when changing settings.
  • New dashboard UI styling is cleaner and easier to navigate.
  • Entity filters are now customizable through the dashboard settings.
  • Added SQS visibility timeout setting.
  • Added SNS message subject setting.
  • Added CloudWatch metric name setting.
  • Passwords and AWS keys are encrypted in the settings.

And some things that were fixed or improved:

  • Added cURL upload example.
  • Fixed document upload when upload parameters are missing.
  • The entity store setting is loaded and shown correctly after saving.
  • Documentation updates.

This is by far the most stable and feature-rich version yet. But with that said, we have already started on version 1.7.0 to offer even more. If you have any feedback or feature requests please let us know!

With the release of Idyl E3 1.6.0 we have also updated the client SDKs. You can find them on GitHub or through Maven Central and NuGet.