Idyl E3 and OpenNLP

As you may know, Idyl E3's entity extraction capabilities is provided by a customized version of OpenNLP. Since the release of OpenNLP 1.7.0, the OpenNLP team has been able to release more often than previously. Because of the more frequent OpenNLP releases we may not incorporate each release into Idyl E3. We will analyze the changes in each new OpenNLP version to decide if the changes should be incorporated into Idyl E3.

Also, we do have on the (distant)  roadmap the ability to make the underlying NLP engine pluggable to allow you to choose which NLP engine to use with Idyl E3.


Idyl E3 2.1.0 Analyst Edition Now Available

Idyl E3 Analyst Edition 2.1.0 is now available on the AWS Marketplace. Idyl E3 Analyst Edition includes everything in the Free and Standard editions plus licenses for all plugins and licenses for unlimited custom models.


Heuristic Confidence Filtering

In Idyl E3 2.2.0 we are introducing a feature we call Heuristic Confidence Filtering. Here's how it works.

As you may (or may not) already know, each entity extraction request can have an associated "confidence threshold value." Any entities that are extracted who have a confidence lower than this value will not be returned in the entity extraction response. This is useful but it is a bit of a sledgehammer approach and can either result in too much noise or missed entities depending on its value.

When enabled, heuristic confidence filtering tracks the confidence values of extracted entities per the entity model that extracted them. Once a large enough sample of confidence values has been collected, Idyl E3 will filter entities by determining if an entity's confidence value is significant to the mean of the collected values. This provides a way to filter out noise but still receive important entities.

It is important to note that the confidence threshold value still plays a part even when heuristic confidence filtering is enabled. Any entity whose confidence value is greater than or equal to the confidence threshold for that request will always be returned even when heuristic confidence filtering is enabled.

Because of the mathematical calculations involved and the memory required to store the confidence values the heuristic confidence filtering does require a bit more computation time but not to the point where it should be noticeable.

We are excited to offer this feature and we hope that it helps with "entity noise." We welcome your feedback on how it performs for you! For more information on this feature you can refer to the Idyl E3 2.2.0 User Documentation or by contacting us. Look for Idyl E3 2.2.0 to be available in February 2017.


Idyl E3 2.1.0

Idyl E3 2.1.0 has been released. This version introduces a new version of the API that includes changes to the extract and ingest endpoints. With version 2 of the API these two endpoints accept text in the body of the request instead of as a query string parameter. Version 1 of the API is still available so you do not need to update your clients unless you just want to or need to for other reasons. The Idyl E3 Java SDK and the Idyl E3 .NET SDK have been updated to use API v2.

Idyl E3 2.1.0 is based on a customized OpenNLP 1.7.0 which was released in early January 2016.  Previous versions of Idyl E3 were based on a customized OpenNLP 1.6.0.

Idyl E3 2.1.0 Analyst Edition will be available on the AWS Marketplace soon. The Analyst Edition includes all plugins and allows for the use of unlimited custom models without separate licensing. (See the Idyl E3 edition comparison.)


Privacy Policy Changes

We want to make you aware of a recent change to our Privacy Policy. We added a paragraph to the “Non-personal identification information” section about product update checks. The new paragraph describes the information that is transmitted when our products perform an updated version check. Remember that update checks can always be enabled or disabled -- please check the product's documentation for instructions or contact us.


Idyl E3 2.0

Update: Idyl E3 2.0 is now available on the AWS Marketplace: https://aws.amazon.com/marketplace/pp/B01BSQUR2K

Today we are announcing Idyl E3 2.0. It has been over a year since version 1.0 was introduced and we'd like to thank our users for helping us to reach this milestone. The main goals of version 2.0 were to make Idyl E3 extensible and increase performance. We would like to thank our users for helping us get to this milestone release. We could not have done it without your feedback and comments.

Idyl E3 is available for download from our website. Look for Idyl E3 2.0 to be available on the AWS Marketplace and other channels shortly thereafter.

Three Editions

Idyl E3 2.0 will be available in three editions:

Idyl E3 Free Edition

This edition of Idyl E3 is free. It includes an English-persons entity model and no plugins. This edition can be customized with plugins and models to meet your requirements.

Idyl E3 Standard Edition

The Standard Edition includes everything in the free edition plus model evaluation tools and priority email technical support.

Idly E3 Analyst Edition

The Analyst Edition includes everything in the standard edition plus all plugins and supports unlimited custom models.

Plugins

In Idyl E3 1.x, things like email addresses and phone numbers were extracted through built-in functionality called extraction modules. In version 2.0 we are introducing plugins. There are two types of plugins - a plugin type that perform an entity extraction and a plugin type that publishes the extracted entities. Plugins can be downloaded from our website and installed in your Idyl E3. The following plugins are currently available or will soon be:

Text Consumption Plugins

  • Consume input text from Kafka topic
  • Consume input text from Kinesis stream

Entity Extraction Plugins

  • Phone numbers extraction plugin
  • Email addresses extraction plugin
  • Hashtags extraction plugin
  • User mentions extraction plugin

Document Processing Plugins

  • Parse text from PDF files

Entity Publisher Plugins

  • AWS Kinesis Firehose publisher plugin
  • EntityDB publisher plugin

Internal changes were made to improve Idyl E3's performance to lower the time to extract entities. One change was the removal of the web-based dashboard. Configuration is now done directly through the properties file.

Custom Sentence and Token Models

Also new in version 2.0 to increase performance is the ability to generate and use custom sentence and token models. In versions 1.x, internal models were used for sentence detection and sentence tokenizing. These models were not always representative of the input text so their performance was degraded. In version 2.0 you have the option to generate sentence and token models from your data or use the legacy internal models just as versions 1.x did. You can still create your own entity models.

UIMA

Idyl E3 2.0 supports integration with UIMA through the Idyl E3 UIMA connector.


EntityDB and AWS CloudWatch Metrics

We added the ability for EntityDB to report metrics to AWS CloudWatch. The metrics reported include the numbers of entities stored and indexed. The screen capture of an AWS CloudWatch graph is shown below. The system that generated the metrics illustrated by the chart was composed of 5 EntityDB t2.micro instances in auto-scaling group behind an elastic load balancer. An SQS queue was used for the entity queue and entities were persisted to a MongoDB database also running on a t2.micro instance. (This architecture was created using the CloudFormation templates in the GitHub repository.)

EntityDB CloudWatch metrics

As the metrics show, the entities are being stored at a rate much faster than the entities are being indexed. We will be working to make the index rate (orange line) more closely follow the stored rate (blue line).


Open Source Updates

In the past week we made the following updates to our open source projects.

Entity Model - Updated to include a new Span class on entity. The Span class identifies the location of the entity in the source text by token and by character indexes. This update was made to the entity-model and entity-model-net projects. Version 1.0.8 of entity-model was published to Maven Central and version 2.0.0 of entity-model-net was published to NuGet.

Idyl E3 UIMA Annotator - An update was made that annotates the entities based on the character index instead of the token index so the entities are properly annotated in UIMA. (The Idyl E3 UIMA Annotator requires Idyl E3 1.13.0 which is not quite ready but look for it soon.)

Idyl E3 Client SDKs - The Idyl E3 client SDK for Java was updated to use entity-model 1.0.8. The Idyl E3 client SDK for .NET was updated to use the new MountainFog.EntityModel 2.0.0 package from NuGet.

Anthology - Anthology was updated to include the ability to load balance Idyl E3 entity extraction requests. You can now specify multiple Idyl E3 endpoints per entity type when defining the routes.

EntityDB - EntityDB was updated to use entity-model 1.0.8.

 


Idyl E3 UIMA Annotator

We have published a new project to our GitHub. The new project is a UIMA annotator that uses Idyl E3 for named entity recognition. When added to a UIMA pipeline, the annotator will send the text that is the subject of analysis to Idyl E3. The project is licensed under the Apache Software License, version 2.0.

The Idyl E3 UIMA annotator requires Idyl E3 1.13.0 which will be available very soon.


Idyl Talk - New Open Source Project

We have pushed a new open source project to our GitHub called Idyl Talk. The goal of Idyl Talk is to replace traditional interface-defined software communication with natural language text.

When software communicates with other software, either internally or with external software, the communication is defined by interfaces. These interfaces tell each side how to communicate. Interfaces are an essential piece of good design. But what happens when two components have to communicate, and for whatever reasons, it is difficult (or impossible) to define the interface? Idyl Talk addresses this problem by letting software components communicate using natural language English text.

Imagine your refrigerator talking to your smartphone app to update your shopping list. The communication might look a bit like this:

{
    inventory: {
        "milk": "low",
        "eggs": 12
    }
}

Your smartphone receives the message and an app notifies you that you need milk. For this to be possible the developers of the refrigerator and the smartphone app have to agree on some interface that dictates the communication between the devices. This requires collaboration, and of course, time and money.

Now, imagine that when you are running low on milk your refrigerator sends the following message to your smartphone app:

You are low on milk.

The agreed-to interface here is the English language. With Idyl Talk can now create devices that are enabled to communicate even if they do not exist yet! The app processes the received message and alerts you that you are low on milk.

Sound interesting? We think so! We welcome your contributions to the project as it matures and grows. Check out Idyl Talk on GitHub.

See a listing of all our open source projects.


AWS CloudFormation Supports YAML

In an exciting update from AWS, it was announced that CloudFormation now supports YAML in addition to JSON. I think most of us will agree this is great. The JSON templates worked, but whew, were they hard to read and the lack of the ability to add comments sometimes made my templates look more like sudokus or word searches than anything else.

They also announced the support for cross-stack references. That means no more duplicating resources between templates! There's a small gotcha with cross-stack references in that the names of the exported values have to be unique in your account and have to be literal string values.

These new features are significant enough that I felt they deserved a mention on this blog. They will definitely have an immediate impact on how we create CloudFormation for ourselves and our clients.


New Open Source Project - Anthology

We have published a new open source project called Anthology to our GitHub. Anthology makes working with multiple instances of Idyl E3 easier. Anthology can route entity extraction requests based on the type of entities to extract. With Anthology you can have a single Idyl E3 endpoint for multiple Idyl E3 clusters.

Anthology is licensed under the Apache Software License, version 2. All contributions are welcome!


EntityDB is Open Source

EntityDB is now open source on GitHub. It is licensed under the AGPLv3. The goal of EntityDB is to provide an integration solution for storing, managing, and querying entities (persons, places, and things). Everyone is welcome to contribute to its development and future as we work toward a first release.

EntityDB provides a choice of underlying databases. MySQL, MongoDB, Cassandra, and DynamoDB are currently supported. The Entity Query Language (EQL) is also included in the open sourced code. EQL provides an abstraction layer for querying the entities regardless of the underlying database.

Proprietary licenses are available for situations where the AGPLv3 is not suitable. Please contact us for more information.


OpenNLP's RegexNameFinder and Tokenizing

OpenNLP's RegexNameFinder takes one or more regular expressions and uses those expressions to extract entities from the input text. This is very useful for instances in which you want to extract things that follow a set format, like phone numbers and email addresses. However, when tokenizing the input to the RegexNameFinder be careful because it can affect the RegexNameFinder's accuracy.

The RegexNameFinder is very simple to use and here's an example borrowed from an OpenNLP testcase.

Pattern testPattern = Pattern.compile("test");
String sentence[] = new String[]{"a", "test", "b", "c"};

Pattern[] patterns = new Pattern[]{testPattern};
Map<String, Pattern[]> regexMap = new HashMap<>();
String type = "testtype";

regexMap.put(type, patterns);

RegexNameFinder finder =
new RegexNameFinder(regexMap);

Span[] result = finder.find(sentence);

The sentence variable is a list of tokens. In the above example the tokens are set manually. In a more likely scenario the string would be received as "a test b c" and it would be up to the application to tokenize the string into {"a", "test", "b", "c"}.

There are three types of tokenizers available in OpenNLP - the WhitespaceTokenizer, the SimpleTokenizer, and a tokenizer (TokenizerME) that uses a token model you have trained. The WhitespaceTokenizer works on, you guessed it, white space. The locations of white space in the string is used to tokenize the string. The SimpleTokenizer looks at character classes, such as letters and numbers.

Let's take the example string "My email address is me@me.com and I like Gmail." Using the WhitespaceTokenizer the tokens are {"My", "email", "address", "is", "me@me.com", "and", "I", "like", "Gmail."}. If we use the RegexNameFinder with a regular expression that matches an email address, OpenNLP will return to us the span covering "me@me.com". Works great!

However, let's consider the sentence "My email address is me@me.com." Using the WhitespaceTokenizer again the tokens are {"My", "email", "address", "is", "me@me.com."}. Notice the last token includes the sentence's period. Our regular expression for an email address will not match "me@me.com." because it is not a valid email address. Using the SimpleTokenizer doesn't give any better results.

How to work around this is up to you. You could make a custom tokenizer by implementing the Tokenizer interface, try using a token model, or massaging your text before it is passed to the tokenizer.


Idyl E3 1.12.0

Idyl E3Look for Idyl E3 1.12.0 to be available on various cloud marketplaces this week. Version 1.12.0 starts the separation from the entity stores we announced in our last post. It also contains some minor fixes and improvements. (See the Idyl E3 Release Notes.)

There will be multiple versions of Idyl E3 1.12.0 available. The versions will differ based on what entity models are included in the version. One version will not have any entity models making it ideal for scenarios when you want to use your own generated entity models. As a reminder, you can create entity models from your own data for use with Idyl E3. Using your own data to generate models will result in models that perform better than our models for your type of data.


Idyl E3's entity store and EntityDB

Along with the ability to extract entities from text, Idyl E3's entity store feature allows you to save the extracted entities to a database of your choice. Supported databases include a relational database like MySQL and the NoSQL databases MongoDB and DynamoDB. In addition to save the entities to a database you can also query the entities using a special language called Entity Query Language (EQL). EQL has a SQL-like syntax letting you select entities based on conditions in the query. Your EQL query is translated into a native query for your selected database. A single EQL query can be executed against MySQL, MongoDB, and DynamoDB.

The entity store feature of Idyl E3 is being separated from Idyl E3 into its own product called EntityDB. This separation will allow Idyl E3 to focus on entity extraction. Idyl E3 will integrate with EntityDB's public API to still provide entity storage services.

EntityDB will continue to support the same databases as well as a new database - Apache Cassandra. Cassandra is ideally suited for storing entities and will allow for large-scale querying and analysis. The Cassandra-based entity store will support EQL queries but you will also have the ability to query it using other tools like SparkSQL.

Look for the first version of EntityDB to be available in the near future. We have a large roadmap for EntityDB and plan to add features incrementally over a series of releases.


Mountain Fog, Inc. Listed in AWS Marketplace for the U.S. Intelligence Community

Mountain Fog, Inc. Listed in AWS Marketplace for the U.S. Intelligence Community

Idyl E3 Entity Extraction Engine now available to 17 US Intelligence Agencies in a Cloud Marketplace

June 20 – Morgantown, WV – Mountain Fog, Inc., a leading provider of natural language processing software for commercial and law enforcement users, today announced it is among the first group of technology vendors to be listed in Amazon Web Services (AWS) Marketplace for the U.S. Intelligence Community (IC). AWS Marketplace for the U.S. IC is designed exclusively for the 17 intelligence agencies to evaluate, purchase, and deploy in minutes via 1-Click® a broad array of common software infrastructure, developer tools, and business software products, with the categories of products and vendors growing over time.

Mountain Fog’s product, Idyl E3 Entity Extraction Engine, analyzes multilingual natural language text and identifies persons, places, and things within the text. Its integrated rules engine and entity persistence capabilities provide a complete solution for processing unstructured text. “Idyl E3 can help government agencies manage unstructured text and turn it into usable information. We are pleased to offer Idyl E3 on the AWS Marketplace for the U.S. IC in order to give more agencies immediate access to its capabilities,” said Mountain Fog president Jeff Zemerick.

AWS Marketplace for the U.S. IC provides the same purchasing convenience, open and transparent license terms and conditions, and variety of pricing models, including hourly usage and annual subscription, as the commercial AWS Marketplace. It also supports Bring-Your-Own-License (BYOL) so that agencies can more easily migrate existing software licenses and applications. AWS Marketplace for the U.S. IC is part of the Commercial Cloud Services (C2S) program, under the Director of National Intelligence (DNI) Intelligence Community (IC) Information Technology Enterprise (IC ITE). For more information on AWS Marketplace for the U.S. IC, contact icmp@amazon.com.

###

About Mountain Fog, Inc.

Mountain Fog was founded in 2011 to develop innovative language processing solutions. Our team of developers and engineers specialize in big-data analysis, natural language processing, and cloud systems. Our philosophy is simple – provide the best products we can and back them up with unparalleled customer support. (412) 206-1079 | sales@mtnfog.com

 

 

 


User-created entity models

Idyl E3 offers can extract many types of entities such as building, cities, and more. In instances where we do not offer the type of entity you need we will soon be offering a tool to create your own entity models. You will be able to create your own entity models from your own text giving you entity models customized for your own use-cases. This ability is expected to be available in an upcoming release so stay tuned for more details.


Idyl E3 on the VMware Solutions Marketplace

Idyl E3 is now available through the VMware Solutions Marketplace. Check it out at https://solutionexchange.vmware.com/store/products/idyl-e3!


Idyl E3 1.7.1 on the AWS Marketplace

Idyl E3 1.7.1 is now available on the AWS Marketplace. Launch it for free now!


Idyl E3 1.7.1 Debian Package Available

Ubuntu LinuxThe debian package for Idyl E3 1.7.1 is available for free download in your account. If you do not have an account you can create one (also free!).

See the Idyl E3 on Ubuntu Server page for more details.


Idyl E3 on Docker Hub

Idyl E3 is now available as a docker container through Docker Hub. Be sure to star the Idyl E3 repository on Docker Hub and to send us your feedback!  Learn more about the Idyl E3 docker containers.


Idyl E3 is Now Free


Idyl E3
is now available for free. And to make it even better, Idyl E3 will soon also be available for Windows Azure, VMware ESXi, and as a standalone download. Idyl E3 will continue to be available through the AWS Marketplace. Deploy Idyl E3 to the platform of your choice.

And to make it even better, the entity models used by Idyl E3 for entity extraction are now configurable and additional models are available for download. Idyl E3 comes with a base model for extracting person entities that is fully functional. Models capable of extracting more entities with higher confidence are available.

We will still provide support for Idyl E3. Priority support is available as well as development and integration support to help you get Idyl E3 integrated into your systems.

We are very excited about the expanded availability of Idyl E3. If you need help getting started with Idyl E3 or have any questions please contact us.


New Website

In case you haven't noticed we have rolled out our website update. The goal of this update was to improve usability. We felt that our previous website was at times hard to navigate with too much text. In the new website we are going for simplicity, especially on the product pages.

The new website also features a new and improved My Account. Soon you will be able to access your downloads and purchase history from your account. The new website also features improved single sign-on with our .

We have also migrated our blog to the new website. No longer do you have to leave our main site to checkout our newest blog posts.

So, please bear with us over the next few days as we iron out any issues and missed 404 errors.


Idyl E3 1.6.0 Available on the AWS Marketplace

Idyl E3 1.6.0 is now available on the AWS Marketplace. This release brings some good fixes and exciting new features.

Here's some of what's new in 1.6.0:

  • Access to the API can now require authentication.
  • Restarts are no longer required when changing settings.
  • New dashboard UI styling is cleaner and easier to navigate.
  • Entity filters are now customizable through the dashboard settings.
  • Added SQS visibility timeout setting.
  • Added SNS message subject setting.
  • Added CloudWatch metric name setting.
  • Passwords and AWS keys are encrypted in the settings.

And some things that were fixed or improved:

  • Added cURL upload example.
  • Fixed document upload when upload parameters are missing.
  • The entity store setting is loaded and shown correctly after saving.
  • Documentation updates.

This is by far the most stable and feature-rich version yet. But with that said, we have already started on version 1.7.0 to offer even more. If you have any feedback or feature requests please let us know!

With the release of Idyl E3 1.6.0 we have also updated the client SDKs. You can find them on GitHub or through Maven Central and NuGet.


Idyl E3 1.6.0 Available on the AWS Marketplace

Idyl E3 1.6.0 is now available on the AWS Marketplace. This release brings some good fixes and exciting new features.

Here's some of what's new in 1.6.0:

  • Access to the API can now require authentication.
  • Restarts are no longer required when changing settings.
  • New dashboard UI styling is cleaner and easier to navigate.
  • Entity filters are now customizable through the dashboard settings.
  • Added SQS visibility timeout setting.
  • Added SNS message subject setting.
  • Added CloudWatch metric name setting.
  • Passwords and AWS keys are encrypted in the settings.

And some things that were fixed or improved:

  • Added cURL upload example.
  • Fixed document upload when upload parameters are missing.
  • The entity store setting is loaded and shown correctly after saving.
  • Documentation updates.

This is by far the most stable and feature-rich version yet. But with that said, we have already started on version 1.7.0 to offer even more. If you have any feedback or feature requests please let us know!

With the release of Idyl E3 1.6.0 we have also updated the client SDKs. You can find them on GitHub or through Maven Central and NuGet.


Idyl E3 1.5.3 now available

Idyl E3 1.5.3 is now available on the AWS Marketplace. Version 1.5.3 adds support for extracting the following entity types in addition to person and place entities:

  • Email addresses
  • Twitter usernames
  • Hashtags
  • US and international phone numbers

Version 1.5.3 also adds MongoDB as a supported entity store and adds an API endpoint for querying entities through the Entity Query Language (EQL).

See the Idyl E3 Release Notes page for the full history. We are very excited about this release! If you have any questions or comments please get in touch at idyl@mtnfog.com.


Idyl Cloud and Entity Extraction

A very large number of our users use Idyl E3 for entity extraction since it can be used in a local network instead of Idyl Cloud. (There are a few reasons for this but a couple big ones that we hear often are because of the sensitive nature of the users' text and for performance.) Because of this we are removing entity extraction from Idyl Cloud so we can fully devote to its development in Idyl E3. One feature on the Idyl E3 roadmap is to allow for custom entity models and this is not a feature that's readily accommodated by Idyl Cloud.

The Idyl Cloud SDKs will be updated to reflect this change.


Idyl E3 1.4 Now Available

Idyl E3 1.4 is now available on the AWS Marketplace. You can see the full Release Notes but here's a summary of what's new in 1.4.

If you have any questions please get in touch. Helpdesk tickets can now be created directly from our website and you can always reach us directly for more production information or general questions.

Idyl Cloud Integration

Idyl E3 1.4 is integrated with Idyl Cloud for entity disambiguation and enrichment. If enabled, all entities extracted by Idyl E3 will be sent to Idyl Cloud for disambiguation and enrichment. To enable this feature provide your Idyl Cloud API key in Idyl E3's settings. Requests made to Idyl Cloud via Idyl E3 will be billed at the rate defined by your Idyl Cloud subscription plan.

Entity Store

New in 1.4 is the Entity Store feature. The Entity Store is a database that stores extracted entities and enrichments. When an entity extraction request is received, Idyl E3 extracts the entities and then persists the entities to the Entity Store. The Entity Store can be any JDBC database, such as MySQL, Oracle, or SQL Server.

API Changes

There is a new query API for performing queries against the entity store. With the query API you can find entities by text, context, and confidence. Since the Entity Store is an RDBMS you can always write more complex queries against it directly.

Two additional optional parameters have been added for the extraction API. The documentId parameter lets you categorize your text by documents. (So now documents can be categorized by context and by document ID.) The value of documentId can be any value that identifies your text.

The second new parameter is refTag. This parameter, also optional, lets you associate a value with the extraction request. This value can be anything and is only for your reference.

SDKs

The Java SDK for Idyl E3 has been updated on GitHub to support Idyl E3 1.4. We will be updating the .NET SDK for Idyl E3 shortly.

Upgrading to 1.4

When running on AWS you can upgrade to 1.4 by replacing any existing Idyl E3 instances with instances running 1.4. Any existing clients for 1.3 will work for 1.4 but will not have the entity querying capabilities.

What's coming?

We have some more exciting features lined up. Coming soon will be the ability to use DynamoDB as an Entity Store and improved settings management.


Idyl Cloud SDKs

Idyl Cloud 1.1.0 SDKs for Java and .NET are now available. The .NET SDK is available through NuGet and the Java SDK is available through Maven Central using the dependency:

<dependency>
<groupId>com.mtnfog.idyl.cloud.sdk</groupId>
<artifactId>idyl-cloud-java-sdk</artifactId>
<version>1.1.0</version>
</dependency>

These versions add support for entity disambiguation and enrichment. Both SDKs support consuming Idyl Cloud through Mashape.

Both SDKs are available on GitHub and are licensed under the Apache 2.0 license.


Idyl E3 1.3.1 Now Available

Idyl 1.3.1 is now available on the AWS Marketplace. This version brings minor changes and comes with a free 30 day trial.


Almost half of WV geotagged tweets are sent from Morgantown and Huntington

Mountain Fog is a West Virginia company, and as such we take an interest in the social media use of West Virginians. From June 9, 2015, to June 19, 2015, we sampled tweets and divided them into two categories - tweets that were sent from West Virginia and tweets that were sent from the other 49 states. Our goal was to survey the tweets between the two categories for similarities and differences.

We captured approximately 209,000 tweets, of those about 800, or about 0.40%, originated in West Virginia. (It is interesting to note that WV's population represents 0.58% of the United States' population according to the 2014 census.)

Tweets by City

Almost half (45.7%) of all WV geotagged tweets were sent from Morgantown and Huntington. Charleston, WV's largest city by population, came in fourth behind Parkersburg. Perhaps the younger, student populations of Morgantown and Huntington helped contribute to the rank of each city since the cities are not ordered by population, but that's just a hypothesis. Other areas of WV represented to a lesser degree are Wheeling and Weirton in the northern panhandle and Martinsburg in the eastern panhandle. Fewer tweets were sent from the Fairmont/Clarksburg and Beckley areas. (The West Virginia tweets that were not geotagged with a city were not considered.)

Tweets by West Virginia City

 

Heat map of tweets by West Virginia city

Sentiment of Tweets

Next, we looked at the sentiment of WV tweets compared to non-WV tweets. We used Idyl's sentiment analyzer. (In case you are not familiar, Idyl is our product for performing text analysis.) We found WV tweets to be more positive than tweets from the rest of the country. 37% of WV tweets were found to have a positive sentiment compared to 31% of the tweets from the rest of the country. WV tweets were also less negative by 1%. The sentiment analysis algorithm determines whether the sentiment of a tweet is positive, negative, or neutral based on the text of the tweet. For example, the tweet "This place is great" has a positive sentiment while "This place is terrible" has a negative sentiment.

Sentiment
Count of WV Tweets
Count of Non-WV Tweets
Negative 172 (20.8%) 46,438 (21.07%)
Neutral 347 (41.96%) 104,308 (47.34%)
Positive 308 (37.24%) 69,604 (31.59%)

Tweet Content

As for the content of the tweets they were all over the board. There were tweets about the NBA finals, school being out, and random conversations. Perhaps a larger sample size would expose more specific topics.

Thanks for reading and stay tuned for further updates.


Idyl Extraction Engine (Places Entities) now on the AWS Marketplace!

The Idyl Extraction Engine (Places Entities) is now available on the AWS Marketplace! This is a turnkey solution for performing extraction of place entities from natural language English text. Instead of having to make requests out of your network you can now extract places right in your own cloud network. A free, no risk 7 day trial is available.


Idyl Cloud API on the ProgrammableWeb

A short post today. The Idyl Cloud API is listed on the ProgrammableWeb. Check it out at http://www.programmableweb.com/api/idyl-cloud.


Idyl Extraction Engine Java SDK on Maven Central

The Idyl Extraction Engine Java SDK is now available in the Maven Central repository:

<dependency>
   <groupId>com.mtnfog.idyl.ami.sdk</groupId>
   <artifactId>idyl-ami-java-sdk</artifactId>
   <version>1.0.0</version>
</dependency>

The SDK is licensed under the Apache Software License, version 2.0 and the source code is available on Bitbucket.

The SDK provides an IdylAmiClient that has functions for submitting text for entity extraction and interacting with the optionally integrated services. An example invocation of entity extraction using the SDK is:


Idyl Extraction Engine.NET SDK Available through NuGet

 

https://www.nuget.org/packages/MountainFog.IdylAMI.SDK/1.0.0

The Idyl Extraction Engine.NET SDK is now available through NuGet. Similar to the Java SDK, the .NET SDK for the Idyl AMI provides the ability to submit text to the Idyl AMI entity extraction engine and parse the returned entities. The Idyl AMI .NET SDK is licensed under the Apache Software License, version 2.0.

Idyl AMI SDK for .NET on NuGet

Use the SDK for easy integration of Idyl's entity extraction capabilities into your .NET applications. The source code of the SDK is available on Bitbucket. We welcome any feedback on the SDK.


Announcing the Idyl Extraction Engine on the AWS Marketplace

We are very excited to announce that Idyl Extraction Engine is now available through the AWS Marketplace. Now you can have person entity extraction capabilities inside your own cloud with no request limits, no contracts, zero initial investment, and the first 7 days are free.

The Idyl AMI for Person Entities is a turn-key person entity extraction solution. Through a simple webservice (REST) interface, Idyl AMI's extraction capabilities can be integrated into your text processing systems and solutions.

Idyl AMI includes support for integrating with other AWS services:

  • DynamoDB integration allows for storing your extracted entities.
  • Automatically put your extracted entities onto an SQS queue for later processing.
  • Trigger SNS notifications when entities are extracted.
  • Submit extraction metrics to CloudWatch to monitor extraction times.

These integrations are all optional and can be used in combination with each other.

Launch the Idyl AMI for Person Entities in your cloud today from the AWS Marketplace.


Entity Extraction for Tweets

We have added entity extraction capabilities for tweets to the Idyl Cloud API. The tweet extraction endpoint can be accessed through Mashape and through Idyl Cloud accounts. Support extracting entities from tweets will be added to the Idyl Cloud SDKs in the coming week.


Idyl Cloud API on Mashape

The Idyl Cloud APIs for language detection and person entity recognition are now available through Mashape. Look for more Idyl Cloud APIs to be added to Mashape soon.

Idyl Cloud is a webservice for performing natural language processing. Learn more about Idyl Cloud at http://www.mtnfog.com/idyl/.


Idyl AMIs on AWS EC2

In the near future we will be making Idyl AMIs (Amazon Machine Images) available through the AWS Marketplace. These AMIs will contain a turnkey named-entity recognition solution. Stay tuned!

Update: The Idyl Extraction Engine is now available from the AWS Marketplace!


Distribution of Entity Confidence Values in a Sample Data Set

In a previous post titled Tuning the Confidence Threshold Parameter we described how the confidence threshold parameter can be used to control the strictness of the entity extraction. We would like to now give a little more insight into the parameter.

We recently extracted entities from more than 500,000 documents with Idyl. These documents were mostly news and news-like articles. (I say "News-like" because some did not follow the traditional format of a news article.) During the extraction we tracked the confidence value of each entity.  When the processing was complete we randomly selected 10,000 of the entities and produced the histogram of the confidence values shown below. (The Y-axis is the number of entities having the confidence value on the X-axis.)

 As the histogram shows, nearly all of the entities extracted had a confidence value greater than 50. In our spot checks, all of the entities with a confidence value less than 50 was not an actual entity and could be discarded. (They included things like abbreviations.) Between 60 and 80 the entities were more reliable, with about 75% of the entities being actual entities. Nearly all entities that were extracted with a confidence level greater than 80 were actual entities. We just spot checked the extracted entities in this investigation but in a follow-up post we will provide numbers and percentages.

The takeaway from all this is that choosing a confidence threshold of 80 is probably a safe value. You can always, of course, tweak the value later if you find that you need to.

Thanks for reading!


OpenSSL "Heartbleed" Vulnerability

Our systems were upgraded to the patched versions of OpenSSL earlier in the week and we re-keyed our SSL certificate. We recommend that all users change their passwords and generate new API keys.


Tuning the Confidence Threshold Parameter

When you start using Idyl you'll see the confidence threshold parameter when extracting entities. In this post we want to shed some light on this parameter

When Idyl looks for entities it is not a binary "yes or no" operation. The Idyl engine will have more confidence that some words or phrases constitute entities and less confidence in others. With the confidence threshold parameter you can tell Idyl to not extract any entities if Idyl's confidence level in the entity is less than the value you provide.

Valid values for the confidence threshold parameter range from 0 to 100. Keep in mind that entities rarely ever (if ever) achieve a 100% confidence level. Most will fall in the 60-90% range but it really depends on your text and can vary (described below). If you need Idyl to return entities with a lower confidence level you can just change the confidence threshold parameter in your API request. If you don't specify a value for the confidence threshold parameter it will default to 0, meaning that all identified entities will be returned.

What confidence threshold value to use depends upon your data. If you start with a value of 60 and notice that some entities are not being detected try lowering the value. The Idyl Demo uses a value of 0 so you can use it to see Idyl's confidence level for a sample of your input.

We hope this provides some insight into the purpose and function of the confidence threshold value. If you have any questions please comment or shoot an email to support@mtnfog.com.


Updates to Idyl SDKs

Recently we announced that a subset of Idyl's APIs are available on Mashape. Currently, only Idy's entity extraction API and language detection API are exposed through Mashape. We have updated the Idyl SDKs to be able to use the Mashape API endpoints. Example code snippets are shown in Mashape's readme for Idyl.

Happy coding!


Idyl Engine Update

In the past week we deployed an update to Idyl's querying engine. This update greatly improves the performance of executing SPARQL queries. Queries on small entity contexts probably won't see much improvement but users with large numbers of entities will notice improvements in query response time.


New Support Help Desk!

As part of our effort to give you a better experience we have just migrated our customer support processes to a new help desk. We believe the capabilities and features provided by the new help desk will help us help you better. (We all win!)

What does this mean to you? You can now create support tickets at https://mtnfog.freshdesk.com. If you email support@mtnfog.com with a support request we will create a helpdesk ticket on your behalf.

Over the next weeks we will be populating the helpdesk's FAQs and solutions with the goal of documenting many common problems, questions, and solutions.

If you have any comments or questions please always feel free to drop us a line at support@mtnfog.com.


Using the Idyl SDK with Maven

The Idyl API is described on the api page. The Idyl interface is just a set of REST webservices. To reduce the time necessary to develop for Idyl we have created wrappers for in both Java and .NET. Here's a quick look at how to use the Java SDK with Maven.

First, add our repository to your pom.xml:

<repository>
<id>mtnfog-repo</id>
<name>Mountain Fog Repository</name>
<url>http://content.mtnfog.com/sdk/idyl-saas/release/</url>
</repository>

Next, add the Idyl SaaS SDK dependency:

<dependency>
<groupId>com.mtnfog.sdk</groupId>
<artifactId>idyl-saas-java-sdk</artifactId>
<version>1.0.0</version>
</dependency>

Now with that done you can move on to the fun stuff. Here's a snippet of using the SDK for extracting entities:

// Set your Idyl API key.
final String apiKey = "HFPL37MZAP03JFXS";

// Set the text to be sent to Idyl.
final String sentence = "John Smith is a person.";

IdylClient idylClient = new IdylClient(apiKey);

ExtractEntitiesRequest request = new ExtractEntitiesRequest(sentence);

// If you want to correlate entities set the context and optionally the doc id:
// request.setContext("contextA");
// request.setDocId("document1");

ExtractEntitiesResponse response = idylClient.extractEntities(request);

// Show the http status code.
System.out.println("Http status code: " + response.getHttpResponseCode());

// Check the extracted entities.
System.out.println("Extracted entities: " + response.getEntities().size());

// Loop over the entities.
for(Entity entity : response.getEntities()) {

System.out.println("Entity: " + entity.getEntity() + ", Type: " + entity.getType());

}

All you need to do is replace the example API key with your Idyl API key and set your sentence value. And that is all. The request will be sent to Idyl and the extracted entities will be printed.

If you want to store the extracted entities to query over them later uncomment the two lines that set the context and document ID and set your values. Think of the context as the name for a collection of documents and the document ID as the name for a single document. For example, if you were extracting entities from books the context could be the type of book (fiction, nonfiction) or the author and the document ID could be each book's title. Now with stored entities you can use the SDK to query those entities. The code follows the same pattern as above:

final String apiKey = "HFPL37MZAP03JFXS";
final String query = "SELECT ?entity WHERE { <http://mtnfog.com/idyl/testcontext/doc1> <http://mtnfog.com/idyl/contains> ?entity . }";

IdylClient idylClient = new IdylClient(apiKey);

QueryRequest request = new QueryRequest(query);

QueryResponse response = idylClient.query(request);

// Show the http status code.
System.out.println("Http status code: " + response.getHttpResponseCode());

// The result of the query with be a RDF/XML string.
System.out.println(response.getRdfXmlOutput());

This code executes a  SPARQL query on your entities that simply returns all entity names under the testcontext and document ID doc1. The query will be sent to Idyl and the returned entities will be printed.

The use of the .NET SDK is very similar but we will describe it soon!


Welcome!

Welcome to our new blog! We're excited to blog and share with you information about services. Right now we are busy getting Idyl ready for prime time. Idyl will soon be open for beta users. If you would like to be notified when it is ready please let us know.

We're also on Twitter! Send us a tweet @mtnfog.