Blog – Mountain Fog

Streaming Text in Idyl E3 2.5.0

The Idyl E3 API has an /extract endpoint that receives text and returns the extracted entities in response. This means you have to make a full HTTP connection for each extraction request. Idyl E3 2.5.0 introduces the ability to accept streaming text through a TCP socket. When Idyl E3 starts it will open a TCP port and listen for incoming text. As text is received the socket will extract entities from the text and return an entity extraction response.

Now you can extract entities from the command line using a tool like netcat:

cat some-file.txt | netcat [idyl-e3-ip-address] [port]

Compare that command with using cURL:

curl -X POST http://idyl-e3-ip-address:port/api/v2/extract -H "Content-Type: plain/text; charset=UTF-8" -d "George Washington was president."

It’s easy to see which command is simpler. Using streaming should make processing text files and other constant sources of text much simpler.

The response to streaming input is identical to the response received from the /extract endpoint. (Both commands above will produce the same output.)

{
   "entities":[
      {
         "text":"George Washington",
         "confidence":0.96,
         "span":{
            "tokenStart":0,
            "tokenEnd":2,
            "characterStart":0,
            "characterEnd":17
         },
         "type":"person",
         "languageCode":"eng",
         "context":"not-set",
         "documentId":"not-set",
         "extractionDate":1502970191843,
         "metadata":{ 
         }
      }
   ],
   "extractionTime":72
}

Streaming is disabled by default. To enable it set the streaming.enabled property to true in Idyl E3’s properties file. Streaming does not currently support authentication. See the Idyl E3 Documentation for more streaming configuration options.

What’s New in Idyl E3 2.5.0

Here’s a quick summary of what’s new in Idyl E3 2.5.0. It’s not available yet but will be soon. For a full list check out the Idyl E3 Release Notes.

What’s New

English-language person entity models can now be trained using the ConLL-2003 format.
You can now create and use deep learning neural network entity models. Check out the previous blog posts for more information!
There’s a new setting that allows you to control how duplicate entities per extraction request are handled. You can choose to retain all duplicates or return only the duplicate entity with the highest probability.
A new TCP endpoint accepts streaming text. This endpoint deprecates the /ingest API endpoint.

What’s Changed

Idyl E3 2.5.0 changes all language codes to 3-letter ISO 3166 codes. While 2-letter codes are still supported we recommend using the 3-letter codes instead.

What’s Fixed

Entities extracted by a non-model method (regular expression, dictionary) used to return a value of 100.0 for the entity’s probability. Extracted entity probabilities should exist within the range 0 to 1 so these entities are now extracted with a probability of 1.0 instead of 100.0.

Deep Learning Entity Models in Idyl E3 2.5.0

As we mentioned in an earlier post, Idyl E3 2.5.0 will include the ability to create and user deep learning neural network entity models. As we get closer to the release of Idyl E3 2.5.0 we wanted to introduce the new capability and compare it with the current entity models.

In Idyl E3 2.4.0 you can create entity models through a perceptron algorithm. This algorithm requires as input annotated training text and a list of features. Feature selection can be a difficult task. Too many features can result in over-fitting the model such that it performs well on the input text but does not generalize well to other text. Feature selection is a crucial part of producing a useful, quality model.

Idyl E3 2.5.0’s ability to create and use deep learning models still requires annotated input text but does not require a list of features. The features are discovered automatically during the execution of the neural network algorithm through the use of word vectors, produced by applications like Word2vec or GloVe. Using a tool like this, generate a file of vectors from your training text to provide to Idyl E3 during model training. In summary, manual feature selection is not required for deep learning models.

While word vectors really helps with deep learning model training, training a deep learning model can still be a challenging task. A neural network has many hyperparameters that tune the underlying algorithms. Small changes to these hyperparameters can have a dramatic effect on the generated model. Hyperparameter optimization is an active area of academic and industry research. Tools and best practices exist to help with hyperparameter selection and we will provide some useful resources to help in the near future.

Idyl E3 2.5.0 and newer versions will continue to support using and creating maximum entropy based models so you can choose which type of model you want to create and use.

Updated English-person Entities Base Model

In the upcoming week we will be posting an updated English-person entities base model on the Models page. The model, like the version it replaces, will be free to use and included in the upcoming Idyl E3 2.5.0 release. To give an idea of the performance of this model, we evaluated the model against the CoNLL-2003 training set and the results are as follows:

Precision: 0.916720
Recall: 0.776873
F-Measure: 0.841023

Please keep in mind that these models are trained on general text and may not provide adequate performance for all text. In these cases it is recommended that you use Idyl E3 Analyst Edition to create a custom entity model from your text. Launch an instance on AWS today.

Ingesting Triples into Rya through a NiFi Dataflow Pipeline

We have a new open source project. The rya-ingest-nifi-processor facilitates ingesting triples into Rya via a NiFi dataflow pipeline. In case you are not familiar, Rya is a project undergoing incubation at the Apache Software Foundation. From the Rya website:

Rya (pronounced “ree-uh” /rēə/) is a cloud-based RDF triple store that supports SPARQL queries. Rya is a scalable RDF data management system built on top of Apache Accumulo®. Rya uses novel storage methods, indexing schemes, and query processing techniques that scale to billions of triples across multiple nodes. Rya provides fast and easy access to the data through SPARQL, a conventional query mechanism for RDF data.

The rya-ingest-nifi-processor is available under the Apache Software License, version 2.0.

https://github.com/mtnfog/rya-ingest-nifi-processor

Deep Learning Entity Extraction in Idyl E3

Idyl E3 Entity Extraction Engine 2.5.0 will introduce entity extraction powered by deep learning neural networks. Neural networks are powerful machine learning algorithms that excel at tasks like natural language processing. Idyl E3 will also support entity model training and usage on GPUs as well as CPUs. Using GPUs provides significant performance improvements. Idyl E3 2.5.0 will add support for AWS’ P2 instance type.

Entity models created by a deep learning neural network will be referred to as “second generation models.” Entity models created by Idyl E3 2.4.0 and earlier will be referred to as “first generation models.”

So how are the current entity models going to be different than the deep learning entity models?

Good question. Training entity models with the Idyl E3 2.4.0 and earlier require you to identify “features” of your text in order to train the model. Some examples of features include where an entity appears in a sentence, what words surround it, if the word is capitalized, and so on. While you can create very powerful models using this method, identifying the features can be a laborious task that requires intimate knowledge of the text. It can also result in over-fitting causing the model to not apply well to non-training text.

When training a deep learning entity model there is no need to identify the features as the algorithm is able to learn the features on its own during the training. It is able to do this through word vectors. Idyl E3 2.5.0 will be able to use word vectors generated by word vector applications such as word2vec and GloVe. To create a deep learning entity model simply provide your input training text and word vectors and Idyl E3 will generate the model.

Can I customize the neural network used to train a model?

There will be many options available to customize the neural network used for model training with a standard set of options to be used out of the box. We will describe all of the available options in the Idyl E3 2.5.0 User’s Guide.

Will there be any other impacts of the new type of model training?

No. You can continue to use your existing first generation models. You can also continue to train new first generation models. In fact, you can use first and second generation models simultaneously in an Idyl E3 pipeline.

Any other questions that we did not cover? Let us know!

English-language “Places” model in Idyl E3 2.4.0 Analyst Edition

Idyl E3 2.4.0 now includes an English-language “Places” model as well as an English-language “Persons” model. Prior to version 2.4.0, only the persons models was included. Idyl E3 2.4.0 Analyst Edition will be available from the AWS Marketplace soon.

The model will be loaded automatically when Idyl E3 2.4.0 Analyst Edition starts. An entity extraction request such as “George Washington was president of the United States.” will return two entities:

George Washington (person)
United States (place)

Idyl E3 2.4.0 comes with a free 30 day trial period in which you can use a single instance of Idyl E3 in AWS by only paying the cost of the underlying instance!

Idyl E3 2.4.0

Idyl E3 2.4.0 is now available for download. It will be available on the AWS Marketplace and DockerHub soon.

The two new features in 2.4.0 are:

The Idyl NLP annotation format that lets you store your annotations outside your training text. See previous post.
You can now configure how duplicate entities are handled. See previous post.

As always, we’re excited to release a new version and welcome your feedback.

Handling Duplicate Entities

When performing entity extraction it is common for an entity extraction request to return duplicate entities. For example, given the input:

George Washington was president. George Washington was married to Martha.

Idyl E3 may return the following entities:

George Washington – person – 86% confidence
George Washington – person – 89% confidence

The entity “George Washington” is a duplicate entity because the entity text and entity type match at least one other entity in the same entity extraction response. New in Idyl E3 2.4.0 you can choose how to handle duplicate entities. The default behavior (and the same in past versions) is to return all entities regardless of whether they are duplicates or not. A new option is to only return the entity having the highest confidence. For example, given the above entities Idyl E3 would only return the entity having 89% confidence. Entities having a confidence lower than 89% will be ignored.

The “Duplicate Entity Handling Strategy” is controlled via the duplicate.entity.handling.strategy property in Idyl E3’s configuration file. The valid values are:

retain – All entities are returned. This is the default behavior.
highest – When duplicate entities are present in a single entity extraction request, only the entity having the highest confidence value will be returned.

In summary, the new duplicate.entity.handling.strategy property controls how duplicate entities are handled on a per-entity extraction request basis. This property will be available in Idyl E3 2.4.0 and is documented in Idyl E3 2.4.0’s configuration documentation.

Training Definition File

In the next release of Idyl E3 Entity Extraction Engine (which will be version 2.4.0) we will introduce the Training Definition File to help alleviate a few problems.

The problems:

When training an entity model there are quite a few command line arguments that you have to provide. The sheer number of arguments doesn’t help with usability.
After training a model, unless you keep excellent documentation it’s easy to lose track of the training parameters. What entity type? Language? Iterations? Features? And so on.
How do you manage the command line arguments and the feature generators XML file?

The Training Definition File offers a solution to these problems. It is an XML file that contains all of the training parameters. Everything. Now you have a record of the parameters used to create the model while also simplifying the command line arguments. Note that you can still use the command line arguments as they will remain available.

Below is an example of a training definition file. Note that the final format may change between now and the release.

<?xml version="1.0" encoding="UTF-8"?>
<trainingdefinition xmlns="http://www.mtnfog.com">
	<algorithm cutoff="1" iterations="1" threads="2" />
	<trainingdata file="person-train.txt" />
	<model file="person.bin" encryptionkey="enckey" language="en" type="person" />	
	<features>
		<generators>
			<cache>
				<generators>
					<window prevLength="2" nextLength="2">
						<tokenclass />
					</window>
					<window prevLength="2" nextLength="2">
						<token />
					</window>
					<definition />
					<prevmap />
					<bigram />
					<sentence begin="true" end="true" />
				</generators>
			</cache>
		</generators>
	</features>
</trainingdefinition>

You can see in the above XML that all of the entity model training parameters are included. The training definition file defines four things:

The training algorithm.
The training data.
The output model.
The feature generators.

This removes the need for a separate feature generators file since it is now included in the training definition file. Now when training an entity model you can use the simpler command:

java -jar idyl-e3-entity-model-generator.jar -td training-definition.xml

Look for the training definition file functionality to be included with Idyl E3 2.4.0. The details may change so check back for updates.

Idyl E3 2.3.0 available on the AWS Marketplace

Idyl E3 2.3.0 is now available on the AWS Marketplace:

For a reminder of what’s new in 2.3.0 check out the Release Notes or the previous blog post.

Idyl E3 2.3.0

We are announ Idyl E3 cing the availability of Idyl E3 2.3.0! This version has a long list of new features. You can see the full list in the Release Notes and we’ll summarize the changes below and they are covered in the documentation.

You can download the new version from our website and look for it to be available on cloud marketplaces in the upcoming week. We love adding new features and supporting our users’ needs. Feel free to let us know how we’re doing!

API Changes

There is a new option for API authentication. You can now use HMACSHA512 instead of plain authorization.
There is a new /sanitize endpoint that takes in text, identifies the entities in the text, and returns the text without the entities. This endpoint is useful for cases where you want to sanitize PII or PHI information from text.
A new sort parameter was added to the /extract endpoint to control how the extracted entities are sorted in the response.
Each API endpoint now responds with HTTP 405 Method Not Allowed when given a HEAD request. This change is to support smoother integration with Idyl E3 and Apache NiFi.
Version 1 of the API has been deprecated and will be removed in Idyl E3 2.4.0.

Entity Extraction

Can now create and use part-of-speech and lemmatization models to improve entity extraction performance.
Added new feature generators to help improve entity extraction performance.
We added a new plugin to complement Idyl E3’s entity extraction capabilities with Google Cloud Natural Language API.

In addition, there were some minor bug fixes and performance improvements. There is also the new Idyl E3 SDK for Go.

Idyl E3 Entity Extraction Engine AWS Reference Architectures

With the popularity of running Idyl E3 Entity Extraction Engine on AWS we wanted to provide some AWS reference architectures to help you get started deploying Idyl E3 to AWS. Don’t forget Idyl E3 is available on the AWS Marketplace for easy launching and we have some Idyl E3 CloudFormation templates available on GitHub. We offer managed Idyl E3 services is you prefer a hands-off approach to Idyl E3 deployment and operation.

A Few Notes Before Starting

Using a Pre-Configured AMI

No matter what architecture you choose we recommend creating a pre-configured Idyl E3 AMI and using it to launch new Idyl E3 instances. This method is recommended instead of relying on user-data scripts to perform the configuration because the time required to spin up a pre-configured AMI can be significantly less than user-data scripts. If you want to have the AMI configuration under source control I highly recommend using Hashicorp’s Packer to build the AMI.

Stateless API

Before we describe the architectures it is helpful to note that the Idyl E3 API is stateless. There is no session data necessary to be shared by multiple instances and as long as all Idyl E3 instances are configured identically (as they should be when behind a load balancer), it does not matter which instance gets routed the entity extraction request. We can take advantage of this stateless architecture to allow us to scale Idyl E3 up (and down) as much as we need to in order to meet the demands of the load.

Load-balanced Architecture

The first architecture is a very simple one yet probably adequate to meet the needs of most users. This architecture has a single VPC that contains two subnets. One subnet is public and it contains an Elastic Load Balancer (ELB) and the other subnet is private and it contains the Idyl E3 instances. In the diagram shown below, the ELB is set to be a public ELB allowing Idyl E3 requests to be received from the internet. However, if your application will also run in the VPC you can change the ELB to an internal ELB. Note that this architecture uses a fixed number of Idyl E3 instances behind the ELB. Any scaling up or down will have to be performed manually when needed. Idyl E3’s API has a /health endpoint that returns HTTP 200 OK when everything is ok and that is perfect for ELB instance health checks.

Load-balanced and Auto-scaling Architecture

The previous architecture is a simple but very functional and it minimizes cost. The first thing that will be noticed in this architecture is the static nature of the Idyl E3 instances. To provide some flexibility we can modify this architecture a bit to put the Idyl E3 instances into an autoscaling group. We can use the group’s Desired Capacity to manually control the number of Idyl E3 instances or we can configure the autoscaling group to automatically scale up and down based on some chosen metrics. The average CPU usage is a good metric for scaling Idyl E3 because entity extraction can cause the CPU usage to rise. With that change here is what our architecture looks like now:

With the autoscaling we don’t have to worry about unexpected surges or decreases in entity extraction requests. The number of Idyl E3 instances will automatically scale up and down based on the average CPU usage of all Idyl E3 instances. Scaling down is important in order to keep costs to a minimum. Nobody wants to pay for more than what they need.

This architecture is available in our GitHub repository of Idyl E3 CloudFormation Templates. The template also contains an optional bastion instance to facilitate SSH access into the Idyl E3 instances from outside the VPC.

Need more?

Got more complicated requirements? Let us know. We have AWS certified engineers on staff and we’ll be glad to help.

Apache NiFi EQL Processor

We have published a new open source project on GitHub that is an Apache NiFi processor that filters entities through an Entity Query Language (EQL) query. When used along with the Idyl E3 NiFi Processor you can perform entity filtering in a NiFi dataflow pipeline.

To add the EQL processor to your NiFi pipeline, clone the project and build it or download the jar file from our website. Then copy the jar to NiFi’s lib directory and restart NiFi. The processor will not be available in the list of processors:

The EQL processor has a single property that holds the EQL query:

For this example our query will look for entities whose text is “George Washington”:

select * from entities where text = "George Washington"

Entities matching the EQL query will be outputted from the processor as JSON. Entities not matching the EQL query will be dropped.

With this capability we can create Apache NiFi dataflows that produce alerts when an entity matches a given set of conditions. Entities matching the EQL query can be published to an SQS queue, a Kafka stream, or any other NiFi processor.

The Entity Query Language previously existed as a component of the EntityDB project. It is now its own project on GitHub and is licensed under the Apache Software License, 2.0. The project’s README.md contains more examples of how to construct EQL queries.

Open Source Project Updates

A few open source project updates to share:

The Entity Query Language (EQL) has been moved out of the EntityDB project and moved into its own entity-query-language project on GitHub. It is also now licensed under the Apache Software License, version 2.0.
A new project, eql-nifi-processor, is an Apache NiFi processor for performing EQL queries on entities extracted using the Idyl E3 NiFi processor.

Apache NiFi and Idyl E3 Entity Extraction Engine

We are happy to let you know how Idyl E3 Entity Extraction Engine can be used with Apache NiFi. First, what is Apache NiFi? From the NiFi homepage: “Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.” Idyl E3 extracts entities (persons, places, things) from natural language text.

That’s a very short description of NiFi but it is very accurate. Apache NiFi allows you to configure simple or complex processes of data processing. For example, you can configure a pipeline to consume files from a file system and upload them to S3. (See Example Dataflow Templates.) There are many operations you can do and they are performed by components called Processors. There are many excellent guides available about NiFi, such as:

There are many processors available for NiFi out of the box. One in particular is the InvokeHttp processor that lets your pipeline send an HTTP request. You can use this processor to send text to Idyl E3 for entity extraction from within your pipeline. However, to make things a bit simpler and more flexible we have created a custom NiFi processor just for Idyl E3. This processor is available on GitHub and its binaries will be included with all editions of Idyl E3 starting with version 2.3.0.

Idyl E3 NiFi Processor

Instructions for how to use the Idyl E3 processor will be added to the Idyl E3 documentation bit they are simple. Here’s a rundown. Copy the idyl-e3-nifi-processor.jar from Idyl E3’s home directory to NiFi’s lib directory. Restart NiFi. Once NiFi is available you will see the Idyl E3 in the list of processors when adding a processor:

Idyl E3 NiFi Processor

There are a few properties you can set but the only required property is the Idyl E3 endpoint. By default, the processor extracts entities from the input text but this can be changed using the action property. The available actions are:

extract (the default) to get a JSON response containing the entities.
annotate to return the input text with the entities annotated.
sanitize to return the input text with the entities removed.
ingest to extract entities from the input text but provide no response. (This is useful if you are letting Idyl E3 plugins handle the publishing of entities to a database or other service outside of the NiFi data flow.)

The available properties are shown in the screen capture below:

And that is it. The processor will send text to Idyl E3 for entity extraction via Idyl E3’s /api/v2/extract endpoint. The response from Idyl E3 containing the entities will be placed in a new idyl-e3-response attribute.

The Idyl E3 NiFi processor is licensed under the Apache Software License, version 2.0. Under the hood, the processor uses the Idyl E3 client SDK for Java which is also licensed under the Apache license.

Idyl NLP Annotation Format

Idyl E3’s entity model training tool expects entities in training text to be annotated in the format used by OpenNLP. This format uses START and END tags to denote entities:

<START:person> George Washington <END> was president.

This works great but it has a drawback. The annotations and text have to be combined in a single file. Once the text is annotated it becomes difficult to use the training text for any other purposes.

New Annotation Format

Idyl E3 2.4.0 is going to introduce an additional method of annotating text that allows the annotations to be stored separate from the training text. In 2.4.0 the annotations will be able to be stored in a separate file (and we plan to eventually support storing the annotations in a database). Even though Idyl E3 2.4.0 is not yet ready for prime time, we wanted to introduce this feature early in case you are in the middle of any annotation efforts and want to use the new format.

It is still required that the input text contain a single sentence per line. Use blank lines to indicate document boundaries. Here’s an example of a simple input training file:

George Washington was president .
He was president of the United States .
George Washington was married to Martha Washington .
In 1755 , Washington became the senior American aide to British General Edward Braddock on the ill-fated Braddock expedition .

And here’s the annotations stored in a separate file:

1 0 2 person
2 5 6 place
3 0 2 person
3 5 7 person
4 11 12 person

Here’s what this means. Each line in the annotations file represents an annotation in the training text. So there are 5 annotations in this example.

The first column is the line number that contains the entity. In this example there is an annotation in each of the 3 lines.
The second column is the token index of the start of the entity. Indexes are zero-based so the first token is zero!
The third column is the token index of the end of the entity.
The last column is the type of the entity.

Note that there are two entities in the third line and each is put on its own separate line in the annotations file. Specifying the entity text in the three column format simplifies the annotation by removing the need to specify the entity’s token start and end positions. This will only annotate the first occurrence of the entity text. (If Edward Braddock had occurred more than once in the input text on line 4 only the first occurrence would be annotated.)

Summary

Now your annotations can be kept separate from your training text allowing you to use your training text for other purposes. Additionally, we hope that this new annotation method helps decrease the time required for annotating and helps with automating the process. As mentioned earlier in the post, currently the only supported means of storing the annotations is in a separate file but we plan to extend this to support databases in a future release of Idyl E3.

The Entity Model Generator tool included in Idyl E3 has been updated to allow for using this new annotation format. You can, however, continue to use the OpenNLP-style annotations when creating entity models. This new annotation format is only available for entity models. Sentence, token, parts-of-speech, and lemma model annotations will remain unchanged in 2.4.0.

[related_post]

Idyl E3 SDK for Go

The Idyl E3 SDK for Go is now available on GitHub. This SDK allows you to integrate Idyl E3’s entity extraction capabilities into your Go projects.

Like the other Idyl E3 SDKs, the project is licensed under the Apache Software License, version 2.0.

It’s easy to use:

endpoint := "http://localhost:9000"
s := "George Washington was president."
confidence := 0
context =: "context"
documentID := "documentID"
language := "en"
key := "your-api-key"

response := Extract(endpoint, s, confidence, context, documentID, language, key)

[related_post]

Amazon EBS Elastic Volumes

On Feb 13, 2017, Amazon Web Services announced elastic EBS volumes! If you have used EC2 much you have undoubtedly been frustrated by the rigidness of EBS volumes. Once created they could not be modified or resized. If your EC2 instance required more disk space your only option was to manually create a new volume of the desired size and attach it to your instance. Now that EBS volumes are more “elastic” you can now simply resize an EBS volume. I put “elastic” in quotes because the volume size can only be increased and not decreased. That’s more elastic than before but sill not completely elastic. In addition to adjusting size, you can now adjust performance and change the volume type even while the volume is in use. These functions are available for your existing EBS volumes.

You can use the AWS CLI to modify a volume:

aws ec2 modify-volume --region us-east-1 --volume-id vol-11111111111111111 --size 200 --volume-type io1 --iops 10000

After enlarging a volume don’t forget to tell your OS to use the newly allocated storage.

This can make like a lot easier is many situation. As described in the AWS blog post, you can use this functionality in combination with CloudWatch and Lamba to automatically enlarge volumes when running low on disk space. You can also use it to simply save money by starting with a smaller EBS volume than what you might need knowing you have the flexibility to increase the capacity of the volumes when needed.

Why do we find this interesting? Our Idyl E3 managed services run in AWS and we encourage all potential customers to launch Idyl E3 from the AWS Marketplace due to its ease of use and turn-key capabilities. So we like to pass interesting and relevant information regarding related services on to our users and readers when it comes available. Learn more about Idyl E3’s entity extraction capabilities.

[related_post]

New Feature Generators in Idyl E3 2.3.0

A feature generator is arguably the most important part of model-based entity extraction. The feature generators create “features” based on aspects of the input text that are used to determine what is and what is not an entity. Choosing the right (or wrong) features when training your entity models can have a significant impact on the performance of the models so we want you to have a good selection of feature generators available for use.

There are some new feature generators in Idyl E3 2.3.0 available to you that we’d like to take a minute to describe. All of the available feature generators and how to apply each one is described in the Idyl E3 2.3.0 Documentation.

New Feature Generators in Idyl E3 2.3.0

Special Character Feature Generator

This feature generator generates features for tokens that contains special characters. For example, the token Hello would not generate a feature but the token He*llo would generate a feature. This feature generator is probably most useful in the domains of science and healthcare, particularly chemical and drug names.

Token Part of Speech Feature Generator

This feature generator generates features based on each token’s part of speech. To use this feature generator you must provide a trained part of speech model. (Idyl E3 2.3.0 includes a tool for creating parts-of-speech models from your text.) This feature generator helps improve entity extraction performance by also being able to consider each entity’s part of speech.

Word Normalization Feature Generator

This feature generator normalizes tokens by replacing all uppercase characters with A, all lowercase characters with a, and all digits with 0. For example, the token HelloWorld25 would be normalized to AaaaaAaaaa00. This feature generator can optionally lemmatize each token prior to the normalization by applying a lemmatization model. (Idyl E3 2.3.0 includes a tool for creating lemmatization models from your text.) Like the special character feature generator, this feature generator is also probably most useful in the domains of science and healthcare, particularly chemical and drug names.

[related_post]

AWS EC2 Metadata Simulator

mtnfog/aws-metadata-simulator

There is a new project on our GitHub that is an EC2 metadata simulator. The project allows for testing applications that depend on EC2 instance metadata in non-AWS environments. It doesn’t (yet) provide complete simulation of all EC2 metadata endpoints but in time it will and in the mean time it should be simple enough to modify to fit your needs.

One important note, the EC2 instance metadata listens on 169.254.169.254 port 80. The simulator will run on your localhost at port 8080 by default. The result of this command will not persist across system restarts, but you can redirect traffic from 169.254.169.254:80 to localhost:8080 using iptables:

iptables -t nat -A OUTPUT -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination 127.0.0.1:8080

Now, all requests to http://169.254.169.254:80 will get redirected to the simulator running at http://localhost:8080.

EC2 instance metadata provides useful information to instances running in EC2. Using a simple curl command like the one below to find the instance’s ID you can retrieve information about the running instance.

curl http://169.254.169.254/latest/meta-data/instance-id

[related_post]

Packer Script for Graphite AMI

Idyl E3 2.2.0 added support for publishing metrics to a Graphite server. To help make it easier to deploy a Graphite server we have added a new project on our GitHub that contains a Packer script for creating a Graphite AMI. Usage instructions are available in the project’s readme file.

mtnfog/graphite-ami

[related_post]

Idyl E3 and Google Cloud Natural Language API

In late 2016 Google announced a new service on their Google Cloud platform called Google Cloud Natural Language API. This service provides various natural language processing capabilities including entity extraction. At first sight it seems as if Google’s Cloud Natural Language’s API is a direct competitor with Idyl E3 but when given a closer look the two products are very different. This blog post compares and contrasts Idyl E3 and Google Cloud Natural Language API’s entity extraction capabilities.

From the Google Cloud Natural Language API website:

Google Cloud Natural Language API reveals the structure and meaning of text by offering powerful machine learning models in an easy to use REST API. You can use it to extract information about people, places, events and much more, mentioned in text documents, news articles or blog posts.

This sounds a lot like Idyl E3. But let’s take a closer look at the similarities and differences between Idyl E3 and the Google Cloud Natural Language API.

Comparison of Idyl E3 and Google Cloud Natural Language API

Idyl E3 Google Cloud Natural Language API and Idyl E3 are similar in that they are both applications that expose entity extraction capabilities for natural language text over an API interface. Both accept text and return the extracted entities. Idyl E3 is an application that you manage and can be installed behind your organization’s firewall. Google Cloud Natural Language API is a software-as-a-service (SaaS) offering and Google manages the application and billing. In addition to entity extraction, Google Cloud Natural Language API also offers sentiment analysis.

Security

Text sent to Google Cloud Natural Language API is transmitted over the public internet. Even though the text is sent using SSL encryption, this may not be acceptable for text containing sensitive information. Some workloads are not allowed to be transmitted outside of the organization. Idyl E3 runs behind a firewall so your text never leaves your network. This makes Idyl E3 ideal for security sensitive workloads.

Entity Types

Google Cloud Natural Language API supports identifying the following entity types: Unknown, Person, Location, Organization, Event, Work of Art, Consumer Good, Other. Idyl E3 is not limited to any set of entities. With Idyl E3 you are in full control of the entity types because you are able to create entity models for any types of entities. For instance, you can train Idyl E3 to extract Hospitals, Buildings, Bridges, Schools, Stadiums, and more.

Types of Text used for Training

The types (news articles, blog posts, encyclopedia articles, etc.) of text that was used to train the engine powering Google Cloud Natural Language API does not seem to be documented. The type of text that was used is important to provide a high-level of accuracy when extracting entities. With Idyl E3’s ability to create custom models, you can create models specifically for your text, whether it be emails, legal documents, or other text.

For optimal performance, it is very important that the text being processed is similar to the text that was used to train the models.

Language Support

Google Cloud Natural Language API only supports English, Spanish, and Japanese for entity analysis (source). Idyl E3 is not limited to by language. Idyl E3 can create and use entity models for any UTF-8 language.

Cost

Google Cloud Natural Language API’s pricing is per API request. This means that the more you use it the higher your bill. This is not the case with Idyl E3. Idyl E3 has flat licensing pricing. You do not pay per request.

20,000,000 Google Cloud Natural Language API requests: Monthly price = $5,000 (20,000,000 / 1,000 * 0.25)

In contrast, with Idyl E3 you could make 20 million or 100 million API requests per month and there is no additional cost. For example, you can launch Idyl E3 Analyst Edition from the AWS Marketplace for $1.50 per hour. If used for a full month the cost would be $1,080 (plus EC2 instance fees) no matter how many extraction requests you submit to Idyl E3. As you can see, Idyl E3 can cost substantially less than Google Cloud Natural Language API.

Control

With Idyl E3 you have full control over the entity extraction process. You can create custom sentence, token, and entity models for your text giving higher accuracy and improved performance. Idyl E3’s heuristic confidence filtering helps remove noise from the identified entities. Google Cloud Natural Language API does not have a concept of entity confidence values.

Additionally, you have full control over Idyl E3’s deployment architecture. You can also use Idyl E3 in an UIMA pipeline with the UIMA Annotator for Idyl E3.

Summary

To conclude, Idyl E3 and Google Cloud Natural Language API are very different products. They both expose an API for entity extraction from natural language text but that’s where the similarities stop. We will be offering an Idyl E3 plugin that supports using Google Cloud Natural Language API to complement Idyl E3’s entity extraction capabilities. By providing this plugin Idyl E3 will be exposing a common API for both services. Look for it to be available soon.

[related_post]

Idyl E3 2.2.0

Today we are announcing the release of Idyl E3 2.2.0. (See the full Release Notes.) This version brings some new exciting features such as heuristic confidence filtering, support for all UTF-8 languages, and statistics reporting.

Idyl E3 2.2.0 can be downloaded from our website today. Look for it to be available on the AWS Marketplace in the upcoming week.

In related news:

entity-model was updated to include entity metadata (used by Idyl E3 2.2.0).
The Idyl E3 Java SDK has been updated for Idyl E3 2.2.0.
The Idyl E3 UIMA Annotator has been updated to use the updated Idyl E3 Java SDK.
The Idyl E3 CloudFormation template has been updated to include autoscaling policies and other resources.

[related_post]

Idyl E3 and OpenNLP

As you may know, Idyl E3’s entity extraction capabilities is provided by a customized version of OpenNLP. Since the release of OpenNLP 1.7.0, the OpenNLP team has been able to release more often than previously. Because of the more frequent OpenNLP releases we may not incorporate each release into Idyl E3. We will analyze the changes in each new OpenNLP version to decide if the changes should be incorporated into Idyl E3.

Also, we do have on the (distant) roadmap the ability to make the underlying NLP engine pluggable to allow you to choose which NLP engine to use with Idyl E3.

Idyl E3 2.1.0 Analyst Edition Now Available

Idyl E3 Analyst Edition 2.1.0 is now available on the AWS Marketplace. Idyl E3 Analyst Edition includes everything in the Free and Standard editions plus licenses for all plugins and licenses for unlimited custom models.

Heuristic Confidence Filtering

In Idyl E3 2.2.0 we are introducing a feature we call Heuristic Confidence Filtering. Here’s how it works.

As you may (or may not) already know, each entity extraction request can have an associated “confidence threshold value.” Any entities that are extracted who have a confidence lower than this value will not be returned in the entity extraction response. This is useful but it is a bit of a sledgehammer approach and can either result in too much noise or missed entities depending on its value.

When enabled, heuristic confidence filtering tracks the confidence values of extracted entities per the entity model that extracted them. Once a large enough sample of confidence values has been collected, Idyl E3 will filter entities by determining if an entity’s confidence value is significant to the mean of the collected values. This provides a way to filter out noise but still receive important entities.

It is important to note that the confidence threshold value still plays a part even when heuristic confidence filtering is enabled. Any entity whose confidence value is greater than or equal to the confidence threshold for that request will always be returned even when heuristic confidence filtering is enabled.

Because of the mathematical calculations involved and the memory required to store the confidence values the heuristic confidence filtering does require a bit more computation time but not to the point where it should be noticeable.

We are excited to offer this feature and we hope that it helps with “entity noise.” We welcome your feedback on how it performs for you! For more information on this feature you can refer to the Idyl E3 2.2.0 User Documentation or by contacting us. Look for Idyl E3 2.2.0 to be available in February 2017.

Idyl E3 2.1.0

Idyl E3 2.1.0 has been released. This version introduces a new version of the API that includes changes to the extract and ingest endpoints. With version 2 of the API these two endpoints accept text in the body of the request instead of as a query string parameter. Version 1 of the API is still available so you do not need to update your clients unless you just want to or need to for other reasons. The Idyl E3 Java SDK and the Idyl E3 .NET SDK have been updated to use API v2.

Idyl E3 2.1.0 is based on a customized OpenNLP 1.7.0 which was released in early January 2016. Previous versions of Idyl E3 were based on a customized OpenNLP 1.6.0.

Idyl E3 2.1.0 Analyst Edition will be available on the AWS Marketplace soon. The Analyst Edition includes all plugins and allows for the use of unlimited custom models without separate licensing. (See the Idyl E3 edition comparison.)

Privacy Policy Changes

We want to make you aware of a recent change to our Privacy Policy. We added a paragraph to the “Non-personal identification information” section about product update checks. The new paragraph describes the information that is transmitted when our products perform an updated version check. Remember that update checks can always be enabled or disabled — please check the product’s documentation for instructions or contact us.

Idyl E3 2.0

Update: Idyl E3 2.0 is now available on the AWS Marketplace: https://aws.amazon.com/marketplace/pp/B01BSQUR2K

Today we are announcing Idyl E3 2.0. It has been over a year since version 1.0 was introduced and we’d like to thank our users for helping us to reach this milestone. The main goals of version 2.0 were to make Idyl E3 extensible and increase performance. We would like to thank our users for helping us get to this milestone release. We could not have done it without your feedback and comments.

Idyl E3 is available for download from our website. Look for Idyl E3 2.0 to be available on the AWS Marketplace and other channels shortly thereafter.

Three Editions

Idyl E3 2.0 will be available in three editions:

Idyl E3 Free Edition

This edition of Idyl E3 is free. It includes an English-persons entity model and no plugins. This edition can be customized with plugins and models to meet your requirements.

Idyl E3 Standard Edition

The Standard Edition includes everything in the free edition plus model evaluation tools and priority email technical support.

Idly E3 Analyst Edition

The Analyst Edition includes everything in the standard edition plus all plugins and supports unlimited custom models.

Plugins

In Idyl E3 1.x, things like email addresses and phone numbers were extracted through built-in functionality called extraction modules. In version 2.0 we are introducing plugins. There are two types of plugins – a plugin type that perform an entity extraction and a plugin type that publishes the extracted entities. Plugins can be downloaded from our website and installed in your Idyl E3. The following plugins are currently available or will soon be:

Text Consumption Plugins

Consume input text from Kafka topic
Consume input text from Kinesis stream

Entity Extraction Plugins

Phone numbers extraction plugin
Email addresses extraction plugin
Hashtags extraction plugin
User mentions extraction plugin

Document Processing Plugins

Parse text from PDF files

Entity Publisher Plugins

AWS Kinesis Firehose publisher plugin
EntityDB publisher plugin

Internal changes were made to improve Idyl E3’s performance to lower the time to extract entities. One change was the removal of the web-based dashboard. Configuration is now done directly through the properties file.

Custom Sentence and Token Models

Also new in version 2.0 to increase performance is the ability to generate and use custom sentence and token models. In versions 1.x, internal models were used for sentence detection and sentence tokenizing. These models were not always representative of the input text so their performance was degraded. In version 2.0 you have the option to generate sentence and token models from your data or use the legacy internal models just as versions 1.x did. You can still create your own entity models.

UIMA

Idyl E3 2.0 supports integration with UIMA through the Idyl E3 UIMA connector.

EntityDB and AWS CloudWatch Metrics

We added the ability for EntityDB to report metrics to AWS CloudWatch. The metrics reported include the numbers of entities stored and indexed. The screen capture of an AWS CloudWatch graph is shown below. The system that generated the metrics illustrated by the chart was composed of 5 EntityDB t2.micro instances in auto-scaling group behind an elastic load balancer. An SQS queue was used for the entity queue and entities were persisted to a MongoDB database also running on a t2.micro instance. (This architecture was created using the CloudFormation templates in the GitHub repository.)

EntityDB CloudWatch metrics

As the metrics show, the entities are being stored at a rate much faster than the entities are being indexed. We will be working to make the index rate (orange line) more closely follow the stored rate (blue line).

Cloud NLP

We have made available some NLP services over a REST API. The services, collectively called Cloud NLP, currently include sentiment analysis and language detection. Additional services will be added over the next few weeks. Cloud NLP requires an API key that you can get for free by contacting us or by consuming Cloud NLP through the Mashape API Marketplace.

The Cloud NLP Java client SDK is now available on GitHub. It is licensed under the Apache Software License, version 2.0. The Maven dependency information is:

<dependency>
    <groupId>com.mtnfog.cloudnlp</groupId>
    <artifactId>cloud-nlp-java-sdk</artifactId>
    <version>1.0.0</version>
</dependency>

The Cloud NLP Java client SDK includes support for accessing the Cloud NLP services directly with a Mountain Fog API key or through Mashape. It’s easy to use:

CloudNlpClient client = new CloudNlpStandardClient(API_KEY, CloudNlpStandardClient.MTNFOG_CLOUDNLP_ENDPOINT);

String language = cloudNlpClient.detectLanguage("This is english text.");
int sentiment = cloudNlpClient.analyzeSentiment("This widget is great!");

Similarly, to use Cloud NLP via Mashape just change to the CloudNlpMashapeClient:

CloudNlpClient cloudNlpClient = new CloudNlpMashapeClient(MASHAPE_API_KEY);

String language = cloudNlpClient.detectLanguage("This is english text.");
int sentiment = cloudNlpClient.analyzeSentiment("This widget is great!");

And that’s it. As mentioned earlier, look for more natural language processing services to be added to Cloud NLP in the near future!

Open Source Updates

In the past week we made the following updates to our open source projects.

Entity Model – Updated to include a new Span class on entity. The Span class identifies the location of the entity in the source text by token and by character indexes. This update was made to the entity-model and entity-model-net projects. Version 1.0.8 of entity-model was published to Maven Central and version 2.0.0 of entity-model-net was published to NuGet.

Idyl E3 UIMA Annotator – An update was made that annotates the entities based on the character index instead of the token index so the entities are properly annotated in UIMA. (The Idyl E3 UIMA Annotator requires Idyl E3 1.13.0 which is not quite ready but look for it soon.)

Idyl E3 Client SDKs – The Idyl E3 client SDK for Java was updated to use entity-model 1.0.8. The Idyl E3 client SDK for .NET was updated to use the new MountainFog.EntityModel 2.0.0 package from NuGet.

Anthology – Anthology was updated to include the ability to load balance Idyl E3 entity extraction requests. You can now specify multiple Idyl E3 endpoints per entity type when defining the routes.

EntityDB – EntityDB was updated to use entity-model 1.0.8.

Idyl E3 UIMA Annotator

We have published a new project to our GitHub. The new project is a UIMA annotator that uses Idyl E3 for named entity recognition. When added to a UIMA pipeline, the annotator will send the text that is the subject of analysis to Idyl E3. The project is licensed under the Apache Software License, version 2.0.

The Idyl E3 UIMA annotator requires Idyl E3 1.13.0 which will be available very soon.

S3 Uploads with Pre-signed URLs

In case you didn’t know we also do consulting services for Amazon Web Services users. (One team member has achieved all 5 AWS certifications!) One thing clients commonly want to do is allow users to upload images (and other files but typically images) directly to S3. By uploading directly to S3, you don’t have to make a custom webservice to handle the upload and then send the file to S3. Direct uploads by the client to S3 are much more efficient.

An easy way to accomplish direct client uploads to S3 is through the use of pre-signed URLs. Originally, pre-signed URLs were available just for downloading files from S3 but they can be used for uploads, too. (A pre-signed URL for a download is a time-expiring link that allows anyone with that URL to download the file. They are very useful.) For uploading, a pre-signed URL allows anyone with the URL to upload a file to your S3 bucket. You define the target location in S3 and set some options and the URL is generated. You can then use that URL in an HTML upload form and the user’s file will upload directly to S3.

Here’s some sample code from the AWS documentation that shows how to generate a pre-signed URL for uploading:

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider()); 

java.util.Date expiration = new java.util.Date();
long msec = expiration.getTime();
msec += 1000 * 60 * 60; // Add 1 hour.
expiration.setTime(msec);

GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);
generatePresignedUrlRequest.setMethod(HttpMethod.PUT); 
generatePresignedUrlRequest.setExpiration(expiration);
             
URL s = s3client.generatePresignedUrl(generatePresignedUrlRequest); 

// Use the pre-signed URL to upload an object.

In the code we set the expiration for the URL and the HTTP method and then we generate the URL. (See the full code example.)

And that’s where the hold up comes. There’s a slight gotcha that is very often missed by our clients. Our clients encounter the problem when a user goes to upload a file using the pre-signed URL and receive a forbidden error. Here’s the important point:

The user who generates the pre-signed URL must have permissions to upload a file to S3.

This piece of information is in the AWS documentation but it is missed a lot. Here’s the text:

A pre-signed URL gives you access to the object identified in the URL, provided that the creator of the pre-signed URL has permissions to access that object. That is, if you receive a pre-signed URL to upload an object, you can upload the object only if the creator of the pre-signed URL has the necessary permissions to upload that object.

Hopefully this will save you a few minutes of head bashing in case you run into this the next time you’re implementing S3 uploads with pre-signed URLs.

Idyl Talk – New Open Source Project

We have pushed a new open source project to our GitHub called Idyl Talk. The goal of Idyl Talk is to replace traditional interface-defined software communication with natural language text.

When software communicates with other software, either internally or with external software, the communication is defined by interfaces. These interfaces tell each side how to communicate. Interfaces are an essential piece of good design. But what happens when two components have to communicate, and for whatever reasons, it is difficult (or impossible) to define the interface? Idyl Talk addresses this problem by letting software components communicate using natural language English text.

Imagine your refrigerator talking to your smartphone app to update your shopping list. The communication might look a bit like this:

{
    inventory: {
        "milk": "low",
        "eggs": 12
    }
}

Your smartphone receives the message and an app notifies you that you need milk. For this to be possible the developers of the refrigerator and the smartphone app have to agree on some interface that dictates the communication between the devices. This requires collaboration, and of course, time and money.

Now, imagine that when you are running low on milk your refrigerator sends the following message to your smartphone app:

You are low on milk.

The agreed-to interface here is the English language. With Idyl Talk can now create devices that are enabled to communicate even if they do not exist yet! The app processes the received message and alerts you that you are low on milk.

Sound interesting? We think so! We welcome your contributions to the project as it matures and grows. Check out Idyl Talk on GitHub.

See a listing of all our open source projects.

AWS CloudFormation Supports YAML

In an exciting update from AWS, it was announced that CloudFormation now supports YAML in addition to JSON. I think most of us will agree this is great. The JSON templates worked, but whew, were they hard to read and the lack of the ability to add comments sometimes made my templates look more like sudokus or word searches than anything else.

They also announced the support for cross-stack references. That means no more duplicating resources between templates! There’s a small gotcha with cross-stack references in that the names of the exported values have to be unique in your account and have to be literal string values.

These new features are significant enough that I felt they deserved a mention on this blog. They will definitely have an immediate impact on how we create CloudFormation for ourselves and our clients.

New Open Source Project – Anthology

We have published a new open source project called Anthology to our GitHub. Anthology makes working with multiple instances of Idyl E3 easier. Anthology can route entity extraction requests based on the type of entities to extract. With Anthology you can have a single Idyl E3 endpoint for multiple Idyl E3 clusters.

Anthology is licensed under the Apache Software License, version 2. All contributions are welcome!

EntityDB is Open Source

EntityDB is now open source on GitHub. It is licensed under the AGPLv3. The goal of EntityDB is to provide an integration solution for storing, managing, and querying entities (persons, places, and things). Everyone is welcome to contribute to its development and future as we work toward a first release.

EntityDB provides a choice of underlying databases. MySQL, MongoDB, Cassandra, and DynamoDB are currently supported. The Entity Query Language (EQL) is also included in the open sourced code. EQL provides an abstraction layer for querying the entities regardless of the underlying database.

Proprietary licenses are available for situations where the AGPLv3 is not suitable. Please contact us for more information.

OpenNLP’s RegexNameFinder and Tokenizing

OpenNLP’s RegexNameFinder takes one or more regular expressions and uses those expressions to extract entities from the input text. This is very useful for instances in which you want to extract things that follow a set format, like phone numbers and email addresses. However, when tokenizing the input to the RegexNameFinder be careful because it can affect the RegexNameFinder’s accuracy.

The RegexNameFinder is very simple to use and here’s an example borrowed from an OpenNLP testcase.

Pattern testPattern = Pattern.compile("test");
String sentence[] = new String[]{"a", "test", "b", "c"};

Pattern[] patterns = new Pattern[]{testPattern};
Map<String, Pattern[]> regexMap = new HashMap<>();
String type = "testtype";

regexMap.put(type, patterns);

RegexNameFinder finder =
new RegexNameFinder(regexMap);

Span[] result = finder.find(sentence);

The sentence variable is a list of tokens. In the above example the tokens are set manually. In a more likely scenario the string would be received as “a test b c” and it would be up to the application to tokenize the string into {“a”, “test”, “b”, “c”}.

There are three types of tokenizers available in OpenNLP – the WhitespaceTokenizer, the SimpleTokenizer, and a tokenizer (TokenizerME) that uses a token model you have trained. The WhitespaceTokenizer works on, you guessed it, white space. The locations of white space in the string is used to tokenize the string. The SimpleTokenizer looks at character classes, such as letters and numbers.

Let’s take the example string “My email address is me@me.com and I like Gmail.” Using the WhitespaceTokenizer the tokens are {“My”, “email”, “address”, “is”, “me@me.com”, “and”, “I”, “like”, “Gmail.”}. If we use the RegexNameFinder with a regular expression that matches an email address, OpenNLP will return to us the span covering “me@me.com”. Works great!

However, let’s consider the sentence “My email address is me@me.com.” Using the WhitespaceTokenizer again the tokens are {“My”, “email”, “address”, “is”, “me@me.com.”}. Notice the last token includes the sentence’s period. Our regular expression for an email address will not match “me@me.com.” because it is not a valid email address. Using the SimpleTokenizer doesn’t give any better results.

How to work around this is up to you. You could make a custom tokenizer by implementing the Tokenizer interface, try using a token model, or massaging your text before it is passed to the tokenizer.

Idyl E3 1.12.0

Idyl E3 Look for Idyl E3 1.12.0 to be available on various cloud marketplaces this week. Version 1.12.0 starts the separation from the entity stores we announced in our last post. It also contains some minor fixes and improvements. (See the Idyl E3 Release Notes.)

There will be multiple versions of Idyl E3 1.12.0 available. The versions will differ based on what entity models are included in the version. One version will not have any entity models making it ideal for scenarios when you want to use your own generated entity models. As a reminder, you can create entity models from your own data for use with Idyl E3. Using your own data to generate models will result in models that perform better than our models for your type of data.

Idyl E3’s entity store and EntityDB

Along with the ability to extract entities from text, Idyl E3’s entity store feature allows you to save the extracted entities to a database of your choice. Supported databases include a relational database like MySQL and the NoSQL databases MongoDB and DynamoDB. In addition to save the entities to a database you can also query the entities using a special language called Entity Query Language (EQL). EQL has a SQL-like syntax letting you select entities based on conditions in the query. Your EQL query is translated into a native query for your selected database. A single EQL query can be executed against MySQL, MongoDB, and DynamoDB.

The entity store feature of Idyl E3 is being separated from Idyl E3 into its own product called EntityDB. This separation will allow Idyl E3 to focus on entity extraction. Idyl E3 will integrate with EntityDB’s public API to still provide entity storage services.

EntityDB will continue to support the same databases as well as a new database – Apache Cassandra. Cassandra is ideally suited for storing entities and will allow for large-scale querying and analysis. The Cassandra-based entity store will support EQL queries but you will also have the ability to query it using other tools like SparkSQL.

Look for the first version of EntityDB to be available in the near future. We have a large roadmap for EntityDB and plan to add features incrementally over a series of releases.

Mountain Fog, Inc. Listed in AWS Marketplace for the U.S. Intelligence Community

Mountain Fog, Inc. Listed in AWS Marketplace for the U.S. Intelligence Community

Idyl E3 Entity Extraction Engine now available to 17 US Intelligence Agencies in a Cloud Marketplace

June 20 – Morgantown, WV – Mountain Fog, Inc., a leading provider of natural language processing software for commercial and law enforcement users, today announced it is among the first group of technology vendors to be listed in Amazon Web Services (AWS) Marketplace for the U.S. Intelligence Community (IC). AWS Marketplace for the U.S. IC is designed exclusively for the 17 intelligence agencies to evaluate, purchase, and deploy in minutes via 1-Click® a broad array of common software infrastructure, developer tools, and business software products, with the categories of products and vendors growing over time.

Mountain Fog’s product, Idyl E3 Entity Extraction Engine, analyzes multilingual natural language text and identifies persons, places, and things within the text. Its integrated rules engine and entity persistence capabilities provide a complete solution for processing unstructured text. “Idyl E3 can help government agencies manage unstructured text and turn it into usable information. We are pleased to offer Idyl E3 on the AWS Marketplace for the U.S. IC in order to give more agencies immediate access to its capabilities,” said Mountain Fog president Jeff Zemerick.

AWS Marketplace for the U.S. IC provides the same purchasing convenience, open and transparent license terms and conditions, and variety of pricing models, including hourly usage and annual subscription, as the commercial AWS Marketplace. It also supports Bring-Your-Own-License (BYOL) so that agencies can more easily migrate existing software licenses and applications. AWS Marketplace for the U.S. IC is part of the Commercial Cloud Services (C2S) program, under the Director of National Intelligence (DNI) Intelligence Community (IC) Information Technology Enterprise (IC ITE). For more information on AWS Marketplace for the U.S. IC, contact icmp@amazon.com.

###

About Mountain Fog, Inc.

Mountain Fog was founded in 2011 to develop innovative language processing solutions. Our team of developers and engineers specialize in big-data analysis, natural language processing, and cloud systems. Our philosophy is simple – provide the best products we can and back them up with unparalleled customer support. (412) 206-1079 | sales@mtnfog.com

User-created entity models

Idyl E3 offers can extract many types of entities such as building, cities, and more. In instances where we do not offer the type of entity you need we will soon be offering a tool to create your own entity models. You will be able to create your own entity models from your own text giving you entity models customized for your own use-cases. This ability is expected to be available in an upcoming release so stay tuned for more details.

Idyl E3 on the VMware Solutions Marketplace

Idyl E3 is now available through the VMware Solutions Marketplace. Check it out at https://solutionexchange.vmware.com/store/products/idyl-e3!

Idyl E3 1.7.1 on the AWS Marketplace

Idyl E3 1.7.1 is now available on the AWS Marketplace. Launch it for free now!

Idyl E3 1.7.1 Debian Package Available

The debian package for Idyl E3 1.7.1 is available for free download in your account. If you do not have an account you can create one (also free!).

See the Idyl E3 on Ubuntu Server page for more details.

Idyl E3 on Docker Hub

Idyl E3 is now available as a docker container through Docker Hub. Be sure to star the Idyl E3 repository on Docker Hub and to send us your feedback! Learn more about the Idyl E3 docker containers.

Idyl E3 is Now Free

Idyl E3 is now available for free. And to make it even better, Idyl E3 will soon also be available for Windows Azure, VMware ESXi, and as a standalone download. Idyl E3 will continue to be available through the AWS Marketplace. Deploy Idyl E3 to the platform of your choice.

And to make it even better, the entity models used by Idyl E3 for entity extraction are now configurable and additional models are available for download. Idyl E3 comes with a base model for extracting person entities that is fully functional. Models capable of extracting more entities with higher confidence are available.

We will still provide support for Idyl E3. Priority support is available as well as development and integration support to help you get Idyl E3 integrated into your systems.

We are very excited about the expanded availability of Idyl E3. If you need help getting started with Idyl E3 or have any questions please contact us.

New Website

In case you haven’t noticed we have rolled out our website update. The goal of this update was to improve usability. We felt that our previous website was at times hard to navigate with too much text. In the new website we are going for simplicity, especially on the product pages.

The new website also features a new and improved My Account. Soon you will be able to access your downloads and purchase history from your account. The new website also features improved single sign-on with our .

We have also migrated our blog to the new website. No longer do you have to leave our main site to checkout our newest blog posts.

So, please bear with us over the next few days as we iron out any issues and missed 404 errors.