Training Definition File

In the next release of Idyl E3 Entity Extraction Engine (which will be version 2.4.0) we will introduce the Training Definition File to help alleviate a few problems.

The problems:

  1. When training an entity model there are quite a few command line arguments that you have to provide. The sheer number of arguments doesn’t help with usability.
  2. After training a model, unless you keep excellent documentation it’s easy to lose track of the training parameters. What entity type? Language? Iterations? Features? And so on.
  3. How do you manage the command line arguments and the feature generators XML file?

The Training Definition File offers a solution to these problems. It is an XML file that contains all of the training parameters. Everything. Now you have a record of the parameters used to create the model while also simplifying the command line arguments. Note that you can still use the command line arguments as they will remain available.

Below is an example of a training definition file. Note that the final format may change between now and the release.

<?xml version="1.0" encoding="UTF-8"?>
<trainingdefinition xmlns="http://www.mtnfog.com">
	<algorithm cutoff="1" iterations="1" threads="2" />
	<trainingdata file="person-train.txt" />
	<model file="person.bin" encryptionkey="enckey" language="en" type="person" />	
	<features>
		<generators>
			<cache>
				<generators>
					<window prevLength="2" nextLength="2">
						<tokenclass />
					</window>
					<window prevLength="2" nextLength="2">
						<token />
					</window>
					<definition />
					<prevmap />
					<bigram />
					<sentence begin="true" end="true" />
				</generators>
			</cache>
		</generators>
	</features>
</trainingdefinition>

You can see in the above XML that all of the entity model training parameters are included. The training definition file defines four things:

  1. The training algorithm.
  2. The training data.
  3. The output model.
  4. The feature generators.

This removes the need for a separate feature generators file since it is now included in the training definition file. Now when training an entity model you can use the simpler command:

java -jar idyl-e3-entity-model-generator.jar -td training-definition.xml

Look for the training definition file functionality to be included with Idyl E3 2.4.0. The details may change so check back for updates.

Leave a Reply