Differences between revisions 11 and 12
Revision 11 as of 2017-05-09 14:26:18
Size: 11704
Comment:
Revision 12 as of 2017-05-15 01:45:06
Size: 5254
Editor: ThammeGowda
Comment: Added wiki page for DL4J
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
## page was copied from TikaAndVision
= Tika and Computer Vision - DL4J =
= Tika and Computer Vision powered by Deeplearning4j(DL4J) =
Line 6: Line 6:
This page describes how to make use of Object (Visual) Recognition capability of Apache Tika.
TIKA-1993 introduced a new parser to perform object recognition on images.
Visit [[https://issues.apache.org/jira/browse/TIKA-1993 | TIKA-1993 issue on Jira ]] or [[ https://github.com/apache/tika/pull/125 | pull request on Github]] to read the related conversation. The model was updated from Inception V3 to Inception V4 with [[https://issues.apache.org/jira/browse/TIKA-2306|TIKA-2306]] ([[https://github.com/apache/tika/pull/163|Pull request on Github]]). Continue reading to get Tika up and running for object recognition.
This page describes a way to make Tika perform image recognition. Tika has many implementations of image recognition parsers. Specifically, this page provides information for an implementation powered by [[https://deeplearning4j.org/|Deeplearning4j]], InceptionNet-V3 model pre-trained on ImageNet dataset.
This model can detect a thousand different objects in the images.
Line 10: Line 9:
Currently, Tika utilises [[https://arxiv.org/abs/1602.07261|Inception-V4]] model from [[https://www.tensorflow.org/|Tensorflow]] for recognizing objects in the JPEG images. The advantage of this particular setting is, this implementation runs inside a Java Virtual Machine (JVM) stack without dependence on any external services.
So it is perfect for the users who are trying to run image recognition on a distributed setup like Apache Hadoop or Apache Spark.
Line 13: Line 13:
== Tika and Tensorflow Image Recognition ==

Tika has two different ways of bindings to Tensorflow:
 1. Using Commandline Invocation -- Recommended for quick testing, not for production use
 2. Using REST API -- Recommended for production use

=== 1. Tensorflow Using Commandline Invocation ===
'''Pros of this approach:'''
  This parser is easy to setup and test
'''Cons:'''
  Very inefficient/slow as it loads and unloads model for every parse call
Note:
 1. This is a work in progress. This feature was added in Tika 1.15
 2. At the time of writing, Tika 1.15 was not released. You have to [clone Tika repository](https://github.com/apache/tika) and do '''mvn clean install'''.
 3. The rest of the page uses version '''1.15-SNAPSHOT''', however, if you are reading this after release, please use '''1.15''' or newer version.
Line 26: Line 19:
==== Step 1. Install the dependencies ====
To install tensorflow, follow the instructions on [[https://www.tensorflow.org/install/|the official site here]] for your environment.
Unless you know what you are doing, you are recommended to follow pip installation.
= Java/Groovy/Scala example =
Line 30: Line 21:
Then clone the repository [[https://github.com/tensorflow/models|tensorflow/models]] or download the [[https://github.com/tensorflow/models/archive/master.zip|zip file]].
  {{{git clone https://github.com/tensorflow/models.git}}}
For maven users:
Add Tika-parsers and tika-dl to your project
Line 33: Line 24:
Add 'models/slim' folder to the environment variable, PYTHONPATH. '''Here is an example for Apache Maven users:'''
Line 35: Line 26:
   {{{$ export PYTHONPATH="$PYTHONPATH:/path/to/models/slim"}}} {{{#!highlight xml
Line 37: Line 28:
To test the readiness of your environment :     <dependencies>
        <dependency>
            <groupId>org.apache.tika</groupId>
            <artifactId>tika-parsers</artifactId>
            <version>1.15-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.apache.tika</groupId>
            <artifactId>tika-dl</artifactId>
            <version>1.15-SNAPSHOT</version>
        </dependency>
    </dependencies>
}}}
Line 39: Line 42:
  {{{$ python -c 'import tensorflow, numpy, datasets; print("OK")'}}} '''A configuration, tika-config.xml, to activate image recognition model:'''
Line 41: Line 44:
If the above command prints the message "OK", then the requirements are satisfied.

==== Step 2. Create a Tika-Config XML to enable Tensorflow parser. ====
 A sample config can be found in Tika source code at [[https://raw.githubusercontent.com/apache/tika/master/tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow.xml|tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow.xml]]

'''Here is an example:'''
Line 49: Line 46:
    <parsers>
    
<parser class="org.apache.tika.parser.recognition.ObjectRecognitionParser">
            <mime>image/jpeg</mime>
            <params>
        
<param name="topN" type="int">2</param>
        
<param name="minConfidence" type="double">0.015</param>
                <param name="class" type="string">org.apache.tika.parser.recognition.tf.TensorflowImageRecParser</param>
            </params>
        </parser>
    </parsers>
  <parsers>
<parser class="org.apache.tika.parser.recognition.ObjectRecognitionParser">
      <mime>image/jpeg</mime>
      <params>
<param name="topN" type="int">10</param>
<param name="minConfidence" type="double">0.015</param>
        <param name="class" type="string">org.apache.tika.dl.imagerec.DL4JInceptionV3Net</param>
      </params>
    </parser>
  </parsers>
Line 61: Line 58:
Note: Refer to a later section for customizing the config.
Line 62: Line 60:
'''Description of parameters :'''
{{{#!csv
  Param Name, Type, Meaning, Range, Example
  topN, int, Number of object names to output, a non-zero positive integer, 1 to receive top 1 object name
  minConfidence, double, Minimum confidence required to output the name of detected objects, [0.0 to 1.0] inclusive, 0.9 for outputting object names iff at least 90% confident
  class, string, Class that implements object recognition functionality, constant string, org.apache.tika.parser.recognition.tf.TensorflowImageRecParser

'''Sample Java code:'''

{{{#!highlight java
    //create parser as per desired parser
    TikaConfig config;
    try (InputStream stream = ImageRecLocal.class.getClassLoader()
            .getResourceAsStream("tika-config.xml")){
        config = new TikaConfig(stream);
    }
    Tika parser = new Tika(config);
    //sample file
    File imageFile = new File("data/gun.jpg");
    Metadata meta = new Metadata();
    parser.parse(imageFile, meta);
    //retrieve objects from the metadata
    System.out.println(Arrays.toString(meta.getValues("OBJECT")));
    // This should print: [assault_rifle (0.78214), rifle (0.18048), revolver (0.02780)]
Line 70: Line 80:
= For Python and Tika Server users =
Line 71: Line 82:
==== Step 3: Demo ====
To use the vision capability via Tensorflow, just supply the above configuration to Tika.
 1. Create file with name '''tika-config.xml''' by using the content shown above.
 2. Clone tika repository and build the tika project
 3. Add '''tika-dl''' to classpath of tika server
Line 75: Line 87:
For example, to use in Tika App (Assuming you have ''tika-app'' JAR and it is ready to run):

{{{#!bash
      $ java -jar tika-app/target/tika-app-1.15-SNAPSHOT.jar \
         --config=tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow.xml \
         https://upload.wikimedia.org/wikipedia/commons/f/f6/Working_Dogs%2C_Handlers_Share_Special_Bond_DVIDS124942.jpg
 }}}

The input image is:

{{https://upload.wikimedia.org/wikipedia/commons/f/f6/Working_Dogs%2C_Handlers_Share_Special_Bond_DVIDS124942.jpg|Germal Shepherd with Military}}

And, the top 2 detections are:
{{{#!highlight xml
...
<meta name="OBJECT" content="German shepherd, German shepherd dog, German police dog, alsatian (0.78435)"/>
<meta name="OBJECT" content="military uniform (0.06694)"/>
...
{{{#!highlight bash
alias tika="java -cp tika-server/target/tika-server-1.15-SNAPSHOT.jar:tika-dl/target/tika-dl-1.15-SNAPSHOT-jar-with-dependencies.jar org.apache.tika.server.TikaServerCli "
tika --config=tika-config.xml # Starts server port 9998
Line 95: Line 92:

=== 2. Tensorflow Using REST Server ===
This is the recommended way for utilizing visual recognition capability of Tika.
This approach uses Tensorflow over REST API.
To get this working, we are going to start a python flask based REST API server and tell tika to connect to it.
All these dependencies and setup complexities are isolated in docker image.
Refer to [[ https://github.com/chrismattmann/tika-python | Tika-python]] for an example usage.
Line 103: Line 95:
Requirements :
  Docker -- Visit [[https://www.docker.com/| Docker.com]] and install latest version of Docker. (Note: tested on docker v17.03.1)
= Large Scale Image Recognition In Spark =
 
Coming soon! It is being tested here https://github.com/thammegowda/tika-dl4j-spark-imgrec
Line 106: Line 99:
==== Step 1. Setup REST Server ====
You can either start the REST server in an isolated docker container or natively on the host that runs tensorflow.
Line 109: Line 100:
===== a. Using docker (Recommended) =====
{{{#!highlight bash
= Configuration options =
Line 112: Line 102:
cd tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/
# alternatively, if you do not have tika's source code, you may simply wget the 'InceptionRestDockerfile' from github link
docker build -f InceptionRestDockerfile -t inception-rest-tika .
docker run -p 8764:8764 -it inception-rest-tika
Previously, we have used the three parameters to this parser:
{{{#!highlight xml
<params>
    <param name="topN" type="int">10</param>
    <param name="minConfidence" type="double">0.015</param>
    <param name="class" type="string">org.apache.tika.dl.imagerec.DL4JInceptionV3Net</param>
</params>
Line 118: Line 111:
Once it is done, test the setup by visiting [[http://localhost:8764/inception/v4/classify?topk=2&url=https://upload.wikimedia.org/wikipedia/commons/f/f6/Working_Dogs%2C_Handlers_Share_Special_Bond_DVIDS124942.jpg]] in your web browser.

'''Sample output from API:'''
{{{#!json
{
   "confidence":[
      0.7843596339225769,
      0.06694009155035019
   ],
   "classnames":[
      "German shepherd, German shepherd dog, German police dog, alsatian",
      "military uniform"
   ],
   "classids":[
      236,
      653
   ],
   "time":{
      "read":7403,
      "units":"ms",
      "classification":470
   }
}
The other important parameters are:
{{{#!highlight xml
    <param name="modelWeightsPath" type="string">VALUE</param>
    <param name="modelJsonPath" type="string">VALUE</param>
Line 143: Line 117:
Note: MAC USERS:
      If you are using an older version, say, 'Docker toolbox' instead of the newer 'Docker for Mac',
you need to add port forwarding rules in your Virtual Box default machine.
The VALUE string can be:
 1. File path relative to class loader (note that these could also be inside jar files)
 2. Absolute file path anywhere on the local filesystem
 3. Any remote URL, including HTTP or HTTPS
Line 147: Line 122:
  1. Open the Virtual Box Manager.
  2. Select your Docker Machine Virtual Box image.
  3. Open Settings -> Network -> Advanced -> Port Forwarding.
  4. Add an appname,Host IP 127.0.0.1 and set both ports to 8764.
For example:
 1. {{{<param name="modelWeightsPath" type="string">inception-model-weights.h5</param>}}}
 2. {{{<param name="modelWeightsPath" type="string">/usr/share/apache-tika/models/tikainception-model-weights.h5</param>}}}
 3. {{{<param name="modelWeightsPath" type="string">https://myserver.com/files/apache-tika/models/tikainception-model-weights.h5</param>}}}
-----
Line 152: Line 128:
===== b. Without Using docker =====
If you chose to setup REST server without a docker container, you are free to manually install all the required tools specified in the [[ https://github.com/apache/tika/blob/master/tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/InceptionRestDockerfile | docker file]].

Note: docker file has setup instructions for Ubuntu, you will have to transform those commands for your environment.

{{{#!highlight bash
   python tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/inceptionapi.py --port 8764
}}}

==== Step 2. Create a Tika-Config XML to enable Tensorflow parser. ====
 A sample config can be found in Tika source code at [[https://github.com/apache/tika/blob/da82df5e9def9698fd32f85fe706660641d7c31f/tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-rest.xml|tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-rest.xml]]

'''Here is an example:'''
{{{#!xml
<properties>
    <parsers>
        <parser class="org.apache.tika.parser.recognition.ObjectRecognitionParser">
            <mime>image/jpeg</mime>
            <params>
                <param name="topN" type="int">2</param>
                <param name="minConfidence" type="double">0.015</param>
                <param name="class" type="string">org.apache.tika.parser.recognition.tf.TensorflowRESTRecogniser</param>
            </params>
        </parser>
    </parsers>
</properties>
}}}

'''Description of parameters :'''
{{{#!csv
  Param Name, Type, Meaning, Range, Example
  topN, int, Number of object names to output, a non-zero positive integer, 1 to receive top 1 object name
  minConfidence, double, Minimum confidence required to output the name of detected objects, [0.0 to 1.0] inclusive, 0.9 for outputting object names iff at least 90% confident
  class, string, Name of class that Implements Object recognition Contract, constant string, org.apache.tika.parser.recognition.tf.TensorflowRESTRecogniser
  healthUri, URI, HTTP URL to check availability of API service, any HTTP URL that gets 200 status code when available, http://localhost:8764/inception/v4/ping
  apiUri, URI, HTTP URL to POST image data, any HTTP URL that returns data in the JSON format as shown in the sample API output, http://localhost:8764/inception/v4/classify?topk=10
}}}


==== Step 3. Demo ====
This demo is same as the Commandline Invocation approach, but this is faster and efficient

{{{#!bash
   $ java -jar tika-app/target/tika-app-1.15-SNAPSHOT.jar \
    --config=tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-rest.xml \
    https://upload.wikimedia.org/wikipedia/commons/f/f6/Working_Dogs%2C_Handlers_Share_Special_Bond_DVIDS124942.jpg
}}}

The input image is:

{{https://upload.wikimedia.org/wikipedia/commons/f/f6/Working_Dogs%2C_Handlers_Share_Special_Bond_DVIDS124942.jpg|Germal Shepherd with Military}}

And, the top 2 detections are:
{{{#!highlight xml
...
<meta name="OBJECT" content="German shepherd, German shepherd dog, German police dog, alsatian (0.78435)"/>
<meta name="OBJECT" content="military uniform (0.06694)"/>
...
}}}

==== Changing the default topN, API port or URL ====
To change the defaults, update the parameters in config XML file accordingly

'''Here is an example scenario:'''

Run REST API on port 3030, and get top 4 object names if the confidence is above 10%. You may also change host to something else than 'localhost' if required.

'''Example Config File'''
{{{#!xml
<properties>
    <parsers>
        <parser class="org.apache.tika.parser.recognition.ObjectRecognitionParser">
            <mime>image/jpeg</mime>
            <params>
                <param name="topN" type="int">4</param>
                <param name="minConfidence" type="double">0.1</param>
                <param name="class" type="string">org.apache.tika.parser.recognition.tf.TensorflowRESTRecogniser</param>
         <param name="healthUri" type="uri">http://localhost:3030/inception/v4/ping</param>
         <param name="apiUri" type="uri">http://localhost:3030/inception/v4/classify?topk=4</param>
            </params>
        </parser>
    </parsers>
</properties>
}}}

'''To Start the service on port 3030:'''

Using Docker:

  {{{docker run -it -p 3030:8764 inception-rest-tika}}}


Without Using Docker:

{{{python tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/inceptionapi.py --port 3030}}}


----

=== Questions / Suggestions / Improvements / Feedback ? ===
= Questions / Suggestions / Improvements / Feedback ? =

Tika and Computer Vision powered by Deeplearning4j(DL4J)

This page describes a way to make Tika perform image recognition. Tika has many implementations of image recognition parsers. Specifically, this page provides information for an implementation powered by Deeplearning4j, InceptionNet-V3 model pre-trained on ImageNet dataset. This model can detect a thousand different objects in the images.

The advantage of this particular setting is, this implementation runs inside a Java Virtual Machine (JVM) stack without dependence on any external services. So it is perfect for the users who are trying to run image recognition on a distributed setup like Apache Hadoop or Apache Spark.

Note:

  1. This is a work in progress. This feature was added in Tika 1.15
  2. At the time of writing, Tika 1.15 was not released. You have to [clone Tika repository](https://github.com/apache/tika) and do mvn clean install.

  3. The rest of the page uses version 1.15-SNAPSHOT, however, if you are reading this after release, please use 1.15 or newer version.

Java/Groovy/Scala example

For maven users: Add Tika-parsers and tika-dl to your project

Here is an example for Apache Maven users:

   1     <dependencies>
   2         <dependency>
   3             <groupId>org.apache.tika</groupId>
   4             <artifactId>tika-parsers</artifactId>
   5             <version>1.15-SNAPSHOT</version>
   6         </dependency>
   7         <dependency>
   8             <groupId>org.apache.tika</groupId>
   9             <artifactId>tika-dl</artifactId>
  10             <version>1.15-SNAPSHOT</version>
  11         </dependency>
  12     </dependencies>

A configuration, tika-config.xml, to activate image recognition model:

   1 <properties>
   2   <parsers>
   3     <parser class="org.apache.tika.parser.recognition.ObjectRecognitionParser">
   4       <mime>image/jpeg</mime>
   5       <params>
   6         <param name="topN" type="int">10</param>
   7         <param name="minConfidence" type="double">0.015</param>
   8         <param name="class" type="string">org.apache.tika.dl.imagerec.DL4JInceptionV3Net</param>
   9       </params>
  10     </parser>
  11   </parsers>
  12 </properties>

Note: Refer to a later section for customizing the config.

Sample Java code:

   1     //create parser as per desired parser
   2     TikaConfig config;
   3     try (InputStream stream = ImageRecLocal.class.getClassLoader()
   4             .getResourceAsStream("tika-config.xml")){
   5         config = new TikaConfig(stream);
   6     }
   7     Tika parser = new Tika(config);
   8     //sample file
   9     File imageFile = new File("data/gun.jpg");
  10     Metadata meta = new Metadata();
  11     parser.parse(imageFile, meta);
  12     //retrieve objects from the metadata
  13     System.out.println(Arrays.toString(meta.getValues("OBJECT")));
  14     // This should print: [assault_rifle (0.78214), rifle (0.18048), revolver (0.02780)]
  15 

For Python and Tika Server users

  1. Create file with name tika-config.xml by using the content shown above.

  2. Clone tika repository and build the tika project
  3. Add tika-dl to classpath of tika server

   1 alias tika="java -cp tika-server/target/tika-server-1.15-SNAPSHOT.jar:tika-dl/target/tika-dl-1.15-SNAPSHOT-jar-with-dependencies.jar org.apache.tika.server.TikaServerCli "
   2 tika --config=tika-config.xml # Starts server port 9998
   3 

Refer to Tika-python for an example usage.

Large Scale Image Recognition In Spark

Coming soon! It is being tested here https://github.com/thammegowda/tika-dl4j-spark-imgrec

Configuration options

Previously, we have used the three parameters to this parser:

   1 <params>
   2     <param name="topN" type="int">10</param>
   3     <param name="minConfidence" type="double">0.015</param>
   4     <param name="class" type="string">org.apache.tika.dl.imagerec.DL4JInceptionV3Net</param>
   5 </params>

The other important parameters are:

   1     <param name="modelWeightsPath" type="string">VALUE</param>
   2     <param name="modelJsonPath" type="string">VALUE</param>

The VALUE string can be:

  1. File path relative to class loader (note that these could also be inside jar files)
  2. Absolute file path anywhere on the local filesystem
  3. Any remote URL, including HTTP or HTTPS

For example:

  1. <param name="modelWeightsPath" type="string">inception-model-weights.h5</param>

  2. <param name="modelWeightsPath" type="string">/usr/share/apache-tika/models/tikainception-model-weights.h5</param>

  3. <param name="modelWeightsPath" type="string">https://myserver.com/files/apache-tika/models/tikainception-model-weights.h5</param>


Questions / Suggestions / Improvements / Feedback ?

  1. If it was useful, let us know on twitter by mentioning @ApacheTika

  2. If you have questions, let us know by using Mailing Lists

  3. If you find any bugs, use Jira to report them

TikaAndVisionDL4J (last edited 2017-07-06 03:20:06 by ChrisMattmann)