The Pooled Time Series algorithm was developed by Michael Ryoo and it allows for video descriptors to be considered over time and in this consideration for videos to be compared based on the activity going on in their scenes. In short, Pooled Time Series is a video comparison metric. An ALv2 licensed version of the Pooled Time Series code is available for use in computing Histogram of Oriented Gradients (HOG) and Histogram of Optical Flows (HOF) which can be useful extracted data and metadata for a Tika Parser.

Metadata Representation

The ultimate goal of the project is to be able to extract metadata and data from videos and to index that information inside of a searh engine like Apache Solr. Videos, like images, are just numbers - or a ordered sequence of number - or matrices. There are many ways in which these numbers can be defined. Some popular visual descriptors are Histogram of Gradients, Optical Flow vectors, RGB or Color Histograms. The challenge is to figure out a way to map this datatype to a datatype that can be understood by Solr. In the case of color based histograms, we can convert the image into a matrix of hex values, where each hex value is the pixel color value and index that as a text_ws field in Solr.

Some Related Efforts

ShutterStock developed an image search tool using a similar approach.

Representation of output data

The data output from the Pooled Time Series parser is an XHTML document of table values, where each <tr>..</tr> would be a row of the feature matrix and <td> would be the corresponding element in that column. When using a search engine like Apache Solr to do ranking or querying we can to compute a distance function on these values (for the dataset and the query video), such as Chi-Squared, which is what the pooled time series algorithm does.

A Tika Parser has been developed that implements the Pooled Time Series algorithm above and that outputs the HOF and HOG data from videos for use in later processing and indexing. Read on below to install and use it!

Pre-requisites

Install Pooled Time Series

mkdir -p $HOME/git && cd $HOME/git && git clone https://github.com/USCDataScience/hadoop-pot.git 2. cd hadoop-pot/hadoop-pot-assembly && mvn install assembly:assembly 3. Follow steps 3, 4 and 5 from the install guide from Pooled Time Series and confirm that pooled_time_series installed correctly. Note the pre-requisites from Pooled Time Series require you to install OpenCV and set some environment variables.

After above steps you must be able to execute pooled_time_series through terminal and get below output

usage: pooled_time_series
 -d,--dir <directory>            A directory with image files in it
 -f,--file <file>                Path to a single file
 -h,--help                       Print this message.
 -j,--json                       Set similarity output format to JSON.
                                 Defaults to .txt
 -o,--outputfile <output file>   File containing similarity results.
                                 Defaults to ./similarity.txt
 -p,--pathfile <path file>       A file containing full absolute paths to
                                 videos. Previous default was
                                 memex-index_temp.txt

Run PooledTimeSeries Parser from Tika-App

Grab an MP4, QuickTime, or other video (supported by your OpenCV implementation). Then run the following command:

java -classpath tika-app/target/tika-app-X.Y.jar org.apache.tika.cli.TikaCLI --config=$HOME/git/pooled_time_series/src/main/resources/tika-config.xml -J yourfile.mov

which should output something like:

[
    {
        "Content-Length": "1926487",
        "Content-Type": "video/quicktime",
        "X-Parsed-By": [
            "org.apache.tika.parser.CompositeParser",
            "org.apache.tika.parser.pot.PooledTimeSeriesParser"
        ],
        "X-TIKA:content": "<html xmlns=\"http://www.w3.org/1999/xhtml\">\n<head>\n<meta name=\"of_frames\" content=\"358\" />\n<meta name=\"Content-Length\" content=\"1926487\" />\n<meta name=\"of_vecSize\" content=\"200\" />\n<meta name=\"og_frames\" content=\"358\" />\n<meta name=\"X-Parsed-By\" content=\"org.apache.tika.parser.CompositeParser\" />\n<meta name=\"X-Parsed-By\" content=\"org.apache.tika.parser.pot.PooledTimeSeriesParser\" />\n<meta name=\"Content-Type\" content=\"video/quicktime\" />\n<meta name=\"resourceName\" content=\"yourfile.mov\" />\n<meta name=\"og_vecSize\" content=\"200\" />\n<title></title>\n</head>\n<body><h3>Histogram of Optical Flows (HOF)</h3>\n<table =\"358\" =\"200\"><tr>\t<td>0</td>\t<td>106.000000</td>\t<td>77.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>95.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>13.000000</td>\t<td>92.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>3.000000</td>\t<td>3.000000</td>\t<td>102.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>65.000000</td>\t<td>20.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>5.000000</td>\t<td>92.000000</td>\t<td>6.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>6.000000</td>\t<td>14.000000</td>\t<td>1.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>5.000000</td>\t<td>12.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>2.000000</td>\t<td>46.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>2.000000</td>\t<td>26.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>4.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>78.000000</td>\t<td>30.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>13.000000</td>\t<td>1.000000</td>\t<td>2.000000</td></tr>\n<tr>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>3.000000</td>\t<td>40.000000</td>\t<td>9.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>1.000000</td>\t<td>1.000000</td>\t<td>2.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>19.000000</td>\t<td>17.000000</td>\t<td>15.000000</td>\t<td>18.000000</td>\t<td>4.000000</td>\t<td>14.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>1.000000</td>\t<td>4.000000</td>\t<td>2.000000</td>\t<td>1.000000</td>\t<td>12.000000</td>\t<td>35.000000</td>\t<td>18.000000</td>\t<td>1.000000</td>\t<td>7.000000</td>\t<td>36.000000</td>\t<td>3.000000</td>\t<td>0.000000</td>\t<td>1.000000</td>\t<td>11.000000</td>\t<td>4.000000</td>\t<td>3.000000</td>\t<td>14.000000</td>\t<td>98.872983</td>\t<td>58.159994</td>\t<td>55.624418</td>\t<td>109.918199</td>\t<td>154.628142</td>\t<td>274.379973</td>\t<td>260.610341</td>\t<td>216.563245</td>\t<td>161.240045</td>\t<td>110.679989</td>\t<td>101.618012</td>\t<td>179.517287</td>\t<td>242.615251</td>\t<td>135.289986</td>\t<td>131.035653</td>\t<td>120.141349</td>\t<td>108.120592</td>\t<td>110.649989</td>\t<td>109.483038</td>\t<td>121.264360</td>\t<td>129.601505</td></tr>\n</table>\n</body></html>",
        "X-TIKA:parse_time_millis": "32054",
        "of_frames": "358",
        "of_vecSize": "200",
        "og_frames": "358",
        "og_vecSize": "200",
        "resourceName": "yourfile.mov"
    }
]

Will this work from Tika Server?

Yes, it will! Start Tika server like so:

java -jar tika-server/target/tika-server-X.Y.jar --config=$HOME/git/pooled_time_series/src/main/resources/tika-config.xml

Then send your movie file to Tika server like so:

curl -T yourfile.mov http://localhost:9998/rmeta

which should produce in response:

[
    {
        "Content-Type": "video/quicktime",
        "X-Parsed-By": [
            "org.apache.tika.parser.CompositeParser",
            "org.apache.tika.parser.pot.PooledTimeSeriesParser"
        ],
        "X-TIKA:content": "<html xmlns=\"http://www.w3.org/1999/xhtml\">\n<head>\n<meta name=\"of_frames\" content=\"358\" />\n<meta name=\"of_vecSize\" content=\"200\" />\n<meta name=\"og_frames\" content=\"358\" />\n<meta name=\"X-Parsed-By\" content=\"org.apache.tika.parser.CompositeParser\" />\n<meta name=\"X-Parsed-By\" content=\"org.apache.tika.parser.pot.PooledTimeSeriesParser\" />\n<meta name=\"Content-Type\" content=\"video/quicktime\" />\n<meta name=\"og_vecSize\" content=\"200\" />\n<title></title>\n</head>\n<body><h3>Histogram of Optical Flows (HOF)</h3>\n<table =\"358\" =\"200\"><tr>\t<td>0</td>\t<td>106.000000</td>\t<td>77.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>95.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>13.000000</td>\t<td>92.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>3.000000</td>\t<td>3.000000</td>\t<td>102.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>65.000000</td>\t<td>20.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>5.000000</td>\t<td>92.000000</td>\t<td>6.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>6.000000</td>\t<td>14.000000</td>\t<td>1.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>5.000000</td>\t<td>12.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>2.000000</td>\t<td>46.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>2.000000</td>\t<td>26.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>4.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>78.000000</td>\t<td>30.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>13.000000</td>\t<td>1.000000</td>\t<td>2.000000</td></tr>\n<tr>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>3.000000</td>\t<td>40.000000</td>\t<td>9.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>0.000000</td>\t<td>245.873308</td>\t<td>153.544600</td>\t<td>103.059990</td>\t<td>162.541708</td>\t<td>252.468921</td>\t<td>319.609416</td>\t<td>292.059971</td>\t<td>274.583075</td>\t<td>228.970014</td>\t<td>164.473069</td>\t<td>108.219989</td>\t<td>146.701403</td>\t<td>219.940923</td>\t<td>271.829481</td>\t<td>268.119973</td>\t<td>239.719107</td>\t<td>189.607921</td>\t<td>144.830008</td>\t<td>121.639988</td>\t<td>152.505474</td>\t<td>213.557055</td>\t<td>255.007528</td>\t<td>70.309993</td>\t<td>65.385228</td>\t<td>70.524468</td>\t<td>86.334997</td>\t<td>99.789990</td>\t<td>99.137078</td>\t<td>91.163868</td>\t<td>79.987551</td>\t<td>77.109992</td>\t<td>72.579427</td>\t<td>70.198524</td>\t<td>69.390141</td>\t<td>73.559993</td>\t<td>72.062835</td>\t<td>71.696460</td>\t<td>73.634280</td>\t<td>94.779991</td>\t<td>85.952229</td>\t<td>70.536712</td>\t<td>53.734593</td>\t<td>51.209995</td>\t<td>59.868269</td>\t<td>76.378587</td>\t<td>89.771828</td>\t<td>114.999988</td>\t<td>106.977985</td>\t<td>89.542021</td>\t<td>67.454518</td>\t<td>51.209995</td>\t<td>55.368092</td>\t<td>81.877135</td>\t<td>104.533636</td>\t<td>177.449982</td>\t<td>170.156562</td>\t<td>141.561883</td>\t<td>98.872983</td>\t<td>58.159994</td>\t<td>55.624418</td>\t<td>109.918199</td>\t<td>154.628142</td>\t<td>274.379973</td>\t<td>260.610341</td>\t<td>216.563245</td>\t<td>161.240045</td>\t<td>110.679989</td>\t<td>101.618012</td>\t<td>179.517287</td>\t<td>242.615251</td>\t<td>135.289986</td>\t<td>131.035653</td>\t<td>120.141349</td>\t<td>108.120592</td>\t<td>110.649989</td>\t<td>109.483038</td>\t<td>121.264360</td>\t<td>129.601505</td></tr>\n</table>\n</body></html>",
        "X-TIKA:parse_time_millis": "34291",
        "of_frames": "358",
        "of_vecSize": "200",
        "og_frames": "358",
        "og_vecSize": "200"
    }
]

Page tree

PooledTimeSeriesParser