GeoTopicParser

The GeoTopicParser combines a Gazetteer (a lookup dictionary of names/places to latitudes, longitudes) and a Named Entity Recognition (NER) modeling technique that identifies names and places in text to provide a way to geo tag documents and text i.e., to identify places in the text, and then to look up the latitude/longitude pairs for those places.

GeoTopicParser uses Geonames.org, Apache Lucene and Apache OpenNLP to provide its capabilities.

The work is based on this paper:

An Automatic Approach for Discovering and Geocoding Locations in Domain-Specific Web Data

In Proceedings of the IEEE International Conference on Information Reuse and Integration, Pittsburgh, Pennsylvania, USA, July 28-30, 2016 | Read this article

Authors: Chris A. Mattmann, Madhav Sharan

Installing the Lucene Gazetteer

First you will need to download the Lucene Geo Gazetteer project and to install it. You can do so by:

$ cd $HOME/src
$ git clone https://github.com/chrismattmann/lucene-geo-gazetteer.git
$ cd lucene-geo-gazetteer
$ mvn install assembly:assembly
$ add $HOME/src/lucene-geo-gazetteer/src/main/bin to your PATH environment variable

Once done, you can verify that the installation worked by running the following command:

$ lucene-geo-gazetteer --help
usage: lucene-geo-gazetteer
 -b,--build <gazetteer file>           The Path to the Geonames
                                       allCountries.txt
 -c,--count <number of results>        Number of best results to be
                                       returned for one location
 -h,--help                             Print this message.
 -i,--index <directoryPath>            The path to the Lucene index
                                       directory to either create or read
 -json,--json                          Formats output in well defined json
                                       structure
 -s,--search <set of location names>   Location names to search the
                                       Gazetteer for
 -server,--server                      Launches Geo Gazetteer Service

You will now need to build a Gazetteer using the Geonames.org dataset. Instructions are provided below. Note that you will need least 1.2 GB disk space for building Lucene Index for the Gazetteer.

$ cd $HOME/src/lucene-geo-gazetteer
$ curl -O http://download.geonames.org/export/dump/allCountries.zip
$ unzip allCountries.zip
$ lucene-geo-gazetteer -i geoIndex -b allCountries.txt

You can verify that the Gazetteer build worked by searching e.g., for Pasadena, and/or Texas:

$ lucene-geo-gazetteer -s Pasadena Texas -json
{
    "Pasadena": [
        {
            "admin1Code": "CA",
            "admin2Code": "037",
            "countryCode": "US",
            "latitude": 34.14778,
            "longitude": -118.14452,
            "name": "Pasadena"
        }
    ],
    "Texas": [
        {
            "admin1Code": "TX",
            "admin2Code": "",
            "countryCode": "US",
            "latitude": 31.25044,
            "longitude": -99.25061,
            "name": "Texas"
        }
    ]
}

Now you need to start REST service of lucene-geo-gazetteer. Tika uses this service internally

$ lucene-geo-gazetteer -server

You can verify that the REST API is responding by searching e.g., for Pasadena, and/or Texas:

$ curl "http://localhost:8765/api/search?s=Pasadena&s=Texas"
{
    "Pasadena": [
        {
            "admin1Code": "CA",
            "admin2Code": "037",
            "countryCode": "US",
            "latitude": 34.14778,
            "longitude": -118.14452,
            "name": "Pasadena"
        }
    ],
    "Texas": [
        {
            "admin1Code": "TX",
            "admin2Code": "",
            "countryCode": "US",
            "latitude": 31.25044,
            "longitude": -99.25061,
            "name": "Texas"
        }
    ]
}

Note that we used the convenience script lucene-geo-gazetteer which assumes that you created an indexed named geoIndex in the $HOME/src/lucene-geo-gazetter/geoIndex directory. We could have also used the pure Java command line to search. The return from the Gazetteer is a JSON List of Object structures in which the structure is a key->Object List map. The key is the location name given and the Object List is a list of most popular location objects in the Gazetteer for that name.

Installing and downloading an NER model

The next thing you'll need is a Named Entity Recognition model for places. The GeoTopicParser uses Apache OpenNLP and with its 1.5 version, OpenNLP provides already trained models for location names in text data. You can download the en-ner-location.bin file already pre-trained by the OpenNLP folks. One thing to note is that OpenNLP's default name finder is not accurate, so building your own NER location model is highly recommended. In this case, please follow these instructions.

The model needs to be placed on the classpath for your Tika installation in the following directory:

org/apache/tika/parser/geo/

The following instructions show how to download the model and place it on the right path:

$ mkdir $HOME/src/location-ner-model && cd $HOME/src/location-ner-model
$ curl -O https://opennlp.sourceforge.net/models-1.5/en-ner-location.bin
$ mkdir -p org/apache/tika/parser/geo/
$ mv en-ner-location.bin org/apache/tika/parser/geo/

Test out the GeoTopicParser

Now you can run Tika and try out the GeoTopicParser. At the moment since it's a Parser and not a Content-Handler (hopefully will develop it later), the parser is mapped to the MIME type application/geotopic which is a sub-class of text/plain. So, there are two steps to try the parser out now.

Create a .geot file, you can use this sample file from the NSF Polar data contributed to TREC. 2. Tell Tika about the application/geotopic MIME type. You can download this file and place it on the classpath in the org/apache/tika/mime directory, e.g., by doing:

$ mkdir $HOME/src/geotopic-mime && cd $HOME/src/geotopic-mime
$ mkdir -p org/apache/tika/mime
$ curl -O https://raw.githubusercontent.com/chrismattmann/geotopicparser-utils/master/mime/org/apache/tika/mime/custom-mimetypes.xml
$ mv custom-mimetypes.xml org/apache/tika/mime

With those files in place, let's use the GeoTopicParser using Tika-App:

$ java -classpath tika-app/target/tika-app-<LATEST-VERSION>-SNAPSHOT.jar:tika-parsers/tika-parsers-ml/tika-parser-nlp-package/target/tika-parser-nlp-package-<LATEST-VERSION>-SNAPSHOT.jar:$HOME/src/location-ner-model:$HOME/src/geotopic-mime org.apache.tika.cli.TikaCLI -m polar.geot

This should output:

Content-Length: 881
Content-Type: application/geotopic
Geographic_LATITUDE: 27.33931
Geographic_LONGITUDE: -108.60288
Geographic_NAME: China
Optional_LATITUDE1: 39.76
Optional_LONGITUDE1: -98.5
Optional_NAME1: United States
X-Parsed-By: org.apache.tika.parser.DefaultParser
X-Parsed-By: org.apache.tika.parser.geo.topic.GeoParser
resourceName: polar.geot

The output will output 3-tuples of {Name, Latitude, Longitude}. The *best* match for the location is the one that occurs most frequently in the text, and that is provided as Geographic_NAME, along with its corresponding Geographic_LATITUDE and Geographic_LONGITUDE. Places also identified as entities by the NER model in the provide text are also listed as Optional_NAME*N*, e.g., Optional_NAME1 for the 1st alternative location identified and its corresponding Optional_LATITUDE1 and Optional_LONGITUDE1.

Will this work from Tika Server?

It sure will! When you start Tika Server, make sure that the NER model file and the custom MIME type are on your classpath, and that the lucene-geo-gazetteer is on the $PATH where Tika-Server is started, and you can post all the .geot files that you'd like and Tika-Server will happily call the GeoTopicParser to provide you location information.

First, start up the Tika server with your NER model and .geot MIME type definition on the classpath:

java -classpath $HOME/src/location-ner-model:$HOME/src/geotopic-mime:tika-server/tika-server-standard/target/tika-server-standard-<LATEST-VERSION>-SNAPSHOT.jar:tika-parsers/tika-parsers-ml/tika-parser-nlp-package/target/tika-parser-nlp-package-<LATEST-VERSION>-SNAPSHOT.jar org.apache.tika.server.core.TikaServerCli

Then, try calling the /rmeta service to get the returned metadata:

curl -T $HOME/src/geotopicparser-utils/geotopics/polar.geot -H "Content-Disposition: attachment; filename=polar.geot" http://localhost:9998/rmeta

And then look for it to return the following, that's it!

[
   {
      "Content-Type":"application/geotopic",
      "Geographic_LATITUDE":"39.76",
      "Geographic_LONGITUDE":"-98.5",
      "Geographic_NAME":"United States",
      "Optional_LATITUDE1":"27.33931",
      "Optional_LONGITUDE1":"-108.60288",
      "Optional_NAME1":"China",
      "X-Parsed-By":[
         "org.apache.tika.parser.DefaultParser",
         "org.apache.tika.parser.geo.topic.GeoParser"
      ],
      "X-TIKA:parse_time_millis":"1634",
      "resourceName":"polar.geot"
   }
]

Page tree

GeoTopicParser

GeoTopicParser

An Automatic Approach for Discovering and Geocoding Locations in Domain-Specific Web Data

Installing the Lucene Gazetteer

Installing and downloading an NER model

Test out the GeoTopicParser

Will this work from Tika Server?