...
The Pig tutorial file (pigtutorial.tar.gz) or the tutorial/pigtutorial.tar.gz file in the pig distribution) includes the Pig JAR file (pig.jar) and the tutorial files (tutorial.jar, Pigs scripts, log files). These files work with Hadoop 0.18 and provide everything you need to run the Pig scripts. To get started, follow these basic steps:
...
- Download the Pig tutorial file to your local directory (pigtutorial.tar.gz)
- Unzip the Pig tutorial file (the files are stored in a newly created directory, pigtmp).
Code Block $ tar -xzf pigtutorial.tar.gz
- Move to the pigtmp directory.
- Review the contents of the Pig tutorial file.
- Copy the pig.jar file to the appropriate directory on your system. For example: /home/me/pig.
- Create an environment variable, PIGDIR, and point it to your directory. For example: export PIGDIR=/home/me/pig (bash, sh) or setenv PIGDIR /home/me/pig (tcsh, csh).
...
Pig Scripts: Local Mode
...
To run the Pig scripts in local mode, do the following:
- Move to the pigtmp directory.
- Review Pig Script 1 and Pig Script 2.
- Execute the following command (using either script1-local.pig or script2-local.pig).
Code Block $ java -cp $PIGDIR/pig.jar org.apache.pig.Main -x local script1-local.pig
- Review the result file (either script1-local-results.txt or script2-local-results.txt):
Code Block $ ls -l script1-local-results.txt $ cat script1-local-results.txt
Pig Scripts: Hadoop Mode
To run the Pig scripts in hadoop (mapreduce) mode, do the following:
- Move to the pigtmp directory.
- Review Pig Script 1 and Pig_Script_2.
- Copy the excite.log.bz2 file from the pigtmp directory to the HDFS directory.
Code Block $ hadoop fs ‚-copyFromLocal excite.log.bz2 . - Set the HADOOPSITEPATH environment variable to the location of your hadoop-site.xml file.
- Execute the following command (using either script1-hadoop.pig or script2-hadoop.pig):
1.#6 Review the result files (located in either the script1-hadoop-results or script2-hadoop-results HDFS directory):Code Block $ java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH org.apache.pig.Main script1-hadoop.pig
| Code Block |
|---|
$ hadoop fs -ls script1-hadoop-results $ hadoop fs -cat 'script1-hadoop-results/*' | less |
<<Anchor(
| Anchor | |||
|---|---|---|---|
|
|
Pig Tutorial File
The contents of the Pig tutorial file (pigtutorial.tar.gz) are described here.
File | Description |
|---|---|
pig.jar | Pig JAR file |
tutorial.jar | User-defined functions (UDFs) and Java classes |
script1-local.pig | Pig Script 1, Query Phrase Popularity (local mode) |
script1-hadoop.pig | Pig Script 1, Query Phrase Popularity (Hadoop cluster) |
script2-local.pig | Pig Script 2, Temporal Query Phrase Popularity (local mode) |
script2-hadoop.pig | Pig Script 2, Temporal Query Phrase Popularity (Hadoop cluster) |
excite-small.log | Log file, Excite search engine (local mode) |
excite.log.bz2 | Log file, Excite search engine (Hadoop cluster) |
A better-documented version of script1-local.pig can be found at https://cwiki.apache.org/confluence/download/attachments/27822259/script1-local-with-added-documentation.pig . It includes comments showing samples from each intermediate relation.
The user-defined functions (UDFs) are described here.
UDF | Description |
|---|---|
! ExtractHour | Extracts the hour from the record. |
N!GramGenerator NGramGenerator | Composes n-grams from the set of words. |
NonURLDetector | Removes the record if the query field is empty or a URL. |
! ScoreGenerator | Calculates a "popularity" score for the n-gram. |
! ToLower | Changes the query field to lowercase. |
! TutorialUtil | Divides the query string into a set of words. |
...
- Call the NonURLDetector UDF to remove records if the query field is empty or a URL.
Code Block clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
- Call the ! ToLower UDF to change the query field to lowercase.
Code Block clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;
- Because the log file only contains queries for a single day, we are only interested in the hour. The excite query log timestamp format is YYMMDDHHMMSS. Call the ! ExtractHour UDF to extract the hour (HH) from the time field.
Code Block houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query;
- Call the N!GramGenerator NGramGenerator UDF to compose the n-grams of the query.
Code Block ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;
...
- For each group, identify the hour in which this n-gram is used with a particularly high frequency. Call the ! ScoreGenerator UDF to calculate a "popularity" score for the n-gram.
Code Block uniq_frequency2 = FOREACH uniq_frequency1 GENERATE flatten($0), flatten(org.apache.pig.tutorial.ScoreGenerator($1));
...
- Call the NonURLDetector UDF to remove records if the query field is empty or a URL.
Code Block clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
- Call the ! ToLower UDF to change the query field to lowercase.
Code Block clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;
- Because the log file only contains queries for a single day, we are only interested in the hour. The excite query log timestamp format is YYMMDDHHMMSS. Call the ! ExtractHour UDF to extract the hour from the time field.
Code Block houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query;
- Call the N!GramGenerator NGramGenerator UDF to compose the n-grams of the query.
Code Block ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;
...