Supported parser types

mnoGoSearch supports four types of external parsers:

• read data from STDIN, send the result to STDOUT

• read data from a file, send the result to STDOUT

• read data from a file, send the result to another file

• read data from STDIN, send the result to a file

Setting up parsers

1. Configure Mime types

   Make sure your HTTP server sends correct Content-Type headers. For Apache, have a look at the file mime.types, the most common Mime types are already defined there.

   If you want to index local files or an FTP server, use the AddType command in indexer.conf to associate file name extensions with their Mime types. For example:

   AddType text/html *.html
   

2. Add parsers

   Add parser definition commands. These commands have the following format with three arguments:

   Mime <from_mime> <to_mime> <command line>
   

   For example, the following command defines a parser for man pages:

   # Use deroff for parsing man pages ( *.man )
   Mime  application/x-troff-man   text/plain   deroff
   

   This parser will take data from STDIN and output results to STDOUT.

   Some parsers can not operate on STDIN and require a file to read from. In this case indexer can create a temporary file in /tmp and remove the file when the parser is done. Use the $1 macro in the parser command line to substitute the temporary file name. For example, the Mime command for the catdoc MS-Word-to-text converter can look like this:

   Mime application/msword text/plain "/usr/bin/catdoc -a $1"
   

   If your parser writes the result into an output file, use the $2 macro. indexer will replace $2 with the output temporary file name, then start the parser, read the result from this temporary file and delete the file. For example:

   Mime application/msword text/plain "/usr/bin/catdoc -a $1 >$2"
   

   The parser above will read data from the first temporary file and write results to the second file. Both temporary files will be deleted after reading parser results. Note that this command is effectively the same with the previous example. They only differ in the execution method used by indexer: file-to-STDOUT versus file-to-file.

Preventing indexer from getting stuck on a parser execution

To prevent indexer from getting stuck on a parser execution you can specify the amount of time (in seconds) indexer waits for an external parser to return results. Use the ParserTimeOut indexer.conf for this purpose. For example:

ParserTimeOut 600

The default value is 300 seconds (5 minutes). If an external parser does not return results within this period of time, indexer will kill the parser process, remove the associated temporary files and continue with the next document in the queue.

Pipes in a parser command line

You can use pipes in a parser command line. For example, these lines will be useful to index gzipped man pages from the local disk:

AddType  application/x-gzipped-man  *.1.gz *.2.gz *.3.gz *.4.gz
Mime     application/x-gzipped-man  text/plain  "zcat | deroff"

Parsers and character sets

Some parsers can produce output in a character sets different from the one given in the LocalCharset command. You can specify the output character set in a parser command line to make indexer convert the parser output to LocalCharset. For example, if catdoc is configured to produce output in windows-1251 character set but LocalCharset is set to koi8-r, you can use this command for parsing MS Word documents:

Mime  application/msword  "text/plain; charset=windows-1251" "catdoc -a $1"

The UDM_URL environment variable

When executing a parser, indexer creates the UDM_URL environment variable with the document URL as a value. You can use this variable in the parser scripts.

External parsers for the most common file types

If you're using an external parser not listed here, please contribute your parser configuration to <general@mnogosearch.org>.