Frequently Asked Questions

ht://Dig Copyright © 1995-2001 The ht://Dig Group
Please see the file COPYING for license information.

This FAQ is compiled by the ht://Dig developers and the most recent version is available at <http://htdig.sourceforge.net/FAQ.html>. Questions (and answers!) are greatly appreciated. Please send questions and/or answers to the ht://Dig user mailing list at: <htdig-general@lists.sourceforge.net>.

Questions

1. General

2. Getting ht://Dig

3. Compiling

4. Configuration

5. Troubleshooting

Answers

1. General

No, ht://Dig is a system for indexing and searching a finite (not necessarily small) set of sites or intranet. It is not meant to replace any of the many internet-wide search engines.

No, as above, ht://Dig is not meant as an internet-wide search engine. While there is theoretically nothing to stop you from indexing as much as you wish, practical considerations (e.g. time, disk space, memory, etc.) will limit this.

The complete ht://Dig package consists of several programs, one of which is called "htdig." This program performs the "digging" or indexing of the web pages. Of course an index doesn't do you much good without a program to sort it, search through it, etc.

Andrew no longer does much work on ht://Dig. He has started a company, called Contigo Software and is quite busy with that. To contact any of the current developers, send mail to <htdig-dev>. This list is intended primarily for the discussion of current and future development of the software.

Geoff and Gilles are currently the maintainers of ht://Dig, but they are both volunteers. So while they do read all the e-mail they receive, they may not respond immediately. Questions about ht://Dig in general, and especially questions or requests for help in configuring the software, should be posted to the <htdig-general> mailing list. When posting a followup to a message on the list, you should use the "reply to all" or "group reply" feature of your mail program, to make sure the mailing list address is included in the reply, rather than replying only to the author of the message. See also question 1.16.

Development of ht://Dig is done by volunteers. Since we all have other jobs, it make take a while before someone gets back to you. Please be patient and don't hound the volunteers with direct or repeated requests. If you don't get a response after 3 or 4 days, then a reminder may help. See also question 1.16.

Great! Development of ht://Dig continues through suggestions and improvements from users. If you have an idea (or even better, a patch), please send it to the ht://Dig mailing list so others can use it. For suggestions on how to submit patches, please check the Guidelines for Patch Submissions. If you'd like to make a feature request, you can do so through the ht://Dig bug database

ht://Dig should be y2k compliant since it never stores dates as two-digit years. Under ht://Dig's copyright (GPL), there is no warranty whatsoever as permitted by law. If you would like an iron-clad, legally-binding guarantee, feel free to check the source code itself. Versions prior to 3.1.2 did have a problem with the parsing of the Last-Modified header returned by the HTTP server, which will cause incorrect dates to be stored for documents modified after February 28, 2000 (yes, it didn't recognize 2000 as a leap year). Versions prior to 3.1.5 didn't correctly handle servers that return two digit years in the Last-Modified header, for years after 99. These problems are fixed in the current release. If you discover something else, please let us know!

Well, there are probably bugs out there. You have two options for bug-reporting. You can either mail the ht://Dig mailing list at <htdig-general@lists.sourceforge.net> or better yet, report it to the bug database, which ensures it won't become lost amongst all of the other mail on the list. Please try to include as much information as possible, including the version of ht://Dig, the OS, and anything else that might be helpful. Often, running the programs with one "-v" or more (e.g. "-vvv") gives useful debugging information. If you are unsure whether the problem is a bug or a configuration problem, you should discuss the problem on <htdig-general> (after carefully reading the FAQ and searching the mail archive and patch archive, of course) to sort out what it is. The mailing list has a wider audience, so you're more likely to get help with configuration problems there than by reporting them to the bug database.

Whether reporting problems to the bug database or mailing list, we cannot stress enough the importance of always indicating which version of ht://Dig you are running. There are still a lot of users, ISPs and software distributors using older versions, and there have been a lot of bug fixes and new features added in recent versions. Knowing which version you're running is absolutely essential in helping to find a solution. If you're unsure if your version is current, or what fixes and features have been added in more recent versions, please see the release notes. See also question 2.1.

Phrase searching has been added for the 3.2 release, which is currently in the beta phase (3.2.0b3 as of this writing). Near or proximity matching will probably be added in a future beta.

The code itself doesn't put any real limit on the number of pages. There are several sites in the hundreds of thousands of pages. As for practical limits, it depends a lot on how many pages you plan on indexing. Some operating systems limit files to 2 GB in size, which can become a problem with a large database. There are also slightly different limits to each of the programs. Right now htmerge performs a sort on the words indexed. Most sort programs use a fair amount of RAM and temporary disk space as they assemble the sorted list. The htdig program stores a fair amount of information about the URLs it visits, in part to only index a page once. This takes a fair amount of RAM. With cheap RAM, it never hurts to throw more memory at indexing larger sites. In a pinch, swap will work, but it obviously really slows things down.

The 3.2 development code helps with many of these limitations. In paticular, it generates the databases on the fly, which means you don't have to sort them before searching. Additionally, the new databases are compressed significantly, making them usually around 50% the size of those in previous versions.

Yes. A list of such ISPs is available here

Sure! The GNU GPL license has no restrictions on use. So you are free to use ht://Dig however you want on your website, personal files, etc. The license only restricts distribution. So if you're planning on a commercial software product that includes ht://Dig, you will have to provide source code including any modifications upon request.

We don't. You can use the "acroread" program to index PDF files, but this is no longer recommended. Initially this program was the only reliable way to extract data from PDF files. However, the xpdf package is a reliable, free software package for indexing and viewing PDF files. See question 4.9 for details on using xpdf to index PDF files. We do not advocate using acroread any longer because it is a proprietary product. Additionally it is no longer reliable at extracting data.

SourceForge is a new service for open source software. You can host your project on SourceForge servers and use many of their services like bug-tracking and the like. The ht://Dig project currently uses SourceForge for a mirror of the main website at htdig.sourceforge.net as well as a mirror of ht://Dig releases and contributed work.

Before you go anywhere else, think of other ways of phrasing your question. Many times people have questions that are very similar to other FAQ and while we try to phrase the queries in the FAQ closely to the most common questions, we obviously can't get them all! The next place to check is the documentation itself. In particular, take a look at the list of configuration attributes, particularly the list by name and by program. There are a lot of them, but chances are there's something that might fit your needs. You should also take a close look at all of htsearch's documentation, especially the section "HTML form" which describes all the CGI input parameters available for controlling the search, including limiting the search to certain subdirectories. You can find the answer yourself to almost all "how can I..." questions by exploring what the various configuration attributes and search form input parameters can do.

Finally, if you've exhausted all the online documentation, there's the htdig-general mailing list. There are hundreds of users subscribed and chances are good that someone has had a similar problem before or can suggest a solution.

The htdig-general mailing list exists for dealing with questions about the software, its installation, configuration, and problems with it. E-mailing the developers directly circumvents this forum and its benefits. Most annoyingly, it puts the onus on an individual to answer, even if that individual is not the best or most qualified person to answer. This is not a one-man show. It also circumvents the archiving mechanism of the mailing list, so not only do subscribers not see these private messages and replies, but future users who may run into the exact same problems won't see them. Remember that the developers are all volunteers, and they don't work for free for your benefit alone. They volunteer for the benefit of the whole ht://Dig user community, so don't expect extra support from them outside of that community. See also questions 1.4 and 1.5.

2. Getting ht://Dig

The latest version is 3.1.5 as of this writing. A beta version of the 3.2 code, 3.2.0b3 is also available, for those who wish to test it. You can find out about the latest version by reading the release notes.

Note that if you're running any version older than 3.1.5 on a public web site, you should upgrade immediately, as older versions have a rather serious security hole which is explained in detail in this advisory which was sent to the BugTraq mailing list.

We're trying to get consistent binary distributions for popular platforms. Contributed binary releases will go in ftp://ftp.htdig.org/pub/htdig/binaries/ and contributions should be mentioned to the htdig-general mailing list.

Anyone who would like to make consistent binary distributions of ht://Dig at least should signup to the htdig-announce mailing list.

Yes, see our mirrors listing. If you'd like to mirror the site, please see the mirroring guide.

Yes. You can find the current versions and several older versions at <ftp.htdig.org> as well as the other locations mentioned in the download page.

Most versions are also distributed as a patch to the previous version's source code. The most recent exception to this was version 3.1.0b1. Since this version switched from the GDBM database to DB2, the new database package needed to be shipped with the distribution. This made the potential patch almost as large as the regular distribution. Update patches resumed with version 3.1.0b2. You can also find archives of patches submitted to the htdig mailing lists, to fix specific bugs or add features, at Joe Jah's htdig-patches ftp site.

The ht://Dig package can be built on the Win32 platform when using the Cygwin package. For details, see the contributed guide, Idiot's Guide to Installing ht://Dig on Win32.

3. Compiling

This usually indicates that either libstdc++ is not installed or is installed incorrectly. To get libstdc++ or any other GNU too, check ftp://ftp.gnu.org/pub/gnu/. Note that the most recent versions of gcc come with libstdc++ included.

This is due to a bug in the Makefile.config.in of version 3.1.0b1. Remove all flags "-ggdb" in Makefile.config.in. Then type "./config.status" to rebuild the Makefiles and recompile. This bug is fixed in version 3.1.0b2.

Answer contributed by George Adams <learningapache@my-dejanews.com>

What you're seeing are problems related to the Berkeley DB library. htdig needs a fairly modern version of db, which is why it ships with one that works. (see that -L../db-2.4.14/dist line? That's where htdig's db library is).
The solution is to modify the c++ command so it explicity references the correct libdb.a . You can do this by replacing the "-ldb" directive in the c++ command with "../db-2.4.14/dist/libdb.a" This problem has been resolved as of version 3.1.0.

Answer contributed by Laura Wingerd <laura@perforce.com>
I got a clean build of htdig-3.1.2 on FreeBSD 2.2.8 by taking -D_THREAD_SAFE out of CPPFLAGS, and setting LIBS to null, in db/dist/configure.

The db/ pacakge, included with ht://Dig seems to be unable to complete on HP/UX 10.20 in particular. After running the top-level configure script, cd into db/dist and type:

Then continue with the normal compilation.

Answer contributed by Adam Rice <adam@newsquest.co.uk>

The problem is that the Solaris loader can't find the library. The best thing to do is set the LD_RUN_PATH environment variable during compile to the directory where libstdc++.so.2.8.1.1 lives. This tells the linker to search that directory at runtime.

Note that LD_RUN_PATH is not to be confused with LD_LIBRARY_PATH. The latter is parsed at run-time, while LD_RUN_PATH essentially compiles in a library path into the executable, so that it doesn't need a LD_LIBRARY_PATH setting to find its libraries. This allows you to avoid all the complexities of setting an environment variable for a CGI program run from the server.

It is not entirely clear why these problems occur, though they seem to only happen when older compilers are used. Several people have reported that the problems go away when using the latest version of gcc.

4. Configuration

There are a variety of reasons ht://Dig won't index a site. To get to the bottom of things, it's advisable to turn on some debugging output from the htdig program. When running from the command-line, try "-vvv" in addition to any other flags. This will add debugging output, including the responses from the server.

Answer contributed by: Malki Cymbalista <Malki.Cymbalista@weizmann.ac.il>

You can change the output format of htsearch by creating different header, footer and result files that specify how you want the output to look. You then create a configuration file that specifies which files to use. In the html document that links to the search, you specify which configuration file to use.

So the configuration file would have the lines:

search_results_header: ${common_dir}/ccheader.html
search_results_footer: ${common_dir}/ccfooter.html
template_map:  Long long builtin-long \
               Short short builtin-short \
               Default default ${common_dir}/ccresult.html
template_name: Default

You would also put into the configuration file any other lines from the default configuration file that apply to htsearch.

The files ${common_dir}/ccheader.html and ${common_dir}/ccfooter.html and ${common_dir}/ccresult.html would be tailored to give the output in the desired format.

Assuming your configuration file is called cc.conf, the html file that links to the search has to set the config parameter equal to cc. The following line would do it:
<input type=hidden name=config value="cc">

ht://Dig should index pages starting with '~' as if it was another web browser. If you are having problems with this, check your server log files to see what file the server is attempting to return.

Yes, though you may find it easier to have one larger database and use restrict or exclude fields on searches. To use multiple databases, you will need a config file for each database. Then each file will set the database_base attribute to change the name of the databases.

As of version 3.1.0, you can do this with the -m option to htmerge.

There are several ways to cut down on disk space. One is not to use the "-a" option, which creates work copies of the databases. Naturally this essentially doubles the disk usage. If you don't need to index and search at the same time, you can ignore this flag. Changing configuration variables can also help cut down on disk usage. Decreasing max_head_length and max_meta_description_length will cut down on the size of the excerpts stored (in fact, if you don't have use_meta_description set, you can set max_meta_description_length to 0!). Other techniques include removing the db.wordlist file and adding more words to the bad_words file.

Not really. Apache will not parse CGI output for SSI statements (See the Apache FAQ). Thus,the htsearch CGI does not understand SSI markup and thus cannot include other CGIs. However, it is possible doing it the other way round: you can have the htsearch results included in your dynamic page.

The Apache project has mentioned that this will be a feature added to the Apache 2.0 version, currently in development.

The easiest approach in the meantime is using SSI with the help of the script_name configuration file attribute. See the contrib/scriptname directory for a small example using SSI.

For CGI and PHP, you need a "wrapper" script to do that. For perl script examples, see the files in contrib/ewswrap. The PHP guide (see contributed guides) not only describes a wrapper script for PHP, but also offers a step by step tutorial to the basics of ht://dig and is well worth reading. For other alternatives, see question 4.11.

This must be done with an external parser or converter. A sample of such a parser is the contrib/parse_doc.pl Perl script. It will parse Word, PostScript and PDF documents, when used with the appropriate document to text converters. It uses catdoc to parse Word documents, and ps2ascii to parse PostScript files. The comments in the Perl script indicate where you can obtain these converters.

As of htdig version 3.1.4, you can use an external converter, such as contrib/conv_doc.pl or the newer and more complete doc2html.pl Perl script, instead of an external parser. These scripts are simpler to write and maintain than a full external parser, as they just convert input documents to text/plain or text/html, and pass that back to htdig to be parsed. Parsing is more consistent across document types as a result.

The most recent versions of parse_doc.pl, conv_doc.pl and the doc2html package are available on our FTP site.
See below for an example of parse_doc.pl, or see the comments in conv_doc.pl or the documentation for doc2html for examples of their usage.

This too can be done with an external parser or converter, in combination with the pdftotext program that is part of the xpdf 0.90 package. A sample of such a parser is the contrib/parse_doc.pl Perl script. It uses pdftotext to parse PDF documents, then processes the text into external parser records. The most recent version of parse_doc.pl is available on our FTP site.

For example, you could put this in your configuration file:

external_parsers: application/msword /usr/local/bin/parse_doc.pl \
                  application/postscript /usr/local/bin/parse_doc.pl \
                  application/pdf /usr/local/bin/parse_doc.pl

You would also need to configure the script to indicate where all of the document to text converters are installed.

As of htdig version 3.1.4, you can use an external converter, such as contrib/conv_doc.pl or the newer and more complete doc2html.pl Perl script, also available on our FTP site, instead of an external parser. These scripts are simpler, and offer more consistent parsing, because the final work is done by htdig's internal parsers. See the comments inside these scripts for an example of their usage.

Whether you use this external parser or converter, or acroread with the pdf_parser attribute, to successfully index PDF files be sure to set the max_doc_size attribute to a value larger than the size of your largest PDF file. PDF documents can not be parsed if they are truncated.

This also raises the questions of why two different methods of indexing PDFs are supported, and which method is preferred. The built-in PDF support, which uses acroread to convert the PDF to PostScript, was the first method which was provided. It had a few problems with it: acroread is not open source, it is not supported on all systems on which ht://Dig can run, and for some PDFs, the PostScript that acroread generated was very difficult to parse into indexable text. Also, the built-in PDF support expected PDF documents to use the same character encoding as is defined in your current locale, which isn't always the case. The external parser, which uses pdftotext, was developed to overcome these problems. xpdf 0.90 is free software, and its pdftotext utility works very well as an indexing tool. It also converts various PDF encodings to the Latin 1 set. It is the opinion of the developers that this is the preferred method. However, some users still prefer to stick with acroread, as it works well for them, and is a little easier to set up if you've already installed Acrobat.

Also, pdftotext still has some difficulty handling text in landscape orientation, even with its new -raw option in 0.90, so if you need to index such text in PDFs, you may still get better results with acroread.

See also question 5.2 below and question 1.13 above.

The first and most important thing you must do, to allow ht://Dig to properly support international characters, is to define the correct locale for the language and country you wish to support. This is done by setting the locale attribute (see question 5.8). The next step is to configure ht://Dig to use dictionary and affix files for the language of your choice. These can be the same dictionary and affix files as are used by the ispell software. A collection of these is available from Geoff Kuenning's International Ispell Dictionaries page, and we're slowly building a collection of word lists on our FTP site.

For example, if you install German dictionaries in common/german, you could use these lines in your configuration file:

locale:               de_DE
lang_dir:             ${common_dir}/german
bad_word_list:        ${lang_dir}/bad_words
endings_affix_file:   ${lang_dir}/german.aff
endings_dictionary:   ${lang_dir}/german.0
endings_root2word_db: ${lang_dir}/root2word.db
endings_word2root_db: ${lang_dir}/word2root.db

You can build the endings database with htfuzzy endings. (This command may actually take days to complete, for releases older than 3.1.2. Current releases use faster regular expression matching, which will speed this up by a few orders of magnitude.) You will also need to redefine the synonyms file if you wish to use the synonyms search algorithm. This file is not included with most of the dictionaries, nor is the bad_words file. Current versions of ht://Dig only support 8-bit characters, so languages such as Chinese and Japanese, which require 16-bit characters, are not currently supported.

Didier Lebrun has written a guide for configuring htdig to support French, entitled Comment installer et configurer HtDig pour la langue française. His "kit de francisation" is also available on our FTP site.

While htsearch doesn't currently provide a means of doing SSI on its output, or calling other CGI scripts, it does have the capability of using environment variables in templates.

The easiest way to get rotating banners in htsearch is to replace htsearch with a wrapper script that sets an environment variable to the banner content, or whatever dynamically generated content you want. Your script can then call the real htsearch to do the work. The wrapper script can be written as a shell script, or in Perl, C, C++, or whatever you like. You'd then need to reference that environment variable in header.html (or wrapper.html if that's what you're using), to indicate where the dynamic content should be placed.

If the dynamic content is generated by a CGI script, your new wrapper script which calls this CGI would then have to strip out the parts that you don't want embedded in the output (headers, some tags) so that only the relevant content gets put into the environment variable you want. You'd also have to make sure this CGI script doesn't grab the POST data or get confused by the QUERY_STRING contents intended for htsearch. Your script should not take anything out of, or add anything to, the QUERY_STRING environment variable.

An alternative approach is to have a cron job that periodically regenerates a different header.html or wrapper.html with the new banner ad, or changes a link to a different pre-generated header.html or wrapper.html file. For other alternatives, see question 4.7.

By default, htdig doesn't treat numbers without letters as words, so it doesn't index them. To change this behavior, you must set the allow_numbers attribute to true, and rebuild your index from scratch using rundig or htdig with the -i option, so that bare numbers get added to the index.

If you change the search.html form to use the GET method rather than POST, you can see the URLs complete with all the arguments that htsearch needs for a query. Here is an example:
http://www.grommetsRus.com/cgi-bin/htsearch?config=htdig&restrict=&exclude=&method=and&format=builtin-long&words=grapple+grommets which can actually be simplified to:
http://www.grommetsRus.com/cgi-bin/htsearch?method=and&words=grapple+grommets with the current defaults. The "&" character acts as a separator for the input parameters, while the "+" character acts as a space character within an input parameter. In versions 3.1.5 or 3.2.0b2, or later, you can use a semicolon character ";" as a parameter separator, rather than "&", for HTML 4.0 compliance. Most non-alphanumeric characters should be hex-encoded following the convention for URL encoding (e.g. "%" becomes "%25", "+" becomes "%2B", etc). Any htsearch input parameter that you'd use in a search form can be added to the URL in this way. This can be embedded into an <a href="..."> tag.
See also question 5.21.

First of all, you do not do this by using the "keywords" field in the search form. This seems to be a frequent cause of confusion. The "keywords" input parameter to htsearch has absolutely nothing to do with searching meta keywords fields. It actually predates the addition of meta keyword support in 3.1.x. A better choice of name for the parameter would have been "requiredwords", because that's what it really means - a list of words that are all required to be found somewhere in the document, in addition to the words the user specifies in the search form.

To restrict a search to meta keywords only, you must set all factors other than keywords_factor to 0, and for 3.1.x, you must then reindex your documents. In 3.2, you'll be able to change factors at search time without needing to reindex, as well as offering the ability to restrict the search in the query itself.

Yes, in each HTML file you want to exclude, add the following between the <HEAD> and </HEAD> tags:

<META NAME="robots" CONTENT="noindex, follow">

Doing so will allow htdig to still follow links to other documents, but will prevent this document from being put into the index itself. You can also use "nofollow" to prevent following of links. See the section on Recognized META information for more details. For documents produced automatically by MhonArc, you can have that line inserted automatically by putting it in the MhonArc resource file, in the sections IDXPGBEGIN and TIDXPGBEGIN.

You can also use the noindex_start and noindex_end attributes to define one set of tags which will mark sections to be stripped out of documents, so they don't get indexed, or you can mark sections with the non-DTD <noindex> and </noindex> tags.

You must set either the image_url_prefix attribute, or both star_blank and star_image in your htdig.conf, to refer to the URL path for these files. You should also set this URL path similarly in in common/header.html and common/wrapper.html, as they also refer to the star.gif file. If you want to relocate other graphics, such as the buttons or the ht://Dig logo, you should change all references to these in htdig.conf and common/*.html.

This can be done by using the url_part_aliases configuration file attribute. You have to set up different configuration files for htdig and htsearch, to define a different setting of this attribute for each one.

In ht://Dig's terminology, the settings in its configuration files are called configuration attributes, to distinguish them from command line options, CGI input parameters and template variables. There are many, many attributes that can be set to control almost all aspects of indexing, searching, customization of output and internationalization. All attributes have a built-in default setting, and only a subset of these appear in the sample htdig.conf file. See the documentation for all default values for attributes not overridden in the configuration file, and for help on using any of them. See also question 1.15.

In version of htsearch before 3.1.4, maximum_pages limited only the number of page list buttons, and not the actual number of pages. This was changed because there was no means of limiting the total number of pages, but this ended up frustrating users who wanted the ability to have more pages than buttons. In 3.2.0b3 we will introduce a maximum_page_buttons attribute for this purpose.

That depends on whether you want to protect certain parts of your site from prying eyes, or just limit the scope of search results to certain relevant areas. For the latter, you just need to set the restrict or exclude input parameter in the search form. This can be done using hidden input fields containing preset values, text input fields, select lists, radio buttons or checkboxes, as you see fit. If you use select lists, you can propagate the choices to select lists in the follow-up search forms using the build_select_lists configuration attribute.

If you wish to keep secure and non-secure areas on your site separate, and avoid having unauthorized users seeing documents from secure areas in their search results, that takes a bit more effort. You certainly can't rely on the restrict and exclude parameters, or even the config parameter, as any parameter in a search form can also be overridden by the user in a URL with CGI parameters. The safest option would be to host the secure and non-secure areas on separate servers with independent installations of htsearch, each with its own ht://Dig database, but that is often too costly or impractical an option. The next best thing is to host them on the same site, but make sure that everything is very clearly separated to prevent any leakage of secure data. You should maintain separate databases for the secure and public areas of your site, by setting up different htdig configuration files for each area. Use different settings of the start_url, limit_urls_to and database_dir configuration attributes, and possibly even different common_dir settings as well. Make sure your database_dir, and even your common_dir, are not in any directories accessible from the web server. Run htdig and htmerge (or rundig) with each separate configuration file, to build your two databases.

The tricky part is to make sure your htsearch program is secure. You don't want to use the same htsearch for the secure and public sites, because otherwise the public site could access the configuration for the secure database, making its data publicly accessible. You must either compile two separate versions of htsearch, with different settings of the CONFIG_DIR make variable, or you must make a simple wrapper script for htsearch that overrides the compiled-in CONFIG_DIR setting by a different setting of the CONFIG_DIR environment variable. Make sure the CONFIG_DIR for the secure area is not a subdirectory of the CONFIG_DIR for the public area. In this way, you can maintain separate directories of config files for the public and secure sites, so that the secure config files are not accessible from the public htsearch.

Put the htsearch binary or wrapper script for the secure site in a different ScriptAlias'ed cgi-bin directory than the public one, and protect the secure cgi-bin with a .htaccess file or in your server configuration. Alternatively, you can put the secure program, let's call it htssearch, in the same cgi-bin, but protect that one CGI program in your server configuration, e.g.:

<Location /cgi-bin/htssearch>
AuthType Basic
AuthName ....
AuthUserFile ...
AuthGroupFile ...
<Limit GET POST>
require group foo
</Limit>
</Location>

This describes the setup for an Apache server. You'd need to work out an equivalent configuration for your server if you're not running Apache.

Answer contributed by Avi Rappoport <avirr@searchtools.com>

If you have enough disk space for two copies of the index database, use -a with the htdig and htmerge processes. This will make use of a copy of the index database with the extension ".work", and update the copy instead of the originals. This way, htsearch can use those originals while the update is going on. When it's done, you can move the .work versions to replace the originals, and htsearch will use them. The current rundig script will do this for you if you supply the -a flag to it. However, rundig builds the database from scratch each time you run it. If you want to update an alternate copy of the database, see the contributed rundig.sh script.

5. Troubleshooting

This usually has to do with the default document size limit. If you set max_doc_size in your config file to something enough to read in the directory index (try 100000 for 100K) this should fix this problem. Of course this will require more memory to read the larger file. Don't set it to a value larger than the amount of memory you have, and never more than about 2 billion, the maximum value of a 32-bit integer. If htdig is missing entire directories, see question 5.25.

As above, this usually has to do with the default document size. What happens is ht://Dig will read in part of a PDF file and try to index it. This usually fails. Try setting max_doc_size in your config file to a larger value than the size of your largest PDF file. Don't go overboard, though, as you don't want to overflow a 32-bit integer (about 2 billion), and you don't want to allocate much more memory than you need to store the largest document.

There is a bug in Adobe Acrobat Reader version 4, in its handling of the -pairs option, which causes a segmentation violation when using it with htdig 3.1.2 or earlier. There is a workaround for this as of version 3.1.3 - you must remove the -pairs option from your pdf_parser definition, if it's there. However, acroread version 4 is still very unstable (on Linux, anyway) so it is not recommended as a PDF parser. An alternative is to use an external parser with the xpdf 0.90 package installed on your system, as described in question 4.9 above.

This is due to a bug in the Makefile.in file in version 3.1.0b1. The easiest fix is to edit the rundig file and change the line "TMPDIR=@DATABASE_DIR@" to set TMPDIR to a directory with a large amount of temporary disk space for htmerge. This bug is fixed in version 3.1.0b2.

This means that htmerge has run out of temporary disk space for sorting. Either in your "rundig" script (if you run htmerge through that) or before you run htmerge, set the variable TMPDIR to a temp directory with lots of space.

This problem commonly occurs on Red Hat Linux 5.0 and 5.1, because of a bug in vixie-cron. It causes htmerge to fail with a "Word sort failed" error. It's fixed in Red Hat 5.2. You can install vixie-cron-3.0.1-26.{arch}.rpm from a 5.2 distribution to fix the problem on 5.0 or 5.1. A quick fix for the problem is to change the first line of rundig to "#!/bin/ash" which will run the script through the ash shell, but this doesn't solve the underlying problem.

Often this is because the databases are corrupt. Try removing them and rebuilding. If this doesn't work, some have found that the solution for question 3.2 works for this as well. This should be fixed in version 3.1.0b2.

If you are running under Solaris, see 3.6.
See also question 5.13.

Most of the time, this is caused by either not setting or incorrectly setting the locale attribute. The default locale for most systems is the "portable" locale, which strips everything down to standard ASCII. Most systems expect something like locale: en_US or locale: fr_FR. Locale files are often found in /usr/share/locale or the $LANGUAGE environment variable. See also question 4.10.

There are three common causes of this. First of all, the sort program may be running out of temporary file space. Fix this by freeing up some space where sort puts its temporary files, or change the setting of the TMPDIR environment variable to a directory on a volume with more space. A second common problem is on systems with a BSD version of the sort program (such as FreeBSD or NetBSD). This program uses the -T option as a record separator rather than an alternate temporary directory. On these systems, you must remove the TMPDIR environment variable from rundig, or change the code in htmerge/words.cc not to use the -T option. A third cause is the cron program on Red Hat Linux 5.0 or 5.1. (See question 5.5 above.)

When you run htsearch with no customization, on a large database, and it gets a lot of hits, it tends to take a long time to process those hits. Some users with large databases have reported much higher performance, for searches that yield lots of hits, by setting the backlink_factor attribute in htdig.conf to 0, and sorting by score. The scores calculated this way aren't quite as good, but htsearch can process hits much faster when it doesn't need to look up the db.docdb record for each hit, just to get the backlink count, date or title, either for scoring or for sorting. This affects versions 3.1.0b3 and up. In version 3.2, currently under development, the databases will be structured differently, so it should perform searches more quickly.

This most commonly happens when you run htsearch while the database is currently being rebuilt or updated by htdig. If htdig and htmerge have run to completion, and the problem still occurs, this is usually an indication of a corrupted database. If it's finding matches, it's because it found the matching words in db.words.db. However, it isn't finding the document records themselves in db.docdb, which would suggest that either db.docdb, or db.docs.index (which maps document IDs used in db.words.db to URLs used to look up records in db.docdb), is incomplete or messed up. You'll likely need to rebuild your database from scratch if it's corrupted. Older versions of ht://Dig were susceptible to database corruption of this sort. Versions 3.1.2 and later are much more stable.

There is a bug in the implementation of the remove_default_doc attribute in htdig versions 3.1.0, 3.1.1 and 3.1.2, which causes it to match more than it should. The default value for this attribute is "index.html", so any URL in which the filename ends with this string (rather than matches it entirely) will have the filename stripped off. This is fixed in version 3.1.3.

This happens when htsearch dies before putting out a "Content-Type" header. If you are running Apache under Solaris, first try the solution described in question 3.6. If that doesn't work, or you're running on another system, try running "htsearch -vvv" directly from the command line to see where and why it's failing. It should prompt you for the search words, as well as the format.
If it works from the command line, but not from the web server, it's almost certainly a web server configuration problem. One increasingly common problem is Apache configurations which expect all CGI scripts to be Perl, rather than binary executables or other scripts, so they use "perl-handler" rather than "cgi-handler".
See also questions 5.7, 5.14 and 5.23.

Despite a great deal of debugging of these programs, we haven't been able to completely eliminate all such problems on all platforms. If you're running htsearch or htfuzzy on a BSDI system, a common cause of core dumps is due to a conflict between the GNU regex code bundled in htdig 3.1.2 and later, and the BSD C or C++ library. The solution is to use the BSD library's own regex code instead, as summarized by Joe Jah:

• make clean
• Remove references to regex.o from htlib/Makefile.
• Remove htlib/regex.h.
• Remove references to htlib/regex.h in htfuzzy/Makefile, which will be there if you have previously done a "make depend".
• make

This solution may work on some other platforms as well (we haven't heard one way or the other), but will definitely not work on some platforms. For instance, on libc5-based Linux systems, the bundled regex code works fine by default, but using libc5's regex code causes core dumps.

Users of Cobalt Raq or Qube servers have complained of segmentation faults in htdig. Apparently this is due to problems in their C++ libraries, which are fixed in their experimental compiler and libraries. The following commands should install the packages you need:

rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/binutils-2.8.1-3C1.mips.rpm
rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/egcs-1.0.2-9.mips.rpm
rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/egcs-c++-1.0.2-9.mips.rpm
rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/egcs-g77-1.0.2-9.mips.rpm
rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/egcs-objc-1.0.2-9.mips.rpm
rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/libstdc++-2.8.0-9.mips.rpm
rpm -Uvh ftp://ftp.cobaltnet.com/pub/experimental/libstdc++-devel-2.8.0-9.mips.rpm
rpm -Uvh --force ftp://ftp.cobaltnet.com/pub/products/current/RPMS/gcc-2.7.2-C2.mips.rpm

You may have to remove the libg++ package, if you have it installed before installing libstdc++, because of conflicts in these packages. Be sure to do a "make clean" before a "make", to remove any object files compiled with the old compiler and headers.

For other causes of segmentation faults, or in other programs, getting a stack backtrace after the fault can be useful in narrowing down the problem. E.g.: try "gdb /path/to/htsearch /path/to/core", then enter the command "bt". You can also try running the program directly under the debugger, rather than attempting a post-mortem analysis of the core dump. Options to the program can be given on gdb's "run" command, and after the program is suspended on fault, you can use the "bt" command. This may give you enough information to find and fix the problem yourself, or at least it may help others on the htdig mailing list to point out what to do next.

This is a known bug in 3.1.3, and is fixed with this patch. You can apply the patch by entering into the main source directory for htdig-3.1.3, and using the command "patch -p0 < /path/to/HTML.cc.0". This is also fixed as of version 3.1.4.

The most common cause of this error is that htdig did not manage to index any documents, and so it did not create a word list. You should repeat the htdig or rundig command with the -vvv option to see where and why it is failing. See question 4.1.

Check your search form. Chances are there is a hidden input field with no value defined. For example, one user had
<input type=hidden name=restrict> in his search form, instead of
<input type=hidden name=restrict value=""> The problem is that Netscape sets the missing value to a default of " " (two spaces), rather than an empty string. For the restrict parameter, this is a problem, because htsearch won't likely find any URLs with two spaces in them. Other input parameters may similarly pose a problem.

Another possibility, if you're running 3.2.0b1 or 3.2.0b2, is that you need to make the db.words.db_weakcmpr file writeable by the user ID under which the web server runs. This is a bug, and will be fixed in the next beta.

There probably isn't any indexing tool in existance that follows JavaScript links, because they don't know how to initiate JavaScript events. Realistically, it would take a full JavaScript parser in order to be able to figure out all the possible URLs that the code could generate, something that's way beyond the means of any search engine. You have a few options:

• Add "backup" links using plain HTML <a href=...> tags to all the pages that could be accessed through JavaScript,
• Add <link> tags to point to all these pages (requires htdig 3.1.3 or greater, but then everyone should be running 3.1.5 anyway),
• Compose a list of all the unreachable documents, or write a program to do so, and feed that list as part of htdig's start_url attribute. See also question 5.25.

Your server is returning the contents of the htsearch binary. Common causes of this are:

• no execute permission on the htsearch binary,
• the binary won't run on this system (it may be compiled for the wrong system type), or
• the web server doesn't recognize the file as a CGI (for Apache, you must have a ScriptAlias directive for the program or the directory in which it's installed, or define a cgi-script handler for some suffix, e.g. .cgi, and add that suffix to the program file name).

By default, Apache is usually configured with one cgi-bin directory as ScriptAlias, so all your CGI programs must go in there, or have a .cgi suffix on them. Your configuration may differ, however.

As the release notes for these versions suggest, they are somewhat unoptimized and are made available for testing Since the 3.2 code indexes all locations of words to support phrase searching and other advanced methods, this additional data slows down the indexer. To compensate, the code has a cache configured by the wordlist_cache_size attribute. As of this writing, the word database code will slow down considerably when the cache fills up. Setting the cache as large as possible provides considerable performance improvement. Development is in progress to improve cache performance.

In versions 3.1.5 and 3.2.0b2, and later, htsearch was changed to use a semicolon character ";" as a parameter separator for page button URLs, rather than "&", for HTML 4.0 compliance. It now allows both the "&" and the ";" as separators for input parameters, because the CGI specification still uses the "&". This change may cause some PHP or CGI wrapper scripts to stop working, but these scripts should be similarly changed to recognize both separator characters.
See also question 4.13.

In version 3.1.5, htsearch was fixed to properly re-encode the characters &, <, >, and " into SGML entities. However, the default value for the translate_amp, translate_lt_gt and translate_quot attributes is still false, so these entities don't get converted by htdig. If you set these three attributes to true in your htdig.conf and reindex, the problem will go away.

In the 3.2 betas there was a bug in the HTML parser that caused it to fail when attempting to translate the "&amp;" entity. This will be fixed in 3.2.0b3. The translate_* attributes are gone as of 3.2.0b2.

An increasingly common problem is Apache configurations which expect all CGI scripts to be Perl, rather than binary executables or other scripts, so they use "perl-handler" rather than "cgi-handler". The fix is to create a separate directory for non-Perl CGI scripts, and define it as such in your httpd.conf file. You should define it the same way as your existing cgi-bin directory, but use "cgi-handler" instead of "perl-handler".
See also questions 5.7, 5.14 and 5.13.

All configuration file attributes have compiled-in, default values. Taking an attribute out of the file is not the same thing as setting it to an empty string, a 0, or a value of false. See question 4.18.

First of all, htdig doesn't look at directories itself. It is a spider, and it follows hypertext links in HTML documents. If htdig seems to be missing some documents or entire directory sub-trees of your site, it is most likely because there are no HTML links to these documents or directories. (See also question 5.18.) If htdig does not come across at least one hypertext link to a document or directory, and it's not explicitly listed in the start_url attribute, then this document or directory is essentially hidden from view to htdig, or to any web browser or spider for that matter. You can only get htdig to index directories, without providing your own files with links to the contents of these directories, by using your web server's automatic index generation feature. In Apache, this is done with the mod_autoindex module, which is usually compiled-in by default, and is enabled with the "Indexes" option for a given directory hierarchy. For example, you can put these directives in your Apache configuration:

<Directory "/path/to/your/document/root">
    Options Indexes FollowSymLinks Includes ExecCGI
</Directory>

This will cause Apache to automatically generate an index for any directory that does not have an index.html or other "DirectoryIndex" file in it. Other web servers will have similar features, which you should look for in your server documentation.

As an alternative to relying on the web server's autoindex feature, you can compose a list of all the unreachable documents, or write a program to do so, and feed that list as part of htdig's start_url attribute. Here is an example of simple shell script to make a file of URLs you can use with a configuration entry like start_url: `/path/to/your/file`:

find /path/to/your/document/root -type f -name \*.html -print | \
    sed -e 's|/path/to/your/document/root/|http://www.yourdomain.com/|' > \
        /path/to/your/file

Other reasons why htdig might be missing portions of your site might be that they fall out of the bounds specified by the limit_urls_to attribute (which takes on the value of start_url by default), they are explicitly excluded using the exclude_urls attribute, or they are disallowed by a robots.txt file (see the htdig documentation for notes about robot exclusion) or by a robots meta tag (see question 4.15). If htdig seems to be missing the last part of a large directory or document, see question 5.1.

Output from htdig -v typically looks like this:

23000:35506:2:http://xxx.yyy.zz/index.html: ***-+****--++***+ size = 4056

The first number is the number of documents parsed so far, the second is the DocID for this document, and the third is the hop count of the document (number of hops from one of the start_url documents). After the URL, it shows a "*" for a link in the document that it already visited (or at least queued for retrieval), a "+" for a new link it just queued, and a "-" for a link it rejected for any of a number of reasons. To find out what those reasons are, you need to run htdig with at least 3 "v" options, i.e. -vvv. If there are no "*", "+" or "-" symbols after the URL, it doesn't mean the document was not parsed or was empty, but only that no links to other documents were found within it.

When htdig parses documents and finds hypertext links to other documents (hrefs), it may reject them for any of several reasons. To find out what those reasons are, you need to run htdig with at least 3 "v" options, i.e. -vvv. Here are the meanings of some of the messages you might see at this verbosity level.

Not an http or relative link!

In versions 3.1.5 and earlier, only "http://" URLs, or URLs relative to those, are allowed.

Item in the exclude list: item # n

A substring of the URL matches one of the items in the exclude_urls attribute. The given item number will indicate which pattern matched, starting at 1. The 3.2.0 betas do not give the item number.

Extension is invalid!

The file name extension or suffix matches one of those listed in the bad_extensions attribute.

Extension is not valid!

The file name extension or suffix does not match one of those listed in the valid_extensions attribute, if any are specified.

Invalid Querystring! or
item in bad query list

The URL contains a query string which matches one of those listed in the bad_querystr attribute.

URL not in the limits!

No substring of the URL entirely matches one of the items in the limit_urls_to attribute. The purpose of this attribute is to keep htdig from attempting to index the entire World Wide Web.

forbidden by server robots.txt!

A substring of the URL matches one of the items disallowed in the servers robots.txt file. See A Standard for Robot Exclusion. This message exists only in the 3.2.0 betas. In 3.1.5 and earlier, this condition is only caught later, resulting in the message "robots.txt: discarding 'URL'" from htdig, and a later "Deleted: no excerpt" message from htmerge.

url rejected: (level 2)

No substring of the URL entirely matches one of the items in the limit_normalized attribute. All the other rejections above will be indicated as level 1. The 3.2.0 betas give the much more meaningful message 'not in "limit_normalized" list!'

See also question 5.25.