Features and System requirements

ht://Dig Copyright © 1995-2002 The ht://Dig Group
Please see the file COPYING for license information.

Features

Here are some of the major features of ht://Dig. They are in no particular order.

Intranet searching

ht://Dig has the ability to search through many servers on a network by acting as a WWW browser.

It is free

The whole system is released under the GNU General Public License

Robot exclusion is supported

The Standard for Robot Exclusion is supported by ht://Dig.

Boolean expression searching

Searches can be arbitrarily complex using boolean expressions.

Configurable search results

The output of a search can easily be tailored to your needs by means of providing HTML templates.

Fuzzy searching

Searches can be performed using various configurable algorithms. Currently the following algorithms are supported (in any combination):

• exact
• soundex
• metaphone
• common word endings (stemming)
• synonyms
• accent stripping
• substring and prefix

Searching of HTML and text files

Both HTML documents and plain text files can be searched. Searching of other file types will be supported in future versions.

Keywords can be added to HTML documents

Any number of keywords can be added to HTML documents which will not show up when the document is viewed. This is used to make a document more like to be found and also to make it appear higher in the list of matches.

Email notification of expired documents

Special meta information can be added to HTML documents which can be used to notify the maintainer of those documents at a certain time. It is handy to get reminded when to remove the "New" images from a certain page, for example.

A Protected server can be indexed

ht://Dig can be told to use a specific username and password when it retrieves documents. This can be used to index a server or parts of a server that are protected by a username and password.

Searches on subsections of the database

It is easy to set up a search which only returns documents whose URL matches a certain pattern. This becomes very useful for people who want to make their own data searchable without having to use a separate search engine or database.

Full source code included

The search engine comes with full source code. The whole system is released under the terms and conditions of the GNU Public License version 2.0

The depth of the search can be limited

Instead of limiting the search to a set of machines, it can also be restricted to documents that are a certain number of "mouse-clicks" away from the start document.

Full support for the ISO-Latin-1 character set

Both SGML entities like 'à' and ISO-Latin-1 characters can be indexed and searched.

Requirements to build ht://Dig

ht://Dig was developed under Unix using C++.

For this reason, you will need a Unix machine, a C compiler and a C++ compiler. (The C compiler is needed to compile some of the GNU libraries)

Unfortunately the developers only have access to a couple of different Unix machines. Most development is done on Linux systems with gcc/g++, but ht://Dig has been tested on these machines (and compilers):

• Sun Solaris SPARC 2.X (using gcc/g++)
• Sun SunOS 4.1.4 SPARC (using gcc/g++ 2.7.0)
• HP/UX 10.X (using gcc/g++)
• IRIX 5.3 and 6.X (SGI C++ compiler.)
• Most Linux Distributions (using gcc/g++)
• Most BSD platforms, including BSDI and Mac OS X (using gcc/g++)

libstdc++

If you plan on using g++ to compile ht://Dig, you have to make sure that libstdc++ has been installed. For older versions of gcc/g++ (2.8.X and prior), libstdc++ is a separate package from gcc/g++. You can get libstdc++ from the GNU software archive.

GNU-style (Berkeley) 'make'

The building relies heavily on the make program. The problem with this is that not all make programs are the same. The requirement for the make program is that it understands the 'include' statement as in

include somefile

The Berkeley 4.4 make program supplied in many BSD-style systems doesn't use this syntax, instead it wants

.include "somefile"

and hence it cannot be used to build ht://Dig. Many systems include a "gmake" command that will work instead.

If your make program doesn't understand the right 'include' syntax, it is best if you get and install GNU make before you try to compile everything. The alternative is to change all the Makefiles.

Disk space requirements

The search engine will require lots of disk space to store its databases. Unfortunately, there is no exact formula to compute the space requirements. It depends on the number of documents you are going to index but also on the various options you use. To give you an idea of the space requirements, here is what I have deduced from our own database size at San Diego State University.

If you keep around the wordlist database (for update digging instead of initial digging) I found that multiplying the number of documents covered by 12,000 will come pretty close to the space required.

We have about 13,000 documents:

         13,000
         12,000 x
    -----------
    156,000,000

Without the wordlist database, the factor drops down to about 7500:

         13,000
          7,500 x
     ----------
     97,500,000

Keep in mind that we keep at most 50,000 bytes of each document. This may seen a lot, but most documents aren't very big and it gives us a big enough chunk to almost always show an excerpt of the matches.

You may find that if you store most of each document, the databases are almost the same size, or even larger than the documents themselves! Remember that if you're storing a significant portion of each document (say 50,000 bytes as above), you have that requirement, plus the size of the word database and all the additional information about each document (size, URL, date, etc.) required for searching.