Release notes

ht://Dig Copyright © 1995-2004 The ht://Dig Group
Please see the file COPYING for license information.

These are notes that go with each release of ht://Dig. There is also a ChangeLog file which has more details on the code changes.

Release notes for htdig-3.2.0b6 20 Jun 2004
The next beta release of ht://Dig, 3.2.0b6, is now available. It fixes several bugs from 3.2.0b5, and runs somewhat faster, although still much slower than 3.1.6. (No significant speed improvements are expected in the near future, although we are working on it.) Calling this release a "beta" simply means that exhausive testing, especially on non-Linux platforms, is not yet complete. However, we consider it stable enough for most production use.

As with 3.2.0b5, if you are upgrading from a previous version, you should read the upgrade guide first.

• Correctly handle empty disallow entries in robots.txt
• No longer compile regular expressions for every URL (improve performances)
• Allow compressed databases on Cygwin
• Fixed bugs in phrase searching
• Improved parsing of the configuration file
• bin/rundig -a handles multiple database directories
• Ellipsis displayed correctly by htsearch
• Allow '-' argument to '-m' ('minimal') runtime option to htdig
• Check validity of first URL from each server
• No longer ignore empty configuration attributes
• fixed bug in handling 'http_proxy', 'http_proxy_authorization', 'authorization attributes'
• remove stale md5_db if '-i' specified
• Make 'server_alias' case insensitive
• fixed bugs with zlib
• Allow € HTML entity
• fixed other minor bugs

• added allow_space_in_url attribute: if set to true, htdig will handle URLs that contain embedded spaces
• added store_phrases attribute: if it is false, htdig only stores the first occurrence of each word in a document
• added an improved version of RTF2HTML into the contrib section
• added OpenOffice.org support to doc2html in contrib section
• improved date factor formula
• improved tests
• improved documentation
• added man pages

Release notes for htdig-3.2.0b5 10 Nov 2003
This version was slated to be 3.2.0rc1, but some final testing is still required. It primarily fixes many bugs in 3.2.0b3, with some limited new functionality. As with 3.2.0b1 and 3.2.0b2, if you are upgrading from a previous version, you should read the upgrade guide first.

• Fixed database bugs. Introduced zlib compression to replace buggy internal compression.
• Forward-ported functionality from 3.1.6 (description_meta_tag_names, use_doc_date, ignore_alt_text, ignore_dead_servers, boolean_keywords, boolean_syntax_errors, multimatch_factor, translate_latin1)
• Fixed bugs in phrase searching
• Fixed compile problems due to deprecated C++ includes
• Fixed bugs handling double slashes in URLs
• Suppress display of matches with weight zero
• Fixed bugs in nesting of tags which turn off indexing

• Added Native Win32 support
• Added http_proxy_authorization attribute
• Improved networking code, with improved cookie handling and accept_language support
• Implemented field-restricted searches (e.g. title:word)
• Handle noindex_start/noindex_end as string lists
• Implemented external converters, text/html->text/html-internal
• Improved support for MIME types
• Changed licence to LGPL from GPL

Release notes for htdig-3.2.0b4
This beta was never issued.

Release notes for htdig-3.2.0b3 22 Feb 2001
This version is still marked beta because it has still only received limited testing and there are still revisions pending for the 3.2 releases. However, it adds more functionality and should address all serious bugs in the 3.2.0b2 release. As with 3.2.0b1 and 3.2.0b2, if you are upgrading from a previous version, you should read the upgrade guide first.

Please note if you are updating from a prior release (3.1 or 3.2), the htmerge program has changed syntax as noted below. You will probably want to change your behavior to call htpurge instead of htmerge after htdig as noted below.

• Fixed several non-exploitable bugs in handling external parsers or transport agents.
• Fix bug where changes in the robots.txt would be ignored. If a URL was indexed and later the robots.txt changed to forbid it, the URL would be checked anyway.
• Fixed scoring bugs introduced in 3.2.0b2.
• Fixed a non-exploitable security issue where content-type headers were passed incorrectly to external parsers or converters.
• Fixed bugs in the accents fuzzy algorithm, cutting down on the size of the accent database.
• Fixed a bug where duplicate documents would be generated when merging a database with itself.
• Fixed a bug in the new regex handling for indexing limits where large patterns could fail and would be silently ignored.
• Fixed minor bugs with the HTTP/1.1 implementation.
• Fix a bug where an extra config= portion of a URL would be output when using collections.
• Fixed a bug with content-type declarations in external parsers with combined content-type; charset declarations.
• Fixed a bug in the config parser that did not correctly handle relative config include statements.
• Fixed a bug in htfuzzy which would append to an existing synonyms database rather than creating it anew.
• Fixed problems with the configure script ignoring --enable-bigfile flags.
• Fixed problems with retrieval order--this could potentially foul things up when limiting indexing by hopcount.
• Fixed some problems with the HTML in the included sample files.
• Make the -l flag to htdig obsolete--this is now the default behavior -- the program will intercept many signals and write a log file for a restart.
• Updated database format from the mifluz/htword project.
• Changed syntax of htmerge. The program now only merges databases. The htpurge program will "clean up" databases after running htdig. The included "rundig" script reflects this.
• htload now properly loads ASCII word databases.
• Enhanced build_select_lists attribute.
• Added support for controlling the number of Page buttons in htsearch with maximum_page_buttons.
• Added the METADESCRIPTION htsearch template variable for displaying the <META> description field in output along with the normal description, instead of using the use_meta_description attribute.
• Added support for permanent URL rewriting with the url_rewrite_rules attribute. (As opposed to the url_part_aliases attribute which can provide a different URL to htsearch and htdig.)
• Added support for restricting a search to match only documents between two dates as specified in the search form as well as the template variables STARTYEAR, STARTMONTH, STARTDAY, ENDYEAR, ENDMONTH, ENDDAY.
• Added support for limiting duplicates based on MD5 signatures with the new attributes check_unique_md5, check_unique_date, md5_db.
• The documentation has been revised to include a block: portion to note if attributes can be included in URL or Server blocks. See the configuration documentation for more information.
• More attributes are set on a per-server or per-URL basis.
• New support for nttp:// protocol.
• Added support for auto-generating directory listings for file:// URLs.
• Set the default compilation to enable tests that can be run with "make check"
• Greatly improved htnotify program with one message per e-mail address and support for message templates using the new attributes htnotify_webmaster, htnotify_replyto, htnotify_prefix_file, htnotify_suffix_file.
• There are the usual variety of other fixes and changes. See the ChangeLog for more details.
• Once again, a huge thank you to everyone who contributed bug reports, fixes and patches!

• Fixed several bugs in the new HTTP/1.1 implementation that would cause problems with so-called "Chunked" data.
• Fixed a bug in the new regex-based configuration options that would ignore the case_sensitive attribute.
• Fixed the robots.txt parsing to more rigorously stick to the standard.
• Fixed a bug where upper-case META robots directives would be ignored.
• Fixed a bug that could leave a connection open when it failed.
• Fixed the timeout in the connection code to ensure that hung connections are killed properly.
• Fixed a bug where duplicates of modified documents could pile up over time.
• Fixed a bug in the SGML entity handling where numeric entities would be ignored. (e.g. &#162; -> ¢)
• Fixed a bug in the new configuration parser that wouldn't accept lists including numbers
• Fixed a potential infinite loop in the phrase searching parser that came up when fuzzy algorithms were used.
• The HTML parser now ignores anything between <script> tags, much like it does for <style> tags.
• Fixed some performance problems in the new word database code.
• Removed the attributes translate_quot, translate_lt, translate_gt and translate_amp since all SGML entities are now encoded and decoded when displayed.
• Removed the attribute uncoded_db_compatible since the 3.2 databases are no longer compatible with previous versions anyway.
• Removed the attribute word_list because the db.wordlist file is no longer generated. To get an ASCII version of the database, use the word_dump attribute.
• Removed the pdf_parser attribute. It is now preferred to use the external parser or external converter support with xpdf.
• The wordlist_compress attribute is now turned on by default.
• The output from htsearch and the default and included templates should now be more HTML-4.0 compliant.
• Added support for searching collections of multiple databases. To use this, supply multiple config fields or config names separated by "|" characters. Also see the collection_names attribute.
• Added a new accents fuzzy algorithm, which treats accented and unaccented words the same. You must create an accents_db with htfuzzy after indexing.
• Added new attributes tcp_max_retries and tcp_wait_time to control how many times a low-level connection is retried and how long to wait on a hung connection.
• Add any_keywords attribute to OR the keywords field in a search form instead of AND-ing them together.
• Add the attributes search_results_order and url_seed_score to control result ranking and scoring based on URL patterns.
• Moved the htnotify program into the new httools directory.
• Added the programs htdump, htload, htstat and htpurge.
• There are the usual variety of other fixes and changes. See the ChangeLog for more details.
• Once again, a huge thank you to everyone who contributed bug reports, fixes and patches!

Release notes for htdig-3.1.5 25 Feb 2000
This version cleans up some remaining bugs in the 3.1.4 release. As the latest stable release of ht://Dig, it is recommended for all production servers.

• Fixed a nasty security hole in htsearch, which would allow users to view any file on your site that had read permission.
• Fixed a bug that could cause problems with 8-bit characters on some systems.
• Made some attempts to get htsearch's output to be more HTML 4.0 compliant. It quotes all HTML tag parameters, and uses ";" instead of "&" as parameter separator in URLs for next pages. Reserved characters in parameters are now encoded. Please note that this may break a variety of CGI wrappers, for example, those written in PHP3.
• Fixed handling of SGML entities: htdig will still decode them to store as single characters in the database, but htsearch now encodes some of them back for compliant results.
• Added two new formats for variables in htsearch templates, $%(var), which escapes the variable for a URL, and $&(var), which HTML-escapes the variable as necessary.
• Fixed htdig's handling of robots.txt, such that only the first applicable User-agent field bearing its name will be used, rather than only the last.
• Fixed htdig's handling of servers that return 2-digit years.
• Fixed handling of embedded quotes in quoted string lists.
• Fixed handling of relative URLs with trailing ".." or leading "//".
• Fixed handling of the valid_extensions attribute, which sometimes failed in the previous version.
• Enhanced the handling of local filesystem indexing with the local_urls, local_user_urls or local_default_doc attributes, which now allow multiple directory or file names to be tried.
• Added the build_select_lists attribute to allow the config file to specify <select> form elements in htsearch output as a template variable, much like $(SORT) and $(METHOD).
• Added support for two additional configuration attributes: max_keywords, and nph.
• A variety of other bug fixes, and many documentation updates. See the ChangeLog for details.
• Once again, thanks to everyone who reported bugs and bug fixes.

Release notes for htdig-3.2.0b1 4 Feb 2000
This marks the first beta version of the 3.2.0 codebase, over a year in the works. Since it has not received as much testing as the 3.1.x series, it is *not* recommended for production environments. A full description of how to upgrade is provided here.

NOTE: Read this document before upgrading. You have been warned.

• Fixed a bug in htdig where hopcounts could be calculated incorrectly between multiple servers.
• Fixed a bug that could cause problems with 8-bit characters on some systems.
• Fixed handling of unreachable servers. First, the new max_retries attribute allows htdig to attempt multiple connections. Secondly, if the server is not available, htdig will stop trying to connect.
• Fixed handling of SGML entities: htdig will still decode them to store as single characters in the database, but htsearch now encodes them back for compliant results.
• Rewrote the database formats, allowing room for more sophisticated searches and compression of the word database using the new attribute wordlist_compress. These changes include the removal of the word_list file (db.wordlist) and the addition of the new doc_excerpt database.
• Cleaned up many parts of the code, including the URL and HTML parsers. Additionally, on platforms that support it, much of the code will be built as shared libraries, which should help memory utilization, especially under high load.
• Removed the modification_time_is_now attribute, which is now on by default. This means the time at indexing is taken as the date of the document if the server does not return a date.
• Added the new attribute use_doc_date to use the date specified in a META date tag.
• Merged all heading_factor attributes into one new attribute, heading_factor.
• As a result of the new database format, all _factor attributes (like title_factor and keywords_factor are now dynamic--you do not have to rebuild your database to change the scaling.
• Changed attributes bad_querystr, exclude_urls, limit_urls_to, limit_normalized, http_proxy_exclude to allow full regular expressions when the regex are surrounded by [ and ].
• Changed htsearch fields restrict and exclude to allow regular expressions when the regex are surrounded by [ and ].
• Added phrase searching support to htsearch--queries enclosed in quotes will be checked to ensure the words occur in that exact order in the documents.
• Added the build_select_lists attribute to allow the config file to specify <select> form elements in htsearch output as a template variable, much like $(SORT) and $(METHOD).
• Added a regex fuzzy method. This will allow searches to include regex that match words. The fuzzy method will return up to regex_max_words matches.
• Added a speling [sic] fuzzy method. This attempts several simple spelling mistakes (like transposed letters and extra letters) to find matches. This adds the new attribute minimum_speling_length to restrict whether small words should be checked. Transposing letters in smaller words can give unrelated correctly-spelled words.
• Added support for external transport methods, using the external_protocols attribute, an analogue of the external_parsers system.
• Added support for HTTP/1.1, including persistent connections. This can be configured using the new attributes persistent_connections, head_before_get, and max_connection_requests.
• Added support for file:// URLs and support for using the mime_types file to decide whether local files are parsable.
• Added two new formats for variables in htsearch templates, $%(var), which escapes the variable for a URL, and $&(var), which HTML-escapes the variable as necessary.
• Added support for reading the list of URLs to index with htdig by supplying the command-line option -.
• Added a flag -m to htdig to index only the files given in the filename.
• There are many more changes especially to the internal code structure, so a huge thank you goes out to everyone who helped make this release!

Release notes for htdig-3.1.4 9 Dec 1999
This version cleans up some remaining bugs in the 3.1.3 release. As the latest stable release of ht://Dig, it is recommended for all production servers.

• Fixed a nasty bug in URL parameter parsing, which was gobbling up bare ampersands (&) and CGI parameter names.
• Fixed a bug where htdig would go into an infinite loop if an entry in local_urls, local_user_urls or server_aliases was missing the "=".
• Fixed a bug in htsearch, where it failed when reading long queries via the POST method.
• Fixed a bug in htdig, where it failed to close the connection after certain errors.
• Fixed a bug that clobbered the hop count of initial documents.
• Fixed bugs in HTML parser's handling of META tags. It no longer continues indexing meta tags when indexing is turned off for the document, and it no longer gets confused by punctuation in META descriptions and keywords.
• Fixed a bug in the handling of the case_sensitive attribute, so that it's not limited to robots.txt parsing. Now, if false, it causes URLs to be mapped to lowercase, to avoid mixed case duplicates as expected.
• HTML parser now indexes text in alt parameter of img tags, and calculates word locations more accurately than before.
• Digging via the local filesystem can now be done even without an HTTP server running, and a few more file types can be indexed locally, without having to rely on the server.
• Sender name in htnotify's e-mail messages is now quoted.
• The external_parsers attribute is now extended to support external converters, to avoid a lot of the complications of writing external parsers.
• Added support for several new configuration attributes: authorization, start_highlight, end_highlight, local_urls_only, page_number_separator, script_name, template_patterns, and valid_extensions.
• The keywords input parameter to htsearch is now propagated to followup searches, as for other input parameters.
• The query string can now be passed to htsearch as a single command line argument, for use in scripts.
• Added better examples and comments in sample htdig.conf, and added boolean match type to sample search.html form.
• The HTML parser in htdig now turns off indexing between <style> and </style> tags.
• A variety of other bug fixes, and many documentation updates. See the ChangeLog for details.
• Once again, thanks to everyone who reported bugs and bug fixes.

Release notes for htdig-3.1.3 22 Sep 1999
This version fixes a number of bugs in the 3.1.2 release and is the latest stable release of ht://Dig. It is the only version recommended for production servers and users of all previous versions are suggested to upgrade.

• Fixed a long-standing bug where search queries containing punctuation would not be highlighted in excerpts.
• Fixed a bug where SGML entities inside HTML tags were not expanded.
• Fixed the server_aliases attribute to default to port 80 if ommitted.
• Fixed a bug in URL parsing, where documents ending in the value used for remove_default_doc were ignored. For example, a URL ending in /left_index.html would become /.
• Fixed META robot parsing to correctly parse multiple directives.
• Fixed a coredump when generating the metaphone fuzzy database on some systems.
• Fixed the behavior of the modification_time_is_now attribute to work as documented.
• Fixed the behavior of htdig to block out the username/password set on the command-line in process listing.
• Fixed a bug with external parsers to prevent shell escapes in filenames.
• Fixed a bug on some systems, where printing a date might crash.
• Handles the ispell endings lists better so that suffixes more closely match grammatical rules.
• Changed the maximum word length to a run-time option, set with the new attribute maximum_word_length.
• Tests for the presence of alloca.h, which would cause problems with compiling the regex code under non-GNU compilers.
• Added support for <EMBED>, <OBJECT>, and <LINK> HTML tags.
• A variety of other bugs were fixed, see the ChangeLog for details.
• When indexing, htdig should now attempt to index compound words as separate words in addition to a compound word. For example, "pdf_parser" would also be indexed as "pdf" and "parser."
• Once again, thanks to everyone who reported bugs and bug fixes.

Release notes for htdig-3.1.2 21 Apr 1999
This version fixes a number of bugs in the 3.1.1 release and is the latest stable release of ht://Dig. It is highly recommended for production servers.

• Fixed a bug that ignored META description tags when they were also added to the meta_keywords attribute.
• Fixed the HTML comment parsing to be more lenient about non-standard comments.
• Fixed problems in the date-parsing code that made it Y2K incompatible. In particular, it forgot that 2000 is a leap year and wouldn't correctly parse dates after 29 Feb 2000.
• Fixed a variety of bugs in the HTML parser.
• Fixed an old bug that would exclude all URLs if the exclude_urls attribute left empty.
• Fixed display of META description tags. Now it always shows the top of a description. If no description exists, it looks for the search terms in the excerpt as usual.
• Fixed some small memory leaks.
• Changed the htfuzzy endings algorithm to use a more efficient regex system. Speed improvements on non-English languages are noted, now taking minutes for generation that would take days!
• Changed the noindex_start and noindex_end attributes to allow case-insensitive matching.
• Added on-disk versions of the builtin templates to make it more obvious how to change the results templates.
• Added date_format attribute to change the format of dates output in search results.
• Added extra_word_characters attribute that defines extra characters that should be considered part of a word, rather than punctuation.
• Several other, relatively minor bugs were also fixed. Many thanks to those who sent in bug reports and to Gilles Detillieux for coordinating this release.

Release notes for htdig-3.1.1 17 Feb 1999
This version cleans up some remaining bugs in the 3.1.0 release. As the latest stable release of ht://Dig, it is recommended for all production servers.

• Fixed a bug in the configure script under IRIX and Solaris 7.
• Fixed a minor bug with the Berkeley database code under AlphaLinux.
• Fixed a serious bug causing bus errors on several platforms, notably Solaris SPARC, caused by unaligned access to database structures.
• Fixed some bugs in the boolean search parser.
• Replaced the contributed parse_word_doc.pl script with a more capable parse_doc.pl script.
• Fixed the htnotify program to parse dates as mentioned in the documentation.
• Cleaned up some minor mistakes in the documentation and moved to HTML 4.0 Transitional syntax.
• Fixed the documentation for the pdf_parser attribute that was changed in version 3.1.0. This attribute must call the parser with all command-line options.

Release notes for htdig-3.1.0 9 Feb 1999
This version marks the "full release" of version 3.1.0. Naturally, this version adds a few new feature and fixes a large number of remaining bugs. This version is the latest stable release of ht://Dig and is recommended for all production servers for current bug-fixes and oft-requested features.

NOTE: You must rebuild your databases from scratch after updating to this version. Several database-related bugs were fixed and will remain unless you rebuild from scratch. We're sorry for any inconvenience.

• Fixed a variety of small memory leaks.
• Fixed a bug that could duplicate documents in the document databases.
• Fixed a bug that would not remove documents marked as deleted.
• Fixed a bug that could dump core with incorrectly defined template_map attributes.
• Fixed a bug that could dump core or produce bogus dates when a server returns the date in an incorrect format.
• Fixed a variety of string-matching bugs that caused problems with restricting indexing and searching.
• Fixed a bug that could dump core if logging searches and CGI environment variables were not set.
• Fixed a bug that would not hilight searches properly if they contained punctuation.
• Fixed PDF parsing to support programs beyond acroread.
• Fixed a bug that caused problems with large robots.txt files.
• Fixed a bug in the sample rundig script from a non-portable test for the age of databases.
• Fixed bugs in the fuzzy matching code that could prevent searches from completing if fuzzy databases were not present.
• Fixed bugs in the soundex and metaphone algorithms that would only return the first word of several matching words. Note that to completely fix this bug, you must rebuild your soundex and metaphone databases.
• Fixed up many compilation warnings and errors.
• Fixed a performance slowdown in htsearch when backlink_factor and date_factor are zero and can be ignored.
• Improved performance when a server ignores the If-Modified-Since request during update digs.
• Added a warning message if the locale: option is set to a locale that is not present.
• Some minor performance improvements.
• Allow "include" keyword in config file to include other config files.
• Uses latest (2.6.4) version of the Berkeley database.
• Two databases may be merged together using htmerge.
• The htdig program can be safely stopped and restarted in the middle of a dig. The dig will write the progress to the file specified by the new url_log option.
• Added support for anchors in excerpts with the add_anchors_to_excerpt option and the ANCHOR template variable.
• Added support for sorting results in increasing or decreasing order of document date, size, title and score using the search form. Note that changing sort from the default of score will result in a performance decrease.
• Added config options sort and sort_names to change the default sort and names used in the SORT template variable.
• Added the option compression_level to compress the document database if the zlib library is present.
• Added the options noindex_start and noindex_stop to delimit sections of HTML documents to be ignored.
• Added the option allow_in_form to allow specific config options to be set in the search form.
• Added the option bad_querystr to ingore URLs containing specified CGI queries.
• Added the option search_results_wrapper to replace separate header and footer files. For mor information, see the general htsearch documentation.
• Added option no_title_text to allow configuration of the text used when no title is found.
• Added option url_part_aliases to allow rewriting portions of URLs.
• Added option common_url_parts to compression common portions of URLs. Requires rebuilding databases when changed.
• Added option remove_default_doc to control whether ht://Dig strips off the default document in a folder. Set to empty will prevent problems with servers that treat / and /index.html as different URLs.
• Of course there are many other bug-fixes and small enhancements. Many thanks to everyone who reported a bug or contributed code for this release!

Release notes for htdig-3.1.0b4 22 Dec 1998
This version fixes a security hole in htnotify. The hole has been present in previous versions but was inadevertently made worse in the 3.1.0 beta releases. Malicious users could contstruct pages that executed commands running under the shell of the user running htnotify. It is highly recommended that users of previous versions switch to this release.

• Fixed a memory leak in htnotify and htsearch.
• Updated the contributed parse_word_doc.pl script.

Release notes for htdig-3.1.0b3 15 Dec 1998
This version adds only a few features and a significant number of bug fixes. This version has been pretty thoroughly tested. Though there are a few remaining issues, it is hoped that this will be near the end of the beta releases before version 3.1.0. Note that it's recommended to update your databases to eliminate the possibility of subtle changes in the database format.

• Fixed a bug which would ignore the proxy settings, introduced in version 3.1.0b2.
• Fixed a bug where words would remain from deleted documents.
• Fixed a bug where SGML < was considered part of a tag in the HTML parser, introduced in verison 3.1.0b2.
• Fixed a bug where empty boolean searches would dump core.
• Fixed a bug where boolean "and," "or," and "not" would be removed from a search string, causing a sytnax error.
• Fixed a bug which wouldn't keep track of the hopcounts correctly.
• Added support for META refresh tags, contributed by Aidas Kasparas
• Added support for using CGI environment variables in the search templates, contributed by Gilles Detillieux.
• Improved memory requirements slightly through fixing a memory leak in htdig and a general system-wide adjustment.
• Improved support for multiple exclude and restrict items through htsearch, contributed by William Rhee and Gilles.
• Improved support to compile under CygWinB20, contributed by Klaus Mueller.
• Upgraded to the latest version (2.5.9) of the Berkeley DB
• Added a new option server_wait_time to give a delay between connections to a server. Currently this can also affect local filesystem digging if set.
• Added a new option server_max_docs to limit the number of documents pulled down from a server in one dig.
• Added a new option http_proxy_exclude to ignore the proxy setting on certain URLs.
• Added a new option no_excerpt_show_topto show the top of a document when there is no excerpt.
• Added new options date_factor, backlink_factor, and description_factor to improve search rankings. Respectively, they can give higher rankings to more recent documents, documents with a high number of links pointing to them, and documents with relevant URL descriptions pointing to them. See the documentation for more information.
• Added a set of contributed scripts called multidig to help work with multiple sets of URLs and databases.
• Fixed many compilation problems under AIX, thanks to Alexander Bergolth!
• Many other bugs were fixed, so a big thanks to everyone who submitted a bug report, patch or gave other feedback! See the ChangeLog for more details.

Release notes for htdig-3.1.0b2 1 Nov 1998
This version adds a few minor features as well as many bugfixes. It is still considered beta as some bug reports have not been fully examined.

• Fixed a major database corruption problem. Since this bug corrupted the document databases, to completely fix it, you will need to rebuild your databases from scratch.
• Fixed many problems with the Makefiles and configure scripts. Using ./configure --prefix= now works.
• Added fixes for connection problems with Digital Alpha-based systems contributed by Paul J. Meyer!
• Added support for syslog-based htsearch logging. See the config documentation for more details. Thanks to Leo Bergolth for this!
• Added fixes to work with DNS aliases (as opposed to virtual hosts) through the server_aliases and limit_normalized options as contributed by Leo Bergolth.
• Added cleanups of the HTML parser and the connection timeout code contributed by René Seindal.
• Now supports case insensitive servers through the case_sensitive option.
• Now supports ISO 8601 date format, using the iso_8601 option.
• Added a wrapper to emulate Exite for Web Servers (EWS) contributed by John Grohol.
• Added fixes to the contrib whatsnew.pl script to work with DB2 contributed by Jacques Reynes.
• Added a new contributed synonyms file from John Banbury
• Added a new template variable: CURRENT, the number of the current match, from a patch by René Seindal.
• Many other minor bugs were fixed, so a big thanks to everyone who submitted a bug report or a patch! See the ChangeLog for more details.

Release notes for htdig-3.1.0b1 8 Sep 1998
This version adds several major new features as well as some bug-fixes. It is considered a beta release since it has only seen limited testing.

It is extremely important that you rebuild all your databases made with previous versions. This version no longer uses the GDBM database format and databases produced with it will be incompatible with other versions. Do not blame me for anything if you didn't do this. You have been warned...

• Added patches made by Pasi Eronen to support local filesystem access
• Added a PDF parser contributed by Sylvain Wallez
• Added support for META description and robots tags
• Converted the database code to use the BerkeleyDB format, contibuted by Esa Ahola and Jesse op den Brouw.
• Added a prefix fuzzy algorithm, contributed by Esa and Jesse.
• Various other bugs were fixed. Thanks for all the patches that were sent to me and the mailing list!

Release notes for htdig-3.0.8b2 15 Aug 1997
This new version contains most of the patches that Pasi Eronen has posted to the list plus some other random fixes.

Release notes for htdig-3.0.8b1 27-Apr-1997
I consider this a beta release since I have not had time to test everything. Use at your own risk...

• Base tag problem fixed
• URL parser somewhat more robust
• Date parsing bug fixed
• Added Substring fuzzy algorithm.
• Various other bugs were fixed. Thanks for all the patches that were sent to me!

Release notes for htdig-3.0.7 12-Jan-1997
More bug fixes and some minor new functionality. Hopefully, I'll be able to finish up work on version 3.1 at some point in the near future.
I have recently received some more patches for various things, but I have not incorporated those, yet. Next version.

• The problem with the missing words has been fixed. This was a problem in the Dictionary class.
• htsearch is a *lot* faster due to a patch by Esa Ahola.
• htfuzzy has some work done to it. With the addition of the new rx-1.4 library, the endings algorithm now actually works for languages other than English... It still takes an awfully long time to build the tables for languages with lots of rules.
• URLs now can be of the dubious form http:foo.html I have never seen this used and think it is bogus, but alas, it works now.
• A search form can now manually add words to any search using the new keywords form attribute.
• A problem in the plaintext parser used to cause bogus HTML in search results. This has been fixed.
• New documentation format. Lots of new documentation, as well.
• New robotstxt_name attribute. Used to match the 'user-agent' lines in robots.txt files.
• The <base> tag is now properly supported.
• Preliminary support for lots of new features, including:
  • External document parsers. You'll be able to write your own document parser for that special document type that ht://Dig doesn't know about.
  • New fuzzy search algorithms: substring, regex, globbing, etc.

Release notes for htdig-3.0.6 26-Oct-1996
Just a single bug fix and one additional feature in this release.

• Fixed the problem that caused frequent crashes with virtual memory exhausted.
• Added a new attribute, keywords_meta_tag_names, which should contain a list of meta tag names for which the content should be used as keywords. The default is set to "keywords htdig-keywords"

Release notes for htdig-3.0.5 13-Oct-1996
This release consists of more bug fixes.
I want to thank Elliot Lee <sopwith@cuc.edu> for his help with tracking down several bugs.

• Fixed problem with accent characters. Words with SGML entities and iso-8859-1 characters will now be indexed correctly.
• Changed the auto configuration to detect the need for a prototype for the gethostname() function. (This was supposed to be fixed before, but wasn't)
• Reduced the memory requirements for all the programs by changing the rehash() method in the Dictionary class. Access to hashes may be a little slower, but the memory requirements were reduced by a factor 10 or so.
• Hopefully fixed a problem with the time related functions on certain platforms. More checks are done to make sure the functions that are used are actually available.

Release notes for htdig-3.0.4 2-Sep-1996
The previous version failed to build under Linux. This should be fixed now.

• Fixed problem with the time stuff which caused the build of htdig to fail.
• Fixed a memory problem in htdig

Release notes for htdig-3.0.3 2-Sep-1996
Bugs bugs bugs... Will they ever all be found?

NOTE: I made extensive changes to the htdig.conf file that gets installed. I would advise you to remove or rename your existing htdig.conf and let the installation process create a new one for you that you can then modify.

Also, since the rundig script has changed, you should remove the old one before installing ht://Dig. (The installation will refuse to overwrite existing files...)

• The problem with htsearch crashing on some machines has been fixed.
• A bug caused the <AREA> tab to be ignored. Fixed.
• A bug in SunOS caused dates to be all screwed up.
• Added lots of comments to the example htdig.conf file. Also added some additional example attributes.
• Fixed a bug in the installation process which caused rundig to be created incorrectly.
• Added a sample synonyms file. Also modified rundig to create a synonyms database for it.

Release notes for htdig-3.0.2 22-Aug-1996
More bug fixes.

• Multiple start URLs now actually work. Before they were just documented to work, but didn't actually work.
• htmerge now will refuse to remove database files if it detects that the call to /bin/sort failed.
• htmerge can now tell /bin/sort to use a specific temporary directory. This is done by setting the TMPDIR environment variable.
• htsearch can now search for words with non-ASCII characters in them.
• Added support for finding URLs in the <frame> and <area> tags.
• There is a problem with htsearch under Linux. It causes a segmentation violation after the first search result is displayed. Don't know what the problem is, yet.
• Fixed bug in the auto configuration which always set the value for NEED_PROTO_GETHOSTNAME to 1. For most systems this actually needs to be 0.
• Release notes for htdig-3.0.1 16-Aug-1996
  This is a maintenance release in response to several bug reports.
  • htdig now will display a list of errors when the statistics option (-s) is used. The list gives the URL that caused the error and a URL that referred to it. Hopefully this information is useful for site maintainers.
  • Some problems with the SGML character entities were fixed. The major symptom was that the ';' that ends an entity used to be included as well.
  • Major problems with htnotify were fixed. There were many hardcoded things in this program that made it very specific to SDSU and to me.
  • malloc.h should not be included anymore. All references to it were replaced with stdlib.h instead. This should make compiles on some platforms work better.
  • htsearch now will use the CONFIG_DIR environment variable to override the compiled in default. (set in the CONFIG file...) This was done so that htsearch can be called from a simple wrapper that sets that environment variable. Only the wrapper needs to be be modified to get different CONFIG_DIR values.

Release notes for htdig-3.0 17-Jul-1996
I decided to make this the official 3.0 release.

It is extremely important that you remove all traces of earlier beta versions of the software before installing this version or that you install in a completely different location. Do not blame me for anything if you didn't do this. You have been warned...

• htwrapper is no more. htsearch is now the CGI program
• htsearch now uses templates to display the results. A template is simply a piece of HTML code for a single match. The HTML code includes variables that will be expanded to the various items that are unique to each match, like URL, EXCERPT, TITLE, etc. The template can be selected at search time (through a menu). There are two builtin templates: builtin-short and builtin-long. The builtin-short template just lists the stars and title while the builtin-long template lists results in a similar fashion to the way Alta Vista displays results.
• Many runtime configuration options have been removed and many new ones have been added. Check the configuration file documentation for details. There are also some enhancements to the format of the configuration file.
  • Attribute values can now span multiple lines by ending each line that needs to be continued with a backslash ('\'). The file that is specified is read in and all newlines and starting and trailing whitespaces are reduced to a single space. If the file is not found, nothing is included and no error is flagged.
    Note that the backquote character is used, not the regular quote character.
  • Attribute values can now include the contents of files. Just put the filename in back-quotes. The filename can use the normal variable expansion so that things like:

    someattribute: `${common_dir}/somefile`

  Notable attribute changes:
  • All the attributes that set the heading text have been removed. These attributes include:
    • accessed_heading_text
    • datesize_heading_text
    • descriptions_heading_text
    • excerpt_heading_text
    • modified_heading_text
    • score_heading_text
    • size_heading_text
    • url_heading_text
    • wordlist_heading_text
    • field_order
  • New attributes added:

    http_proxy

    Added to support the use of a HTTP proxy server to index documents

    locale

    Added to support international character sets

    match_method

    New way of specifying if a search is an 'or', 'and', or 'boolean' search

    matches_per_page

    The new paged results uses this

    max_doc_size

    Limit the size of documents retrieved

    next_page_text

    Used in the navigation between pages

    no_excerpt_text

    Text displayed if no excerpt was available (this used to be hard-coded)

    no_next_page_text

    Used in the navigation between pages

    no_prev_page_text

    Used in the navigation between pages

    prev_page_text

    Used in the navigation between pages

    star_patterns

    Allow different star images to be used depending on the match URL

    synonym_dictionary

    Support for the new synonyms fuzzy algorithm

    synonym_db

    Support for the new synonyms fuzzy algorithm

    syntax_error_file

    HTML file displayed if there was a boolean expression syntax error

    template_map

    Used in the support for the new result display templates

    template_name

    Sets the default template name

    text_factor

    Added to allow normal text to have a variable weight (0, for example...)

  • Some form tag names have changed. The list of recognized form tags are in the htsearch documentation.
  • Multiple start urls can be specified as a value to the 'start_url' attribute. This could be combined with the file inclusion to read in a file of URLs to start with.
  • htdig now sends the 'Referer:' header in HTTP requests so that any link errors will be logged in the server's log files.
  • In addition to the "htdig-keywords" META tag name, htdig now also supports just "keywords". This is to make it more compatible with the Alta Vista search engine.
  • The verbose display of htdig was enhanced to show '+' for a link that will be followed and '-' for a link that was discarded.
  • htmerge was changed to use the Unix sort program instead of doing its own sorting. It no longer uses mmap() to map the words into memory. This was causing problems on systems with limited virtual memory available. (What??? You mean you DON'T have at least a 1GB disk dedicated to swap???)
  • The Endings algorithm was fixed up to work properly now. There were several well hidden bugs that made the algorithm come up with illegal words.
  • The synonyms fuzzy algorithm was added. This is simply a mapping of words to other words. The input file is just a list of words which causes the first word on a line to be mapped to the rest of the words on that line. (We use this to map course abbreviations to full course names)
  • SGML entities are now supported. They are translated to their equivalent ISO-8859-1 encoding.