Contents

Last modification: by Holmes Maintainers. No counters, no frames, no syntax errors.

Sherlock Holmes Search Engine

What's that?

Sherlock Holmes is a universal search engine – a system for gathering and indexing of textual, image, and audio data (text files, web pages, ...), both locally and over the network. Better than explaining what does such a thing really mean, let's look at some of its features:

Features

• Gathers files via HTTP or from local files.
• Parses text files, HTML, PDF, MS Word, MS Excel, and several other formats using external parsers (such as PostScript). Parses JPEG, PNG, and GIF images, stores their thumbnails, detects duplicates, and searches for similar images. Parses MP3 and OGG files, and stores their ID3 tags.
• The whole system is modular, so adding your own data sources or parsers is just a matter of plugging in the right module (well, usually also writing it :) ).
• Works well in mixed charset environment.
• Considers multiple occurences of the same file (even with minor changes) a single document with multiple URL's.
• Site compression (the search server may only show the best 2 pages from one domain).
• Automatic page ranking (based on the links between the pages).
• New intelligent gatherer (it chooses which pages to gather according to their ranks, it is much faster and more scalable, and it can refresh pages much more often than the old gatherer from before v4.0).
• Splitting the index into independent areas (one search engine for a number of domains).
• Everything is highly configurable. You can write filtering rules in a special language which allows to tweak configuration variables depending on the document being processed.
• Searching of words, phrases, and boolean expressions. Searching in file names and link texts.
• Proximity search and proximity weighting of regular searches.
• Recognition of languages, easy integration of stemmers and synonymic dictionaries.
• Spelling checker based on word frequencies observed in the indexed data, hinting the user that his query might be misspelled.
• Search results include context in each document.
• Scales well to tens of millions of documents on normal PC hardware.
• User interface (the front-end) is completely separated from the rest of the system, making it easy to modify and also to embed the search engine in existing applications.
• Downloaded files and indices are compressed to save space.

License

Currently, two versions of Holmes exist. The freely distributable version is licenced under the GNU General Public License (GPL). Some libraries and support modules are distributed under the GNU Lesser General Public License (LGPL) and some example programs are public domain. In all such cases, it's clearly stated in comments at the start of the module.

The commercial version of the search engine contains some additional features and you can get paid support for it. If you are interested in it, please contact us. The feature list follows:

• Distributed gatherer (downloading documents much faster than any single machine could do).
• Distributed search server (serving documents at arbitrary load).
• Connection with a catalog (human-administrated titles and keywords are taken into account during the search, and displayed with the results).

Also, donations for development of features you'd like to use are welcome. So much for advertisements :)

If you want to incorporate Holmes in your applications or to use it for whatever non-standard purposes you'd like, please contact the authors – licenses for such cases are available as well and in case of academic research and non-profit projects, they are usually provided free of charge.

Download

The current version of Holmes is 4.0. You can download it via HTTP and FTP. For instructions on installation, see doc/install in the archive. Debian packages are available as well.

There is currently no mailing list of Holmes users, but you can find information about new releases at these pages and also in FreshMeat.

Several libraries from the Holmes project can be also used separately, so we have prepared a stand-alone library package. See the libUCW home page for more information.

Authors

In short: Holmes has been written by Martin Mareš and Robert Špalek. The development has been heavily sponsored by Netcentrum s.r.o. which is also one of the most important users and works as an exclusive distributor of the commercial version.

Here is the full story: Back in 1997, Martin Mareš wrote the first version called Sherlock 1.0 as his term project at MFF UK, but it somehow escaped from his control soon – in October 1997 it was indexing the whole .cz domain in cooperation with the Bajt company.

The time slowly passed by, the author was busy working on other stuff, Bajt had its own problems and the whole project would have been almost forgotten weren't it for people from Netcentrum who were building a new Czech portal, wanted to use Sherlock for searching and were willing to sponsor its further development.

After several years of successfully running Sherlock 1.2 on a couple of servers, Robert Špalek joined the "team" and together we decided to rewrite the whole project from scratch and change the whole architecture (confirming the ancient wisdom that every good program including TeX has to be rewritten at least once in its lifetime :) ). Unfortunately, we have been forced to delay the public release of this version for some time. So was it back in 2001.

In September 2002, we have resurrected the freeely distributable version of Sherlock, but in the meantime Apple started distributing another program of the same name as part of their OS X, so we decided to rename the whole package to Sherlock Holmes (or Holmes) to avoid both confusion and trademark problems.

In April 2009, we have moved most of the features from the commerial version to the free version, and released the package as version 4.0. The main distinction between the two versions is now that the commercial version can work distributed over many computers, whereas the free version only supports a single machine.

Reporting bugs

All reports of bugs or inconveniences and also ideas for new features (or patches implementing them :-) ) are welcome at holmes-bugs@ucw.cz.

If you use Holmes to your pleasure, the authors will be glad to hear about your experience at the same address.

Libraries

Many pieces of code developed originally for Holmes are now available as separate libraries for use in other projects. If you are interested, take a peek at the UCW Library.