12.6. 字典

Dictionaries are used to eliminate words that should not be considered in a search (stop words), and tonormalize_words so that different derived forms of the same word will match. A successfully normalized word is called a_lexeme. Aside from improving search quality, normalization and removal of stop words reduce the size of thetsvectorrepresentation of a document, thereby improving performance. Normalization does not always have linguistic meaning and usually depends on application semantics.

Some examples of normalization:

  • Linguistic - Ispell dictionaries try to reduce input words to a normalized form; stemmer dictionaries remove word endings
  • URLlocations can be canonicalized to make equivalent URLs match:
  • Color names can be replaced by their hexadecimal values, e.g.,red, green, blue, magenta -> FF0000, 00FF00, 0000FF, FF00FF
  • If indexing numbers, we can remove some fractional digits to reduce the range of possible numbers, so for example_3.14_159265359,_3.14_15926,_3.14_will be the same after normalization if only two digits are kept after the decimal point.

A dictionary is a program that accepts a token as input and returns:

  • an array of lexemes if the input token is known to the dictionary (notice that one token can produce more than one lexeme)
  • a single lexeme with theTSL_FILTERflag set, to replace the original token with a new token to be passed to subsequent dictionaries (a dictionary that does this is called afiltering dictionary)
  • an empty array if the dictionary knows the token, but it is a stop word
  • NULLif the dictionary does not recognize the input token

PostgreSQLprovides predefined dictionaries for many languages. There are also several predefined templates that can be used to create new dictionaries with custom parameters. Each predefined dictionary template is described below. If no existing template is suitable, it is possible to create new ones; see thecontrib/area of thePostgreSQLdistribution for examples.

A text search configuration binds a parser together with a set of dictionaries to process the parser’s output tokens. For each token type that the parser can return, a separate list of dictionaries is specified by the configuration. When a token of that type is found by the parser, each dictionary in the list is consulted in turn, until some dictionary recognizes it as a known word. If it is identified as a stop word, or if no dictionary recognizes the token, it will be discarded and not indexed or searched for. Normally, the first dictionary that returns a non-NULLoutput determines the result, and any remaining dictionaries are not consulted; but a filtering dictionary can replace the given word with a modified word, which is then passed to subsequent dictionaries.

The general rule for configuring a list of dictionaries is to place first the most narrow, most specific dictionary, then the more general dictionaries, finishing with a very general dictionary, like aSnowballstemmer orsimple, which recognizes everything. For example, for an astronomy-specific search (astro_enconfiguration) one could bind token typeasciiword(ASCII word) to a synonym dictionary of astronomical terms, a general English dictionary and aSnowballEnglish stemmer:

  1. ALTER TEXT SEARCH CONFIGURATION astro_en
  2.     ADD MAPPING FOR asciiword WITH astrosyn, english_ispell, english_stem;

A filtering dictionary can be placed anywhere in the list, except at the end where it’d be useless. Filtering dictionaries are useful to partially normalize words to simplify the task of later dictionaries. For example, a filtering dictionary could be used to remove accents from accented letters, as is done by theunaccentmodule.

12.6.1. Stop Words

Stop words are words that are very common, appear in almost every document, and have no discrimination value. Therefore, they can be ignored in the context of full text searching. For example, every English text contains words likeaandthe, so it is useless to store them in an index. However, stop words do affect the positions intsvector, which in turn affect ranking:

  1. SELECT to_tsvector('english','in the list of stop words');
  2.         to_tsvector
  3. ----------------------------
  4.  'list':3 'stop':5 'word':6

The missing positions 1,2,4 are because of stop words. Ranks calculated for documents with and without stop words are quite different:

  1. SELECT ts_rank_cd (to_tsvector('english','in the list of stop words'), to_tsquery('list 
  2. &
  3.  stop'));
  4.  ts_rank_cd
  5. ------------
  6.        0.05
  8. SELECT ts_rank_cd (to_tsvector('english','list stop words'), to_tsquery('list 
  9. &
  10.  stop'));
  11.  ts_rank_cd
  12. ------------
  13.         0.1

It is up to the specific dictionary how it treats stop words. For example,ispelldictionaries first normalize words and then look at the list of stop words, whileSnowballstemmers first check the list of stop words. The reason for the different behavior is an attempt to decrease noise.

12.6.2. Simple Dictionary

Thesimpledictionary template operates by converting the input token to lower case and checking it against a file of stop words. If it is found in the file then an empty array is returned, causing the token to be discarded. If not, the lower-cased form of the word is returned as the normalized lexeme. Alternatively, the dictionary can be configured to report non-stop-words as unrecognized, allowing them to be passed on to the next dictionary in the list.

Here is an example of a dictionary definition using thesimpletemplate:

  1. CREATE TEXT SEARCH DICTIONARY public.simple_dict (
  2.     TEMPLATE = pg_catalog.simple,
  3.     STOPWORDS = english
  4. );

Here,englishis the base name of a file of stop words. The file’s full name will be$SHAREDIR/tsearch_data/english.stop, where$SHAREDIRmeans thePostgreSQLinstallation’s shared-data directory, often/usr/local/share/postgresql(usepg_config --sharedirto determine it if you’re not sure). The file format is simply a list of words, one per line. Blank lines and trailing spaces are ignored, and upper case is folded to lower case, but no other processing is done on the file contents.

Now we can test our dictionary:

  1. SELECT ts_lexize('public.simple_dict','YeS');
  2.  ts_lexize
  3. -----------
  4.  {yes}
  6. SELECT ts_lexize('public.simple_dict','The');
  7.  ts_lexize
  8. -----------
  9.  {}

We can also choose to returnNULL, instead of the lower-cased word, if it is not found in the stop words file. This behavior is selected by setting the dictionary’sAcceptparameter tofalse. Continuing the example:

  1. ALTER TEXT SEARCH DICTIONARY public.simple_dict ( Accept = false );
  3. SELECT ts_lexize('public.simple_dict','YeS');
  4.  ts_lexize
  5. -----------
  8. SELECT ts_lexize('public.simple_dict','The');
  9.  ts_lexize
  10. -----------
  11.  {}

With the default setting ofAccept=true, it is only useful to place asimpledictionary at the end of a list of dictionaries, since it will never pass on any token to a following dictionary. Conversely,Accept=falseis only useful when there is at least one following dictionary.

Caution

Most types of dictionaries rely on configuration files, such as files of stop words. These files_must_be stored in UTF-8 encoding. They will be translated to the actual database encoding, if that is different, when they are read into the server.

Caution

Normally, a database session will read a dictionary configuration file only once, when it is first used within the session. If you modify a configuration file and want to force existing sessions to pick up the new contents, issue anALTER TEXT SEARCH DICTIONARYcommand on the dictionary. This can be a“dummy”update that doesn’t actually change any parameter values.

12.6.3. Synonym Dictionary

This dictionary template is used to create dictionaries that replace a word with a synonym. Phrases are not supported (use the thesaurus template (Section 12.6.4) for that). A synonym dictionary can be used to overcome linguistic problems, for example, to prevent an English stemmer dictionary from reducing the word“Paris”to“pari”. It is enough to have aParis parisline in the synonym dictionary and put it before theenglish_stemdictionary. For example:

  1. SELECT * FROM ts_debug('english', 'Paris');
  2.    alias   |   description   | token |  dictionaries  |  dictionary  | lexemes 
  3. -----------+-----------------+-------+----------------+--------------+---------
  4.  asciiword | Word, all ASCII | Paris | {english_stem} | english_stem | {pari}
  6. CREATE TEXT SEARCH DICTIONARY my_synonym (
  7.     TEMPLATE = synonym,
  8.     SYNONYMS = my_synonyms
  9. );
  11. ALTER TEXT SEARCH CONFIGURATION english
  12.     ALTER MAPPING FOR asciiword
  13.     WITH my_synonym, english_stem;
  15. SELECT * FROM ts_debug('english', 'Paris');
  16.    alias   |   description   | token |       dictionaries        | dictionary | lexemes 
  17. -----------+-----------------+-------+---------------------------+------------+---------
  18.  asciiword | Word, all ASCII | Paris | {my_synonym,english_stem} | my_synonym | {paris}

The only parameter required by thesynonymtemplate isSYNONYMS, which is the base name of its configuration file —my_synonymsin the above example. The file’s full name will be$SHAREDIR/tsearch_data/my_synonyms.syn(where$SHAREDIRmeans thePostgreSQLinstallation’s shared-data directory). The file format is just one line per word to be substituted, with the word followed by its synonym, separated by white space. Blank lines and trailing spaces are ignored.

Thesynonymtemplate also has an optional parameterCaseSensitive, which defaults tofalse. WhenCaseSensitiveisfalse, words in the synonym file are folded to lower case, as are input tokens. When it istrue, words and tokens are not folded to lower case, but are compared as-is.

An asterisk (*) can be placed at the end of a synonym in the configuration file. This indicates that the synonym is a prefix. The asterisk is ignored when the entry is used into_tsvector(), but when it is used into_tsquery(), the result will be a query item with the prefix match marker (seeSection 12.3.2). For example, suppose we have these entries in$SHAREDIR/tsearch_data/synonym_sample.syn:

  1. postgres        pgsql
  2. postgresql      pgsql
  3. postgre pgsql
  4. gogle   googl
  5. indices index*

Then we will get these results:

  1. mydb=# CREATE TEXT SEARCH DICTIONARY syn (template=synonym, synonyms='synonym_sample');
  2. mydb=# SELECT ts_lexize('syn','indices');
  3.  ts_lexize
  4. -----------
  5.  {index}
  6. (1 row)
  8. mydb=# CREATE TEXT SEARCH CONFIGURATION tst (copy=simple);
  9. mydb=# ALTER TEXT SEARCH CONFIGURATION tst ALTER MAPPING FOR asciiword WITH syn;
  10. mydb=# SELECT to_tsvector('tst','indices');
  11.  to_tsvector
  12. -------------
  13.  'index':1
  14. (1 row)
  16. mydb=# SELECT to_tsquery('tst','indices');
  17.  to_tsquery
  18. ------------
  19.  'index':*
  20. (1 row)
  22. mydb=# SELECT 'indexes are very useful'::tsvector;
  23.             tsvector             
  24. ---------------------------------
  25.  'are' 'indexes' 'useful' 'very'
  26. (1 row)
  28. mydb=# SELECT 'indexes are very useful'::tsvector @@ to_tsquery('tst','indices');
  29.  ?column?
  30. ----------
  31.  t
  32. (1 row)

12.6.4. Thesaurus Dictionary

A thesaurus dictionary (sometimes abbreviated asTZ) is a collection of words that includes information about the relationships of words and phrases, i.e., broader terms (BT), narrower terms (NT), preferred terms, non-preferred terms, related terms, etc.

Basically a thesaurus dictionary replaces all non-preferred terms by one preferred term and, optionally, preserves the original terms for indexing as well.PostgreSQL’s current implementation of the thesaurus dictionary is an extension of the synonym dictionary with added_phrase_support. A thesaurus dictionary requires a configuration file of the following format:

  1. # this is a comment
  2. sample word(s) : indexed word(s)
  3. more sample word(s) : more indexed word(s)
  4. ...

where the colon (:) symbol acts as a delimiter between a phrase and its replacement.

A thesaurus dictionary uses asubdictionary(which is specified in the dictionary’s configuration) to normalize the input text before checking for phrase matches. It is only possible to select one subdictionary. An error is reported if the subdictionary fails to recognize a word. In that case, you should remove the use of the word or teach the subdictionary about it. You can place an asterisk (*) at the beginning of an indexed word to skip applying the subdictionary to it, but all sample words_must_be known to the subdictionary.

The thesaurus dictionary chooses the longest match if there are multiple phrases matching the input, and ties are broken by using the last definition.

Specific stop words recognized by the subdictionary cannot be specified; instead use?to mark the location where any stop word can appear. For example, assuming thataandtheare stop words according to the subdictionary:

  1. ? one ? two : swsw

matchesa one the twoandthe one a two; both would be replaced byswsw.

Since a thesaurus dictionary has the capability to recognize phrases it must remember its state and interact with the parser. A thesaurus dictionary uses these assignments to check if it should handle the next word or stop accumulation. The thesaurus dictionary must be configured carefully. For example, if the thesaurus dictionary is assigned to handle only theasciiwordtoken, then a thesaurus dictionary definition likeone 7will not work since token typeuintis not assigned to the thesaurus dictionary.

Caution

Thesauruses are used during indexing so any change in the thesaurus dictionary’s parameters_requires_reindexing. For most other dictionary types, small changes such as adding or removing stopwords does not force reindexing.

12.6.4.1. Thesaurus Configuration

To define a new thesaurus dictionary, use thethesaurustemplate. For example:

  1. CREATE TEXT SEARCH DICTIONARY thesaurus_simple (
  2.     TEMPLATE = thesaurus,
  3.     DictFile = mythesaurus,
  4.     Dictionary = pg_catalog.english_stem
  5. );

Here:

  • thesaurus_simpleis the new dictionary’s name
  • mythesaurusis the base name of the thesaurus configuration file. (Its full name will be$SHAREDIR/tsearch_data/mythesaurus.ths, where$SHAREDIRmeans the installation shared-data directory.)
  • pg_catalog.english_stemis the subdictionary (here, a Snowball English stemmer) to use for thesaurus normalization. Notice that the subdictionary will have its own configuration (for example, stop words), which is not shown here.

Now it is possible to bind the thesaurus dictionarythesaurus_simpleto the desired token types in a configuration, for example:

  1. ALTER TEXT SEARCH CONFIGURATION russian
  2.     ALTER MAPPING FOR asciiword, asciihword, hword_asciipart
  3.     WITH thesaurus_simple;

12.6.4.2. Thesaurus Example

Consider a simple astronomical thesaurusthesaurus_astro, which contains some astronomical word combinations:

  1. supernovae stars : sn
  2. crab nebulae : crab

Below we create a dictionary and bind some token types to an astronomical thesaurus and English stemmer:

  1. CREATE TEXT SEARCH DICTIONARY thesaurus_astro (
  2.     TEMPLATE = thesaurus,
  3.     DictFile = thesaurus_astro,
  4.     Dictionary = english_stem
  5. );
  7. ALTER TEXT SEARCH CONFIGURATION russian
  8.     ALTER MAPPING FOR asciiword, asciihword, hword_asciipart
  9.     WITH thesaurus_astro, english_stem;

Now we can see how it works.ts_lexizeis not very useful for testing a thesaurus, because it treats its input as a single token. Instead we can useplainto_tsqueryandto_tsvectorwhich will break their input strings into multiple tokens:

  1. SELECT plainto_tsquery('supernova star');
  2.  plainto_tsquery
  3. -----------------
  4.  'sn'
  6. SELECT to_tsvector('supernova star');
  7.  to_tsvector
  8. -------------
  9.  'sn':1

In principle, one can useto_tsqueryif you quote the argument:

  1. SELECT to_tsquery('''supernova star''');
  2.  to_tsquery
  3. ------------
  4.  'sn'

Notice thatsupernova starmatchessupernovae starsinthesaurus_astrobecause we specified theenglish_stemstemmer in the thesaurus definition. The stemmer removed theeands.

To index the original phrase as well as the substitute, just include it in the right-hand part of the definition:

  1. supernovae stars : sn supernovae stars
  3. SELECT plainto_tsquery('supernova star');
  4.        plainto_tsquery
  5. -----------------------------
  6.  'sn' 
  7. &
  8.  'supernova' 
  9. &
  10.  'star'

12.6.5. IspellDictionary

TheIspelldictionary template supportsmorphological dictionaries, which can normalize many different linguistic forms of a word into the same lexeme. For example, an EnglishIspelldictionary can match all declensions and conjugations of the search termbank, e.g.,banking,banked,banks,banks', andbank's.

The standardPostgreSQLdistribution does not include anyIspellconfiguration files. Dictionaries for a large number of languages are available fromIspell. Also, some more modern dictionary file formats are supported —MySpell(OO < 2.0.1) andHunspell(OO >= 2.0.2). A large list of dictionaries is available on theOpenOffice Wiki.

To create anIspelldictionary perform these steps:

  • download dictionary configuration files.OpenOfficeextension files have the.oxtextension. It is necessary to extract.affand.dicfiles, change extensions to.affixand.dict. For some dictionary files it is also needed to convert characters to the UTF-8 encoding with commands (for example, for a Norwegian language dictionary):

    1. iconv -f ISO_8859-1 -t UTF-8 -o nn_no.affix nn_NO.aff
    2. iconv -f ISO_8859-1 -t UTF-8 -o nn_no.dict nn_NO.dic

  • copy files to the$SHAREDIR/tsearch_datadirectory

  • load files into PostgreSQL with the following command:

    1. CREATE TEXT SEARCH DICTIONARY english_hunspell (
    2.     TEMPLATE = ispell,
    3.     DictFile = en_us,
    4.     AffFile = en_us,
    5.     Stopwords = english);

Here,DictFile,AffFile, andStopWordsspecify the base names of the dictionary, affixes, and stop-words files. The stop-words file has the same format explained above for thesimpledictionary type. The format of the other files is not specified here but is available from the above-mentioned web sites.

Ispell dictionaries usually recognize a limited set of words, so they should be followed by another broader dictionary; for example, a Snowball dictionary, which recognizes everything.

The.affixfile ofIspellhas the following structure:

  1. prefixes
  2. flag *A:
  3.     .           
  4. >
  5.    RE      # As in enter 
  6. >
  7.  reenter
  8. suffixes
  9. flag T:
  10.     E           
  11. >
  12.    ST      # As in late 
  13. >
  14.  latest
  15.     [^AEIOU]Y   
  16. >
  17.    -Y,IEST # As in dirty 
  18. >
  19.  dirtiest
  20.     [AEIOU]Y    
  21. >
  22.    EST     # As in gray 
  23. >
  24.  grayest
  25.     [^EY]       
  26. >
  27.    EST     # As in small 
  28. >
  29.  smallest

And the.dictfile has the following structure:

  1. lapse/ADGRS
  2. lard/DGRS
  3. large/PRTY
  4. lark/MRS

Format of the.dictfile is:

  1. basic_form/affix_class_name

In the.affixfile every affix flag is described in the following format:

  1. condition 
  2. >
  3.  [-stripping_letters,] adding_affix

Here, condition has a format similar to the format of regular expressions. It can use groupings[...]and[^...]. For example,[AEIOU]Ymeans that the last letter of the word is"y"and the penultimate letter is"a","e","i","o"or"u".[^EY]means that the last letter is neither"e"nor"y".

Ispell dictionaries support splitting compound words; a useful feature. Notice that the affix file should specify a special flag using thecompoundwords controlledstatement that marks dictionary words that can participate in compound formation:

  1. compoundwords  controlled z

Here are some examples for the Norwegian language:

  1. SELECT ts_lexize('norwegian_ispell', 'overbuljongterningpakkmesterassistent');
  2.    {over,buljong,terning,pakk,mester,assistent}
  3. SELECT ts_lexize('norwegian_ispell', 'sjokoladefabrikk');
  4.    {sjokoladefabrikk,sjokolade,fabrikk}

MySpellformat is a subset ofHunspell. The.affixfile ofHunspellhas the following structure:

  1. PFX A Y 1
  2. PFX A   0     re         .
  3. SFX T N 4
  4. SFX T   0     st         e
  5. SFX T   y     iest       [^aeiou]y
  6. SFX T   0     est        [aeiou]y
  7. SFX T   0     est        [^ey]

The first line of an affix class is the header. Fields of an affix rules are listed after the header:

  • parameter name (PFX or SFX)
  • flag (name of the affix class)
  • stripping characters from beginning (at prefix) or end (at suffix) of the word
  • adding affix
  • condition that has a format similar to the format of regular expressions.

The.dictfile looks like the.dictfile ofIspell:

  1. larder/M
  2. lardy/RT
  3. large/RSPMYT
  4. largehearted

Note

MySpelldoes not support compound words.Hunspellhas sophisticated support for compound words. At present,PostgreSQLimplements only the basic compound word operations of Hunspell.

12.6.6. SnowballDictionary

TheSnowballdictionary template is based on a project by Martin Porter, inventor of the popular Porter’s stemming algorithm for the English language. Snowball now provides stemming algorithms for many languages (see theSnowball sitefor more information). Each algorithm understands how to reduce common variant forms of words to a base, or stem, spelling within its language. A Snowball dictionary requires alanguageparameter to identify which stemmer to use, and optionally can specify astopwordfile name that gives a list of words to eliminate. (PostgreSQL’s standard stopword lists are also provided by the Snowball project.) For example, there is a built-in definition equivalent to

  1. CREATE TEXT SEARCH DICTIONARY english_stem (
  2.     TEMPLATE = snowball,
  3.     Language = english,
  4.     StopWords = english
  5. );

The stopword file format is the same as already explained.

ASnowballdictionary recognizes everything, whether or not it is able to simplify the word, so it should be placed at the end of the dictionary list. It is useless to have it before any other dictionary because a token will never pass through it to the next dictionary.