• Splits words at punctuation characters, removing punctuation. However, a dot that’s not followed by whitespace is considered part of a token.
• Not suitable for file names because the .extension is treated as part of token.
• Splits words at hyphens, unless there’s a number in the token, in which case the whole token is interpreted as a product number and is not split.
• Recognizes email addresses and internet hostnames as one token.

Example: This sentencet can’t be “tokenized_Correctly” by www.google.com or IBM  or NATO 10.1.9.5 test@email.org product-number 123-456949 file.txt is tokenized as This sentence can’t be tokenized Correctly by www.google.com or IBM or NATO 10.1.9.5 test@email.org product number 123-456949 file.txt

• “pattern” is the regular expression.
• “group” says which group to extract into tokens.

group=-1 (the default) is equivalent to “split”. In this case, the tokens will be equivalent to the output from (without empty tokens): String.split(java.lang.String) Using group >= 0 selects the matching group as the token. For example, if you have:

  pattern = '([^']+)'
  group = 0
  input = aaa 'bbb' 'ccc'

the output will be two tokens: ‘bbb’ and ‘ccc’ (including the ‘ marks). With the same input but using group=1, the output would be: bbb and ccc (no ‘ marks)

• split on intra-word delimiters (all non alpha-numeric characters).
  • "Wi-Fi" -> "Wi", "Fi"
• split on case transitions (can be turned off – see splitOnCaseChange parameter)
  • "PowerShot" -> "Power", "Shot"
• split on letter-number transitions (can be turned off – see splitOnNumerics parameter)
  • "SD500" -> "SD", "500"
• leading and trailing intra-word delimiters on each subword are ignored
  • "//hello---there, 'dude'" -> "hello", "there", "dude"
• trailing “‘s” are removed for each subword (can be turned off – see stemEnglishPossessive parameter)
  • "O'Neil's" -> "O", "Neil"
    • Note: this step isn’t performed in a separate filter because of possible subword combinations.

Splitting is affected by the following parameters:

• splitOnCaseChange=”1″causes lowercase => uppercase transitions to generate a new part [Solr 1.3]:
  • "PowerShot" => "Power" "Shot"
  • "TransAM" => "Trans" "AM"
  • default is true (“1″); set to 0 to turn off
• splitOnNumerics=”1″causes alphabet => number transitions to generate a new part [Solr 1.3]:
  • "j2se" => "j" "2" "se"
  • default is true (“1″); set to 0 to turn off
• stemEnglishPossessive=”1″causes trailing “‘s” to be removed for each subword.
  • "Doug's" => "Doug"
  • default is true (“1″); set to 0 to turn off

There are also a number of parameters that affect what tokens are present in the final output and if subwords are combined:

• generateWordParts=”1″causes parts of words to be generated:
  • "PowerShot" => "Power" "Shot" (if splitOnCaseChange=1)
  • "Power-Shot" => "Power" "Shot"
  • default is 0
• generateNumberParts=”1″causes number subwords to be generated:
  • "500-42" => "500" "42"
  • default is 0
• catenateWords=”1″causes maximum runs of word parts to be catenated:
  • "wi-fi" => "wifi"
  • default is 0
• catenateNumbers=”1″causes maximum runs of number parts to be catenated:
  • "500-42" => "50042"
  • default is 0
• catenateAll=”1″causes all subword parts to be catenated:
  • "wi-fi-4000" => "wifi4000"
  • default is 0
• preserveOriginal=”1″causes the original token to be indexed without modifications (in addition to the tokens produced due to other options)
  • default is 0

Example of generateWordParts=”1″ and catenateWords=”1″:

• "PowerShot" -> 0:"Power", 1:"Shot" 1:"PowerShot"(where 0,1,1 are token positions)
• "A's+B's&C's" -> 0:"A", 1:"B", 2:"C", 2:"ABC"
• "Super-Duper-XL500-42-AutoCoder!" -> 0:"Super",
• 1:"Duper",
• 2:"XL",
• 2:"SuperDuperXL",
• 3:"500"
• 4:"42",
• 5:"Auto",
• 6:"Coder",
• 6:"AutoCoder"

• The synonyms parameter names an external file defining the synonyms.
• If ignoreCase is true, matching will lowercase before checking equality.
• If expand is true, a synonym will be expanded to all equivalent synonyms. If it is false, all equivalent synonyms will be reduced to the first in the list.

Synonym file format: i-pod, i pod => ipod, sea biscuit, sea biscit => seabiscuit

• withOriginal – if true, then produce both original and reversed tokens at the same positions. If false, then produce only reversed tokens.
• maxPosAsterisk – maximum position (1-based) of the asterisk wildcard (‘*’) that triggers the reversal of query term. Asterisk that occurs at positions higher than this value will not cause the reversal of query term. Defaults to 2, meaning that asterisks on positions 1 and 2 will cause a reversal.
• maxPosQuestion – maximum position (1-based) of the question mark wildcard (‘?’) that triggers the reversal of query term. Defaults to 1. Set this to 0, and maxPosAsteriskto 1 to reverse only pure suffix queries (i.e. ones with a single leading asterisk).
• maxFractionAsterisk – additional parameter that triggers the reversal if asterisk (‘*’) position is less than this fraction of the query token length. Defaults to 0.0f (disabled).
• minTrailing – minimum number of trailing characters in query token after the last wildcard character. For good performance this should be set to a value larger than 1. Defaults to 2.

Note 1: This filter always reverses input tokens during indexing. Note 2: Query tokens without wildcard characters will never be reversed.

<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->

<!-- Case insensitive stop word removal. add enablePositionIncrements=true in both the index and query analyzers to leave a ‘gap’ for more accurate phrase queries. -->
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>

- Tokenizes at whitespaces

- Stop words are removed

- Words delimiters are used to generate word tokens.

generateWordParts=1 => wi-fi will generate wi and fi

generateNumberParts = 1 => 3.5 will generate 3 and 5

catenateWords=1 => wi-fi will generate wi, fi and wifi

catenateNumbers = 1 => 3.5 will generate 3,5 and 35

catenateAll = 1 => wi-fi-35 will generate wi, fi, wifi, 35 and wifi35.

catenateAll = 0 => wi-fi-35 will generate wi, fi, wifi and 35, but not wifi35.

splitOnCaseChange=1 => camelCase will generate camel and case.

- All text is changed to lower case.

- The Snowball porter stemmer will convert running to “run”

Querying behaviour:

<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>

In querying, only the synonym filter is additional. So something like TV which is in the synonym group “Television, Televisions, TV, TVs” results in this query token stream: televis televis tv tvs (“televis” is because “television” has been stemmed by Snowball Porter).

<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- Case insensitive stop word removal. add enablePositionIncrements=true in both the index and query analyzers to leave a ‘gap’ for more accurate phrase queries. -->
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>

- Tokenizes at whitespaces

- Stop words are removed

- Words delimiters are used to generate word tokens.

generateWordParts=1 => wi-fi will generate wi and fi

generateNumberParts = 1 => 3.5 will generate 3 and 5

catenateWords=1 => wi-fi will generate wi, fi and wifi

catenateNumbers = 1 => 3.5 will generate 3,5 and 35

catenateAll = 1 => wi-fi-35 will generate wi, fi, wifi, 35 and wifi35.

catenateAll = 0 => wi-fi-35 will generate wi, fi, wifi and 35, but not wifi35.

splitOnCaseChange=1 => camelCase will generate camel and case.

- All text is changed to lower case.

- Note that there is no stemmer, which is what makes this different from “text” type.

Querying behaviour:

<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>

In querying, only the synonym filter is additional. So something like TV which is in the synonym group “Television, Televisions, TV, TVs” results in this query token stream: television televisions tv tvs

For file paths and filenames, “textgen” seems to give the most appropriate results.