Naoya Murakami
null+****@clear*****
Fri May 20 16:06:03 JST 2016
Naoya Murakami 2016-04-23 14:12:44 +0900 (Sat, 23 Apr 2016) New Revision: 033f48a7e69d6c904175bffecb2b7563959d4261 https://github.com/groonga/groonga/commit/033f48a7e69d6c904175bffecb2b7563959d4261 Merged 429e423: Merge pull request #533 from naoa/overlap-token-skip Message: test: add test for GRN_NGRAM_TOKENIZER_REMOVE_BLANK_DISABLE Added files: test/command/suite/tokenizers/bigram/env/remove_blank_disable/include_blank.expected test/command/suite/tokenizers/bigram/env/remove_blank_disable/include_blank.test Added: test/command/suite/tokenizers/bigram/env/remove_blank_disable/include_blank.expected (+50 -0) 100644 =================================================================== --- /dev/null +++ test/command/suite/tokenizers/bigram/env/remove_blank_disable/include_blank.expected 2016-04-23 14:12:44 +0900 (6a7e3db) @@ -0,0 +1,50 @@ +tokenize TokenBigramSplitSymbolAlpha "Hong Kong" NormalizerAuto --mode GET +[ + [ + 0, + 0.0, + 0.0 + ], + [ + { + "value": "ho", + "position": 0, + "force_prefix": false + }, + { + "value": "on", + "position": 1, + "force_prefix": false + }, + { + "value": "ng", + "position": 2, + "force_prefix": false + }, + { + "value": "g ", + "position": 3, + "force_prefix": false + }, + { + "value": " k", + "position": 4, + "force_prefix": false + }, + { + "value": "ko", + "position": 5, + "force_prefix": false + }, + { + "value": "on", + "position": 6, + "force_prefix": false + }, + { + "value": "ng", + "position": 7, + "force_prefix": false + } + ] +] Added: test/command/suite/tokenizers/bigram/env/remove_blank_disable/include_blank.test (+3 -0) 100644 =================================================================== --- /dev/null +++ test/command/suite/tokenizers/bigram/env/remove_blank_disable/include_blank.test 2016-04-23 14:12:44 +0900 (f965cd5) @@ -0,0 +1,3 @@ +#@omit "need enable `GRN_NGRAM_TOKENIZER_REMOVE_BLANK_DISABLE` environment variable." + +tokenize TokenBigramSplitSymbolAlpha "Hong Kong" NormalizerAuto --mode GET -------------- next part -------------- HTML����������������������������...Download
