refactored singularize() to use SnowballStemmer; fixed the smartJoin() #11

ramji-c · 2017-06-08T06:35:38Z

Summary of changes:

refactored singularize() function in utils.py to make use of SnowballStemmer to convert plural words to singular form
fixed the smartJoin() function to use regex for whitespace removal
fixed a minor bug in use of convert_to_json.py, wherein the output of roundtrip.sh script wasn't compatible. The import_data() now checks if confidence score is available during the conversion process

…(); updated convert_to_json.py to work with output of roundtrip.sh

tisdall · 2017-10-24T17:35:54Z

ingredient_phrase_tagger/training/utils.py

@@ -96,41 +99,11 @@ def getFeatures(token, index, tokens):

 def singularize(word):
    """
-    A poor replacement for the pattern.en singularize function, but ok for now.
+    use nltk/Snowballstemmer to singularize word.


a stemmer will do a lot more than singularize words.. ex "larger" becomes "large"

Wouldn't it be better to use nltk.WordNetLemmatizer?

refactored singularize() to use SnowballStemmer; fixed the smartToken…

cb2ad27

…(); updated convert_to_json.py to work with output of roundtrip.sh

tisdall reviewed Oct 24, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactored singularize() to use SnowballStemmer; fixed the smartJoin() #11

refactored singularize() to use SnowballStemmer; fixed the smartJoin() #11

ramji-c commented Jun 8, 2017

tisdall Oct 24, 2017

generalic Dec 20, 2017

refactored singularize() to use SnowballStemmer; fixed the smartJoin() #11

Are you sure you want to change the base?

refactored singularize() to use SnowballStemmer; fixed the smartJoin() #11

Conversation

ramji-c commented Jun 8, 2017

tisdall Oct 24, 2017

Choose a reason for hiding this comment

generalic Dec 20, 2017

Choose a reason for hiding this comment