If you don't like the output of clean-text, consider adding a test with your specific input and desired output. Pull requests are especially welcomed when they fix bugs or improve the code quality. If you have a question, found a bug or want to propose a new feature, have a look at the issues page. sklearn import CleanTransformer cleaner = CleanTransformer( no_punct = False, lower = False)Ĭleaner. There is also scikit-learn compatible API to use in your pipelines.Īll of the parameters above work here as well.įrom cleantext. If you need some special handling for your language, feel free to contribute. It should work for the majority of western languages. So far, only English and German are fully supported. For this, take a look at the source code. You may also only use specific functions for cleaning. Lang = "en" # set to 'de' for German special handlingĬarefully choose the arguments that fit your task. From cleantext import clean clean( "some input",įix_unicode = True, # fix various unicode errors to_ascii = True, # transliterate to closest ASCII representation lower = True, # lowercase text no_line_breaks = False, # fully strip line breaks as opposed to only normalizing them no_urls = False, # replace all URLs with a special token no_emails = False, # replace all email addresses with a special token no_phone_numbers = False, # replace all phone numbers with a special token no_numbers = False, # replace all numbers with a special token no_digits = False, # replace all digits with a special token no_currency_symbols = False, # replace all currency symbols with a special token no_punct = False, # remove punctuations replace_with_punct = "", # instead of removing punctuations you may replace them replace_with_url = "",
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |