GitHub - ssun32/clir

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.gitignore		.gitignore
README		README
check_relevance.py		check_relevance.py
filter.py		filter.py
filter_rules.txt		filter_rules.txt
fix_query.py		fix_query.py
fix_rel.py		fix_rel.py

Repository files navigation

Used tools
----------
convert traditional chinese to simplied
http://github.com/berniey/hanziconv

stanford segmenter
tokenize chinese language

filter.py
---------
filter out ill-formed documents, such as Wikipedia:, Help: PortalA

fix_query.py
------------
Regenerate the query from raw wikipedia documents. 
Previously, title words were not deleted from queries properly.

fix_rel.py:
----------
Scan all the rel and documents files and remove the lines with deleted wikipedia articles