Does gensim.corpora.Dictionary have term frequency saved?
No, gensim.corpora.Dictionary
does not save term frequency. You can see the source code here. The class only stores the following member variables:
self.token2id = {} # token -> tokenId
self.id2token = {} # reverse mapping for token2id; only formed on request, to save memory
self.dfs = {} # document frequencies: tokenId -> in how many documents this token appeared
self.num_docs = 0 # number of documents processed
self.num_pos = 0 # total number of corpus positions
self.num_nnz = 0 # total number of non-zeroes in the BOW matrix
This means everything in the class defines frequency as document frequency, never term frequency, as the latter is never stored globally. This applies to filter_n_most_frequent(remove_n)
as well as every other method.
I had the same simple question. It appears that the frequency of the word is hidden and not accessible in the object. Not sure why it makes testing and validation a pain. What I did was export the dictionary as text..
dictionary.save_as_text('c:\\research\\gensimDictionary.txt')
In that text file they have three columns.. For example here are the words "summit" "summon" and "sumo"
Key Word Frequency
10 summit 1227
3658 summon 118
8477 sumo 40
I found a solution the .cfs are the word frequencies.. see https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary
print(str(dictionary[10]), str(dictionary.cfs[10]))
summit 1227
simple