NotFittedError: TfidfVectorizer - Vocabulary wasn't fitted
You've fitted a vectorizer, but you throw it away because it doesn't exist past the lifetime of your vectorize
function. Instead, save your model in vectorize
after it's been transformed:
self._vectorizer = vectorizer
Then in your classify
function, don't create a new vectorizer. Instead, use the one you'd fitted to the training data:
input_counts = self._vectorizer.transform(input_text)
Save vectorizer
as a pickle
or joblib
file and load it when you want to predict.
pickle.dump(vectorizer, open("vectorizer.pickle", "wb")) //Save vectorizer
pickle.load(open("models/vectorizer.pickle", 'rb')) // Load vectorizer
You can save both the model and the vectorizer and use them later on as well: here is how I did it:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.svm import LinearSVC
import pickle
# Train the classification model
def train_model():
df = pd.read_json('intent_data.json')
X_train, X_test, y_train, y_test = train_test_split(df['Utterance'], df['Intent'], random_state=0)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
model = LinearSVC().fit(X_train_tfidf, y_train)
# Save the vectorizer
vec_file = 'vectorizer.pickle'
pickle.dump(count_vect, open(vec_file, 'wb'))
# Save the model
mod_file = 'classification.model'
pickle.dump(model, open(mod_file, 'wb'))
# Load the classification model from disk and use for predictions
def classify_utterance(utt):
# load the vectorizer
loaded_vectorizer = pickle.load(open('vectorizer.pickle', 'rb'))
# load the model
loaded_model = pickle.load(open('classification.model', 'rb'))
# make a prediction
print(loaded_model.predict(loaded_vectorizer.transform([utt])))