1.9. Naive Bayes

Naive Bayes methods are a set of supervised learning algorithmsbased on applying Bayes’ theorem with the “naive” assumption ofconditional independence between every pair of features given thevalue of the class variable. Bayes’ theorem states the followingrelationship, given class variable

and dependent featurevector through, :

Using the naive conditional independence assumption that

for all

, this relationship is simplified to

Since

is constant given the input,we can use the following classification rule:

and we can use Maximum A Posteriori (MAP) estimation to estimate

and;the former is then the relative frequency of classin the training set.

The different naive Bayes classifiers differ mainly by the assumptions theymake regarding the distribution of

.

In spite of their apparently over-simplified assumptions, naive Bayesclassifiers have worked quite well in many real-world situations, famouslydocument classification and spam filtering. They require a small amountof training data to estimate the necessary parameters. (For theoreticalreasons why naive Bayes works well, and on which types of data it does, seethe references below.)

Naive Bayes learners and classifiers can be extremely fast compared to moresophisticated methods.The decoupling of the class conditional feature distributions means that eachdistribution can be independently estimated as a one dimensional distribution.This in turn helps to alleviate problems stemming from the curse ofdimensionality.

On the flip side, although naive Bayes is known as a decent classifier,it is known to be a bad estimator, so the probability outputs frompredict_proba are not to be taken too seriously.

References:

• H. Zhang (2004). The optimality of Naive Bayes.Proc. FLAIRS.

1.9.1. Gaussian Naive Bayes

GaussianNB implements the Gaussian Naive Bayes algorithm forclassification. The likelihood of the features is assumed to be Gaussian:

The parameters

andare estimated using maximum likelihood.

>>>

>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.naive_bayes import GaussianNB
>>> X, y = load_iris(return_X_y=True)
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
>>> gnb = GaussianNB()
>>> y_pred = gnb.fit(X_train, y_train).predict(X_test)
>>> print("Number of mislabeled points out of a total %d points : %d"
...       % (X_test.shape[0], (y_test != y_pred).sum()))
Number of mislabeled points out of a total 75 points : 4

1.9.2. Multinomial Naive Bayes

MultinomialNB implements the naive Bayes algorithm for multinomiallydistributed data, and is one of the two classic naive Bayes variants used intext classification (where the data are typically represented as word vectorcounts, although tf-idf vectors are also known to work well in practice).The distribution is parametrized by vectors

for each class, where is the number of features(in text classification, the size of the vocabulary)and is the probabilityof feature appearing in a sample belonging to class.

The parameters

is estimated by a smoothedversion of maximum likelihood, i.e. relative frequency counting:

where

isthe number of times feature appears in a sample of classin the training set,and is the total count ofall features for class.

The smoothing priors

accounts forfeatures not present in the learning samples and prevents zero probabilitiesin further computations.Setting is called Laplace smoothing,while is called Lidstone smoothing.

1.9.3. Complement Naive Bayes

ComplementNB implements the complement naive Bayes (CNB) algorithm.CNB is an adaptation of the standard multinomial naive Bayes (MNB) algorithmthat is particularly suited for imbalanced data sets. Specifically, CNB usesstatistics from the complement of each class to compute the model’s weights.The inventors of CNB show empirically that the parameter estimates for CNB aremore stable than those for MNB. Further, CNB regularly outperforms MNB (oftenby a considerable margin) on text classification tasks. The procedure forcalculating the weights is as follows:

where the summations are over all documents

not in class, is either the count or tf-idf value of term in document, is a smoothing hyperparameter like that found inMNB, and. The second normalization addressesthe tendency for longer documents to dominate parameter estimates in MNB. Theclassification rule is:

i.e., a document is assigned to the class that is the poorest complementmatch.

References:

• Rennie, J. D., Shih, L., Teevan, J., & Karger, D. R. (2003).Tackling the poor assumptions of naive bayes text classifiers.In ICML (Vol. 3, pp. 616-623).

1.9.4. Bernoulli Naive Bayes

BernoulliNB implements the naive Bayes training and classificationalgorithms for data that is distributed according to multivariate Bernoullidistributions; i.e., there may be multiple features but each one is assumedto be a binary-valued (Bernoulli, boolean) variable.Therefore, this class requires samples to be represented as binary-valuedfeature vectors; if handed any other kind of data, a BernoulliNB instancemay binarize its input (depending on the binarize parameter).

The decision rule for Bernoulli naive Bayes is based on

which differs from multinomial NB’s rulein that it explicitly penalizes the non-occurrence of a feature

that is an indicator for class,where the multinomial variant would simply ignore a non-occurring feature.

In the case of text classification, word occurrence vectors (rather than wordcount vectors) may be used to train and use this classifier. BernoulliNBmight perform better on some datasets, especially those with shorter documents.It is advisable to evaluate both models, if time permits.

References:

• C.D. Manning, P. Raghavan and H. Schütze (2008). Introduction toInformation Retrieval. Cambridge University Press, pp. 234-265.

• A. McCallum and K. Nigam (1998).A comparison of event models for Naive Bayes text classification.Proc. AAAI/ICML-98 Workshop on Learning for Text Categorization, pp. 41-48.

• V. Metsis, I. Androutsopoulos and G. Paliouras (2006).Spam filtering with Naive Bayes – Which Naive Bayes?3rd Conf. on Email and Anti-Spam (CEAS).

1.9.5. Categorical Naive Bayes

CategoricalNB implements the categorical naive Bayesalgorithm for categorically distributed data. It assumes that each feature,which is described by the index

, has its own categoricaldistribution.

For each feature

in the training set,CategoricalNB estimates a categorical distribution for each feature iof X conditioned on the class y. The index set of the samples is defined as, with as the number of samples.

The probability of category

in feature given class is estimated as:

where

is the numberof times category appears in the samples, which belongto class, is the numberof samples with class c, is a smoothing parameter and is the number of available categories of feature.

CategoricalNB assumes that the sample matrix

is encoded(for instance with the help of OrdinalEncoder) such that allcategories for each feature are represented with numbers where is the number of available categoriesof feature.

1.9.6. Out-of-core naive Bayes model fitting

Naive Bayes models can be used to tackle large scale classification problemsfor which the full training set might not fit in memory. To handle this case,MultinomialNB, BernoulliNB, and GaussianNBexpose a partial_fit method that can be usedincrementally as done with other classifiers as demonstrated inOut-of-core classification of text documents. All naive Bayesclassifiers support sample weighting.

Contrary to the fit method, the first call to partial_fit needs to bepassed the list of all the expected class labels.

For an overview of available strategies in scikit-learn, see also theout-of-core learning documentation.

Note

The partial_fit method call of naive Bayes models introduces somecomputational overhead. It is recommended to use data chunk sizes that are aslarge as possible, that is as the available RAM allows.