本文章使用的数据集是“Census Income Data Set”，来自UCI Machine Learning Repository（https://link.zhihu.com/?target=https%3A//archive.ics.uci.edu/ml/datasets/Census%2BIncome）。我们的目标是使用多种机器学习算法预测收入，具体来说，预测一个人的收入是否超过50000美金.
# Import libraries necessary for this project import numpy as np import pandas as pd from time import time from IPython.display import display # Allows the use of display() for DataFrames # Import supplementary visualization code visuals.py import visuals as vs # Pretty display for notebooks %matplotlib inline # Load the Census dataset data = pd.read_csv("census.csv") # Success - Display the first record display(data.head(n=3))
# TODO: Total number of records n_records = data.shape[0] # TODO: Number of records where individual's income is more than $50,000 #n_greater_50k = sum(data['income'] == '>50K') # TODO: Number of records where individual's income is at most $50,000
#n_at_most_50k = sum(data['income'] == '<=50K') n_at_most_50k, n_greater_50k = data.income.value_counts() # TODO: Percentage of individuals whose income is more than $50,000 greater_percent = np.true_divide(n_greater_50k , n_records) * 100 # Print the results print"Total number of records: {}".format(n_records) print"Individuals making more than $50,000: {}".format(n_greater_50k) print"Individuals making at most $50,000: {}".format(n_at_most_50k) print"Percentage of individuals making more than $50,000: {:.2f}%".format(greater_percent) Total number of records: 45222 Individuals making more than $50,000: 11208 Individuals making at most $50,000: 34014 Percentage of individuals making more than $50,000: 24.78%
# Split the data into features and target label income_raw = data['income'] features_raw = data.drop('income', axis = 1) # Visualize skewed continuous features of original data vs.distribution(data)
# Import sklearn.preprocessing.StandardScaler from sklearn.preprocessing import MinMaxScaler # Initialize a scaler, then apply it to the features scaler = MinMaxScaler() # default=(0, 1) numerical = ['age', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week'] features_log_minmax_transform = pd.DataFrame(data = features_log_transformed) features_log_minmax_transform[numerical] = scaler.fit_transform(features_log_transformed[numerical]) # Show an example of a record with scaling applied display(features_log_minmax_transform.head(n = 5)) vs.distribution(features_log_minmax_transform)
观察结果，所有连续特征均被转换到[0,1]区间。
总结以上步骤：
1. 观察所有数值特征
2. 对高偏差的特征进行对数转换
3. 对所有数值特征进行正则化
2.3 数据预处理
接下来我们来考虑离非数值征值，当然，我们需要将其转换为数值特征。一种常用的方法是使用one-hot编码。One-hot编码为非数值特征的每一种可能创建一个_"dummy"_ 变量. 例如, 假设 someFeature 有三种可能的取值: A, B, 或 C. 我们可以将其编码为someFeature_A, someFeature_B 和 someFeature_C.
同样，我们也需要对标签'income'进行转换。由于我们的标签只有两种取值("<=50K" and ">50K"), 我们可以不使用one-hot编码，而是直接使用0或1分别代表.
下面是实战代码
from sklearn.preprocessing import LabelEncoder # TODO: One-hot encode the 'features_log_minmax_transform' data using pandas.get_dummies() features_final = pd.get_dummies(features_log_minmax_transform) # TODO: Encode the 'income_raw' data to numerical values
encoder = LabelEncoder() income = encoder.fit_transform(income_raw) # Print the number of features after one-hot encoding encoded = list(features_final.columns) print "{} total features after one-hot encoding.".format(len(encoded)) # Uncomment the following line to see the encoded feature names # print encoded 103 total features after one-hot encoding.
# Import train_test_split from sklearn.cross_validation import train_test_split # Split the 'features' and 'income' data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(features_final, income, test_size = 0.2, random_state = 0) # Show the results of the split print"Training set has {} samples.".format(X_train.shape[0]) print"Testing set has {} samples.".format(X_test.shape[0])
''' TP = np.sum(income) # Counting the ones as this is the naive case. Note that 'income' is the 'income_raw' data encoded to numerical values done in the data preprocessing step. FP = income.count() - TP # Specific to the naive case TN = 0 # No predicted negatives in the naive case FN = 0 # No predicted negatives in the naive case ''' # TODO: Calculate accuracy, precision and recall TP = np.sum(income) FP = len(income) - TP accuracy = np.true_divide(TP,TP + FP) recall = 1 precision = accuracy
# TODO: Calculate F-score using the formula above for beta = 0.5 and correct values for precision and recall. # HINT: The formula above can be written as (1 + beta**2) * (precision * recall) / ((beta**2 * precision) + recall) fscore = (1 + 0.5**2) * (precision * recall) / ((0.5**2 * precision) + recall) # Print the results print "Naive Predictor: [Accuracy score: {:.4f}, F-score: {:.4f}]".format(accuracy, fscore) Naive Predictor: [Accuracy score: 0.2478, F-score: 0.2917]
# TODO: Import two metrics from sklearn - fbeta_score and accuracy_score from sklearn.metrics import fbeta_score,accuracy_score deftrain_predict(learner, sample_size, X_train, y_train, X_test, y_test): ''' inputs: - learner: the learning algorithm to be trained and predicted on - sample_size: the size of samples (number) to be drawn from training set - X_train: features training set - y_train: income training set - X_test: features testing set - y_test: income testing set ''' results = {} # TODO: Fit the learner to the training data using slicing with 'sample_size' using .fit(training_features[:], training_labels[:]) start = time() # Get start time learner = learner.fit(X_train[:sample_size], y_train[:sample_size]) end = time() # Get end time # TODO: Calculate the training time results['train_time'] = end - start # TODO: Get the predictions on the test set(X_test), # then get predictions on the first 300 training samples(X_train) using .predict() start = time() # Get start time predictions_test = learner.predict(X_test) predictions_train = learner.predict(X_train[:300]) end = time() # Get end time # TODO: Calculate the total prediction time results['pred_time'] = end-start # TODO: Compute accuracy on the first 300 training samples which is y_train[:300] results['acc_train'] = accuracy_score(predictions_train, y_train[:300]) # TODO: Compute accuracy on test set using accuracy_score() results['acc_test'] = accuracy_score(predictions_test, y_test) # TODO: Compute F-score on the the first 300 training samples using fbeta_score() results['f_train'] = fbeta_score(y_train[:300], predictions_train, beta= 0.5) # TODO: Compute F-score on the test set which is y_test results['f_test'] = fbeta_score(y_test, predictions_test, beta= 0.5) # Success print"{} trained on {} samples.".format(learner.__class__.__name__, sample_size) # Return the results return results
# TODO: Import the three supervised learning models from sklearn from sklearn.naive_bayes import GaussianNB from sklearn.tree import DecisionTreeClassifier from sklearn.svm import SVC
# TODO: Initialize the three models clf_A = GaussianNB() clf_B = DecisionTreeClassifier(random_state=0) clf_C = SVC(kernel = 'rbf') # TODO: Calculate the number of samples for 1%, 10%, and 100% of the training data # HINT: samples_100 is the entire training set i.e. len(y_train) # HINT: samples_10 is 10% of samples_100 # HINT: samples_1 is 1% of samples_100 samples_100 = len(y_train) samples_10 = int(len(y_train)*0.1) samples_1 = int(len(y_train)*0.01) # Collect results on the learners results = {} for clf in [clf_A, clf_B, clf_C]: clf_name = clf.__class__.__name__ results[clf_name] = {} for i, samples in enumerate([samples_1, samples_10, samples_100]): results[clf_name][i] = train_predict(clf, samples, X_train, y_train, X_test, y_test) # Run metrics visualization for the three supervised learning models chosen vs.evaluate(results, accuracy, fscore) GaussianNB trained on361 samples. GaussianNB trained on3617 samples. GaussianNB trained on36177 samples. DecisionTreeClassifier trained on361 samples. DecisionTreeClassifier trained on3617 samples. DecisionTreeClassifier trained on36177 samples. /anaconda3/envs/py2/lib/python2.7/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples. 'precision', 'predicted', average, warn_for) SVC trained on361 samples. SVC trained on3617 samples. SVC trained on36177 samples.
# TODO: Import 'GridSearchCV', 'make_scorer', and any other necessary libraries from sklearn.grid_search import GridSearchCV from sklearn.metrics import fbeta_score, make_scorer # TODO: Initialize the classifier clf = DecisionTreeClassifier(random_state=0) # TODO: Create the parameters list you wish to tune, using a dictionary if needed. # HINT: parameters = {'parameter_1': [value1, value2], 'parameter_2': [value1, value2]} parameters = {'max_depth':(2,3,4,5,6), 'criterion': ['gini','entropy']} # TODO: Make an fbeta_score scoring object using make_scorer() scorer = make_scorer(fbeta_score, beta=0.5) # TODO:
Perform grid search on the classifier using 'scorer' as the scoring method using GridSearchCV() grid_obj = GridSearchCV(clf, parameters, scorer) # TODO: Fit the grid search object to the training data and find the optimal parameters using fit() grid_fit = grid_obj.fit(X_train, y_train) # Get the estimator best_clf = grid_fit.best_estimator_ # Make predictions using the unoptimized and model predictions = (clf.fit(X_train, y_train)).predict(X_test) best_predictions = best_clf.predict(X_test) # Report the before-and-afterscores print"Unoptimized model\n------" print"Accuracy score on testing data: {:.4f}".format(accuracy_score(y_test, predictions)) print"F-score on testing data: {:.4f}".format(fbeta_score(y_test, predictions, beta = 0.5)) print"\nOptimized Model\n------" print"Final accuracy score on the testing data: {:.4f}".format(accuracy_score(y_test, best_predictions)) print"Final F-score on the testing data: {:.4f}".format(fbeta_score(y_test, best_predictions, beta = 0.5)) Unoptimized model ------ Accuracy score on testing data: 0.8186 F-score on testing data: 0.6279 Optimized Model ------ Final accuracy score on the testing data: 0.8523 Final F-score on the testing data: 0.7224
# TODO: Import a supervised learning model that has 'feature_importances_'
from sklearn.ensemble import AdaBoostClassifier # TODO: Train the supervised model on the training set using .fit(X_train, y_train) model = AdaBoostClassifier() model.fit(X_train, y_train) # TODO: Extract the feature importances using .feature_importances_ importances = model.feature_importances_ # Plot vs.feature_plot(importances, X_train, y_train)
这里展示了前5个影响最大的特征及其影响权重。
6. 特征选择
最后，我们来看一下，如果仅使用最有影响力的5个特征进行训练，得到什么结果。
# Import functionality for cloning a model from sklearn.base import clone # Reduce the feature space X_train_reduced = X_train[X_train.columns.values[(np.argsort(importances)[::-1])[:5]]] X_test_reduced = X_test[X_test.columns.values[(np.argsort(importances)[::-1])[:5]]] # Train on the "best" model found from grid search earlier clf = (clone(best_clf)).fit(X_train_reduced, y_train) # Make new predictions reduced_predictions = clf.predict(X_test_reduced) # Report scores from the final model using both versions of data print"Final Model trained on full data\n------" print"Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, best_predictions)) print"F-score on testing data: {:.4f}".format(fbeta_score(y_test, best_predictions, beta = 0.5)) print"\nFinal Model trained on reduced data\n------" print"Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, reduced_predictions)) print"F-score on testing data: {:.4f}".format(fbeta_score(y_test, reduced_predictions, beta = 0.5))
Final Model trained on full data ------ Accuracy on testing data: 0.8523 F-score on testing data: 0.7224 Final Model trained on reduced data ------ Accuracy on testing data: 0.8278 F-score on testing data: 0.6587
