100 days to get the machine learning | Day1 data preprocessing

---

Data preprocessing is part of machine learning the most basic and most troublesome
before we put effort relent derivation of various algorithms, the best thing to do is to first get the data preprocessing
each algorithm after the realization process and case practice hand in step, which is essential for
the students do not take the trouble to come to blows
basis of good students can also learn new things, practice Kazakhstan

Without further ado, Syria, here we are six steps to complete the data preprocessing
fact, I feel a little step here: the observed data
! [Enter image description here] [1]

This is a ten groups of nationality, age, income, whether the purchase data

There classification data, there is numerical data, there are some missing values

It appears to be a class prediction problem

Predicting the basis of nationality, age, income is enough to buy

OK, have a general understanding, he began performing.

Step 1: Import library

import numpy as np

import pandas as pd

Step 2: Import Data Set

dataset = pd.read_csv('Data.csv')

X = dataset.iloc[ : , :-1].values
Y = dataset.iloc[ : , 3].values
print("X")
print(X)
print("Y")
print(Y)

The purpose of this step is the independent variable and the dependent variable is split into a matrix and a vector.
The results are as follows

X
[['France' 44.0 72000.0]
 ['Spain' 27.0 48000.0]
 ['Germany' 30.0 54000.0]
 ['Spain' 38.0 61000.0]
 ['Germany' 40.0 nan]
 ['France' 35.0 58000.0]
 ['Spain' nan 52000.0]
 ['France' 48.0 79000.0]
 ['Germany' 50.0 83000.0]
 ['France' 37.0 67000.0]]
Y
['No' 'Yes' 'No' 'No' 'Yes' 'Yes' 'No' 'Yes' 'No' 'Yes']

Step 3: handle missing data

from sklearn.preprocessing import Imputer
imputer = Imputer(missing_values = "NaN", strategy = "mean", axis = 0)
imputer = imputer.fit(X[ : , 1:3])
X[ : , 1:3] = imputer.transform(X[ : , 1:3])

Imputer class venue and specific usage

http://scikit-learn.org/stable/modules/preprocessing.html#preprocessing

In this example we use the average value substitution fill in missing

Results are as follows

Step 3: Handling the missing data
step2
X
[['France' 44.0 72000.0]
 ['Spain' 27.0 48000.0]
 ['Germany' 30.0 54000.0]
 ['Spain' 38.0 61000.0]
 ['Germany' 40.0 63777.77777777778]
 ['France' 35.0 58000.0]
 ['Spain' 38.77777777777778 52000.0]
 ['France' 48.0 79000.0]
 ['Germany' 50.0 83000.0]
 ['France' 37.0 67000.0]]

Step 4: converting digital data classified

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[ : , 0] = labelencoder_X.fit_transform(X[ : , 0])

onehotencoder = OneHotEncoder(categorical_features = [0])
X = onehotencoder.fit_transform(X).toarray()
labelencoder_Y = LabelEncoder()
Y =  labelencoder_Y.fit_transform(Y)
print("X")
print(X)

print("Y")
print(Y)

LabelEncoder usage venue

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html

X
[[1.00000000e+00 0.00000000e+00 0.00000000e+00 4.40000000e+01
  7.20000000e+04]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00 2.70000000e+01
  4.80000000e+04]
 [0.00000000e+00 1.00000000e+00 0.00000000e+00 3.00000000e+01
  5.40000000e+04]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00 3.80000000e+01
  6.10000000e+04]
 [0.00000000e+00 1.00000000e+00 0.00000000e+00 4.00000000e+01
  6.37777778e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 3.50000000e+01
  5.80000000e+04]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00 3.87777778e+01
  5.20000000e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 4.80000000e+01
  7.90000000e+04]
 [0.00000000e+00 1.00000000e+00 0.00000000e+00 5.00000000e+01
  8.30000000e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 3.70000000e+01
  6.70000000e+04]]
Y
[0 1 0 0 1 1 0 1 0 1]

Step 5: The data set into training and test sets
from sklearn.cross_validation Import train_test_split
X_train, X_test, Y_train, android.permission.FACTOR. Train_test_split = (X-, the Y, test_size = 0.2, random_state = 0)

X_train
[[0.00000000e+00 1.00000000e+00 0.00000000e+00 4.00000000e+01
  6.37777778e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 3.70000000e+01
  6.70000000e+04]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00 2.70000000e+01
  4.80000000e+04]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00 3.87777778e+01
  5.20000000e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 4.80000000e+01
  7.90000000e+04]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00 3.80000000e+01
  6.10000000e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 4.40000000e+01
  7.20000000e+04]
 [1.00000000e+00 0.00000000e+00 0.00000000e+00 3.50000000e+01
  5.80000000e+04]]
X_test
[[0.0e+00 1.0e+00 0.0e+00 3.0e+01 5.4e+04]
 [0.0e+00 1.0e+00 0.0e+00 5.0e+01 8.3e+04]]
step2
Y_train
[1 1 1 0 1 0 0 1]
Y_test
[0 0]

Step 6: feature scaling

from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)

Most machine learning algorithm using the Euclidean distance between the two data points in the calculations

Characterized by great changes in magnitude and scope of the unit, which caused problems

Wherein a high weight value in the distance calculation feature significant low value

Characterized by standardized or normalized to Z-scores complete

Import sklearn.preprocessing library StandardScala

Usage: http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html

X_train
[[-1.          2.64575131 -0.77459667  0.26306757  0.12381479]
 [ 1.         -0.37796447 -0.77459667 -0.25350148  0.46175632]
 [-1.         -0.37796447  1.29099445 -1.97539832 -1.53093341]
 [-1.         -0.37796447  1.29099445  0.05261351 -1.11141978]
 [ 1.         -0.37796447 -0.77459667  1.64058505  1.7202972 ]
 [-1.         -0.37796447  1.29099445 -0.0813118  -0.16751412]
 [ 1.         -0.37796447 -0.77459667  0.95182631  0.98614835]
 [ 1.         -0.37796447 -0.77459667 -0.59788085 -0.48214934]]
X_test
[[-1.          2.64575131 -0.77459667 -1.45882927 -0.90166297]
 [-1.          2.64575131 -0.77459667  1.98496442  2.13981082]]