进度24/12/15

昨日复盘
Intermediate Mechine Learning之类型变量
读两篇讲解如何提问的文章，在提问区里发起一次提问
实战：自己从头到尾首先Housing Prices Competition for Kaggle Learn Users并成功提交

Intermediate Mechine Learning之管道（pipeline之前一直错译为工作流）

今日进度
Intermediate Mechine Learning之交叉验证
Intermediate Mechine Learning之XGBoost
Intermediate Mechine Learning之数据泄露
利用以上所学刷一遍分数。

Cross-Validation

交叉验证用来更好的测评模型表现。
验证集越大，我们得到的测评结果约可靠，但是在数据集大小确定的情况下，验证集越大意味着训练集越小，这是我们不想面对的情况。

交叉验证将数据分为多个fold，进行多次实验，每次实验使用其中一个fold作为验证集，最终确保每一个已知数据都被当作验证集使用过。

优点是足够可靠，缺点是开销翻倍。如果运行一次时间可以接收，采用交叉验证无疑是一个不错的选择，但如果运行时间较长，且数据量足够大，则不宜采用交叉验证。

利用交叉验证选择最优参数：

#数据只保留了数字类型
numeric_cols = [cname for cname in train_data.columns if train_data[cname].dtype in ['int64', 'float64']]
X = train_data[numeric_cols].copy()
X_test = test_data[numeric_cols].copy()

from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score

def get_score(n_estimators):
    """Return the average MAE over 3 CV folds of random forest model.
    
    Keyword argument:
    n_estimators -- the number of trees in the forest
    """
    # Replace this body with your own code
    my_pipeline = Pipeline(steps=[
        ('preprocessor', SimpleImputer()),
        ('model', RandomForestRegressor(n_estimators=n_estimators, random_state=0))
    ])

    scores = -1 * cross_val_score(my_pipeline, X, y,
                              cv=3,
                              scoring='neg_mean_absolute_error')
    return scores.mean()

n_list = list(range(50, 401, 50))
results = {}
for ns in n_list:
    mean_s = get_score(ns)
    results[ns] = mean_s
    print(results)
    
import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(list(results.keys()), list(results.values()))
plt.show()

后续可以学习超参数优化课程，可以从网格搜索grid search开始

XGBoost

对于结构化数据最准确的建模技术

gradient boosting梯度迭代模型是Kaggle比赛中实现了多种数据集的SOTA

对于随机森林方法，它本质上使用了多个单独的决策树进行学习，可以称作ensemble methods集成学习方法。另外一种集成学习方法叫做graient boosting

基本流程：先使用一个基本模型做出预测，计算损失函数。利用这个损失值去训练新的模型。具体来说，我们决定了模型参数以便新的模型加入后可以降低损失。

XGBoost代表了极致的梯度迭代，专注于表现和效率。

from xgboost import XGBRegressor
my_model = XGBRegressor()
my_model.fit(X_train, y_train)

# 更多参数
my_model = XGBRegressor(n_estimators=500, learning_rate=0.05, n_jobs=4)  # 迭代次数，学习率和并行数
my_model.fit(X_train, y_train, 
             early_stopping_rounds=5,       #自动停止
             eval_set=[(X_valid, y_valid)], #测试用集合
             verbose=False)

Data Leakage

数据泄露使得模型在训练时看起来非常准确，但是用来预测时准确率不高。
两种类型的数据泄露：target leakage和train-test contamination 训练、测试污染

Target leakage

目标泄露发生在时间或时间顺序类型的数据上。
任何在目标产生那一刻以后生成的数据都不应该出现在已知变量集合中。

示例：生病的人会用抗生素，如果是否服用抗生素信息出现在训练数据中，在训练和验证时依据这个信息就可以准确地判断一个人是否生病。但是实际用来预测时，一个人未来是否会生病和当前是否服用抗生素没有直接的必然联系，原本学习到的经验变成了错误的。

Train-test Contamination

如果验证和测试数据通过某种方式影响了模型的训练过程，就会导致这种泄露。这种泄露的发生有时是不易察觉的，需要注意数据预处理的时间。
一个建议是：When using cross-validation, it’s even more critical that you do your preprocessing inside the pipeline!

观察这样一组数据

• card: 1 if credit card application accepted, 0 if not
• reports: Number of major derogatory reports
• age: Age n years plus twelfths of a year
• income: Yearly income (divided by 10,000)
• share: Ratio of monthly credit card expenditure to yearly income
• expenditure: Average monthly credit card expenditure
• owner: 1 if owns home, 0 if rents
• selfempl: 1 if self-employed, 0 if not
• dependents: 1 + number of dependents
• months: Months living at current address
• majorcards: Number of major credit cards held
• active: Number of active credit accounts

expenditures_cardholders = X.expenditure[y]
expenditures_noncardholders = X.expenditure[~y]

print('Fraction of those who did not receive a card and had no expenditures: %.2f' \
      %((expenditures_noncardholders == 0).mean()))
print('Fraction of those who received a card and had no expenditures: %.2f' \
      %(( expenditures_cardholders == 0).mean()))
"""
Fraction of those who did not receive a card and had no expenditures: 1.00
Fraction of those who received a card and had no expenditures: 0.02
"""

potential_leaks = ['expenditure', 'share', 'active', 'majorcards']     #排除潜在可能的泄露
X2 = X.drop(potential_leaks, axis=1)

# Evaluate the model with leaky predictors removed
cv_scores = cross_val_score(my_pipeline, X2, y, 
                            cv=5,
                            scoring='accuracy')

print("Cross-val accuracy: %f" % cv_scores.mean())   # 准确率大大下降

一般只会发生在自己构建的数据集上，标准数据集一般不会有这种情况。如果不能详尽的了解每一项数据的由来，排除所有可能的泄露也许是更好的选择。

另一个好用的方法是：在实际的预测场景中能用相同的方法获取到的数据用在训练中都不算泄露。

实际应用场景中，还要考虑预测结果是否真的有效。

一个加深理解的例子

Step 4: Preventing Infections
An agency that provides healthcare wants to predict which patients from a rare surgery are at risk of infection, so it can alert the nurses to be especially careful when following up with those patients.
You want to build a model. Each row in the modeling dataset will be a single patient who received the surgery, and the prediction target will be whether they got an infection.
Some surgeons may do the procedure in a manner that raises or lowers the risk of infection. But how can you best incorporate the surgeon information into the model?
You have a clever idea.

1. Take all surgeries by each surgeon and calculate the infection rate among those surgeons.
2. For each patient in the data, find out who the surgeon was and plug in that surgeon’s average infection rate as a feature.
   Does this pose any target leakage issues?
   Does it pose any train-test contamination issues?

This poses a risk of both target leakage and train-test contamination (though you may be able to avoid both if you are careful).
You have target leakage if a given patient’s outcome contributes to the infection rate for his surgeon, which is then plugged back into the prediction model for whether that patient becomes infected. You can avoid target leakage if you calculate the surgeon’s infection rate by using only the surgeries before the patient we are predicting for. Calculating this for each surgery in your training data may be a little tricky.
You also have a train-test contamination problem if you calculate this using all surgeries a surgeon performed, including those from the test-set. The result would be that your model could look very accurate on the test set, even if it wouldn’t generalize well to new patients after the model is deployed. This would happen because the surgeon-risk feature accounts for data in the test set. Test sets exist to estimate how the model will do when seeing new data. So this contamination defeats the purpose of the test set.

非常有帮助的例子。直觉上没有问题，但是考虑到手术数据本身就很少，感觉结果又会对某些变量有影响。（当数据量很大时，某个病人是否感染对比例产生的影响微乎其微）
但是从原理上将，只要结果参与到某个用于预测的变量的计算中，这就叫数据泄露，本例中毫无疑问是发生了数据泄露的。

实战XGBoost–pipelien

# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session


# Load original data
from sklearn.model_selection import train_test_split

X_full = pd.read_csv("/kaggle/input/home-data-for-ml-course/train.csv")
X_test = pd.read_csv("/kaggle/input/home-data-for-ml-course/test.csv")

X_full.dropna(axis=0, subset=['SalePrice'], inplace=True)
y = X_full.SalePrice
X_full.drop(['SalePrice'], axis=1, inplace=True)

# X_train, X_valid, y_train, y_valid = train_test_split(X_full, y, train_size=0.8, test_size=0.2,
                                                      # random_state=0)

print("Load data successfully.")


# print(X_full.isnull().sum()[X_full.isnull().sum()>0])
# # 对于缺失值过多的列，采用丢弃策略
# X_drop_cols = [col for col in X_full.columns if X_full[col].isnull().sum() > 100]
# X_full.drop(X_drop_cols, axis=1, inplace=True)

numerical_cols = [col for col in X_full.columns if X_full[col].dtype in ["int64", "float64"]]
categorical_cols = [col for col in X_full.columns if X_full[col].dtype == "object"]

# print(X_drop_cols)




# define pipelinefrom sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import cross_val_score

numerical_transformer = SimpleImputer(strategy="constant")

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy="most_frequent")),
    ('one_hot', OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numerical_transformer, numerical_cols),
        ('cat', categorical_transformer, categorical_cols)
    ]
)

def get_score(model):
    my_pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor),
        ('model', model)
    ])
    
    scores = -1 * cross_val_score(my_pipeline, X_full, y,
                                 cv=3,
                                 scoring='neg_mean_absolute_error')
    return scores.mean()

print("get_score defined.")

# 挑选最佳模型
from xgboost import XGBRegressor
# my_model = XGBRegressor(n_estimators=2000, 
#                         learning_rate=0.01,
#                         random_state=0,
#                        n_jobs=4)
# s = get_score(my_model)
# print(f"MAE is {s}")

• 最原始模型：17468
• 丢弃缺失值超过10的：17562
• 丢弃缺失值超过40的：17524
• 丢弃缺失值超过100的：17516

不丢弃（原始500）：

• epoch-200: 17489
• epoch-300: 17467
• epoch-400: 17463
• epoch-450: 17467

学习率–0.05–>0.01

• 轮次450: 17818
• 轮次600：17504
• 轮次700：17403
• 轮次800：17343
• 轮次900：17319
• 轮次1000：17307
• 轮次1500：17271
• 轮次2000：17268

final_model = XGBRegressor(n_estimators=2000, 
                        learning_rate=0.01,
                        random_state=0,
                       n_jobs=4)
final_pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor),
        ('model', final_model)
    ])
    
final_pipeline.fit(X_full, y)

predictions = final_pipeline.predict(X_test)
print("Predictions on test set:", predictions)

output = pd.DataFrame({'Id': X_test.Id,
                      'SalePrice': predictions})
output.to_csv("submission.csv", index=False)
print("Sub saved")

最终损失14898，排名到了140/4711