[PyTorch][chapter 6][李宏毅深度学习][Logistic Regression]

前言：

logistic回归又称logistic回归分析，是一种广义的线性回归分析模型，常用于数据挖掘，疾病自动诊断，经济预测等领域。逻辑回归根据给定的自变量数据集来估计事件的发生概率，由于结果是一个概率，因此因变量的范围在 0 和 1 之间。 [3]例如，探讨引发疾病的危险因素，并根据危险因素预测疾病发生的概率等。

训练样本特别小的时候用 Generative Model会有较好的效果，大的样本使用Discriminative Model，Discriminative Model里面常用的二分类模型sigmoid ，多分类模型softmax

一 sigmoid 简介（Discriminative Model）

二分类模型

1.1 模型定义

使用了sigmoid 函数作为激活函数

$f(x)=\sigma(z)=\frac{1}{1+e^{-z}}$

$z=wx+b=\sum_i w_ix_i+b$

输出 (0,1)

1.2 损失函数

假设有N个二分类样本

$\left\{\begin{matrix} \hat{y}=1\, \, \, \, ,if \, c_1 \\ \hat{y}=0\, \, \, \, \, , if \, c_2 \end{matrix}\right.$

损失函数定义为

$L(w,b)=f(x^1)f(x^2)(1-f(x^3))..$

我们要找到参数w,b使得上面概率最大

$w^{*},b^{*}=argmax_{w,b}L(w,b)$

根据交叉熵原理：我们对式子取对数。因为是求式子的最大值，可以转换成式子乘以负1,之后求最小值

$w^*,b^*=argmin_{w,b} -lnL(w,b)$

$L(w,b)=-\sum_{i}^{N}-\begin{Bmatrix} \hat{y^i}ln f(x^i)+(1-\hat{y^i})ln (1-f(x^i)) \end{Bmatrix}$

1.3 梯度

对w的求导分为两部分

$\frac{\partial lnf}{\partial f}\frac{\partial f}{\partial z}\frac{\partial z}{\partial w}=\frac{1}{f}f(1-f)x$

$=(1-f)x$

$\frac{\partial ln1-f}{\partial f}\frac{\partial f}{\partial z}\frac{\partial z}{\partial w}=\frac{-1}{1-f}f(1-f)x$

$=-fx$

合并起来

$\frac{\partial L}{\partial w}=\sum_i -\begin{Bmatrix} \hat{y^{i}}(1-f)x^{i}-(1-\hat{y^{i}})fx^{i} \end{Bmatrix}$

$=\sum_{i}-(\hat{y^{i}}-f)x^{i}$

$=\sum_{i}(f-\hat{y^{i}})x^{i}$

1.4 跟Linear 区别

二 Multi-class Classification（softmax）

多分类模型

2.1 模型定义

使用了 softmax 作为激活函数

$y=\sigma(z_i)=\frac{e^{z_i}}{\sum_{j=1}^{K}e^{z_j}}$

2.2 损失函数

使cross Entropy

标签是一个one-hot 向量,非零项代表其类别

$L_{w,b}(\hat{y},y)=\sum_{i=1}^{K}\hat{y_i}logy_i$

2.3 梯度

$y_i=softmax(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}}$

$\left\{\begin{matrix} \frac{\partial y_i}{\partial z_j}=y_i*(1-y_j),j=i\\ \frac{\partial y_i}{\partial z_j}=-y_iy_j ,j\neq i \end{matrix}\right.$

损失函数为

$L=-\sum_{i}^{K}\hat{y_k}logy_k$

只跟其中的非零项有关系，假设非零项为 $y_i$

$\frac{\partial L}{\partial z_j}=\left\{\begin{matrix} \frac{-1}{y_i}*y_i*(1-y_i)=y_i-1,j=i\\ \frac{-1}{y_i}*(-y_i y_j)=y_j-0,j\neq i \end{matrix}\right.$

因为标签值是one-hot

$\left\{\begin{matrix} \hat{y_j}=1,i=j\\ \hat{y_j}=0,i \neq j \end{matrix}\right.$

所以

$\frac{\partial L}{\partial z_j}=y_j-\hat{y_j}$

三代码

任务：

给定的个人资料，预测此人的年收入是否大于50k

数据集说明：
共有32561训练集数据,16281 测试集数据
(8140 in private test set and 8141 in public test set)

数据集情况：共14个feature

? 代表不确定性
1 age 年龄: continuous.
2 workclass 工作性质: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
3 fnlwgt: continuous. ＊The number of people the census takers believe that observation represents.人口普查员认为这一观察结果所代表的人数。

4 education 教育水平:
Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.

5 education-num: continuous.

6 marital-status 婚姻状况:
Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.

7 occupation 工作:
Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.

8 relationship 关系:
Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.

9 race 种族: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.
10 sex 性别: Female, Male.
11 capital-gain 资本收益: continuous.
12 capital-loss资本损失: continuous.
13 hours-per-week 每周工作时长: continuous.
14 native-country原国际: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.

针对非数值型的属性，采用了one-hot 编码

分为两个文件：

dataLoader.py: csv文件读取,特征工程

lr.py: 模型训练 $y=xw$

其中

$x=[x,1]$ 增广矩阵，

$w =[b,w]$ 增广矩阵

# -*- coding: utf-8 -*-
"""
Created on Tue Dec 12 14:51:45 2023

@author: chengxf2
"""

import numpy as np
import pandas as pd
from random import shuffle
from math import floor, log


def sample(X, Y):                                 #X and Y are np.array
    randomize = np.arange(X.shape[0])
    np.random.shuffle(randomize)
    return (X[randomize], Y[randomize])


def split_valid_set(X, Y, percentage):
    m = X.shape[0]
    valid_size = int(floor(m * percentage))

    X, Y = sample(X, Y)
    X_valid, Y_valid = X[ : valid_size], Y[ : valid_size]
    X_train, Y_train = X[valid_size:], Y[valid_size:]

    return X_train, Y_train, X_valid, Y_valid

def dataProcess_Y(rawData):
    
    df_y = rawData['income']
    y = pd.DataFrame((df_y==' >50K').astype("int64"), columns=["income"])
    print('\n y',y.shape)
    return y

def dataProcess_X(rawData):

    #axis=1, 删除列 axis=0 删除 index
    if "income" in rawData.columns:
        Data = rawData.drop(["sex", 'income'], axis=1)
        #(32561, 13) 
    else:
        Data = rawData.drop(["sex"], axis=1)
    
    #读取非数字的column
    listObjectColumn = [col for col in Data.columns if Data[col].dtypes == "object"] 
    #数字的column
    listNonObjedtColumn = [x for x in list(Data) if x not in listObjectColumn] 
   

    ObjectData = Data[listObjectColumn]
    NonObjectData = Data[listNonObjedtColumn]

    #insert set into nonobject data with male = 0 and female = 1
    NonObjectData.insert(0 ,"sex", (rawData["sex"] == " Female").astype(int))
    #set every element in object rows as an attribute,相当于one-hot 编码
    ObjectData = pd.get_dummies(ObjectData)

    Data = pd.concat([NonObjectData, ObjectData], axis=1)
    Data_x = Data.astype("int64")
    # Data_y = (rawData["income"] == " <=50K").astype(np.int)
    print("\n data_x: ",Data_x.shape)
    #normalize
    Data_x = (Data_x - Data_x.mean()) / Data_x.std()

    return Data_x


def data_loader():
    
    trainData =  pd.read_csv("data/train.csv")
    testData =  pd.read_csv("data/test.csv")
    test_label = pd.read_csv("data/correct_answer.csv")
 
    # here is one more attribute in trainData
    x_train = dataProcess_X(trainData).drop(['native_country_ Holand-Netherlands'], axis=1).values
    x_test = dataProcess_X(testData).values
    
    
    y_train = dataProcess_Y(trainData).values
    y_test =  test_label['label'].values

    #x=>x[1,x]
    x_train = np.concatenate((np.ones((x_train.shape[0], 1)), x_train), axis=1)
    x_test = np.concatenate((np.ones((x_test.shape[0], 1)), x_test), axis=1)

    valid_set_percentage = 0.1
    X_train, Y_train, X_valid, Y_valid = split_valid_set(x_train, y_train, valid_set_percentage)
    
    return X_train, Y_train, X_valid, Y_valid ,x_test,y_test


import numpy as np

from numpy.linalg import inv
import matplotlib.pyplot as plt
from dataLoader import data_loader
from dataLoader import sample
import os
from math import floor, log
import pandas as pd


output_dir = "output/"





def sigmoid(z):
    res = 1 / (1.0 + np.exp(-z))
    return np.clip(res, 1e-8, (1-(1e-8)))






def valid(X, Y, w):
    a = np.dot(w,X.T)
    y = sigmoid(a)
    y_ = np.around(y)
    result = (np.squeeze(Y) == y_)
    print('Valid acc = %f' % (float(result.sum()) / result.shape[0]))
    return y_

def train(X_train, Y_train):
  
    n= len(X_train[0])
    print("\n n ",n)
    w = np.zeros(n)

    l_rate = 0.001
    batch_size = 32
    m = len(X_train)
    step_num = int(floor(m / batch_size))
    epoch_num = 30
    list_cost = []
    total_loss = 0.0
    
    
    for epoch in range(1, epoch_num):
        total_loss = 0.0
        X_train, Y_train = sample(X_train, Y_train)

        for idx in range(1, step_num):
            X = X_train[idx*batch_size:(idx+1)*batch_size]
            Y = Y_train[idx*batch_size:(idx+1)*batch_size]

            s_grad = np.zeros(len(X[0]))


            z = np.dot(X, w)
            y = sigmoid(z)
            #squeeze 即把shape中为1的维度去掉
            loss = y - np.squeeze(Y)
            cross_entropy = -1 * (np.dot(np.squeeze(Y.T), np.log(y)) + np.dot((1 - np.squeeze(Y.T)), np.log(1 - y)))/ len(Y)
            total_loss += cross_entropy

            grad = np.sum( X * (y-np.squeeze(Y)).reshape((batch_size, 1)), axis=0)
            # grad = np.dot(X.T, loss)
            w = w - l_rate * grad
            
        #print("\n epoch :%d, total_loss: %7.3f"%(epoch, total_loss/batch_size))

       

        list_cost.append(total_loss)

    # valid(X_valid, Y_valid, w)
    plt.plot(np.arange(len(list_cost)), list_cost)
    plt.title("Train Process")
    plt.xlabel("epoch_num")
    plt.ylabel("Cost Function (Cross Entropy)")
    plt.savefig(os.path.join(os.path.dirname(output_dir), "TrainProcess"))
    plt.show()

    return w

if __name__ == "__main__":
   
    X_train, Y_train, X_valid, Y_valid,x_test,y_test  = data_loader()
    w_train = train(X_train, Y_train)
    valid(X_valid, Y_valid, w_train)

    print("\n x_test",x_test.shape, "\t y_test ",y_test.shape,"\t w",w_train.shape)

    valid(x_test, y_test, w_train)

    df = pd.DataFrame({"id": np.arange(1, 16282), "label": y_test})
    if not os.path.exists(output_dir):
        os.mkdir(output_dir)
    df.to_csv(os.path.join(output_dir + 'lr_output.csv'), sep='\t', index=False)

https://github.com/maplezzz/ML2017S_Hung-yi-Lee_HW
动手学深度学习——softmax回归（原理解释+代码详解）-CSDN博客

https://www.cnblogs.com/hider/p/15431858.html