1. What is EDA？

  探索性数据分析（Exploratory Data Analysis，简称EDA）是上世纪六十年代美国统计学家John Tukey提出的。它指的是对已有的数据（特别是调查或观察得来的原始数据）在尽量少的先验假定下进行探索，通过作图、制表、拟合、计算特征指标等手段探索数据的结构和规律的一种数据分析方法。
当大数据时代到来的时候，面对各种杂乱的“脏数据”，我们往往不知所措，不知道从哪里开始了解手上的数据，这时候，探索性数据分析就非常有效。

维基百科–EDA：
In statistics, exploratory data analysis(EDA) is an approach to analyzing data sets to summarize their maincharacteristics, often with visual methods. A statistical model can be used ornot, but primarily EDA is for seeing what the data can tell us beyond theformal modeling or hypothesis testing task. Exploratory data analysis waspromoted by John Tukey to encourage statisticians to explore the data, andpossibly formulate hypotheses that could lead to new data collection andexperiments.
EDA is different from initial data analysis (IDA), which focusesmore narrowly on checking assumptions required for model fitting and hypothesistesting, and handling missing values and making transformations of variables asneeded. EDA encompasses IDA.

2. Why EDA？

3. How EDA？

3.1 基本设置

Package Preparation

• pd 表格处理; np 数据运算;
• matplotlib.pyplot/plt, seaborn/sns, missingno/msno 数据可视化;
• os path setting: the original path is that where u set the code files 路径设置
• 输出cell里面所有的运行结果
• 忽略警告

#basic packages
import numpy as np #计算作用
import pandas as pd #表格作用import matplotlib.pyplot as plt #制图1
%matplotlib inline  #在jupyter中显示运行结果import seaborn as sns #制图2
color = sns.color_palette() #绘制网格背景
sns.set(style="whitegrid", color_codes=True) #灰色网格挺漂亮的
sns.set(font_scale=1)import missingno as msno #制图3#basic settings
import warnings #忽略红色警告
warnings.filterwarnings("ignore")import os #获得当前的路径
os.path
os.getcwd()  
os.chdir('/Users/**/CreditScoring')# 设置当前途径
os.getcwd()from IPython.core.interactiveshell import InteractiveShell #显示每一行代码的运行结果(否则默认只显示最后一行)
InteractiveShell.ast_node_interactivity = "all"

3.2 读取数据

• 删除无意义的行列，如序号等
• 更换变量名称，便于操作

# read data 
trainset_orig = pd.read_csv(r'/Users/**/CreditScoring/dataset/cs-training.csv')
trainset = trainset_orig.drop(trainset_orig.columns[0], axis=1) #删除第一列序号，无意义#Rename Variables 
states={'SeriousDlqin2yrs':'是否坏客户','RevolvingUtilizationOfUnsecuredLines':'剩余可用额度的比例','age':'年龄','NumberOfTime30-59DaysPastDueNotWorse':'两年内逾期30-59天的笔数','DebtRatio':'月负债百分比','MonthlyIncome':'月收入','NumberOfOpenCreditLinesAndLoans':'普通信贷的笔数','NumberOfTimes90DaysLate':'贷款以来逾期90天的笔数','NumberRealEstateLoansOrLines':'固定资产贷款的笔数','NumberOfTime60-89DaysPastDueNotWorse':'两年内逾期60-89天的笔数','NumberOfDependents':'家属人数'}
states2={'SeriousDlqin2yrs':'Serious Dlqin 2yrs','RevolvingUtilizationOfUnsecuredLines':'Revolving Utilization','age':'Age','NumberOfTime30-59DaysPastDueNotWorse':'# of 30-59 Days Past Due','DebtRatio':'Debt Ratio','MonthlyIncome':'Monthly Income','NumberOfOpenCreditLinesAndLoans':'# of Open Credit Lines And Loans','NumberOfTimes90DaysLate':'# of 90 Days Late','NumberRealEstateLoansOrLines':'# of Real Estate Loans/Lines','NumberOfTime60-89DaysPastDueNotWorse':'# of 60-89 Days Past Due','NumberOfDependents':'# of Dependents'}
trainset.rename(columns=states2,inplace=True)

3.3 探索与分析

3.31 数据概览

• 样本类别(y)是否均衡
• 是否含有缺失值(x)：两种图。msno.matrix显示变量缺失关联，msno.bar显示变量缺失百分比

# Overview data 
'样本类别不均衡'
def add_freq():  #定义函数，添加数据标签：比例（百分数）ncount=len(trainset) #数据量ax2=ax.twinx()ax2.yaxis.tick_left()ax.yaxis.tick_right()ax.yaxis.set_label_position('right') ax2.yaxis.set_label_position('left')ax2.set_ylabel('Frequency [%]',fontsize=21) #左边坐标轴名称for p in ax.patches:x=p.get_bbox().get_points()[:,0]y=p.get_bbox().get_points()[1,1]ax.annotate('{:.1f}%'.format(100.*y/ncount),(x.mean(),y),ha='center',va='bottom',fontsize=17,color='k')ax2.set_ylim(0,100)ax2.grid(None)#plot
fig1= plt.figure(figsize=(16,10))
ax=sns.countplot(x = 'Serious Dlqin 2yrs',data = trainset,alpha=0.8,palette='Greys_r') #底部标签
plt.xlabel(u'Y/ N: Serious Diqin 2yrs',fontsize=19)
plt.ylabel(u'Count [ int ]',fontsize=17) plt.title("Data Imbalanced: Bad or Not",fontsize=23)
add_freq() #图表名称plt.savefig('./bad_or_not.png') #图片存储路径
plt.show()

# Missing value overview
fig2= plt.figure(figsize=(40,20)) #尺寸
ax=sns.set(style = "ticks")
msno.matrix(trainset)
plt.savefig('./MissingPatterns.png', dpi=100, bbox_inches='tight') #dpi像素大小，数字越大越清晰
plt.show() #先储存再show

fig= plt.figure(figsize=(40,20))
sns.set(style = "ticks")
ax=msno.bar(trainset)
ax.grid(True, linestyle='-.') #网格图plt.savefig('./Missingpercent.png', dpi=100, bbox_inches='tight')
plt.show()

3.32 缺失值处理

• 对于缺失较少的变量，删除缺失值所在的行/填补
• 对于缺失较多的变量，直接删除该变量/填补

'采用先对缺失较多的变量，填补;之后对缺失较少的变量，删除缺失值所在的行(先补后删，可以利用更多信息)'# Missing Values 
trainset11=trainset.fillna({"Monthly Income":trainset["Monthly Income"].median()})
trainset12=trainset11.dropna()
trainset1=trainset12
trainset1.info()
print('after deleting the missing values of [# of Dependents] ','the rest :' ,146076,',',146076/150000*100,"%")'删除之后还剩146076条 , 97.384 %'

3.32 详细分析

3.22

4. Tips