“智慧政务”中的文本挖掘应用

 

摘要

 

    在这个信息时代，以怎样的方式去了解民意、汇集民智、凝聚民气是城市政府以及相关工作部门日益关心的重大问题之一。但随着各类社情相关的文本数据量的不断攀升，以人工的方式对群众的留言进行分类以及对热点问题的整理往往存在工作效率低下的问题。因此，本文将基于数据挖掘技术对“智慧政务“中的文本即群众的留言数、群众关心的热点问题、以及相关工作部门的解决方案数据进行内在信息的挖掘与分析。

   首先，在本次数据挖掘过程中，我们首先对获取得到的留言数据利用基于Python的fastText原理对数据预处理、分词以及停用过滤操作，实现对留言数据的分类，并提升了可建模度，并使用F-Score对训练模型进行评价，经过不断的训练，最终得到评价值约为0.8835。

    其次，对热点问题的挖掘，我们首先对留言语料进行文本预处理，使用TF-IDF方法进行分词及去除停用词操作，用欧氏距离来得到相似的度量报道与话题的相关性，最后通过文本聚类k-means算法，把相似的高的留言加到对应的话题簇中，得到了话题簇，因此得到了排名前5的热点问题和相应热点问题对应的留言信息。

    最后，问题3主要从答复意见文本的相关性、完整性和可解性、时效性和信息量等角度出发，来建立相关工作部门对留言答复意见质量的评价指标。本文运用预先相似度计算方法来计算留言主题与相关工作部门的答复意见之间的相似度， 用自动化可读性指数ARI来表示可解释性，ARI的计算公式为：API=4.71*（总字符数/总字数）+0.5*（总字数/总句数）-21.43。基于主成分分析之权值计算方法，算出为接下来即将要构建的评价模型中各个评价指标的权重。其中相关性、可解释性、实效性和信息量分别所占的权重为0.26、0.28、0.18、0.28，最后得到答复意见质量评价模型为 Q=0.28Words+0.26Relevancy+0.28Credibility+0.18Timeliness+0.01。

 

 

 

关键词：自然语言、fastText原理、n-gram特征、F-Score评价、k-means算法、欧氏距离、TD-IDF方法，广义线性回归

 

Question C: Text Mining Application in "Smart Government Affairs"

 

Abstract

       In this information age, how to understand public opinion, gather people's wisdom, and gather people's popularity is one of the major issues that the city government and related work departments are increasingly concerned about. However, as the amount of text data related to various social conditions continues to rise, there is often a problem of low work efficiency in manually categorizing the masses' messages and sorting out hot issues. Therefore, based on the data mining technology, this article will mine and analyze the intrinsic information of the text in the "smart government affairs", that is, the number of messages of the masses, the hot issues that the masses care about, and the solution data of the relevant work departments.

       First of all, in this data mining process, we first use the Python-based fastText principle to preprocess the data, segment the words, and disable the filtering operation on the obtained message data, classify the message data, and improve the modelability And use F-Score to evaluate the training model. After continuous training, the final evaluation value is about 0.8835.

Secondly, for the mining of hot issues, we first perform text preprocessing on the message corpus, use the TF-IDF method to perform word segmentation and remove stop words, and use Euclidean distance to obtain similar measurement reports and topic relevance, and finally pass The text clustering k-means algorithm adds similar high messages to the corresponding topic clusters to obtain topic clusters. Therefore, the top 5 hotspot questions and the message information corresponding to the corresponding hotspot questions are obtained.

       Finally, from the perspective of the relevance, completeness and solvability, timeliness and amount of information of the reply opinion text, to establish related work The evaluation index of the quality of the department's response to the message. This article uses the pre-similarity calculation method to calculate the similarity between the subject of the message and the reply of the relevant work department, and uses the automated readability index ARI to indicate the interpretability. The calculation formula of ARI is: API = 4.71 * Number / Total Words) + 0.5 * (Total Words / Total Sentences) -21.43. Based on the weight calculation method of principal component analysis, the weight of each evaluation index in the evaluation model to be constructed next is calculated. Among them, the weights of relevance, interpretability, effectiveness and amount of information are 0.26, 0.28, 0.18 and 0.28 respectively, and the quality evaluation model of the final opinion is Q = 0.28Words + 0.26Relevancy + 0.28Credibility + 0.18Timeliness + 0.01 .

 

Keywords: natural language, fastText principle, n-gram features, F-Score evaluation, k-means algorithm, Euclidean distance, TD-IDF method, generalized linear regression

evaluation, k-means algorithm, Euclidean distance, TD-IDF method, generalized linear regression

 

 

 

 

 

 

 

 

 

 

 

 

 

目录

1 引言... 1

1.1研究背景... 1

2 挖掘目标... 1

3 模型构建过程及结果分析... 2

3.1 问题1分析方法与过程... 2

3.1.1 流程图... 2

3.1.2 数据预处理... 2

3.1.3 留言文本分词... 2

3.1.4停用过滤词... 4

3.1.5 方法实现过程... 5

3.2 问题2方法与过程... 10

3.2.1问题分析：... 10

3.2.2 话题发现基本流程图... 10

3.2.3 文本预处理... 11

3.2.4 留言信息特征提取... 12

3.2.5 话题的表示模型... 13

3.2.6 热值计算... 15

3.2.7 文本聚类话题提取... 16

3.3 问题3方法与过程... 17

3.3.1 问题分析... 17

3.3.2 预处理... 18

3.3.3 指标提取... 19

3.3.6 构建答复意见质量评价指标体系... 25

4 结论... 26

参考文献... 27

附录... 28

 

对于虚词，比如文本中的“了”、“啊”、“无论”、“比如”等，特殊符号如“#”、“γ”、“μ”、“φ”等，英文中的“is”、“are”、“the”、“that”等。于此同时，由于在不同文本的应用中，构建的停用词表对文本数据分类的精确度以及维度有着不同程度的影响。

因此我们结和中文分词所分出的词的集合进行人工选取拟定停用此表，我们选取分类词频中前200的词，再通过统计这些分类在其他分类中出现的情况，即一个分词词频在各个类标签中出现词频是200，且该分词在超过四个类中同时出现，我们则将此定义为停用词，具体停用此表详见附录一

使用停用词表的效果示例如下：

原始留言：关于预防先天缺陷的建议

结果：预防 先天 缺陷 建议