基于sklearn实现文本摘要思考

和各位小伙伴分享一下使用sklearn进行文本摘要的思考。

第一版本

原理

提取式文本摘要的基本原理是：

1. 将文本分割成句子

2. 计算每个句子的重要性(权重)

3. 选择权重最高的几个句子组成摘要

常用的句子权重计算方法：

• TF-IDF：基于词频-逆文档频率

• 文本相似度：计算句子与全文的相似度

• 句子位置：考虑句子在文中的位置(开头/结尾通常更重要)

• 句子长度：适中的句子长度可能更重要

代码实现

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
import nltk
from nltk.tokenize import sent_tokenize
​
nltk.download('punkt')
​
class SklearnSummarizer:def __init__(self, language='english'):self.language = languagedef summarize(self, text, num_sentences=3):"""基于TF-IDF和余弦相似度的文本摘要参数:text: 要摘要的文本num_sentences: 摘要中包含的句子数量返回:摘要文本"""# 分割句子sentences = sent_tokenize(text, self.language)if len(sentences) <= num_sentences:return text# 计算TF-IDF矩阵tfidf = TfidfVectorizer(stop_words=self.language)tfidf_matrix = tfidf.fit_transform(sentences)# 计算句子相似度矩阵sim_matrix = cosine_similarity(tfidf_matrix, tfidf_matrix)# 计算句子重要性得分(与所有其他句子的平均相似度)scores = np.zeros(len(sentences))for i in range(len(sentences)):scores[i] = sim_matrix[i].mean()# 获取得分最高的句子索引top_sentence_indices = scores.argsort()[-num_sentences:][::-1]top_sentence_indices.sort()  # 保持原文顺序# 生成摘要summary = ' '.join([sentences[i] for i in top_sentence_indices])return summary
​
# 使用示例
if __name__ == "__main__":text = """Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language. It focuses on how to program computers to process and analyze large amounts of natural language data. The result is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves. Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural language generation."""summarizer = SklearnSummarizer()summary = summarizer.summarize(text, num_sentences=2)print("摘要结果:")print(summary)

优化

1. 加入句子位置特征：

   # 在计算得分时加入位置权重
   position_weights = [1/(i+1) for i in range(len(sentences))]  # 前面的句子权重更高
   scores = scores * position_weights

2. 加入句子长度特征：

   # 过滤掉过短或过长的句子
   avg_length = np.mean([len(s.split()) for s in sentences])
   length_weights = [1 - abs(len(s.split())-avg_length)/avg_length for s in sentences]
   scores = scores * length_weights

3. 使用更复杂的特征：

   • 命名实体数量

   • 包含数字或特定关键词

   • 句子与标题的相似度

第二版本

第一个版本还是有点问题的：

• 只是简单提取句子，无法生成新句子

• 对长文档效果可能不佳

• 依赖句子分割质量

所以后续又采用Transfromaer进行了重新思考和编写，后续再分享吧。嘿嘿嘿