高级RAG：揭秘PDF解析

原文地址：https://pub.towardsai.net/advanced-rag-02-unveiling-pdf-parsing-b84ae866344e

2024 年 2 月 3 日

附加内容：揭秘PDF解析：如何从科学pdf论文中提取公式

对于RAG，从文档中提取信息是一个不可避免的场景。确保从源头提取内容的有效性对于提高最终输出的质量至关重要。

重要的是不要低估这个过程。在实现RAG时，解析过程中的信息提取不佳可能导致对PDF文件中包含的信息的理解和利用受到限制。

pass过程在RAG中的位置如图1所示:

图1：通过过程在RAG中的位置(红框)。图片来自作者。

在实际工作中，非结构化数据要比结构化数据丰富得多。如果这些海量的数据不能被解析，它们的巨大价值就无法实现。

在非结构化数据中，PDF文档占多数。有效地处理PDF文档还可以**极大地帮助管理其他类型的非结构化文档

本文主要介绍解析PDF文件的方法。它提供了有效解析PDF文档和提取尽可能多的有用信息的算法和建议。

解析PDF的挑战

PDF文档是非结构化文档的代表，但是，从PDF文档中提取信息是一个具有挑战性的过程。

与其说PDF是一种数据格式，不如将其描述为打印指令的集合更为准确。PDF文件由一系列指令组成，这些指令指示PDF阅读器或打印机在屏幕或纸张上显示符号的位置和方式。这与HTML和docx等文件格式形成对比，后者使用<p>、<w:p>、<table>和 <w:tbl>>来组织不同的逻辑结构，如图2所示:

图2：Html与PDF。图片来自作者。

解析PDF文档的挑战在于准确地提取整个页面的布局，并将内容(包括表格、标题、段落和图像)翻译成文档的文本表示形式。该过程涉及处理文本提取、图像识别和表中行-列关系的混淆中的不准确性。

如何解析PDF文档

一般来说，有三种解析pdf的方法:

• 基于规则的方法:根据文件的组织特征确定每个部分的风格和内容。然而，这种方法不是很通用，因为pdf有许多类型和布局，因此不可能用预定义的规则覆盖它们。
• 基于深度学习模型的方法:如目前流行的结合物体检测和OCR模型的解决方案。
• 基于多模态大模型传递复杂结构或提取pdf中的关键信息。

基于规则的方法

最具代表性的工具之一是pypdf，它是一种广泛使用的基于规则的解析器。它是Langchain和LlamaIndex中解析PDF文件的标准方法。

下面是尝试使用pypdf解析“（Attention Is All You Need）”论文的第6页。原始页面如图3所示。

图3:“Attention Is All You Need”论文的原始第6页。

代码如下:

1 2 3 4 5 6 7 8 9 10 11 12 13 14

import PyPDF2 filename = "/Users/Florian/Downloads/1706.03762.pdf" pdf_file = open(filename, 'rb')reader = PyPDF2.PdfReader(pdf_file)page_num = 5 page = reader.pages[page_num] text = page.extract_text()print('--------------------------------------------------') print(text)pdf_file.close()

执行的结果是(为简洁起见，省略了其余部分):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

(py) Florian:~ Florian$ pip list | grep pypdf pypdf 3.17.4 pypdfium2 4.26.0(py) Florian:~ Florian$ python /Users/Florian/Downloads/pypdf_test.py -------------------------------------------------- Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations for different layer types. nis the sequence length, dis the representation dimension, kis the kernel size of convolutions and rthe size of the neighborhood in restricted self-attention. Layer Type Complexity per Layer Sequential Maximum Path Length Operations Self-Attention O(n2·d) O(1) O(1) Recurrent O(n·d2) O(n) O(n) Convolutional O(k·n·d2) O(1) O(logk(n)) Self-Attention (restricted) O(r·n·d) O(1) O(n/r) 3.5 Positional Encoding Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel as the embeddings, so that the two can be summed. There are many choices of positional encodings, learned and fixed [9]. In this work, we use sine and cosine functions of different frequencies: PE(pos,2i)=sin(pos/100002i/d model) PE(pos,2i+1)=cos(pos/100002i/d model) where posis the position and iis the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2πto10000 ·2π. We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k,PEpos+kcan be represented as a linear function of PEpos. ... ... ...

基于PyPDF检测的结果，观察到它将PDF中的字符序列序列化为单个长序列，而不保留结构信息。换句话说，它将文档的每行视为由换行字符“\n”分隔的序列，这妨碍了对段落或表格的准确识别。

这种限制是基于规则的方法的固有特征。

基于深度学习模型的方法。

这种方法的优点是它能够准确地识别整个文档的布局，包括表格和段落。它甚至可以理解表中的结构。这意味着它可以将文档划分为定义良好的完整信息单元，同时保留预期的含义和结构。

然而，也有一些限制。目标检测和OCR阶段可能很耗时。因此，建议使用GPU或其他加速设备，并使用多个进程和线程进行处理。

这种方法涉及到对象检测和OCR模型，我已经测试了几个代表性的开源框架:

• 非结构化:已集成到Langchain。 infer_table_structure=True的hi_res策略的表识别效果较好。然而，fast策略表现不佳，因为它没有使用目标检测模型，错误地识别了许多图像和表格。

• Layout-parser:如果您需要识别复杂的结构化pdf，建议使用最大的模型以获得更高的精度，尽管它可能会稍微慢一些。此外，似乎Layout-parser的模型在过去两年中没有更新过。

• PP-StructureV2:采用多种模型组合进行文档分析，性能高于平均水平。架构如图4所示:

图4：拟议PP-StructureV2的框架。它包含布局信息提取和关键信息提取两个子系统。来源:PP-StructureV2。

除了开源工具之外，还有一些付费工具，比如ChatDOC，利用基于布局的识别+ OCR方法来解析PDF文档。

接下来，我们将解释如何使用开源**非结构化 **框架解析pdf，解决三个关键挑战

挑战1:如何从表和图像中提取数据

在这里，我们将使用非结构化框架作为示例。检测到的表数据可以直接导出为HTML。其代码如下:

1 2 3 4 5 6 7 8 9 10 11

from unstructured.partition.pdf import partition_pdffilename = "/Users/Florian/Downloads/Attention_Is_All_You_Need.pdf"# infer_table_structure=True automatically selects hi_res strategy elements = partition_pdf(filename=filename, infer_table_structure=True) tables = [el for el in elements if el.category == "Table"]print(tables[0].text) print('--------------------------------------------------') print(tables[0].metadata.text_as_html)

我已经跟踪了partition_pdf 函数的内部过程。图5是一个基本流程图。

图5：**partition_pdf**函数的内部过程。图片来自作者。

代码运行结果如下:

1 2 3

Layer Type Self-Attention Recurrent Convolutional Self-Attention (restricted) Complexity per Layer O(n2 · d) O(n · d2) O(k · n · d2) O(r · n · d) Sequential Maximum Path Length Operations O(1) O(n) O(1) O(1) O(1) O(n) O(logk(n)) O(n/r) -------------------------------------------------- <table><thead><th>Layer Type</th><th>Complexity per Layer</th><th>Sequential Operations</th><th>Maximum Path Length</th></thead><tr><td>Self-Attention</td><td>O(n? - d)</td><td>O(1)</td><td>O(1)</td></tr><tr><td>Recurrent</td><td>O(n- d?)</td><td>O(n)</td><td>O(n)</td></tr><tr><td>Convolutional</td><td>O(k-n-d?)</td><td>O(1)</td><td>O(logy(n))</td></tr><tr><td>Self-Attention (restricted)</td><td>O(r-n-d)</td><td>ol)</td><td>O(n/r)</td></tr></table>

复制HTML标记并将其保存为HTML文件。然后，使用Chrome打开它，如图6所示:

图6：图3中表1的可视化表示。图片来自作者。

可以看出，非结构化算法在很大程度上恢复了整个表。

挑战2:如何重新排列检测到的方块?特别是对于双列pdf文件

在处理双列pdf时，让我们以论文“BERT:用于语言理解的深度双向Transformers的预训练”为例。阅读顺序用红色箭头表示:

图7：双列页面。

在确定布局之后，非结构化框架将把每个页面划分为几个矩形块，如图8所示。

图8:布局检测结果的可视化。图片来自作者。

每个矩形块的详细信息可以通过以下格式获得:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

[LayoutElement(bbox=Rectangle(x1=851.1539916992188, y1=181.15073777777613, x2=1467.844970703125, y2=587.8204599999975), text='These approaches have been generalized to coarser granularities, such as sentence embed- dings (Kiros et al., 2015; Logeswaran and Lee, 2018) or paragraph embeddings (Le and Mikolov, 2014). To train sentence representations, prior work has used objectives to rank candidate next sentences (Jernite et al., 2017; Logeswaran and Lee, 2018), left-to-right generation of next sen- tence words given a representation of the previous sentence (Kiros et al., 2015), or denoising auto- encoder derived objectives (Hill et al., 2016). ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.9519357085227966, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=196.5296173095703, y1=181.1507377777777, x2=815.468994140625, y2=512.548237777777), text='word based only on its context. Unlike left-to- right language model pre-training, the MLM ob- jective enables the representation to fuse the left and the right context, which allows us to pre- In addi- train a deep bidirectional Transformer. tion to the masked language model, we also use a “next sentence prediction” task that jointly pre- trains text-pair representations. The contributions of our paper are as follows: ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.9517233967781067, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=200.22352600097656, y1=539.1451822222216, x2=825.0242919921875, y2=870.542682222221), text='• We demonstrate the importance of bidirectional pre-training for language representations. Un- like Radford et al. (2018), which uses unidirec- tional language models for pre-training, BERT uses masked language models to enable pre- trained deep bidirectional representations. This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs. ', source=<Source.YOLOX: 'yolox'>, type='List-item', prob=0.9414362907409668, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=851.8727416992188, y1=599.8257377777753, x2=1468.0499267578125, y2=1420.4982377777742), text='ELMo and its predecessor (Peters et al., 2017, 2018a) generalize traditional word embedding re- search along a different dimension. They extract context-sensitive features from a left-to-right and a right-to-left language model. The contextual rep- resentation of each token is the concatenation of the left-to-right and right-to-left representations. When integrating contextual word embeddings with existing task-specific architectures, ELMo advances the state of the art for several major NLP benchmarks (Peters et al., 2018a) including ques- tion answering (Rajpurkar et al., 2016), sentiment analysis (Socher et al., 2013), and named entity recognition (Tjong Kim Sang and De Meulder, 2003). Melamud et al. (2016) proposed learning contextual representations through a task to pre- dict a single word from both left and right context using LSTMs. Similar to ELMo, their model is feature-based and not deeply bidirectional. Fedus et al. (2018) shows that the cloze task can be used to improve the robustness of text generation mod- els. ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.938507616519928, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=199.3734130859375, y1=900.5257377777765, x2=824.69873046875, y2=1156.648237777776), text='• We show that pre-trained representations reduce the need for many heavily-engineered task- specific architectures. BERT is the first fine- tuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outper- forming many task-specific architectures. ', source=<Source.YOLOX: 'yolox'>, type='List-item', prob=0.9461237788200378, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=195.5695343017578, y1=1185.526123046875, x2=815.9393920898438, y2=1330.3272705078125), text='• BERT advances the state of the art for eleven NLP tasks. The code and pre-trained mod- els are available at https://github.com/ google-research/bert. ', source=<Source.YOLOX: 'yolox'>, type='List-item', prob=0.9213815927505493, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=195.33956909179688, y1=1360.7886962890625, x2=447.47264000000007, y2=1397.038330078125), text='2 Related Work ', source=<Source.YOLOX: 'yolox'>, type='Section-header', prob=0.8663332462310791, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=197.7477264404297, y1=1419.3353271484375, x2=817.3308715820312, y2=1527.54443359375), text='There is a long history of pre-training general lan- guage representations, and we briefly review the most widely-used approaches in this section. ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.928022563457489, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=851.0028686523438, y1=1468.341394166663, x2=1420.4693603515625, y2=1498.6444497222187), text='2.2 Unsupervised Fine-tuning Approaches ', source=<Source.YOLOX: 'yolox'>, type='Section-header', prob=0.8346447348594666, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=853.5444444444446, y1=1526.3701822222185, x2=1470.989990234375, y2=1669.5843488888852), text='As with the feature-based approaches, the first works in this direction only pre-trained word em- (Col- bedding parameters from unlabeled text lobert and Weston, 2008). ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.9344717860221863, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=200.00000000000009, y1=1556.2037353515625, x2=799.1743774414062, y2=1588.031982421875), text='2.1 Unsupervised Feature-based Approaches ', source=<Source.YOLOX: 'yolox'>, type='Section-header', prob=0.8317819237709045, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=198.64227294921875, y1=1606.3146266666645, x2=815.2886352539062, y2=2125.895459999998), text='Learning widely applicable representations of words has been an active area of research for decades, including non-neural (Brown et al., 1992; Ando and Zhang, 2005; Blitzer et al., 2006) and neural (Mikolov et al., 2013; Pennington et al., 2014) methods. Pre-trained word embeddings are an integral part of modern NLP systems, of- fering significant improvements over embeddings learned from scratch (Turian et al., 2010). To pre- train word embedding vectors, left-to-right lan- guage modeling objectives have been used (Mnih and Hinton, 2009), as well as objectives to dis- criminate correct from incorrect words in left and right context (Mikolov et al., 2013). ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.9450697302818298, image_path=None, parent=None), LayoutElement(bbox=Rectangle(x1=853.4905395507812, y1=1681.5868488888855, x2=1467.8729248046875, y2=2125.8954599999965), text='More recently, sentence or document encoders which produce contextual token representations have been pre-trained from unlabeled text and fine-tuned for a supervised downstream task (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018). The advantage of these approaches is that few parameters need to be learned from scratch. At least partly due to this advantage, OpenAI GPT (Radford et al., 2018) achieved pre- viously state-of-the-art results on many sentence- level tasks from the GLUE benchmark (Wang language model- Left-to-right et al., 2018a). ', source=<Source.YOLOX: 'yolox'>, type='Text', prob=0.9476840496063232, image_path=None, parent=None)]

式中(x1, y1)为左上顶点的坐标，(x2, y2)为右下顶点的坐标:

1 2 3 4 5

(x_1, y_1) --------| || || |---------- (x_2, y_2)

此时，您可以选择重新调整页面的阅读顺序。Unstructured自带内置排序算法，但我发现在处理双列情况时排序结果不是很令人满意。

因此，有必要设计一种算法。最简单的方法是先按左上角顶点的水平坐标排序，如果水平坐标相同，再按垂直坐标排序。伪代码如下:

1	layout.sort(key=lambda z: (z.bbox.x1, z.bbox.y1, z.bbox.x2, z.bbox.y2))

然而，我们发现即使在同一列中的块也可能在其水平坐标上有变化。如图9所示，紫色线块的水平坐标bbox。X1实际上更靠左。排序时，它将被放置在绿行块之前，这显然违反了读取顺序。

图9：同一列的水平坐标可能有变化。图片来自作者。

在这种情况下，一个可能使用的算法如下:

• 首先，对所有左上角的x坐标x1进行排序，我们可以得到x1_min
• 然后，对所有右下角的x坐标x2进行排序，我们可以得到x2_max
• 接下来，确定页面中心线的x坐标为:

1 2 3

x1_min = min([el.bbox.x1 for el in layout]) x2_max = max([el.bbox.x2 for el in layout]) mid_line_x_coordinate = (x2_max + x1_min) / 2

接下来，**if bbox.x1 < mid_line_x_coordinate **，该块被分类为左列的一部分。否则，它被认为是右列的一部分。

分类完成后，根据列中的y坐标对每个块进行排序。最后，将右列连接到左列的右侧。

1 2 3 4 5 6 7 8 9 10 11

left_column = [] right_column = [] for el in layout:if el.bbox.x1 < mid_line_x_coordinate:left_column.append(el)else:right_column.append(el)left_column.sort(key = lambda z: z.bbox.y1) right_column.sort(key = lambda z: z.bbox.y1) sorted_layout = left_column + right_column

值得一提的是，这种改进还与单列pdf兼容。

挑战3：如何提取多级标题

抽取题目（包括多级题目）的目的是为了提高LLM答案的准确性。

例如，如果用户想知道图9中section 2.1的主要思想，通过准确提取section 2.1的标题，并将其与相关内容作为上下文一起发送给LLM，最终答案的准确性将大大提高。

该算法仍然依赖于图9所示的布局块。我们可以使用**type='Section-header提取块并计算高度差(bbox.y2 - bbox.y1 **)。高差最大的块对应第一级标题，其次是第二级标题，最后是第三级标题。

基于多模态大模型在pdf中传递复杂结构

在多模态模型爆发之后，也可以使用多模态模型来解析表。有几个选项:

• 检索相关图像(PDF页)，并将其发送给GPT4-V，以回应查询。
• 将每个PDF页面视为图像，让GPT4-V对每个页面进行图像推理。为图像推理建立文本矢量存储索引。根据图像推理向量库查询答案。
• 使用表转换器从检索到的图像中裁剪表信息，然后将这些裁剪后的图像发送到GPT4-V进行查询响应。
• 对裁剪的表格图像应用OCR，并将数据发送到GPT4/ GPT-3.5来回答查询。

经过测试，确定第三种方法最有效

此外，我们可以使用多模态模型从图像中提取或总结关键信息(PDF文件可以很容易地转换为图像)，如图10所示。

图10：从图像中提取或总结关键信息。来源:GPT-4 with Vision: Complete Guide and Evaluation。

结论

一般来说，非结构化文档提供了高度的灵活性，需要各种解析技术。然而，目前还没有达成共识的最佳方法使用。

在这种情况下，建议选择最适合您项目需求的方法。建议根据不同类型的pdf文件采用特定的处理方法。例如，文件、书籍和财务报表可能根据其特点有独特的设计

然而，如果情况允许，仍然建议选择基于深度学习或基于多模态的方法。这些方法可以有效地将文档分割成定义良好且完整的信息单元，从而最大限度地保留文档的预期含义和结构。