github项目--crawl4ai

crawl4ai github上这个项目，没记错的话，昨天涨了3000多的star，今天又新增2000star。一款抓取和解析工具，简单写个demo感受下

这里我们使用crawl4ai抓取github每日趋势，每天通过邮件发到自己邮箱

输出html

async def github_trend_html():async with AsyncWebCrawler(verbose=True) as crawler:result = await crawler.arun(url="https://github.com/trending",)assert result.success, "github 数据抓取失败"return result.cleaned_html

输出的还是html，但对原始页面做了处理，比如移除不相关元素，动态元素，简化html结构。

输出markdown格式

async def github_trend_md():async with AsyncWebCrawler(verbose=True) as crawler:result = await crawler.arun(url="https://github.com/trending",)assert result.success, "github 数据抓取失败"return result.markdown

用md软件打开看一下效果：

输出结构化数据

async def github_trend_json():schema = {"name": "Github trending","baseSelector": ".Box-row","fields": [{"name": "repository","selector": ".lh-condensed a[href]","type": "text",},{"name": "description","selector": "p","type": "text",},{"name": "lang","type": "text","selector": "span[itemprop='programmingLanguage']",},{"name": "stars","type": "text","selector": "a[href*='/stargazers']"},{"name": "today_star","type": "text","selector": "span.float-sm-right",},],}extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)async with AsyncWebCrawler(verbose=True) as crawler:result = await crawler.arun(url="https://github.com/trending",extraction_strategy=extraction_strategy,bypass_cache=True,)assert result.success, "github 数据抓取失败"github_trending_json = json.loads(result.extracted_content)for ele in github_trending_json:ele['repository'] = 'https://github.com/' + ''.join(ele['repository'].split())return github_trending_json

与前两种不同的是，结构化输出需要通过自定义schema来定义解析的数据结构。控制台按照我们定义的schema输出了标准了JSON数据。将数据放入html模版，通过邮件每日发送。看一下邮件显示：

与BeautifulSoup的对比

记得第一次用soup的时候，对于只用过Java sax解析xml的我来说，soup真的太方便了。今天简单测试了下crawl4ai，和soup相比

• crawl4ai数据采集分析更方便
• soup需要配合使用request进行网页抓取，BeautifulSoup负责html解析
• html解析有点类似，都是通过CSS选择器，但crawl4ai通过定义schema，解析更方便
• 数据解析方面，crawl4ai除了提供了markdown和简化版的html，还提供了通过集成OpenAI提取结构化数据的能力(尚未体验)