Qwen2.5-Coder-1.5B应用案例：快速生成网页爬虫代码实战

Qwen2.5-Coder-1.5B应用案例快速生成网页爬虫代码实战1. 引言为什么选择Qwen2.5-Coder生成爬虫代码在日常开发工作中网页爬虫是数据采集和分析的重要工具。传统编写爬虫代码需要开发者熟悉HTTP请求、HTML解析、反爬机制处理等多个技术点耗时且容易出错。Qwen2.5-Coder-1.5B作为专为代码生成优化的AI模型能够快速生成可运行的爬虫代码大幅提升开发效率。本文将展示如何利用Qwen2.5-Coder-1.5B模型通过简单的自然语言描述快速生成Python爬虫代码。您将学习到如何准备Qwen2.5-Coder-1.5B的开发环境编写有效的提示词(Prompt)来生成爬虫代码生成的代码解析与优化建议实际案例演示从电商网站抓取商品信息2. 环境准备与模型调用2.1 安装基础依赖首先确保您的Python环境版本在3.9以上然后安装必要的依赖包pip install transformers torch2.2 加载Qwen2.5-Coder-1.5B模型使用以下代码加载模型和分词器from transformers import AutoModelForCausalLM, AutoTokenizer model_name Qwen/Qwen2.5-Coder-1.5B tokenizer AutoTokenizer.from_pretrained(model_name) model AutoModelForCausalLM.from_pretrained( model_name, device_mapauto, torch_dtypeauto ).eval()3. 生成基础爬虫代码3.1 简单网页抓取示例让我们从最简单的需求开始抓取一个网页的标题和主要内容。prompt 请用Python编写一个爬虫使用requests和BeautifulSoup抓取网页的标题和正文内容。 要求处理HTTP错误和超时并输出结构化结果。 inputs tokenizer(prompt, return_tensorspt).to(model.device) outputs model.generate(**inputs, max_new_tokens512) print(tokenizer.decode(outputs[0], skip_special_tokensTrue))模型生成的代码可能如下import requests from bs4 import BeautifulSoup from urllib.parse import urlparse def scrape_webpage(url): try: headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 } response requests.get(url, headersheaders, timeout10) response.raise_for_status() soup BeautifulSoup(response.text, html.parser) result { url: url, title: soup.title.string if soup.title else None, content: soup.get_text(separator , stripTrue)[:500] ... } return result except requests.exceptions.RequestException as e: print(fError fetching {url}: {e}) return None # 使用示例 if __name__ __main__: webpage_data scrape_webpage(https://example.com) if webpage_data: print(webpage_data)3.2 代码解析与优化建议生成的代码已经具备基本功能我们可以进一步优化增加随机延迟避免被目标网站封禁添加代理支持应对IP限制更精细的内容提取使用CSS选择器优化后的版本import random import time from bs4 import BeautifulSoup import requests def enhanced_scraper(url, proxyNone): try: headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36, Accept-Language: en-US,en;q0.9 } proxies {http: proxy, https: proxy} if proxy else None # 随机延迟1-3秒 time.sleep(random.uniform(1, 3)) response requests.get(url, headersheaders, proxiesproxies, timeout15) response.raise_for_status() soup BeautifulSoup(response.text, html.parser) # 使用CSS选择器提取主要内容 main_content soup.select_one(main, .main-content, #content) or soup.body return { url: url, title: soup.title.string if soup.title else None, description: soup.find(meta, attrs{name: description})[content] if soup.find(meta, attrs{name: description}) else None, content: main_content.get_text(separator , stripTrue)[:1000] ... } except Exception as e: print(fError: {e}) return None4. 实战案例电商商品信息抓取4.1 生成电商爬虫代码让我们尝试更复杂的需求抓取电商网站的商品列表。ecommerce_prompt 请编写一个Python爬虫从电商网站(如淘宝、京东)抓取商品列表 包括商品名称、价格、销量、商家名称等信息。 需要考虑反爬机制使用随机User-Agent和请求延迟。 输出为JSON格式。 inputs tokenizer(ecommerce_prompt, return_tensorspt).to(model.device) outputs model.generate(**inputs, max_new_tokens768) print(tokenizer.decode(outputs[0], skip_special_tokensTrue))生成的代码可能包含以下关键部分import requests from bs4 import BeautifulSoup import json import random import time USER_AGENTS [ Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15, Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 ] def scrape_ecommerce(keyword, pages1): base_url https://search.jd.com/Search products [] for page in range(1, pages 1): params { keyword: keyword, page: page, s: (page - 1) * 30 1 } headers { User-Agent: random.choice(USER_AGENTS), Referer: https://www.jd.com/ } try: # 随机延迟避免被封 time.sleep(random.uniform(1, 3)) response requests.get(base_url, paramsparams, headersheaders, timeout10) response.raise_for_status() soup BeautifulSoup(response.text, html.parser) items soup.select(.gl-item) for item in items: product { name: item.select_one(.p-name em).get_text(stripTrue), price: item.select_one(.p-price i).get_text(stripTrue), shop: item.select_one(.p-shop a).get_text(stripTrue), comments: item.select_one(.p-commit a).get_text(stripTrue) } products.append(product) except Exception as e: print(fError on page {page}: {e}) continue return json.dumps(products, ensure_asciiFalse, indent2)4.2 处理动态内容与反爬机制现代电商网站大量使用JavaScript动态加载内容我们需要扩展爬虫能力from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC def scrape_dynamic_ecommerce(keyword): options Options() options.add_argument(--headless) options.add_argument(--disable-gpu) options.add_argument(fuser-agent{random.choice(USER_AGENTS)}) driver webdriver.Chrome(optionsoptions) driver.get(fhttps://search.jd.com/Search?keyword{keyword}) try: # 等待商品列表加载 WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.CSS_SELECTOR, .gl-item)) ) # 模拟滚动加载更多 for _ in range(3): driver.execute_script(window.scrollTo(0, document.body.scrollHeight);) time.sleep(2) soup BeautifulSoup(driver.page_source, html.parser) items soup.select(.gl-item) products [] for item in items: try: products.append({ name: item.select_one(.p-name em).get_text(stripTrue), price: item.select_one(.p-price i).get_text(stripTrue), shop: item.select_one(.p-shop a).get_text(stripTrue) }) except: continue return products finally: driver.quit()5. 高级技巧与最佳实践5.1 提升爬虫稳定性的技巧请求头管理模拟真实浏览器行为headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64), Accept: text/html,application/xhtmlxml, Accept-Language: en-US,en;q0.9, Referer: https://www.google.com/, DNT: 1 }代理轮换使用代理池避免IP封禁PROXY_POOL [http://proxy1:port, http://proxy2:port] def get_with_proxy(url): proxy random.choice(PROXY_POOL) try: return requests.get(url, proxies{http: proxy, https: proxy}, timeout10) except: return None异常处理完善的错误恢复机制def robust_request(url, max_retries3): for attempt in range(max_retries): try: response requests.get(url, timeout15) response.raise_for_status() return response except requests.exceptions.RequestException as e: print(fAttempt {attempt 1} failed: {e}) if attempt max_retries - 1: time.sleep(2 ** attempt) # 指数退避 else: raise5.2 数据存储方案将爬取的数据保存到文件或数据库import csv import sqlite3 # CSV存储 def save_to_csv(data, filename): with open(filename, w, newline, encodingutf-8) as f: writer csv.DictWriter(f, fieldnamesdata[0].keys()) writer.writeheader() writer.writerows(data) # SQLite存储 def save_to_sqlite(data, db_file): conn sqlite3.connect(db_file) c conn.cursor() c.execute(CREATE TABLE IF NOT EXISTS products (name TEXT, price TEXT, shop TEXT, comments TEXT)) for item in data: c.execute(INSERT INTO products VALUES (?,?,?,?), (item[name], item[price], item[shop], item.get(comments, ))) conn.commit() conn.close()6. 总结与下一步建议通过本文的实践我们展示了如何利用Qwen2.5-Coder-1.5B快速生成功能完善的网页爬虫代码。这种AI辅助开发方式可以大幅提升开发效率特别是在需要快速原型开发的场景中。关键收获Qwen2.5-Coder能够理解自然语言需求并生成可运行的Python代码生成的代码基础功能完整但需要根据实际场景进行优化电商类网站爬取需要考虑反爬机制和动态内容加载下一步建议尝试更复杂的爬虫场景如登录认证、验证码处理等探索分布式爬虫架构提升爬取效率学习使用Scrapy等专业爬虫框架与AI生成代码结合注意事项遵守目标网站的robots.txt协议控制请求频率避免对目标网站造成过大负担注意数据使用的合法性尊重版权和隐私获取更多AI镜像想探索更多AI镜像和应用场景访问 CSDN星图镜像广场提供丰富的预置镜像覆盖大模型推理、图像生成、视频生成、模型微调等多个领域支持一键部署。