myaijarvis - 爬虫
type: channel
link: http://myaijarvis.com/tag/爬虫
items: Python用Pyspider爬取TripAdvisor的景点信息
items: Python用selenium爬取淘宝美食信息
items: Python分析Ajax请求抓取今日头条街拍美图
items: Python用正则爬取猫眼之Top100榜的电影信息
items: Python用正则爬取豆瓣读书之新书速递的书籍信息
items: Python爬取豆瓣读书Top100的书籍信息

Python用Pyspider爬取TripAdvisor的景点信息
type: item
date: 2020-03-10T13:17:00+08:00
description: 先上效果图上代码：from pyspider.libs.base_handler import * import pymongo class Handler(BaseHandler): crawl_config = { } client = pymongo.MongoClient('localhost') db = client['TripAdvise'] @every(minutes=24 * 60) def on_start(self): self.crawl('https://www.tripadvisor.cn/Attractions-g186338-Activities-c47-London_England.html',callback=self.index_page, validate_cert=False) @config(age=10 * 24 * 60 * 60) def index_page(self, response): for each in response.doc('div.listing_title > a').items(): self.crawl(each.attr.href, callback=self.detail_page, validate_cert=False) # 翻页 next = response.doc('.pagination .nav.next').attr.href self.crawl(next, callback=self.index_page, validate_cert=False) @config(priority=2) def detail_page(self, response): url = response.url name = response.doc('.h1').text() rating = response.doc(' div.ratingContainer > a > span').text() garde = response.doc('div.section.rating > span').text() address = response.doc('.contactInfo > .address').text() phone = response.doc('div.contact > div.contactType.phone.is-hidden-mobile > div').text() opening = response.doc('div.prw_rup.prw_common_atf_header_bl_responsive.headerBL > div > span').text() introduction = response.doc('.centerWell > div > div > div > div > div > span').text() return { "url": url, "name": name, "rating": rating, 'garde': garde, "address": address, "phone": phone, "opening": opening, "introduction": introduction } def on_result(self, result): if result: self.save_to_mongo(result) def save_to_mongo(self, result): if self.db['london'].insert(result): print('保存到MongoDB成功', result)参考：https://cuiqingcai.com/8317.html

Python用selenium爬取淘宝美食信息
type: item
date: 2020-03-09T11:32:00+08:00
description: 先奉上代码：# 如果使用PhantomJS则selenium版本只能2.48.0及以下 from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC import time import re from pyquery import PyQuery as py import pymongo MONGO_URL='localhost' MONGO_DB='taobao' MONGO_TABLE='product' SERVICE_ARGS=['--load-images=false','--disk-cache=true'] # 不加载图片，开启缓存 client = pymongo.MongoClient(MONGO_URL, connect=False) db=client[MONGO_DB] # browser = webdriver.Chrome() browser = webdriver.PhantomJS(service_args=SERVICE_ARGS) browser.set_window_size(1400,900) #设置窗口大小，防止元素位置改变 wait = WebDriverWait(browser, 15) def search(): # 执行搜索后跳转到登录界面--选择微博登录-（淘宝要绑定微博，微博登录要取消双重登录验证） try: browser.get("https://www.taobao.com/") print('正在搜索') input = wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#q")) ) # 输入框 submit = wait.until( EC.element_to_be_clickable((By.CSS_SELECTOR, "#J_TSearchForm > div.search-button > button")) ) # 搜索按钮 input.send_keys('美食') time.sleep(1) submit.click() # 登录 print('正在登录') wait.until( EC.element_to_be_clickable((By.CSS_SELECTOR, "#J_Quick2Static")) ).click() # 点击密码登录 time.sleep(2) wait.until( EC.element_to_be_clickable((By.CSS_SELECTOR, "#J_OtherLogin > a.weibo-login")) ).click() # 选择微博登录 wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#pl_login_logged > div > div:nth-child(2) > div > input")) ).send_keys(' ') # 输入账号 time.sleep(2) wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#pl_login_logged > div > div:nth-child(3) > div > input")) ).send_keys(' ') # 输入密码 time.sleep(2) wait.until( EC.element_to_be_clickable( (By.CSS_SELECTOR, "#pl_login_logged > div > div:nth-child(7) > div:nth-child(1) > a > span")) ).click() # 点击登录按钮 time.sleep(5) total = wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#mainsrp-pager > div > div > div > div.total")) ) print('登录成功') get_prodects() return total.text except TimeoutError: search() # 翻页 def ne…

Python分析Ajax请求抓取今日头条街拍美图
type: item
date: 2020-03-08T14:29:22+08:00
description: 结果如下：程序运行结果"C:\Program Files\Python\Python38\python.exe" E:/PycharmProjects/test/爬虫/今日头条街拍/Spider.py 请求文章详情页出错 None 正在下载1d47b4dceb635bff92ef296ce530cdc2.jpg 正在下载bff2d27b1d4ed77cc0a575513b9eeea4.jpg 正在下载f30c57b85c75fe655612450c993b320b.jpg 正在下载2475f2ad97bdcb6a45e6d64ce1650ca6.jpg 正在下载52fc1cd305b52366532f3ec7a9729a71.jpg 正在下载dc59cae9ed2859454c461b2944b819b1.jpg 正在下载bf1b48c4ca5cfeffec3bee95b1da6b64.jpg 正在下载7b8c8a28c037eac669988e34e90e37de.jpg 正在下载ad0fb9dbfcc0f15981fd701339e13bb8.jpg 正在下载98486366369a8c0b79f4c6ced3fb51b0.jpg E:/PycharmProjects/test/爬虫/今日头条街拍/Spider.py:125: DeprecationWarning: insert is deprecated. Use insert_one or insert_many instead. if db[MONGO_TABLE].insert(result): 存储到MongoDB成功 {'url': 'http://toutiao.com/group/6760980286252515853/', 'title': '三里屯街拍', 'images': ['http://p9.pstatp.com/origin/pgc-image/f23cd60127824cb88b11256cf9ffe951', 'http://p9.pstatp.com/origin/pgc-image/0c54ffb46fdd4553ad99e9273a85643f', 'http://p9.pstatp.com/origin/pgc-image/da0dd5a6d8314f699de340c06a9093ec', 'http://p3.pstatp.com/origin/pgc-image/7fff7dff1863466883d0920481bc9ffe', 'http://p1.pstatp.com/origin/pgc-image/9ddcb34586514ab7ad6427c73c416f51', 'http://p3.pstatp.com/origin/pgc-image/a406850542ca496aa1f68333e42aea87', 'http://p1.pstatp.com/origin/pgc-image/50bb8781363e4e3e8e9e45137ded07f6', 'http://p3.pstatp.com/origin/pgc-image/6dfc140a4d814cd49295f210aa1499bd', 'http://p1.pstatp.com/origin/pgc-image/014679ab63584e0ca7f8612b8b120651', 'http://p9.pstatp.com/origin/pgc-image/be0226872a864d2aac76c28987938cdf'], '_id': ObjectId('5e648f564d5a25526d8630e9')} 请求文章详情页出错 None 正在下载289d58776687584857ea8d2cec8e3266.jpg 正在下载4d89bd5efc7166322207e31e737cd6d2.jpg 正在下载bcfd11725613120a2f6f20145521bf70.jpg 正在下载2256eed522bab696d578be7bd021a79d.jpg 正在下载9f4b6fac29acae9d0d3964a33922cd76.jpg 正在下载6226b26aacb6111fa5bb6ab4ecb360ca.jpg 正在下载22e469ecf6277bf6631e9053b0782656.jpg 正在下载fb6e9bba0afd3880fd5acb418c08af06.jpg 正在下载9e418d8f5aa93beefe64005a941b0b8d.jpg 正在下载dc98a060a3ee9abe1583bc304de1dfd1.jpg…

Python用正则爬取猫眼之Top100榜的电影信息
type: item
date: 2020-03-06T21:59:03+08:00
description: 爬取信息的主要出处 2 肖申克的救赎 主演：蒂姆·罗宾斯,摩根·弗里曼,鲍勃·冈顿 上映时间：1994-09-10(加拿大) 9.5 代码如下：import requests import json import re import time class Spider: def __init__(self): self.url_temp = 'https://maoyan.com/board/4?offset={}' self.headers = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'} # 1.根据地址的规律构造url_list def get_url_list(self): url_list = [self.url_temp.format(i) for i in range(0, 91, 10)] return url_list # 2.发送请求，获取响应 def req_url_resp(self, url): print('start:' + url) response = requests.get(url, headers=self.headers) # print(response.text) return response.content.decode() # 3.提取数据 def get_content_list(self, resp_content): content_list = [] # 用列表存储信息 pattern = re.compile( ' .*?board-index.*?>(\d+).*?<img.*?data-src="(.*?)".*?name".*?><a.*?>(.*?).*?star">(.*?) ' '.*?releasetime">(.*?) .*?integer">(.*?).*?fraction">(.*?).*? ', re.S) contents = re.findall(pattern, resp_content) for content in contents: item = {} item["id"] = content[0] item["img_url"] = content[1] item["name"] = content[2] item["star"] = content[3].strip()[3:] item["releasetime"] = content[4][5:] item["grade"] = content[5] + content[6] + "分" # print(id,name,star,releasetime,grade,img_url) content_list.append(item) return content_list # 4.保存数据 def save_content(self, content_list): with open("猫眼Top100.txt", 'a', encoding='utf-8') as f: for content in content_list: f.write(json.dumps(content, ensure_ascii=False)) f.write('\n') print('保存成功！') def run(self): # 1.根据地址的规律构造url_list url_list = self.get_url_list() # 2.发送请求，获取响应 for url in url_list: time.sleep(1) resp_content = self.req_url_resp(url) # 3.提取数据 content_list = self.get_content_list(resp_content) # 4.保存数据 self.save_content(content_list) if __name__ == '__main__': spider = Spider() spider.run() 结果如下：{"id": "1", "img_url": "https://p0.meituan.net/movie/ce4da3e03e655b5b88ed31b5cd7896cf62472.jpg@160w_220h_1e_1c", "name": "霸王别姬", "star": "张国荣,张丰毅,巩俐", "releasetime": "1993-07-26"…

Python用正则爬取豆瓣读书之新书速递的书籍信息
type: item
date: 2020-03-05T13:14:00+08:00
description: 新书速递的书籍信息数据的主要采集出处 夏先生的故事 [德] 帕特里克·聚斯金德 夏先生的故事 [德] 帕特里克·聚斯金德 / 2020-1 / 上海译文出版社 《香水》作者聚斯金德回归纯真之作。 《纽约客》御用画手、法国国宝级插画大师桑贝经典配图。 “我”是一个普通的小男孩，上学，爬树，骑车，幻想着朦胧的初恋，害怕严厉的钢琴教师，常常觉得自己会飞。“我”的邻居夏先生是个“怪人”，沉默地拄着拐杖，背着空空的行囊，没有目的地奔走不休。我们之间只有为数不多的几次接触，但这个敏感易怒、行色匆匆、似乎总在忍受... 代码如下import requests import re url = "https://book.douban.com/" headers = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'} content = requests.get(url, headers=headers).text # 使用re.S参数以后，正则表达式会将这个字符串作为一个整体，将“\n”当做一个普通的字符加入到这个字符串中，在整体中进行匹配。 pattern = re.compile( '<li.*?cover.*?href="(.*?)\stitle="(.*?)">.*?src="(.*?)".*?more-meta.*?author">(.*?).*?year">(.*?).*?publisher">(.*?).*? ', re.S) ##这里应该要在URL和title之间匹配空格，用\s results = re.findall(pattern, content) for result in results: url, name, img_url, author, date, publisher = result author = re.sub('\s', '', author) # 把前后的空白符去掉 author=author.strip()也可以 date = re.sub('\s', '', date) publisher = re.sub('\s', '', publisher) print(name, author, date, publisher, url, img_url) 结果数据如下人类向何处去 [英]奥拉夫·斯特普尔顿 2020-1 北京航空航天大学出版社 https://book.douban.com/subject/34934004/?icn=index-latestbook-subject" https://img9.doubanio.com/view/subject/s/public/s33549854.jpg 戏剧游戏：儿童戏剧游戏 [英]凯瑟琳·扎切斯特 / KatherineZachest 2020-1 后浪丨上海文化出版社 https://book.douban.com/subject/34935067/?icn=index-latestbook-subject" https://img3.doubanio.com/view/subject/s/public/s33550710.jpg 算计 [日]米泽穗信 2020-1 人民文学出版社 https://book.douban.com/subject/34866078/?icn=index-latestbook-subject" https://img1.doubanio.com/view/subject/s/public/s33542087.jpg 从催眠的世界中不断醒来 姜涛 2020-1 华东师范大学出版社 https://book.douban.com/subject/34951625/?icn=index-latestbook-subject" https://img9.doubanio.com/view/subject/s/public/s33562486.jpg 十三封自杀告别信 [德]科斯汀•吉尔 2020-1-1 新星出版社 https://book.douban.com/subject/34908017/?icn=index-latestbook-subject" https://img3.doubanio.com/view/subject/s/public/s33531850.jpg 论爱与死亡 [德]帕·聚斯金德 2020-1 上海译文出版社 https://book.douban.com/subject/348…

Python爬取豆瓣读书Top100的书籍信息
type: item
date: 2020-03-02T23:01:00+08:00
description: 爬取豆瓣读书Top100的书籍信息（实则只有79本书）import requests import json from lxml import etree import re import time class Spider: def __init__(self): self.url_temp = 'https://www.douban.com/doulist/118410371/?start={}' self.headers = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'} # 1.根据地址的规律构造url_list def get_url_list(self): url_list = [self.url_temp.format(i) for i in range(0, 76, 25)] return url_list # 2.发送请求，获取响应 def req_url_resp(self, url): print('start:' + url) response = requests.get(url, headers=self.headers) return response.content.decode() # 3.提取数据 def get_content_list(self, resp_content): content = etree.HTML(resp_content) content_list = [] # 用列表存储信息 div_list = content.xpath("//div[@class='article']//div[@class='doulist-item']") for div in div_list: item = {} item['title'] = div.xpath(".//div[@class='title']/a/text()")[0].strip() item['author'] = div.xpath(".//div[@class='abstract']/text()")[0].strip() # 作者：江一帆 try: result = re.search(".*:(.*)", item['author']) item['author'] = result.group(1) # 江一帆 except Exception as e: print(e) item['press'] = div.xpath(".//div[@class='abstract']/text()")[1].strip() if len( div.xpath(".//div[@class='abstract']/text()")) > 1 else None # 出版社: 你叫啥出版社 try: result = re.search(".*:(.*)", item['press']) item['press'] = result.group(1) # 你叫啥出版社 except Exception as e: print(e) item['time'] = div.xpath(".//div[@class='abstract']/text()")[2].strip() if len( div.xpath(".//div[@class='abstract']/text()")) > 2 else None # 出版年: 2020-3-2 try: result = re.search(".*:(.*)", item['time']) item['time'] = result.group(1) # 2020-3-2 except Exception as e: print(e) item['rating_nums'] = div.xpath(".//div[@class='rating']/span/text()")[0] # 评分 item['rating_people_nums'] = div.xpath(".//div[@class='rating']/span/text()")[1].strip('(').strip(')') # 评分人数 item['img'] = div.xpath(".//div[@class='post']/a/img/@src") content_list.append(item) return content_list # 4.保存数据 def save_…