myaijarvis - Python
type: channel
link: http://myaijarvis.com/category/python
items: Python用Pyspider爬取TripAdvisor的景点信息
items: Python用selenium爬取淘宝美食信息
items: Python分析Ajax请求抓取今日头条街拍美图
items: Python用正则爬取猫眼之Top100榜的电影信息
items: Python用正则爬取豆瓣读书之新书速递的书籍信息
items: Python爬取豆瓣读书Top100的书籍信息
items: Markdown语法测试

Python用Pyspider爬取TripAdvisor的景点信息
type: item
date: 2020-03-10T13:17:00+08:00
description: 先上效果图上代码：from pyspider.libs.base_handler import * import pymongo class Handler(BaseHandler): crawl_config = { } client = pymongo.MongoClient('localhost') db = client['TripAdvise'] @every(minutes=24 * 60) def on_start(self): self.crawl('https://www.tripadvisor.cn/Attractions-g186338-Activities-c47-London_England.html',callback=self.index_page, validate_cert=False) @config(age=10 * 24 * 60 * 60) def index_page(self, response): for each in response.doc('div.listing_title > a').items(): self.crawl(each.attr.href, callback=self.detail_page, validate_cert=False) # 翻页 next = response.doc('.pagination .nav.next').attr.href self.crawl(next, callback=self.index_page, validate_cert=False) @config(priority=2) def detail_page(self, response): url = response.url name = response.doc('.h1').text() rating = response.doc(' div.ratingContainer > a > span').text() garde = response.doc('div.section.rating > span').text() address = response.doc('.contactInfo > .address').text() phone = response.doc('div.contact > div.contactType.phone.is-hidden-mobile > div').text() opening = response.doc('div.prw_rup.prw_common_atf_header_bl_responsive.headerBL > div > span').text() introduction = response.doc('.centerWell > div > div > div > div > div > span').text() return { "url": url, "name": name, "rating": rating, 'garde': garde, "address": address, "phone": phone, "opening": opening, "introduction": introduction } def on_result(self, result): if result: self.save_to_mongo(result) def save_to_mongo(self, result): if self.db['london'].insert(result): print('保存到MongoDB成功', result)参考：https://cuiqingcai.com/8317.html

Python用selenium爬取淘宝美食信息
type: item
date: 2020-03-09T11:32:00+08:00
description: 先奉上代码：# 如果使用PhantomJS则selenium版本只能2.48.0及以下 from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC import time import re from pyquery import PyQuery as py import pymongo MONGO_URL='localhost' MONGO_DB='taobao' MONGO_TABLE='product' SERVICE_ARGS=['--load-images=false','--disk-cache=true'] # 不加载图片，开启缓存 client = pymongo.MongoClient(MONGO_URL, connect=False) db=client[MONGO_DB] # browser = webdriver.Chrome() browser = webdriver.PhantomJS(service_args=SERVICE_ARGS) browser.set_window_size(1400,900) #设置窗口大小，防止元素位置改变 wait = WebDriverWait(browser, 15) def search(): # 执行搜索后跳转到登录界面--选择微博登录-（淘宝要绑定微博，微博登录要取消双重登录验证） try: browser.get("https://www.taobao.com/") print('正在搜索') input = wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#q")) ) # 输入框 submit = wait.until( EC.element_to_be_clickable((By.CSS_SELECTOR, "#J_TSearchForm > div.search-button > button")) ) # 搜索按钮 input.send_keys('美食') time.sleep(1) submit.click() # 登录 print('正在登录') wait.until( EC.element_to_be_clickable((By.CSS_SELECTOR, "#J_Quick2Static")) ).click() # 点击密码登录 time.sleep(2) wait.until( EC.element_to_be_clickable((By.CSS_SELECTOR, "#J_OtherLogin > a.weibo-login")) ).click() # 选择微博登录 wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#pl_login_logged > div > div:nth-child(2) > div > input")) ).send_keys(' ') # 输入账号 time.sleep(2) wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#pl_login_logged > div > div:nth-child(3) > div > input")) ).send_keys(' ') # 输入密码 time.sleep(2) wait.until( EC.element_to_be_clickable( (By.CSS_SELECTOR, "#pl_login_logged > div > div:nth-child(7) > div:nth-child(1) > a > span")) ).click() # 点击登录按钮 time.sleep(5) total = wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, "#mainsrp-pager > div > div > div > div.total")) ) print('登录成功') get_prodects() return total.text except TimeoutError: search() # 翻页 def ne…

Python分析Ajax请求抓取今日头条街拍美图
type: item
date: 2020-03-08T14:29:22+08:00
description: 结果如下：程序运行结果"C:\Program Files\Python\Python38\python.exe" E:/PycharmProjects/test/爬虫/今日头条街拍/Spider.py 请求文章详情页出错 None 正在下载1d47b4dceb635bff92ef296ce530cdc2.jpg 正在下载bff2d27b1d4ed77cc0a575513b9eeea4.jpg 正在下载f30c57b85c75fe655612450c993b320b.jpg 正在下载2475f2ad97bdcb6a45e6d64ce1650ca6.jpg 正在下载52fc1cd305b52366532f3ec7a9729a71.jpg 正在下载dc59cae9ed2859454c461b2944b819b1.jpg 正在下载bf1b48c4ca5cfeffec3bee95b1da6b64.jpg 正在下载7b8c8a28c037eac669988e34e90e37de.jpg 正在下载ad0fb9dbfcc0f15981fd701339e13bb8.jpg 正在下载98486366369a8c0b79f4c6ced3fb51b0.jpg E:/PycharmProjects/test/爬虫/今日头条街拍/Spider.py:125: DeprecationWarning: insert is deprecated. Use insert_one or insert_many instead. if db[MONGO_TABLE].insert(result): 存储到MongoDB成功 {'url': 'http://toutiao.com/group/6760980286252515853/', 'title': '三里屯街拍', 'images': ['http://p9.pstatp.com/origin/pgc-image/f23cd60127824cb88b11256cf9ffe951', 'http://p9.pstatp.com/origin/pgc-image/0c54ffb46fdd4553ad99e9273a85643f', 'http://p9.pstatp.com/origin/pgc-image/da0dd5a6d8314f699de340c06a9093ec', 'http://p3.pstatp.com/origin/pgc-image/7fff7dff1863466883d0920481bc9ffe', 'http://p1.pstatp.com/origin/pgc-image/9ddcb34586514ab7ad6427c73c416f51', 'http://p3.pstatp.com/origin/pgc-image/a406850542ca496aa1f68333e42aea87', 'http://p1.pstatp.com/origin/pgc-image/50bb8781363e4e3e8e9e45137ded07f6', 'http://p3.pstatp.com/origin/pgc-image/6dfc140a4d814cd49295f210aa1499bd', 'http://p1.pstatp.com/origin/pgc-image/014679ab63584e0ca7f8612b8b120651', 'http://p9.pstatp.com/origin/pgc-image/be0226872a864d2aac76c28987938cdf'], '_id': ObjectId('5e648f564d5a25526d8630e9')} 请求文章详情页出错 None 正在下载289d58776687584857ea8d2cec8e3266.jpg 正在下载4d89bd5efc7166322207e31e737cd6d2.jpg 正在下载bcfd11725613120a2f6f20145521bf70.jpg 正在下载2256eed522bab696d578be7bd021a79d.jpg 正在下载9f4b6fac29acae9d0d3964a33922cd76.jpg 正在下载6226b26aacb6111fa5bb6ab4ecb360ca.jpg 正在下载22e469ecf6277bf6631e9053b0782656.jpg 正在下载fb6e9bba0afd3880fd5acb418c08af06.jpg 正在下载9e418d8f5aa93beefe64005a941b0b8d.jpg 正在下载dc98a060a3ee9abe1583bc304de1dfd1.jpg…

Python用正则爬取猫眼之Top100榜的电影信息
type: item
date: 2020-03-06T21:59:03+08:00
description: 爬取信息的主要出处 2 肖申克的救赎 主演：蒂姆·罗宾斯,摩根·弗里曼,鲍勃·冈顿 上映时间：1994-09-10(加拿大) 9.5 代码如下：import requests import json import re import time class Spider: def __init__(self): self.url_temp = 'https://maoyan.com/board/4?offset={}' self.headers = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'} # 1.根据地址的规律构造url_list def get_url_list(self): url_list = [self.url_temp.format(i) for i in range(0, 91, 10)] return url_list # 2.发送请求，获取响应 def req_url_resp(self, url): print('start:' + url) response = requests.get(url, headers=self.headers) # print(response.text) return response.content.decode() # 3.提取数据 def get_content_list(self, resp_content): content_list = [] # 用列表存储信息 pattern = re.compile( ' .*?board-index.*?>(\d+).*?<img.*?data-src="(.*?)".*?name".*?><a.*?>(.*?).*?star">(.*?) ' '.*?releasetime">(.*?) .*?integer">(.*?).*?fraction">(.*?).*? ', re.S) contents = re.findall(pattern, resp_content) for content in contents: item = {} item["id"] = content[0] item["img_url"] = content[1] item["name"] = content[2] item["star"] = content[3].strip()[3:] item["releasetime"] = content[4][5:] item["grade"] = content[5] + content[6] + "分" # print(id,name,star,releasetime,grade,img_url) content_list.append(item) return content_list # 4.保存数据 def save_content(self, content_list): with open("猫眼Top100.txt", 'a', encoding='utf-8') as f: for content in content_list: f.write(json.dumps(content, ensure_ascii=False)) f.write('\n') print('保存成功！') def run(self): # 1.根据地址的规律构造url_list url_list = self.get_url_list() # 2.发送请求，获取响应 for url in url_list: time.sleep(1) resp_content = self.req_url_resp(url) # 3.提取数据 content_list = self.get_content_list(resp_content) # 4.保存数据 self.save_content(content_list) if __name__ == '__main__': spider = Spider() spider.run() 结果如下：{"id": "1", "img_url": "https://p0.meituan.net/movie/ce4da3e03e655b5b88ed31b5cd7896cf62472.jpg@160w_220h_1e_1c", "name": "霸王别姬", "star": "张国荣,张丰毅,巩俐", "releasetime": "1993-07-26"…

Python用正则爬取豆瓣读书之新书速递的书籍信息
type: item
date: 2020-03-05T13:14:00+08:00
description: 新书速递的书籍信息数据的主要采集出处 夏先生的故事 [德] 帕特里克·聚斯金德 夏先生的故事 [德] 帕特里克·聚斯金德 / 2020-1 / 上海译文出版社 《香水》作者聚斯金德回归纯真之作。 《纽约客》御用画手、法国国宝级插画大师桑贝经典配图。 “我”是一个普通的小男孩，上学，爬树，骑车，幻想着朦胧的初恋，害怕严厉的钢琴教师，常常觉得自己会飞。“我”的邻居夏先生是个“怪人”，沉默地拄着拐杖，背着空空的行囊，没有目的地奔走不休。我们之间只有为数不多的几次接触，但这个敏感易怒、行色匆匆、似乎总在忍受... 代码如下import requests import re url = "https://book.douban.com/" headers = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'} content = requests.get(url, headers=headers).text # 使用re.S参数以后，正则表达式会将这个字符串作为一个整体，将“\n”当做一个普通的字符加入到这个字符串中，在整体中进行匹配。 pattern = re.compile( '<li.*?cover.*?href="(.*?)\stitle="(.*?)">.*?src="(.*?)".*?more-meta.*?author">(.*?).*?year">(.*?).*?publisher">(.*?).*? ', re.S) ##这里应该要在URL和title之间匹配空格，用\s results = re.findall(pattern, content) for result in results: url, name, img_url, author, date, publisher = result author = re.sub('\s', '', author) # 把前后的空白符去掉 author=author.strip()也可以 date = re.sub('\s', '', date) publisher = re.sub('\s', '', publisher) print(name, author, date, publisher, url, img_url) 结果数据如下人类向何处去 [英]奥拉夫·斯特普尔顿 2020-1 北京航空航天大学出版社 https://book.douban.com/subject/34934004/?icn=index-latestbook-subject" https://img9.doubanio.com/view/subject/s/public/s33549854.jpg 戏剧游戏：儿童戏剧游戏 [英]凯瑟琳·扎切斯特 / KatherineZachest 2020-1 后浪丨上海文化出版社 https://book.douban.com/subject/34935067/?icn=index-latestbook-subject" https://img3.doubanio.com/view/subject/s/public/s33550710.jpg 算计 [日]米泽穗信 2020-1 人民文学出版社 https://book.douban.com/subject/34866078/?icn=index-latestbook-subject" https://img1.doubanio.com/view/subject/s/public/s33542087.jpg 从催眠的世界中不断醒来 姜涛 2020-1 华东师范大学出版社 https://book.douban.com/subject/34951625/?icn=index-latestbook-subject" https://img9.doubanio.com/view/subject/s/public/s33562486.jpg 十三封自杀告别信 [德]科斯汀•吉尔 2020-1-1 新星出版社 https://book.douban.com/subject/34908017/?icn=index-latestbook-subject" https://img3.doubanio.com/view/subject/s/public/s33531850.jpg 论爱与死亡 [德]帕·聚斯金德 2020-1 上海译文出版社 https://book.douban.com/subject/348…

Python爬取豆瓣读书Top100的书籍信息
type: item
date: 2020-03-02T23:01:00+08:00
description: 爬取豆瓣读书Top100的书籍信息（实则只有79本书）import requests import json from lxml import etree import re import time class Spider: def __init__(self): self.url_temp = 'https://www.douban.com/doulist/118410371/?start={}' self.headers = { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36'} # 1.根据地址的规律构造url_list def get_url_list(self): url_list = [self.url_temp.format(i) for i in range(0, 76, 25)] return url_list # 2.发送请求，获取响应 def req_url_resp(self, url): print('start:' + url) response = requests.get(url, headers=self.headers) return response.content.decode() # 3.提取数据 def get_content_list(self, resp_content): content = etree.HTML(resp_content) content_list = [] # 用列表存储信息 div_list = content.xpath("//div[@class='article']//div[@class='doulist-item']") for div in div_list: item = {} item['title'] = div.xpath(".//div[@class='title']/a/text()")[0].strip() item['author'] = div.xpath(".//div[@class='abstract']/text()")[0].strip() # 作者：江一帆 try: result = re.search(".*:(.*)", item['author']) item['author'] = result.group(1) # 江一帆 except Exception as e: print(e) item['press'] = div.xpath(".//div[@class='abstract']/text()")[1].strip() if len( div.xpath(".//div[@class='abstract']/text()")) > 1 else None # 出版社: 你叫啥出版社 try: result = re.search(".*:(.*)", item['press']) item['press'] = result.group(1) # 你叫啥出版社 except Exception as e: print(e) item['time'] = div.xpath(".//div[@class='abstract']/text()")[2].strip() if len( div.xpath(".//div[@class='abstract']/text()")) > 2 else None # 出版年: 2020-3-2 try: result = re.search(".*:(.*)", item['time']) item['time'] = result.group(1) # 2020-3-2 except Exception as e: print(e) item['rating_nums'] = div.xpath(".//div[@class='rating']/span/text()")[0] # 评分 item['rating_people_nums'] = div.xpath(".//div[@class='rating']/span/text()")[1].strip('(').strip(')') # 评分人数 item['img'] = div.xpath(".//div[@class='post']/a/img/@src") content_list.append(item) return content_list # 4.保存数据 def save_…

Markdown语法测试
type: item
date: 2020-02-16T12:02:00+08:00
description: 这只是一个测试一级标题二级标题三级标题四级标题加粗文字斜体文字有序列表有序列表列表项目列表项目列表项目列表项目列表项目无序列表列表项目列表项目列表项目列表项目列表项目引用文字private String userName; private int userPassword; public User(){ }/* **123 **dgdgsggdfgdsg */ //sgdgsdgsdg public class User { private String userName; private int userPassword; public User(){ } public User(String userName, int userPassword) { this.userName = userName; this.userPassword = userPassword; } public String getUserName() { return userName; } public void setUserName(String userName) { this.userName = userName; } public void setUserPassword(int userPassworduserPassworduserPassworduserPassworduserPassworduserPassword) { this.userPassword = userPassworduserPassworduserPassworduserPassworduserPassworduserPassword; } public int getUserPassword() { return userPassword; } }#include<iostream> using namespace std; int main(){ int x; float y; cout<<"Please input an int number:"<<endl; cin>>x; cout<<"The int number is x= "<<x<<endl; cout<<"Please input a float number:"<<endl; cin>>y; cout<<"The float number is y= "<<y<<endl; return 0; }#include<iostream> using namespace std; int main(){ int x; float y; cout<<"Please input an int number:"<<endl; cin>>x; cout<<"The int number is x= "<<x<<endl; cout<<"Please input a float number:"<<endl; cin>>y; cout<<"The float number is y= "<<y<<endl; return 0; }count = 0 while (count < 9): print 'The count is:', count count = count + 1 print "Good bye!"分割线摘要分割线