scrapy模拟登录

为什么需要模拟登录

获取登录后的页面的信息
获取cookie

回顾：

requests模块是如何模拟登录的？

1.直接携带cookie请求页面

headers={"User-Agent": "",'Cookie':''
}response=requests.get(url,headers=headers)

2.找接口发送post请求存储cookie

# 将cookie字符串转化为一个字典形式
response=requests.get(url,headers=headers,cookies=cookies)

3.session实现会话保持

import sessionsession=requests.session()data={"email":"","password":""
}
response=session.post(url,headers=headers,data=data)response2=session.get("http://www.renren.com/973016491/profile")

selenium是如何实现登录的？

找到对应的input标签，输入文字后点击登录按钮就可以实现登录

scrapy模拟登录

1.直接携带cookie

应用场景

1.cookie过期时间很长，常见于一些不规范的网站
2.能够在cookie过期之前把所有的数据拿到
3.配合其他的程序使用，比如先使用selenium把登录之后的cookie信息获取并保存在本地（数据库中），然后我们使用scrapy发送请求之前先读取本地的cookie信息

start_urls地址的请求是谁发送呢？

为什么我们要知道srart_urls对应的请求是谁发送的呢？

如果说最开始srart_urls这个地方放置的url地址是我们登陆之后才能够访问的url地址，比如说是我们12306的个人主页，我们直接请求一定是拿不到12306的个人主页的，那么我们在构造start_urls之前需要额外做一些事情。
比如说我们需要请求人人网的个人主页，如果我们直接去请求肯定是不成功的，所以在请求之前，在构造请求之前需要对start_urls请求自定义一些设置，自己尝试着通过某一些方法去构造关于url地址的请求，比如说加上cookie信息，加上之后就可以成功了。

srart_urls对应的请求是谁发送的呢？

我们定义在spider下的start_urls=[]默认都是交给start_requests处理的，所以说如果必要的话，我们可以重写start_requests方法

    def start_requests(self):cls = self.__class__if not self.start_urls and hasattr(self, 'start_url'):raise AttributeError("Crawling could not start: 'start_urls' not found ""or empty (but found 'start_url' attribute instead, ""did you miss an 's'?)")if method_is_overridden(cls, Spider, 'make_requests_from_url'):warnings.warn("Spider.make_requests_from_url method is deprecated; it ""won't be called in future Scrapy releases. Please ""override Spider.start_requests method instead (see %s.%s)." % (cls.__module__, cls.__name__),)# 遍历我们的start_urlsfor url in self.start_urls:yield self.make_requests_from_url(url)else:for url in self.start_urls:yield Request(url, dont_filter=True)# 构造一个request对象def make_requests_from_url(self, url):""" This method is deprecated. """warnings.warn("Spider.make_requests_from_url method is deprecated: ""it will be removed and not be called by the default ""Spider.start_requests method in future Scrapy releases. ""Please override Spider.start_requests method instead.")return Request(url, dont_filter=True)

我们可以自己发送关于start_urls的请求

def start_requests(self):

爬虫实战：人人网

目的

通过爬取人人网中的数据来学习scrapy直接携带cookie登录

人人网网站分析

这个是个人主页的url地址

http://www.renren.com/973016491/profile

这个是个人资料页的url地址

http://www.renren.com/973016491/profile?v=info_timeline

renren.py

# -*- coding: utf-8 -*-
import scrapy
import reclass RenrenSpider(scrapy.Spider):name = 'renren'allowed_domains = ['renren.com']start_urls = ['http://www.renren.com/973016491/profile']  # 个人主页的url地址# 使用个人主页的url地址作为start_urls的时候，直接请求是请求不成功的# 必须自定义请求def start_requests(self):"""相当于就是把父类中的start_requests方法覆盖了然后我们发送关于start_urls的请求的时候，就会通过我们自己定义的start_requests方法来进行请求"""# 将cookies参数由字符串的形式变为字典的形式cookies="anonymid=kbuw6z9aw83akl; taihe_bi_sdk_uid=d54968e5cf9f16f982336c2560990309; _r01_=1; jebe_key=738fbde4-dd4d-4bf5-ae45-c0ebaa97acf3%7Cd602d7aea178b0018804f4f7bc8b6cb7%7C1593074554808%7C1%7C1594481624320; depovince=AH; jebecookies=53f99b69-0eed-4d99-b012-7f8d38319808|||||; JSESSIONID=abcsOG7ykWPXw9yT-TSux; taihe_bi_sdk_session=a775b6d1d8cf4cc51a2287e6ed0b6555; ick_login=d17818ab-0632-46bd-8bf7-cc5d0cbb21d7; _de=E0E61E21D5EC0F38439512C7FDDF72C8; p=ff857a88f8178e0859504d14ab4026991; first_login_flag=1; ln_uact=18669319099; ln_hurl=http://head.xiaonei.com/photos/0/0/men_main.gif; t=2aecb03935d4703940a6efb9b1ef546f1; societyguester=2aecb03935d4703940a6efb9b1ef546f1; id=973016491; xnsid=d33d79f4; ver=7.0; loginfrom=null"# 使用字典推导式cookies={i.split("=")[0]:i.split("=")[1] for i in cookies.split("; ")}yield scrapy.Request(self.start_urls[0],callback=self.parse,  # 表示当前我们发送的这个请求对应的响应交给parse函数处理cookies=cookies  # 当我们要添加cookies参数的时候，这个cookies参数也是一个字典的形式)def parse(self, response):print(re.findall("用户名",response.body.decode()))"""判断一下：请求这个url地址是否请求成功怎么判断？使用正则，匹配一下里面的用户名打印：如果里面有内容  说明请求成功如果里面没有内容  说明请求失败""""""个人主页已经请求并成功了，现在又想要请求个人资料页"""yield scrapy.Request("http://www.renren.com/973016491/profile?v=info_timeline",  # 个人资料页的url地址callback=self.parse_detail)def parse_detail(self,response):"""解析个人资料页的数据"""print(re.findall("用户名",response.body.decode()))"""现在能够打印出我们的用户名的信息吗？能的因为我们的COOKIES_ENABLED默认设置为True的，即处于一个开启的状态，表明下一次发送请求时会带上上一次发送请求时对方服务器设置在本地的cookie信息"""

settings.py

# -*- coding: utf-8 -*-# Scrapy settings for login project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.htmlBOT_NAME = 'login'SPIDER_MODULES = ['login.spiders']
NEWSPIDER_MODULE = 'login.spiders'# 如何知道我们的cookie确实是在不同的解析函数中传递的呢？
COOKIES_DEBUG = True  # 我们能够看见这个cookies在我们不同的请求的过程是一个什么样子的
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'login (+http://www.yourdomain.com)'# Obey robots.txt rules
ROBOTSTXT_OBEY = False  # 设置不遵守robots协议# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16# Disable cookies (enabled by default)
#COOKIES_ENABLED = False# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'login.middlewares.LoginSpiderMiddleware': 543,
#}# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'login.middlewares.LoginDownloaderMiddleware': 543,
#}# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'login.pipelines.LoginPipeline': 300,
#}# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'