一、常见的反爬介绍
基于身份识别的反爬：1.User-agent 2.Referer 3.Captcha 验证码 4.必备参数
基于爬虫行为的反爬：1.单位时间内请求数量超过一定阈值 2.相邻两次请求之间间隔小于一定阈值3.蜜罐陷阱
通过对数据加密进行反爬：1.对文字加密 2.对请求参数加密 3.对响应数据加密
二、文字加解密
（一）常见文字加解密方式
1.通过对数据图片化进行反爬
2.对文字进行编码 unicode, HTML 字符实体
（二）二进制与十六进制
以下为用代码表示方法：
num = 20
# 16进制表示
print(hex(num))
# 二进制表示
print(bin(num))
num2 = 0b10001
print(num2)
num3 = 0xA2
print(num3)（三）ASCII&UTF-8&Unicode编码
1.字符集：（1）每一个字符在计算机里都被保存为二进制的编码（2）成千上万组这种关系就组成了字符集
2.常见的字符集：ASCII，UTF-8，Unicode， GBK
3.ASCII：(1) English (2) 7-bits
# 转为ASCII编码形式
num = ord("A")
print(num)
# 查看二进制形式
print(bin(num))
# 传进去ASCII码转为字符
ch1 = chr(65)
print(ch1)
ch2 = chr(0b1000010)
print(ch2)4.Unicode:(1) 世界各地语言 （2）固定长度， 2 bytes
5.UTF-8:（1）基于unicode编码的可变长度的格式 （2）1 bytes - 4 bytes
6.GBK:中文
（四）Python实现EncodeDecode编码转换
s1 = "中"
s2 = s1.encode("utf-8")
print(s2)
s3 = s1.encode("unicode-escape")
print(s3)
s4 = b'\xe4\xb8\xad'
# 将二进制的格式转成文字
s5 = s4.decode("utf-8")
print(s5)
s6 = b'\\u4e2d'
s7 = s6.decode("unicode-escape")
print(s7)（五）破解Unicode与HTML字符实体加密
import requests
url = "http://127.0.0.1:5000/api/C12L06"
headers = {
"User-Agent": "xxxxxxxxxxxxxx"
}
res = requests.get(url, headers=headers)
print(res.text)
# 进行解码
print(res.content.decode("unicode-escape"))
import requests
import html
url = "http://127.0.0.1:5000/api/C12L06b"
headers = {
"User-Agent": "xxxxxxxxxxxxxx"
}
res = requests.get(url, headers=headers)
print(res.text)
# 进行解码
print(html.unescape(res.text))
（六）破解CSS偏移文字加密
import requests
from lxml import etree
url = "http://127.0.0.1:5000/C12L07"
headers = {
"User-Agent":"xxx"
}
res = requests.get(url, headers=headers)
mapping = {
"0":"0",
"-15px":"1",
"-33px":"2",
"-50px":"3",
"-66px":"4",
"-83px":"5",
"-100px":"6",
"-118px":"7",
"-135px":"8",
"-152px":"9"
}
tree = etree.HTML(res.text)
items = tree.xpath('/html/body/div/div/div[1]/div/@style')
number = ""
for item in items:
value = item.split(" ")[1]
value = mapping[value]
number += value
print(number)
items2 = tree.xpath('/html/body/div/div/div[2]/div/@style')
number2 = ""
for item in items2:
value = item.split(" ")[1]
value = mapping[value]
number2 += value
print(number2)（七）实践：自如ZiRoom获取租房信息
上述图片右侧的红框是链接：点开链接为图片形式：
import requests
import re
from lxml import etree
import base64
import json
url = 'https://www.ziroom.com/z/'
headers = {
"User-Agent":"xxxx"
}
res = requests.get(url, headers=headers)
tree = etree.HTML(res.text)
style = tree.xpath('/html/body/section/div[3]/div[2]/div[1]/div[2]/div[2]/span[2]/@style')[0]
exp = re.compile('background-image: url\((.*?)\)', re.S)
img_url = "https:"+exp.findall(style)[0]
img_res = requests.get(img_url, headers=headers)
with open('./C12LO8/img.png', 'wb') as f:
f.write(img_res.content)
url = "https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=xxxxxx&client_secret=xxxxx"
payload = ""
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
response = requests.request("POST", url, headers=headers, data=payload)
access_token = response.json()["access_token")
request_url = "https://aip.baidubce.com/rest/2.0/ocr/v1/accurate_basic"
f = open('./C12LO8/img.png', 'rb')
img = base64.b64encode(f.read())
params = {"image":img}
request_url = request_url + "?access_token=" + access_token
headers = {'content-type': 'application/x-www-form-urlencoded'}
response = requests.post(request_url, data=params, headers=headers)
if response:
result = response.json()["words_result"][0]["words"]
print(result)
mappings = {
"-0px": result[0],
"-21.4px": result[1]
"-42.8px": result[2],
"-64.2px": result[3],
"-85.6px": result[4],
"-107px": result[5],
"-128.4px": result[6],
"-149.8px": result[7],
"-171.2px": result[8],
"-192.6px": result[9]
}
items = tree.xpath('div[@class="Z_list-box"]/div')
for item in items:
try:
title = item.xpath('./div[@class="info-box"]//a/text()')[0]
nums = item.xpath('.//span[@class="num"]/@style')
price = ""
for num in nums:
price += mappings[num.split("background-position:")[1].strip()]
print(title, price)
except:
print("bannar")
