Python爬虫——解析常用三大方式之JsonPath

JsonPath

安装

使用

我们的json数据

基本使用

案例

总结

JsonPath

主要适用于解析一些json的数据

安装

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ jsonpath

使用

obj = json.load(open( ' json文件 ' , ' r ' , encoding = ' utf-8 ' ) )

ret = jsonpath.jsonpath ( obj , ' jsonpath 语法 ' )

可以看看这个文章：JSONPath-简单入门_jsonpath简单入门-CSDN博客

注意：jsonpath和xpath不一样，jsonpath只能解析本地的文件，无法解析服务器的文件

我们的json数据

{
  "store": {
    "book": [
      {
        "category": "爽文",
        "author": "发飙的蜗牛",
        "title": "妖神记",
        "price": 8.95
      },
      {
        "category": "战争",
        "author": "超神影业",
        "title": "雄兵连",
        "price": 12.99
      },
      {
        "category": "爽文",
        "author": "唐家三少",
        "title": "斗罗大陆",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      {
        "category": "爽文",
        "author": "南派三叔",
        "title": "星辰变",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "蓝色",
      "price": 19.95
    }
  }
}

基本使用

import jsonpath
import json

obj = json.load(open('20_解析_jsonpath.json', 'r', encoding='utf-8'))
# print(obj)
# 书店所有的书
author_book_list = jsonpath.jsonpath(obj, '$.store.book[*].author')
print(author_book_list)

# 所有的作者
author_list = jsonpath.jsonpath(obj, '$..author')
print(author_list)

# store的所有元素
store_list = jsonpath.jsonpath(obj, '$.store.*')
print(store_list)

# store里所有的price
store_price_list = jsonpath.jsonpath(obj, '$.store..price')
print(store_price_list)

# 第三本书
book_3 = jsonpath.jsonpath(obj, '$.store.book[2]')
print(book_3)

# 最后一本书
book_last = jsonpath.jsonpath(obj, '$..book[(@.length-1)]')
print(book_last)

# 前两本书
book_12_way1 = jsonpath.jsonpath(obj, '$.store.book[0,1]')
book_12_way2 = jsonpath.jsonpath(obj, '$.store.book[:2]')
print(book_12_way1)
print(book_12_way2)

# 条件过滤需要在 () 的前面添加 ?
# 过滤出所有包含isbn（版本号）的书
book_isbn = jsonpath.jsonpath(obj, '$..book[?(@.isbn)]')
print(book_isbn)

# 哪本书超过了十块
book_moreThan_10 = jsonpath.jsonpath(obj, '$..book[?(@.price>10)]')
print(book_moreThan_10)

案例

获取淘淘票支持的全部城市

案例代码：

import urllib.request
import jsonpath
import json

url = 'https://dianying.taobao.com/cityAction.json?activityId&_ksTS=1709386130547_108&jsoncallback=jsonp109&action=cityAction&n_s=new&event_submit_doGetAllRegion=true'

headers = {
    # ':authority': 'dianying.taobao.com',
    # ':method': 'GET',
    # ':path': '/cityAction.json?activityId&_ksTS=1709386130547_108&jsoncallback=jsonp109&action=cityAction&n_s=new&event_submit_doGetAllRegion=true',
    # ':scheme': 'https',
    'Accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01',
    # 'Accept-Encoding': 'gzip, deflate, br, zstd',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Bx-V': '2.5.11',
    'Cookie': 'cna=ckPSHCe/+WACAW8q8CaeQGRA; t=d1bf17f4ff9b1a4f81050b57987cfd21; cookie2=107a2f127e9443a844b590a9101e6a01; v=0; _tb_token_=3a373550bdf81; xlly_s=1; isg=BNPTBeN8shk43n7NaOUT023XYlf9iGdKzTNYOIXw1fIpBPKmDVtGm_XaPnRqo79C',
    'Referer': 'https://dianying.taobao.com/',
    'Sec-Ch-Ua': '"Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"',
    'Sec-Ch-Ua-Mobile': '?0',
    'Sec-Ch-Ua-Platform': '"Windows"',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',
    'X-Requested-With': 'XMLHttpRequest',
}

request = urllib.request.Request(url=url, headers=headers)

response = urllib.request.urlopen(request)

content = response.read().decode('utf-8')

# print(content)
# 处理掉jsonp109
content = content.split('(')[1].split(')')[0]

# 保存到本地
with open('TaoPiaoPiao_city.json' , 'w', encoding='utf-8') as fp:
    fp.write(content)

obj = json.load(open('TaoPiaoPiao_city.json', 'r', encoding='utf-8'))

city_list = jsonpath.jsonpath(obj, '$..regionName')
print(city_list)