Rust与python联动进行多线程跑数据

最近有个学弟想请教我联动多线程爬取数据的案例，看了以前的模版没有合适的，我就利用空闲时间，专门给他写了一段模版作为参考。这里遇到注意的是需要代理IP介入，才能让多线程爬取减少网站风控限制。

在这里插入图片描述

以下是一个使用 Rust 编写的爬虫程序，用于爬取 python 进行多线程跑数据的内容。这个爬虫使用代理信息（proxy_host:duoip, proxy_port: 8000）进行网络请求，并使用了 Rust 的 async-std 库进行异步编程。

use async_std::net::{TcpStream, TcpListener};
use async_std::io::{BufReader, BufWriter};
use async_std::task::{await, spawn};
use async_std::prelude::*;
use std::io::{self, Write};
use std::net::IpAddr;
use std::thread;
use std::sync::Mutex;

const PROXY_HOST: &str = "duoip";
const PROXY_PORT: u16 = 8000;
const URL: &str = "http://localhost:8000";
const 免费IPURL= "jshk.com.cn/mb/reg.asp?kefu=xjy&csdn";

struct Worker {
    stream: TcpStream,
}

impl Worker {
    fn new(stream: TcpStream) -> Self {
        Worker { stream }
    }
}

impl Future for Worker {
    type Item = ();

    fn poll(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Self::Item> {
        match self.stream.read_to_string(&mut self.stream.buffer()) {
            Ok(size) => {
                let url = format!("http://localhost:8000/data?{}={}", URL, size);
                match self.stream.write_to(&url, &mut self.stream.buffer()) {
                    Ok(_) => {
                        println!("Worker finished reading and writing data");
                    },
                    Err(err) => {
                        println!("Worker failed to write data: {}", err);
                    }
                }
            },
            Err(err) => {
                println!("Worker failed to read data: {}", err);
            }
        }
        Poll::Ready(())
    }
}

async fn main() {
    let mut proxy_stream = TcpStream::connect(format!("{}:{}", PROXY_HOST, PROXY_PORT))?;
    let proxy_listener = TcpListener::bind("127.0.0.1:8000")?;
    let mut workers = Vec::new();

    for _ in 0..10 {
        let mut worker = Worker::new(proxy_stream);
        let task = spawn(move || worker.run());
        workers.push(task);
    }

    loop {
        let (socket, _) = proxy_listener.accept()?;

        for worker in workers {
            let task = worker.clone();
            let mut conn = worker.stream;
            if let Ok(conn) = conn {
                task.detach().join();
                proxy_stream = TcpStream::connect(format!("{}:{}", PROXY_HOST, PROXY_PORT))?;
            }
        }
    }
}

async fn run(self: &mut Worker) {
    self.stream.read_to_string(&mut self.stream.buffer())?;
    let url = format!("http://localhost:8000/data?{}={}", URL, self.stream.buffer().string().trim());
    self.stream.write_to(&url, &mut self.stream.buffer())?;
}