Fastdfs V6.12.1集群部署(arm/x86均可用)

文章目录

一、 Fastdfs 介绍
二、部署信息
三、步骤
- - tracker/storage 机器的 compose 内容
  - storage 机器的 compose
  - tracker 与 storage 启动
  - 目录层级与配置文件
  - 测试
  - 测试集群扩容与缩减
  - 注意事项

一、 Fastdfs 介绍

FastDFS 是一款高性能的分布式文件系统，特别适合用于存储和管理大量的文件

二、部署信息

使用 docker 部署 Fastdfs 集群

角色	服务器 IP	服务器系统	服务器配置	docker /docker-compose版本
tracker/storage	10.10.11.35	麒麟 V10	127G + 96C	18.09.0/v2.25.0
tracker/storage	10.10.11.56	麒麟 V10	254G + 64C	20.10.17/v2.5.0

默认 tracker 端口为 22122， storage 端口为 23000，文件访问端口为 8080
此文档使用镜像在 dockerhub 上有，x86 与 arm 双架构均存在，可以直接 pull

三、步骤

tracker/storage 机器的 compose 内容

由于此容器内部启动使用 sed，所以根据内部命令来启动，编辑环境变量以及域名解析，多个 tracker 使用 \n 换行符实现

version: '3.8'
services:
  tracker:
    container_name: tracker
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: tracker
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/tracker:/var/fdfs
      - /etc/localtime:/etc/localtime
  storage:
    container_name: storage
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: storage
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/storage:/var/fdfs/
      - /etc/localtime:/etc/localtime

storage 机器的 compose

version: '3.8'
services:
  storage:
    container_name: storage
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: storage
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/storage:/var/fdfs/
      - /etc/localtime:/etc/localtime

tracker 与 storage 启动

# 在142上查看 storage 日志，可以看到会输出其他节点连接成功的信息
docker  logs -f storage
[2024-12-16 16:50:46] INFO - file: storage_func.c, line: 338, tracker_client_ip: 10.10.11.35, my_server_id_str: 10.10.11.35, g_server_id_in_filename: 587926026
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.56:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 2077, tracker server: #0. 10.10.11.35:22122, my_report_result: -1
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.35:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:50:47] INFO - file: tracker_client_thread.c, line: 2077, tracker server: #0. 10.10.11.35:22122, my_report_result: -1
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 1441, tracker server 10.10.11.56:22122, set tracker leader: 10.10.11.56:22122
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 790, my status: 6 (ONLINE) from tracker #0  != my status: 0 (INIT) from leader tracker #1, set report_my_status to true
[2024-12-16 16:51:16] INFO - file: storage_sync_func.c, line: 126, successfully connect to storage server 10.10.11.56:23000
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 2484, tracker server: 10.10.11.35:22122, try to set storage status from 6 (ONLINE) to 7 (ACTIVE)
[2024-12-16 16:51:17] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.35:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:51:46] INFO - file: storage_sync_func.c, line: 126, successfully connect to storage server 10.10.11.56:23000

# 在139查看tracker日志
[2024-12-16 16:50:45] INFO - FastDFS v6.12.1, base_path=/var/fdfs, max_connections=1024, connect_timeout=5, network_timeout=60, thread_stack_size=256 KB, max_pkg_size=256 KB, min_buff_size=8 KB, max_buff_size=256 KB, tcp_quick_ack=1, log_level=INFO, run_by_group=, run_by_user=, error-log: {sync_log_buff_interval=1, rotate_everyday=1, rotate_time=00:00, rotate_on_size=0, compress_old=0, compress_days_before=1, keep_days=0, delete_old_time=01:30}, port=22122, bind_addr=, address_family=auto, accept_threads=1, work_threads=4, store_lookup=2, store_group=, store_server=0, store_path=0, reserved_storage_space=1.00%, download_server=0, allow_ip_count=-1, check_active_interval=120s, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=1024 KB, trunk_alloc_alignment_size=256, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_free_space_merge=1, delete_unused_trunk_files=0, trunk_compress_binlog_min_interval=86400, trunk_compress_binlog_interval=86400, trunk_compress_binlog_time_base=03:00, trunk_binlog_max_backups=7, use_storage_id=0, id_type_in_filename=ip, trust_storage_server_id=1, storage_id/ip_count=0 / 0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2024-12-16 16:50:46] ERROR - file: tracker_service.c, line: 686, client ip: 10.10.11.56, leader 10.10.11.56:22122 not exist
[2024-12-16 16:50:47] INFO - file: tracker_mem.c, line: 4393, sys files loaded from tracker server 10.10.11.56:22122
[2024-12-16 16:50:47] INFO - file: tracker_relationship.c, line: 356, the tracker leader is 10.10.11.56:22122
[2024-12-16 16:51:15] WARNING - file: tracker_mem.c, line: 4963, storage server: 10.10.11.56:23000, dest status: 2, my status: 1, should change my status!
[2024-12-16 16:51:15] WARNING - file: tracker_mem.c, line: 4963, storage server: 10.10.11.56:23000, dest status: 5, my status: 2, should change my status!
[2024-12-16 16:51:16] INFO - file: tracker_service.c, line: 1229, client ip: 10.10.11.35, do NOT change storage status, old status: 7 (ACTIVE), new status: 7 (ACTIVE)

目录层级与配置文件

# 目录层级如下
data
  ├── storage
  └── tracker
      ├── data
      └── logs

查看集群信息

# 查看当前集群信息，任意节点的 storage 容器查看即可	
docker exec -it storage fdfs_monitor /etc/fdfs/client.conf  
# server_count = 2 表示 tracker 有两个
server_count=2, server_index=0

tracker server is 10.10.11.35:22122

group count: 1

Group 1:
group name = group111
disk total space = 400,088 MB
disk free space = 15,674 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
。。。。。。
Storage 1:  # 表示第一个storage的信息
                id = 10.10.11.35
                ip_addr = 10.10.11.35  ACTIVE
。。。。。。
Storage 2:  # 第二个storage的信息
                id = 10.10.11.56
                ip_addr = 10.10.11.56  ACTIVE

测试

# 通过访问tracker接口上传，但有可能会报错文件过大
curl -F "file=@./test.txt" http://10.10.11.140:22122/upload

# 通过容器内部命令上传
docker exec -it storage bash 
cd                      
echo 11111 > test.txt 
fdfs_test /etc/fdfs/client.conf upload test.txt

This is FastDFS client test program v6.12.1

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.fastken.com/ 
for more detail.

tracker_query_storage_store_list_without_group: 
        server 1. group_name=, ip_addr=10.10.11.35, port=23000
        server 2. group_name=, ip_addr=10.10.11.56, port=23000

group_name=group111, ip_addr=10.10.11.56, port=23000
storage_upload_by_filename
group_name=group111, remote_filename=M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497.txt
source ip address: 10.10.11.56
file timestamp=2024-12-16 16:53:02
file size=6
file crc32=2836601426
example file url: http://10.10.11.56/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497.txt
storage_upload_slave_by_filename
group_name=group111, remote_filename=M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
source ip address: 10.10.11.56
file timestamp=2024-12-16 16:53:02
file size=6
file crc32=2836601426
example file url: http://10.10.11.56/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
root@localhost:~# curl http://10.10.11.56:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
11111
root@localhost:~# curl http://10.10.11.35:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
11111

访问地址 http://10.10.11.35:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt

fastdfs 集群直接的 storage 会自动同步数据，更换 ip 再次访问，也可以成功访问

测试集群扩容与缩减

扩容：目前有两种方法，扩容容量需要增加硬盘，挂载分区，扩容节点需要增加新的 group，修改新节点的配置后就可以加入集群，原有集群有数据后是无法直接增加节点的，除了上面增加 group 的方法就只能清除数据重新部署节点，所以推荐在部署集群之前最好先确认架构，防止后续因为架构调整带来的不必要的麻烦
缩减：由于集群会自动同步数据，要缩减直接直接关闭storage、修改nginx代理即可

nginx 负载配置（示例配置）

upstream storage {
   server 10.10.11.56:23000;
   server 10.10.11.35:23000;
}

upstream storage_file {
   server 10.10.11.56:8080;
   server 10.10.11.35:8080;
}
upstream tracker {
   server 10.10.11.56:22122;
   server 10.10.11.35:22122;
}

server {
    listen 23001 ;
    server_name _;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://storage;
    }
}

server {
    listen 8889;
    server_name _;
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://storage_file;
    }
}

server {
    listen 22123;
    server_name _;
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://tracker;
    }
}

注意事项

fastdfs 默认保留的磁盘空间为 10%，当至少有一个节点的磁盘存储小于或等于 10%时，文件无法写入，可以更改保留空间，修改 tracker 容器中 /etc/fdfs/tracker.conf，进入容器使用 echo “reserved_storage_space = 1%” >> /etc/fdfs/tracker.conf 追加，所有的 tracker 都需要更改，然后单独重启 tracker，清空原有数据，即 data 下的所有内容，重启 storage 即可
如果遇到一直不同步的清空，则删除等待节点，重新启动，或删除全部数据，重新创建容器