FastDFS分布式文件系统

QQ图片20200212005403.png
FastDFS是一个开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务,如相册网站、视频网站等等。
FastDFS为互联网量身定制,充分考虑了冗余备份、负载均衡、线性扩容等机制,并注重高可用、高性能等指标,使用FastDFS很容易搭建一套高性能的文件服务器集群提供文件上传、下载等服务。

提示:一定要清楚你所操作的目录及文件的位置,以及linux常用命令。为了方便大家回顾Linux命令,特地总结了一篇linux常用命令以供参考

1.下载所需安装包

libfastcommon-master.zip

FastDFS_v5.05.tar.gz

fastdfs-nginx-module_v1.16.tar.gz

nginx-1.8.0.tar.gz

链接:https://pan.baidu.com/s/1v35bfk8y5pp77-omddTU_A
提取码:wm5r

2.安装libfastcommon包

① 安装libfastcommon依赖(系统如果有带有,就不需要安装):

yum install -y gcc
yum install -y gcc-c++

②解压libfastcommon-master.zip

unzip libfastcommon-master.zip

③编译

cd libfastcommon-master
./make.sh

④安装

./make.sh install

3.安装FastDFS

①解压FastDFS安装包(这里选择解压到/home/admin/目录下)

tar -zxvf FastDFS_v5.05.tar.gz

②编译

cd FastDFS
./make.sh

③安装

./make.sh install

4.配置Tracker服务

首先新建tracker的目录

mkdir -p /opt/qinzh/fastdfs_tracker

①进入FastDFS配置文件所在目录:

cd /etc/fdfs/

②修改配置文件名

mv client.conf.sample client.conf
mv storage.conf.sample storage.conf
mv tracker.conf.sample tracker.conf

③打开tracker.conf,修改如下配置:

#the base path to store data and log files
base_path=/opt/qinzh/fastdfs_tracker

设置tracker的数据目录(data)和日志目录(logs)

# the tracker server port
port=22122

注意:配置之后去服务器开启22122端口

④启动tracker

 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

⑤查看服务是否启动成功

netstat -unltp|grep fdfs

⑥Tracker添加到开机自启动

vi /etc/rc.local

将如下命令添加到exit 0行的前面

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

启动Tracker时可能遇到的报错:

/usr/bin/fdfs_trackerd: error while loading shared libraries: libfastcommon.so: cannot open shared object file: No such file or directory

解决方案:建立libfastcommon.so软链接

ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so

5.配置Storage服务

首先新建storage的目录

mkdir -p /opt/qinzh/fastdfs_storage_log
mkdir -p /opt/qinzh/fastdfs_storage_data

①配置storage.conf文件

# the base path to store data and log files
base_path=/opt/qinzh/fastdfs_storage_log

注:日志目录

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
store_path0=/opt/qinzh/fastdfs_storage_data

注:储存目录

# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=127.0.0.1:22122

注:Tracker服务器IP和端口(IP换为自己服务器的IP)

②启动Storage服务

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

③查看服务启动是否成功

netstat -unltp|grep fdfs

④启动成功后,通过fdfs_monitor查看集群情况,即storage是否已经注册到了racker服务器中

/usr/bin/fdfs_monitor /etc/fdfs/storage.conf

⑤tracker添加到开机自启动

vi /etc/rc.local

将如下命令添加到文件中

/usr/bin/fdfs_monitor /etc/fdfs/storage.conf restart

⑥配置client.conf文件

# the base path to store log files
base_path=/opt/qinzh/fastdfs_tracker

# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=127.0.0.1:22122

注:客户端client的IP和端口(IP换为自己服务器的IP)

6.安装和配置Nginx和fastdfs-nginx-module模块

①安装所需其他依赖

yum install -y pcre pcre-devel
yum install -y zlib zlib-devel
yum install -y openssl openssl-devel

②创建软链接(因为 FastDFS 主程序设置的 lib 目录是/usr/local/lib,所以需要创建软链接)

ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so

③解压nginx和fastdfs-nginx-module

/usr/local/nginx-1.8.0.tar.gz

/usr/local/fastdfs-nginx-module_v1.16.tar.gz

解压nginx

 tar -zxvf /usr/local/nginx-1.8.0.tar.gz

解压fastdfs-nginx-module

 tar -zxvf /usr/local/fastdfs-nginx-module_v1.16.tar.gz

④编辑vi

/home/admin/fastdfs-nginx-module/src/config
修改CORE_INCS行,如下如,将路径中的local全部去掉,即:

CORE_INCS=”$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/”

⑤nginx加入fastdfs-nginx-module模块

cd nginx-1.8.0
 ./configure –prefix=/usr/local/nginx –add-module=/usr/local/fastdfs-nginx-module/src/

看到如下结果表示模块加入成功

⑥安装nginx
a)/usr/local/nginx-1.8.0/src下编译nginx

make

出现如上图所示表示编译成功

b)并在此目录下安装nginx

 make install

安装完后,进入/usr/local/目录,看见nginx文件目录表示nginx安装成功

输入

/usr/local/nginx/sbin/nginx -V

显示如下信息,表示fastdfs-nginx-module安装成功
这里写图片描述

⑦编辑nginx.conf

 vi /usr/local/nginx/conf/nginx.conf

加入如下配置:

server {
        listen       8881;
        server_name  localhost;
...
        location /group1/M00 {
            root /opt/qinzh/fastdfs_storage_data/data;
            ngx_fastdfs_module;
        }

⑧复制配置文件到/etc/fdfs下,编辑mod_fastdfs.conf文件
a)复制FastDFS下的配置文件http.conf和mime.types到/etc/fdfs下

cp /home/admin/FastDFS/conf/http.conf /etc/fdfs/
cp /home/admin/FastDFS/conf/mime.types /etc/fdfs/

b)复制fastdfs-nginx-module模块的配置文件mod_fastdfs.conf到/etc/fdfs下

cp /usr/local/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/

c)编辑mod_fastdfs.conf

1)保存日志目录

# the base path to store log files
base_path=/opt/qinzh/fastdfs_storage_info

2)tracker服务器地址

# FastDFS tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
# valid only when load_fdfs_parameters_from_tracker is true
tracker_server=127.0.0.1:22122

IP换为自己服务器的IP

3)storage服务器端口号(配置后去服务器开启23000端口)

# the port of the local storage server
# the default value is 23000
storage_server_port=23000

4)当前服务器group名

# the group name of the local storage server
group_name=group1

5)文件url是否有group名

# if the url / uri including the group name
# set to false when uri like /M00/00/00/xxx
# set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx
# default value is false
url_have_group_name = true

6)存储路径个数,store_path个数匹配

# path(disk or mount point) count, default value is 1
# must same as storage.conf
store_path_count=1

7)存储路径

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
# must same as storage.conf
store_path0=/opt/qinzh/fastdfs_storage_data

8)从文件扩展名找到文件类型

http.need_find_content_type=true

9)设置组的个数

# set the group count
# set to none zero to support multi-group
# set to 0  for single group only
# groups settings section as [group1], [group2], ..., [groupN]
# default value is 0
# since v1.14
group_count = 1

10)放开注释掉的[group1],并修改路径及配置如下

# group settings for group #1
# since v1.14
# when support multi-group, uncomment following section
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/opt/qinzh/fastdfs_storage_data
# store_path1=/home/yuqing/fastdfs1

⑨启动nginx
建立软链接

ln -s /opt/qinzh/fastdfs_storage_data/data /opt/qinzh/fastdfs_storage_data/data/M00
cd /opt/qinzh/fastdfs_storage_data/data/M00
ls

00 0D 1A 27 34 41 4E 5B 68 75 82 8F 9C A9 B6 C3 D0 DD EA F7
01 0E 1B 28 35 42 4F 5C 69 76 83 90 9D AA B7 C4 D1 DE EB F8
02 0F 1C 29 36 43 50 5D 6A 77 84 91 9E AB B8 C5 D2 DF EC F9
03 10 1D 2A 37 44 51 5E 6B 78 85 92 9F AC B9 C6 D3 E0 ED FA
04 11 1E 2B 38 45 52 5F 6C 79 86 93 A0 AD BA C7 D4 E1 EE FB
05 12 1F 2C 39 46 53 60 6D 7A 87 94 A1 AE BB C8 D5 E2 EF FC
06 13 20 2D 3A 47 54 61 6E 7B 88 95 A2 AF BC C9 D6 E3 F0 FD
07 14 21 2E 3B 48 55 62 6F 7C 89 96 A3 B0 BD CA D7 E4 F1 FE
08 15 22 2F 3C 49 56 63 70 7D 8A 97 A4 B1 BE CB D8 E5 F2 FF
09 16 23 30 3D 4A 57 64 71 7E 8B 98 A5 B2 BF CC D9 E6 F3 M00
0A 17 24 31 3E 4B 58 65 72 7F 8C 99 A6 B3 C0 CD DA E7 F4
0B 18 25 32 3F 4C 59 66 73 80 8D 9A A7 B4 C1 CE DB E8 F5
0C 19 26 33 40 4D 5A 67 74 81 8E 9B A8 B5 C2 CF DC E9 F6

重启nginx

cd /usr/local/nginx/sbin
nginx -s reload

重新编译nginx

nginx -c /usr/local/nginx/conf/nginx.conf

去服务器放开8881端口

访问nginx,查看nginx启动是否成功
这里写图片描述
7.测试

①使用命令方式上传一张本地图片到FastDFS。(my.gif是之前就上传到服务器的一张图片)

/usr/bin/fdfs_test /etc/fdfs/client.conf upload /home/admin/my.gif
This is FastDFS client test program v5.05

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/ 
for more detail.

[2020-02-09 14:22:36] DEBUG - base_path=/opt/qinzh/fastdfs_tracker, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

tracker_query_storage_store_list_without_group: 
    server 1. group_name=, ip_addr=ip, port=23000

group_name=group1, ip_addr=ip, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/rBEg0V4_pSyADAFAAAAv-vFvNjs830.gif
source ip address: 172.17.32.209
file timestamp=2020-02-09 14:22:36
file size=12282
file crc32=4050597435
example file url: http://ip:8881/group1/M00/00/00/rBEg0V4_pSyADAFAAAAv-vFvNjs830.gif
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/rBEg0V4_pSyADAFAAAAv-vFvNjs830_big.gif
source ip address: 172.17.32.209
file timestamp=2020-02-09 14:22:36
file size=12282
file crc32=4050597435
example file url: http://ip:8881/group1/M00/00/00/rBEg0V4_pSyADAFAAAAv-vFvNjs830_big.gif

浏览器输入即可访问到

http://ip:8881/group1/M00/00/00/rBEg0V4_pSyADAFAAAAv-vFvNjs830_big.gif

fastdfs

我来吐槽

*

*