CentOS 6.x系统下安装配置及测试分布式文件服务器FastDFS+Nginx+cache+PHP集群详细教程

FastDFS简介

  FastDFS是一个开源的轻量级分布式文件系统,她对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务,如相册网站、视频网站等等。

  FastDFS服务端有两个角色:跟踪器(tracker)和存储节点(storage)。跟踪器主要做调度工作,在访问上起负载均衡的作用。

  存储节点存储文件,完成文件管理的所有功能:存储、同步和提供存取接口,FastDFS同时对文件的metadata进行管理。所谓文件的meta data就是文件的相关属性,以键值对(key valuepair)方式表示,如:width=1024,其中的key为width,value为1024。文件metadata是文件属性列表,可以包含多个键值对。

  FastDFS系统结构如下图所示:

1

  跟踪器和存储节点都可以由一台多台服务器构成。跟踪器和存储节点中的服务器均可以随时增加或下线而不会影响线上服务。其中跟踪器中的所有服务器都是对等的,可以根据服务器的压力情况随时增加或减少。

  为了支持大容量,存储节点(服务器)采用了分卷(或分组)的组织方式。存储系统由一个或多个卷组成,卷与卷之间的文件是相互独立的,所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务器组成,一个卷下的存储服务器中的文件都是相同的,卷中的多台存储服务器起到了冗余备份和负载均衡的作用。

  在卷中增加服务器时,同步已有的文件由系统自动完成,同步完成后,系统自动将新增服务器切换到线上提供服务。

  当存储空间不足或即将耗尽时,可以动态添加卷。只需要增加一台或多台服务器,并将它们配置为一个新的卷,这样就扩大了存储系统的容量。

  FastDFS中的文件标识分为两个部分:卷名和文件名,二者缺一不可。

2

FastDFS file upload

上传文件交互过程:

1. client询问tracker上传到的storage,不需要附加参数;

2. tracker返回一台可用的storage;

3. client直接和storage通讯完成文件上传。

3

FastDFS file download

下载文件交互过程:

1. client询问tracker下载文件的storage,参数为文件标识(卷名和文件名);

2. tracker返回一台可用的storage;

3. client直接和storage通讯完成文件下载。

  需要说明的是,client为使用FastDFS服务的调用方,client也应该是一台服务器,它对tracker和storage的调用均为服务器间的调用。

系统环境

测试环境:最小化安装的CentOS 6.5 x86_64版操作系统
实验拓扑图:

4

系统操作环境的设置

#软件安装包存储
/usr/local/src
#libevent安装目录
/usr/local/libevent
#fastdfs安装目录
/usr/local/fastdfs
#nginx安装目录
/usr/local/nginx

环境准备

所有服务器上操作

为了速度,配置使用sohu的源

cd /etc/yum.repo.d/
wget http://mirrors.sohu.com/help/CentOS-Base-sohu.repo
安装EPEL源
rpm -ihv http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
因为系统是最小化安装,所以需要安装开发工具和编译nginx所需的开发包
yum -y install gcc gcc-c++ perl
或执行以下命令
yum groupinstall "Development Tools"
yum install libevent-devel.x86_64 pcre-devel.x86_64  zlib-devel.x86_64

安装FastDFS

首先安装libevent

官方推荐稳定的1.4.14b版本的,2.x的可能有不稳定性

获取下载
wget https://github.com/downloads/libevent/libevent/libevent-1.4.14b-stable.tar.gz
 解压并编译安装
tar zvxf libevent-1.4.14b-stable.tar.gz
./configure –prefix=/usr
make clean;
make
make install
     
测试libevent是否安装成功
ls -al /usr/lib | grep libevent
出来一大列东西,就说明安装成功了。

如果libevent的安装目录为/usr/local/lib下,则还需要建立 libevent- 2.0 .so. 5 到 /usr/lib 的软连接,这样其他程序运行时才能找到libevent库:
ln -s /usr/local/lib/libevent- 2.0 .so. 5  /usr/lib/libevent- 2.0 .so. 5  
否则会出现
./tracker/fdfs_trackerd: error while loading shared libraries: 
libevent-2.0.so.5: cannot open shared object file: No such file or directory)

下载FastDFS安装包

注意:tracker、storage、php端需要安装FastDFS(即所有机器)
FastDFS官方论坛:http://www.csource.org
下载1:http://sourceforge.net/projects/fastdfs/files/
下载2:https://code.google.com/p/fastdfs/downloads/list (目前这上面基本已停止更新)

所需要下载的压缩包有:
注:以下括号中的地址为下载的列表地址,在下载列表中可以选择当前最新的开发版本或最新的稳定版本
FastDFS源代码:FastDFS_v5.01.tar.gz(http://sourceforge.net/projects/fastdfs/files/FastDFS%20Server%20Source%20Code/)
nginx模块源代码:fastdfs-nginx-module_v1.15.tar.gz(http://sourceforge.net/projects/fastdfs/files/FastDFS%20Nginx%20Module%20Source%20Code/)
nginx服务器源代码:nginx-1.4.7.tar.gz(http://nginx.org/en/download.html)
nginx cache purge插件源代码:ngx_cache_purge-2.1.tar.gz(http://wiki.nginx.org/CachePurgeChs)
nginx依赖的pcre库源代码:pcre-8.34.tar.gz(http://ftp.exim.llorien.org/pcre/)
nginx依赖的zlib库源代码:zlib-1.2.8.tar.gz(http://pkgs.fedoraproject.org/repo/pkgs/mingw-zlib/)

安装tracker

安装

首先在172.16.1.202上安装FastDFS tracker,使用FastDFS_v5.01.tar.gz源代码包。

下载源代码包:wget http://hivelocity.dl.sourceforge.net/project/fastdfs/FastDFS%20Server%20Source%20Code/FastDFS%20Server%20with%20PHP%20Extension%20Source%20Code%20V5.01/FastDFS_v5.01.tar.gz

首先将代码包复制到系统的/usr/local/src内(这一步可选),然后使用tar命令解压

[root@tracker opt]# cp FastDFS_v5.01.tar.gz /usr/local/src/
[root@tracker opt]# cd /usr/local/src/
[root@tracker src]# tar zxf FastDFS_v5.01.tar.gz
[root@tracker src]# cd FastDFS
[root@tracker FastDFS]# ll
总用量 128
drwxrwxr-x. 3 500 500  4096 2月   6 18:07 client
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 common
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 conf
-rw-rw-r--. 1 500 500 35067 7月  28 2008 COPYING-3_0.txt
-rw-rw-r--. 1 500 500 29691 2月   2 13:17 HISTORY
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 init.d
-rw-rw-r--. 1 500 500  7639 1月   5 14:08 INSTALL
-rwxrwxr-x. 1 500 500  5531 12月  7 15:19 make.sh
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 php_client
-rw-rw-r--. 1 500 500  2380 7月  28 2008 README
-rwxrwxr-x. 1 500 500  1768 4月  12 2010 restart.sh
-rwxrwxr-x. 1 500 500  1680 4月  10 2010 stop.sh
drwxrwxr-x. 4 500 500  4096 2月   6 18:07 storage
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 test
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 tracker

运行make.sh,确认make成功。期间如果有错误,可能会是缺少依赖的软件包,需安装后再次make。

[root@tracker FastDFS]# ./make.sh
注意:我们这里使用的是最新版本的fastdfs,由于最新版的fastdfs取消了自带的http功能,所以这里我们就不对make.sh做修改。
运行make.sh install,确认install成功。

[root@tracker FastDFS]# ./make.sh install

安装完成后,所有可执行文件在/usr/local/bin下,以fdfs开头:
 
[root@tracker FastDFS]# ll /usr/local/bin/fdfs*
-rwxr-xr-x. 1 root root  522918 3月  25 14:57 /usr/local/bin/fdfs_appender_test
-rwxr-xr-x. 1 root root  522871 3月  25 14:57 /usr/local/bin/fdfs_appender_test1
-rwxr-xr-x. 1 root root  514023 3月  25 14:57 /usr/local/bin/fdfs_append_file
-rwxr-xr-x. 1 root root  513433 3月  25 14:57 /usr/local/bin/fdfs_crc32
-rwxr-xr-x. 1 root root  513967 3月  25 14:57 /usr/local/bin/fdfs_delete_file
-rwxr-xr-x. 1 root root  514377 3月  25 14:57 /usr/local/bin/fdfs_download_file
-rwxr-xr-x. 1 root root  514133 3月  25 14:57 /usr/local/bin/fdfs_file_info
-rwxr-xr-x. 1 root root  525064 3月  25 14:57 /usr/local/bin/fdfs_monitor
-rwxr-xr-x. 1 root root 1179682 3月  25 14:57 /usr/local/bin/fdfs_storaged
-rwxr-xr-x. 1 root root  529845 3月  25 14:57 /usr/local/bin/fdfs_test
-rwxr-xr-x. 1 root root  527774 3月  25 14:57 /usr/local/bin/fdfs_test1
-rwxr-xr-x. 1 root root  655809 3月  25 14:57 /usr/local/bin/fdfs_trackerd
-rwxr-xr-x. 1 root root  514213 3月  25 14:57 /usr/local/bin/fdfs_upload_appender
-rwxr-xr-x. 1 root root  514999 3月  25 14:57 /usr/local/bin/fdfs_upload_file

所有配置文件在/etc/fdfs下:

[root@tracker FastDFS]# ll /etc/fdfs/
总用量 60
-rw-r--r--. 1 root root  1461 3月  13 15:15 client.conf
-rw-r--r--. 1 root root   858 3月  13 15:15 http.conf
-rw-r--r--. 1 root root 31172 3月  13 15:15 mime.types
-rw-r--r--. 1 root root  3837 3月  25 10:03 mod_fastdfs.conf
-rw-r--r--. 1 root root  7515 3月  24 10:36 storage.conf
-rw-r--r--. 1 root root  6989 3月  13 15:15 tracker.conf

至此tracker安装完成。

配置

编辑配置文件目录下的tracker.conf,设置相关信息并保存。
创建tracker目录保存运行日志

[root@tracker FastDFS]# mkdir -m 777 -p /fdfs/tracker
[root@tracker FastDFS]# vim /etc/fdfs/tracker.conf

一般只需改动以下几个参数即可:
disabled=false            #启用配置文件
port=22122                #设置tracker的端口号
base_path=/fdfs/tracker   #设置tracker的数据文件和日志目录(需预先创建)
http.server_port=8080     #设置http端口号

如需要进行性能调优,可以参照《FastDFS 配置文件详解(修订版1)》:http://bbs.chinaunix.net/thread-1941456-1-1.html

运行

运行tracker之前,先要把防火墙中对应的端口打开(本例中为22122)。

[root@tracker FastDFS]# iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 22122 -j ACCEPT
[root@tracker FastDFS]# /etc/init.d/iptables save
iptables:将防火墙规则保存到 /etc/sysconfig/iptables:[确定]

启动tracker,确认启动是否成功。(查看是否对应端口22122是否开始监听)

[root@tracker FastDFS]# /usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart
telnet 看是否启动成功
[root@tracker FastDFS]# telent 172.16.1.202 22122
也可以netstat看是否启动成功
[root@tracker FastDFS]# netstat -unltp | grep fdfs
tcp    0    0.0.0.0:22122       0.0.0.0:*           LISTEN      1766/fdfs_trackerd

也可查看tracker的日志是否启动成功或是否有错误。

[root@tracker FastDFS]# cat /fdfs/tracker/logs/trackerd.log

tracker server 运行时目录结构
 ${base_path}                                  
    |__data                                      
    |     |__storage_groups.dat:存储分组信息      
    |     |__storage_servers.dat:存储服务器列表   
    |__logs                                        
          |__trackerd.log:tracker server日志文件  
		  
关于目录结构的更详细内容,请参考:http://www.cnblogs.com/chinacloud/archive/2011/01/19/1939066.html

设置开机自动启动。

[root@tracker FastDFS]# vim /etc/rc.d/rc.local

将运行命令行添加进文件:/usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

安装storage

安装

首先在172.16.1.203上安装FastDFS storage,使用FastDFS_v5.01.tar.gz源代码包。

下载源代码包:wget http://hivelocity.dl.sourceforge.net/project/fastdfs/FastDFS%20Server%20Source%20Code/FastDFS%20Server%20with%20PHP%20Extension%20Source%20Code%20V5.01/FastDFS_v5.01.tar.gz
首先将代码包复制到系统的/usr/local/src内(这一步可选),然后使用tar命令解压

[root@storage1 opt]# cp FastDFS_v5.01.tar.gz /usr/local/src/
[root@storage1 opt]# cd /usr/local/src/
[root@storage1 src]# tar zxf FastDFS_v5.01.tar.gz
[root@storage1 src]# cd FastDFS
[root@storage1 FastDFS]# ll
总用量 128
drwxrwxr-x. 3 500 500  4096 2月   6 18:07 client
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 common
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 conf
-rw-rw-r--. 1 500 500 35067 7月  28 2008 COPYING-3_0.txt
-rw-rw-r--. 1 500 500 29691 2月   2 13:17 HISTORY
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 init.d
-rw-rw-r--. 1 500 500  7639 1月   5 14:08 INSTALL
-rwxrwxr-x. 1 500 500  5531 12月  7 15:19 make.sh
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 php_client
-rw-rw-r--. 1 500 500  2380 7月  28 2008 README
-rwxrwxr-x. 1 500 500  1768 4月  12 2010 restart.sh
-rwxrwxr-x. 1 500 500  1680 4月  10 2010 stop.sh
drwxrwxr-x. 4 500 500  4096 2月   6 18:07 storage
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 test
drwxrwxr-x. 2 500 500  4096 2月   6 18:07 tracker

运行make.sh,确认make成功。期间如果有错误,可能会是缺少依赖的软件包,需安装后再次make。

[root@storage1 FastDFS]# ./make.sh

运行make.sh install,确认install成功。

[root@storage1 FastDFS]# ./make.sh install

安装完成后,所有可执行文件在/usr/local/bin下,以fdfs开头:
 
[root@storage1 FastDFS]# ll /usr/local/bin/fdfs*
-rwxr-xr-x. 1 root root  522918 3月  25 14:57 /usr/local/bin/fdfs_appender_test
-rwxr-xr-x. 1 root root  522871 3月  25 14:57 /usr/local/bin/fdfs_appender_test1
-rwxr-xr-x. 1 root root  514023 3月  25 14:57 /usr/local/bin/fdfs_append_file
-rwxr-xr-x. 1 root root  513433 3月  25 14:57 /usr/local/bin/fdfs_crc32
-rwxr-xr-x. 1 root root  513967 3月  25 14:57 /usr/local/bin/fdfs_delete_file
-rwxr-xr-x. 1 root root  514377 3月  25 14:57 /usr/local/bin/fdfs_download_file
-rwxr-xr-x. 1 root root  514133 3月  25 14:57 /usr/local/bin/fdfs_file_info
-rwxr-xr-x. 1 root root  525064 3月  25 14:57 /usr/local/bin/fdfs_monitor
-rwxr-xr-x. 1 root root 1179682 3月  25 14:57 /usr/local/bin/fdfs_storaged
-rwxr-xr-x. 1 root root  529845 3月  25 14:57 /usr/local/bin/fdfs_test
-rwxr-xr-x. 1 root root  527774 3月  25 14:57 /usr/local/bin/fdfs_test1
-rwxr-xr-x. 1 root root  655809 3月  25 14:57 /usr/local/bin/fdfs_trackerd
-rwxr-xr-x. 1 root root  514213 3月  25 14:57 /usr/local/bin/fdfs_upload_appender
-rwxr-xr-x. 1 root root  514999 3月  25 14:57 /usr/local/bin/fdfs_upload_file

所有配置文件在/etc/fdfs下:

[root@storage1 FastDFS]# ll /etc/fdfs/
总用量 60
-rw-r--r--. 1 root root  1461 3月  13 15:15 client.conf
-rw-r--r--. 1 root root   858 3月  13 15:15 http.conf
-rw-r--r--. 1 root root 31172 3月  13 15:15 mime.types
-rw-r--r--. 1 root root  3837 3月  25 10:03 mod_fastdfs.conf
-rw-r--r--. 1 root root  7515 3月  24 10:36 storage.conf
-rw-r--r--. 1 root root  6989 3月  13 15:15 tracker.conf

至此storage安装完成。

配置

#创建Storage目录保存运行日志及其data数据

[root@storage1 FastDFS]# mkdir -m 777 -p /fdfs/storage

编辑配置文件目录下的storage.conf,设置相关信息并保存。

[root@storage1 FastDFS]# vim /etc/fdfs/storage.conf

一般只需改动以下几个参数即可:
disabled=false                    #启用配置文件
group_name=group1                 #组名,根据实际情况修改
port=23000                        #设置storage的端口号
base_path=/fdfs/storage           #设置storage的日志目录(需预先创建)
store_path_count=1                #存储路径个数,需要和store_path个数匹配
store_path0=/fdfs/storage         #存储路径
tracker_server=172.16.1.202:22122 #tracker服务器的IP地址和端口号
http.server_port=8080     #设置http端口号

如需要进行性能调优,可以参照《FastDFS 配置文件详解(修订版1)》:http://bbs.chinaunix.net/thread-1941456-1-1.html

运行

运行storage之前,先要把防火墙中对应的端口打开(本例中为23000)。

[root@storage1 FastDFS]# iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 23000 -j ACCEPT
[root@storage1 FastDFS]# /etc/init.d/iptables save
iptables:将防火墙规则保存到 /etc/sysconfig/iptables:[确定]

启动storage,会根据配置文件的设置自动创建多级存储目录,确认启动是否成功。(查看是否对应端口23000是否开始监听)

[root@storage1 FastDFS]# /usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf restart
data path: /fdfs/storage/data, mkdir sub dir...
mkdir data path: 00 ...
mkdir data path: 01 ...
mkdir data path: 02 ...
mkdir data path: 03 ...
...
data path: /fdfs/storage/data, mkdir sub dir done.
[root@storage1 FastDFS]# netstat -unltp | grep fdfs
tcp    0    0.0.0.0:23000       0.0.0.0:*           LISTEN      1766/fdfs_storaged

也可查看storage的日志是否启动成功或是否有错误。

[root@storage1 FastDFS]# cat /fdfs/storage/logs/storaged.log

确认启动成功后,可以运行fdfs_monitor查看storage服务器是否已经登记到tracker服务器。

[root@storage1 FastDFS]# /usr/local/bin/fdfs_monitor /etc/fdfs/storage.conf
[2014-03-26 01:51:20] DEBUG - base_path=/fdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 172.16.1.202:22122

group count: 1

Group 1:
group name = group1
disk total space = 27789 MB
disk free space = 23920 MB
trunk free space = 0 MB
storage server count = 1
active server count = 1
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

	Storage 1:
		id = 172.16.1.203
		ip_addr = 172.16.1.203  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 01:48:19
		up time = 2014-03-26 01:48:19
		total storage = 27789 MB
		free storage = 23920 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 
		if_trunk_server= 0
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2014-03-26 01:51:03
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00

看到“172.16.1.203  ACTIVE”即可确认storage运行正常。

storage server 运行时目录结构                                        
 ${base_path}                                                           
 |__data                                                               
 |__.data_init_flag:当前storage server初始化信息                      
 |  |__storage_stat.dat:当前storage server统计信息                    
 |  |__sync:存放数据同步相关文件                                      
 |  |  |__binlog.index:当前的binlog文件索引号                         
 |  |  |__binlog.###:存放更新操作记录(日志)                         
 |  |  |__${ip_addr}_${port}.mark:存放同步的完成情况                  
 |                                                                     
 |  | |__一级目录:256个存放数据文件的目录,如:00, 1F                 
 |  |__二级目录:256个存放数据文件的目录                               
 |__logs                                                              
    |__storaged.log:storage server日志文件     

关于目录结构的更详细内容,请参考:http://www.cnblogs.com/chinacloud/archive/2011/01/19/1939066.html

设置开机自动启动。

[root@storage1 FastDFS]# vim /etc/rc.d/rc.local

将运行命令行添加进文件:/usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf restart

之后依次在172.16.1.204~208上全部安装上storage并确认运行正常。注意配置文件中group名参数需要根据实际情况调整,本例中group是这样分配的:
group1:172.16.1.203,172.16.1.204
group2:172.16.1.205,172.16.1.206
group3:172.16.1.207,172.16.1.208
另外每个group中所有storage的端口号必须一致。

在storage上安装nginx

曾经有过疑问,既然文件是直接被保持在storage服务器上,那直接用nginx就可以访问了,为什么还要添加这个模块?
答:Nginx服务器是安装在Storeage服务器上的,也就是说,一般情况下它只提供对本Storeage服务器上的资源提供服务。只有在本机并没有需要获取的资源的情况下,才会通过文件ID反解出源storage,直接请求源storage。
当同组内其他Storeage新上传的图片本机并没有及时同步的时候,Nginx会根据url寻找上传的Storeage源服务器,进行redirect或者proxy两种方式之一的转发,从而解决文件同步延迟问题。redirect和proxy,大家大概能猜到,不累述了。
在这里Nginx因为只是访问Storeage的本地资源,所以并没有充分的利用tracker进行读取信息的负载均衡,在下一节内容中我们也将配置tracker的负载均衡。

tracker server上不需要任何web server
在每台storage server上部署web server,直接对外提供HTTP服务
目前已提供apache和nginx扩展模块
FastDFS默认自带的http服务器性能不好,所以一般建议用外置的apache或者nginx来解决http下载,以应付大并发的情况
注意nginx扩展模块只支持GET和HEAD模式获取文件,需要开发那边配合修改程序

在storage上安装的nginx主要为了提供http的访问服务,同时解决group中storage服务器的同步延迟问题。这个模块nginx默认的编译模块中没有,需手动下载安装。

安装

首先在172.16.1.203上安装nginx,使用nginx-1.4.7.tar.gz源代码包以及FastDFS的nginx插件fastdfs-nginx-module_v1.15.tar.gz。
下载源代码包(如果下载失败,多重新偿试下载几次就可以了):

wget http://nginx.org/download/nginx-1.6.0.tar.gz
wget http://softlayer-dal.dl.sourceforge.net/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz
wget http://labs.frickle.com/files/ngx_cache_purge-2.1.tar.gz
wget http://ftp.exim.llorien.org/pcre/pcre-8.35.tar.gz
wget http://pkgs.fedoraproject.org/repo/pkgs/mingw-zlib/zlib-1.2.8.tar.gz/44d667c142d7cda120332623eab69f40/zlib-1.2.8.tar.gz
首先将代码包和插件复制到系统的/usr/local/src内(可选),然后使用tar命令解压

[root@storage1 opt]# cp nginx-1.4.7.tar.gz /usr/local/src
[root@storage1 opt]# cp fastdfs-nginx-module_v1.15.tar.gz /usr/local/src
[root@storage1 opt]# cp pcre-8.34.tar.gz /usr/local/src
[root@storage1 opt]# cp zlib-1.2.8.tar.gz /usr/local/src
[root@storage1 opt]# cd /usr/local/src/
[root@storage1 src]# tar zxf nginx-1.4.7.tar.gz
[root@storage1 src]# tar zxf fastdfs-nginx-module_v1.15.tar.gz
只需解压开就可,这里我们再解释一下,好些朋友,在加载这个模块的时候,会报一些很莫名其妙的错,这个本人在编译的时候也是亲身体会,百思不得其解,后来经过点拨才发现,版本不匹配,这里fastdfs官网给有详细说明,请看模块解压目录下的HISTORY、INSTALL问件,这里面很详细的安装和版本匹配说明(吃一堑一定要长一智)
[root@storage1 src]# tar zxf pcre-8.34.tar.gz
[root@storage1 src]# tar zxf zlib-1.2.8.tar.gz
[root@storage1 src]# cd nginx-1.4.7
[root@storage1 nginx-1.4.7]# ll
总用量 592
drwxr-xr-x. 6 1001 1001   4096 3月  26 02:34 auto
-rw-r--r--. 1 1001 1001 225213 3月  18 21:17 CHANGES
-rw-r--r--. 1 1001 1001 343040 3月  18 21:17 CHANGES.ru
drwxr-xr-x. 2 1001 1001   4096 3月  26 02:34 conf
-rwxr-xr-x. 1 1001 1001   2369 3月  18 21:17 configure
drwxr-xr-x. 3 1001 1001   4096 3月  26 02:34 contrib
drwxr-xr-x. 2 1001 1001   4096 3月  26 02:34 html
-rw-r--r--. 1 1001 1001   1397 3月  18 21:17 LICENSE
drwxr-xr-x. 2 1001 1001   4096 3月  26 02:34 man
-rw-r--r--. 1 1001 1001     49 3月  18 21:17 README
drwxr-xr-x. 8 1001 1001   4096 3月  26 02:34 src

运行./configure进行安装前的设置,主要设置安装路径、FastDFS插件模块目录、pcre库目录、zlib库目录。
如果提示错误,可能缺少依赖的软件包,需先安装依赖包,再次运行./configure

[root@storage1 nginx-1.4.7]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/fastdfs-nginx-module/src --with-pcre=/usr/local/src/pcre-8.34/ --with-zlib=/usr/local/src/zlib-1.2.8

提示:假设nginx已经在系统被安装过,执行命令:/usr/local/nginx/sbin/nginx -V
后得到原来的编译参数,如:
--prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-openssl=/root/openssl-1.0.0d
cd nginx-1.0.10
然后再重新编译安装一次就可以了。
 
运行make进行编译,确保编译成功。

[root@storage1 nginx-1.4.7]# make

运行make install进行安装。

[root@storage1 nginx-1.4.7]# make install

将FastDFS的nginx插件模块的配置文件copy到FastDFS配置文件目录。

[root@storage1 nginx-1.4.7]# cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/

安装完成后,nginx所有文件在/usr/local/nginx下:
 
[root@storage1 nginx-1.4.7]# ll /usr/local/nginx/
总用量 16
drwxr-xr-x. 2 root root 4096 3月  26 03:11 conf
drwxr-xr-x. 2 root root 4096 3月  26 03:11 html
drwxr-xr-x. 2 root root 4096 3月  26 03:11 logs
drwxr-xr-x. 2 root root 4096 3月  26 03:11 sbin

至此nginx以及FastDFS的nginx插件模块安装完成。
<h3>配置</h3>
编辑/usr/local/nginx/conf配置文件目录下的nginx.conf,设置添加storage信息并保存。

[root@storage1 nginx-1.4.7]# vim /usr/local/nginx/conf/nginx.conf

将server段中的listen端口号改为8080:
listen       8080;
在server段中添加:

访问路径不带group名(storage只有一个group的情况),如/M00/00/00/xxx:
location /M00 {
	root /fdfs/storage/data;
	ngx_fastdfs_module;
}
访问路径带group名(storage对应有多个group的情况),如/group1/M00/00/00/xxx:
location ~ /group[0-9]/M00 {
	root /fdfs/storage/data;
	ngx_fastdfs_module;
}

编辑/etc/fdfs配置文件目录下的mod_fastdfs.conf,设置storage信息并保存。

[root@storage1 nginx-1.4.7]# vim /etc/fdfs/mod_fastdfs.conf

一般只需改动以下几个参数即可:
base_path=/fdfs/storage           #保存日志目录
tracker_server=172.16.1.202:22122 #tracker服务器的IP地址以及端口号
storage_server_port=23000         #storage服务器的端口号
group_name=group1                 #当前服务器的group名
url_have_group_name = true        #文件url中是否有group名,访问路径带group时为true否则为false,多个group的情况下,必须为true,为false时,group_count必须为0
store_path_count=1                #存储路径个数,需要和store_path个数匹配
store_path0=/fdfs/storage         #存储路径
http.need_find_content_type=true  #从文件扩展名查找文件类型(nginx时为true)
group_count = 3                   #设置组的个数
在末尾增加3个组的具体信息:
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/fdfs/storage

 [group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/fdfs/storage

[group3]
group_name=group3
storage_server_port=23000
store_path_count=1
store_path0=/fdfs/storage

建立M00至存储目录的符号连接。

[root@storage1 nginx-1.4.7]# ln -s /fdfs/storage/data /fdfs/storage/data/M00
[root@storage1 nginx-1.4.7]# ll /fdfs/storage/data/M00
lrwxrwxrwx. 1 root root 19 3月  26 03:44 /fdfs/storage/data/M00 -> /fdfs/storage/data/

至此,nginx以及FastDFS插件模块设置完成。
<h3>运行</h3>
运行nginx之前,先要把防火墙中对应的端口打开(本例中为8080)。

[root@storage1 nginx-1.4.7]# iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
[root@storage1 nginx-1.4.7]# /etc/init.d/iptables save
iptables:将防火墙规则保存到 /etc/sysconfig/iptables:[确定]

启动nginx,确认启动是否成功。(查看是否对应端口8080是否开始监听)

[root@storage1 nginx-1.4.7]# /usr/local/nginx/sbin/nginx
ngx_http_fastdfs_set pid=40638
[root@storage1 nginx-1.4.7]# netstat -unltp | grep nginx
tcp    0    0.0.0.0:8080            0.0.0.0:*              LISTEN      40639/nginx

也可查看nginx的日志是否启动成功或是否有错误。

[root@storage1 nginx-1.4.7]# cat /usr/local/nginx/logs/error.log
ngx_http_fastdfs_process_init pid=40640
[2014-03-26 03:47:17] INFO - local_host_ip_count: 2,  127.0.0.1  172.16.1.203
[2014-03-26 03:47:17] INFO - fastdfs apache / nginx module v1.15, response_mode=proxy, base_path=/tmp, url_have_group_name=1, group_count=3, connect_timeout=2, network_timeout=30, tracker_server_count=1, if_alias_prefix=, local_host_ip_count=2, need_find_content_type=1, default_content_type=application/octet-stream, anti_steal_token=0, token_ttl=0s, anti_steal_secret_key length=0, token_check_fail content_type=, token_check_fail buff length=0, load_fdfs_parameters_from_tracker=1, storage_sync_file_max_delay=86400s, use_storage_id=0, storage server id count=0, flv_support=1, flv_extension=flv
[2014-03-26 03:47:17] INFO - group 1. group_name=group1, storage_server_port=23000, path_count=1, store_path0=/fdfs/storage
[2014-03-26 03:47:17] INFO - group 2. group_name=group2, storage_server_port=23000, path_count=1, store_path0=/fdfs/storage
[2014-03-26 03:47:17] INFO - group 3. group_name=group3, storage_server_port=23000, path_count=1, store_path0=/fdfs/storage

在error.log中没有错误,既启动成功。可以打开浏览器,直接访问http://172.16.1.203:8080,查看是否弹出nginx欢迎页面。

5

之后依次在172.16.1.204~208上全部安装上nginx并确认运行正常。

设置开机自动启动。

[root@storage1 nginx-1.4.7]# vim /etc/rc.d/rc.local

将运行命令行添加进文件:/usr/local/nginx/sbin/nginx

在tracker上安装nginx

在tracker上安装的nginx主要为了提供http访问的反向代理、负载均衡以及缓存服务。

安装

首先将代码包和插件复制到系统的/usr/local/src内(可选),然后使用tar命令解压

[root@tracker opt]# cp nginx-1.4.7.tar.gz /usr/local/src
[root@tracker opt]# cp ngx_cache_purge-2.1.tar.gz /usr/local/src
[root@tracker opt]# cp pcre-8.34.tar.gz /usr/local/src
[root@tracker opt]# cp zlib-1.2.8.tar.gz /usr/local/src
[root@tracker opt]# cd /usr/local/src/
[root@tracker src]# tar zxf nginx-1.4.7.tar.gz
[root@tracker src]# tar zxf ngx_cache_purge-2.1.tar.gz
[root@tracker src]# tar zxf pcre-8.34.tar.gz
[root@tracker src]# tar zxf zlib-1.2.8.tar.gz 
[root@tracker src]# cd nginx-1.4.7
[root@tracker nginx-1.4.7]# ll
总用量 592
drwxr-xr-x. 6 1001 1001   4096 3月  26 02:34 auto
-rw-r--r--. 1 1001 1001 225213 3月  18 21:17 CHANGES
-rw-r--r--. 1 1001 1001 343040 3月  18 21:17 CHANGES.ru
drwxr-xr-x. 2 1001 1001   4096 3月  26 02:34 conf
-rwxr-xr-x. 1 1001 1001   2369 3月  18 21:17 configure
drwxr-xr-x. 3 1001 1001   4096 3月  26 02:34 contrib
drwxr-xr-x. 2 1001 1001   4096 3月  26 02:34 html
-rw-r--r--. 1 1001 1001   1397 3月  18 21:17 LICENSE
drwxr-xr-x. 2 1001 1001   4096 3月  26 02:34 man
-rw-r--r--. 1 1001 1001     49 3月  18 21:17 README
drwxr-xr-x. 8 1001 1001   4096 3月  26 02:34 src

运行./configure进行安装前的设置,主要设置安装路径、nginx cache purge插件模块目录、pcre库目录、zlib库目录。
如果提示错误,可能缺少依赖的软件包,需先安装依赖包,再次运行./configure

[root@storage1 nginx-1.4.7]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/ngx_cache_purge-2.1 --with-pcre=/usr/local/src/pcre-8.34/ --with-zlib=/usr/local/src/zlib-1.2.8

运行make进行编译,确保编译成功。

[root@storage1 nginx-1.4.7]# make

运行make install进行安装。

[root@storage1 nginx-1.4.7]# make install

至此nginx以及nginx cache purge插件模块安装完成。
<h3>配置</h3>
编辑/usr/local/nginx/conf配置文件目录下的nginx.conf,设置负载均衡以及缓存。

[root@storage1 nginx-1.4.7]# vim /usr/local/nginx/conf/nginx.conf

worker_processes  4;                  #根据CPU核心数而定
events {
    worker_connections  65535;        #最大链接数
    use epoll;                        #新版本的Linux可使用epoll加快处理性能
}
http {
#设置缓存参数
    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
    client_max_body_size 300m;
    sendfile        on;
    tcp_nopush      on;
    proxy_redirect off;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffer_size 16k;
    proxy_buffers 4 64k;
    proxy_busy_buffers_size 128k;
    proxy_temp_file_write_size 128k;
#设置缓存存储路径、存储方式、分配内存大小、磁盘最大空间、缓存期限
    proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:500m max_size=10g inactive=30d;
    proxy_temp_path /var/cache/nginx/proxy_cache/tmp;
#设置group1的服务器
    upstream fdfs_group1 {
        server 172.16.1.203:8080 weight=1 max_fails=2 fail_timeout=30s;
        server 172.16.1.204:8080 weight=1 max_fails=2 fail_timeout=30s;
    }
#设置group2的服务器
    upstream fdfs_group2 {
        server 172.16.1.205:8080 weight=1 max_fails=2 fail_timeout=30s;
        server 172.16.1.206:8080 weight=1 max_fails=2 fail_timeout=30s;
    }
#设置group3的服务器
    upstream fdfs_group3 {
        server 172.16.1.207:8080 weight=1 max_fails=2 fail_timeout=30s;
        server 172.16.1.208:8080 weight=1 max_fails=2 fail_timeout=30s;
    }

   server {
#设置服务器端口
        listen       8080;
#设置group1的负载均衡参数
        location /group1/M00 {
            proxy_next_upstream http_502 http_504 error timeout invalid_header;
            proxy_cache http-cache;
            proxy_cache_valid  200 304 12h;
            proxy_cache_key $uri$is_args$args;
            proxy_pass http://fdfs_group1;
            expires 30d;
        }
#设置group2的负载均衡参数
        location /group2/M00 {
            proxy_next_upstream http_502 http_504 error timeout invalid_header;
            proxy_cache http-cache;
            proxy_cache_valid  200 304 12h;
            proxy_cache_key $uri$is_args$args;
            proxy_pass http://fdfs_group2;
            expires 30d;
        }
#设置group3的负载均衡参数
        location /group3/M00 {
            proxy_next_upstream http_502 http_504 error timeout invalid_header;
            proxy_cache http-cache;
            proxy_cache_valid  200 304 12h;
            proxy_cache_key $uri$is_args$args;
            proxy_pass http://fdfs_group3;
            expires 30d;
        }
#设置清除缓存的访问权限
        location ~ /purge(/.*) {
            allow 127.0.0.1;
            allow 172.16.1.0/24;
            deny all;
            proxy_cache_purge http-cache  $1$is_args$args;
        }   
	}
}

至此,nginx以及nginx cache purge插件模块设置完成。

运行

运行nginx之前,先要把防火墙中对应的端口打开(本例中为8080)。

[root@tracker nginx-1.4.7]# iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
[root@tracker nginx-1.4.7]# /etc/init.d/iptables save
iptables:将防火墙规则保存到 /etc/sysconfig/iptables:[确定]

启动nginx,确认启动是否成功。(查看是否对应端口8080是否开始监听)

[root@tracker nginx-1.4.7]# /usr/local/nginx/sbin/nginx
???ngx_http_fastdfs_set pid=40638
[root@tracker nginx-1.4.7]# netstat -unltp | grep nginx
tcp    0    0.0.0.0:8080            0.0.0.0:*              LISTEN      40639/nginx

尝试上传一个文件到FastDFS,然后访问试试。先配置client.conf文件。

[root@tracker nginx-1.4.7]# vim /etc/fdfs/client.conf

修改以下参数:
base_path=/fdfs/tracker                    #日志存放路径
tracker_server=172.16.1.202:22122          #tracker服务器IP地址和端口号
http.tracker_server_port=8080              #tracker服务器的http端口号

使用/usr/local/bin/fdfs_upload_file上传一个文件,程序会自动返回文件的URL。

[root@tracker nginx-1.4.7]# /usr/local/bin/fdfs_upload_file /etc/fdfs/client.conf /mnt/monk.jpg
group3/M00/00/00/rBABz1MzKG6Ad_hBAACKLsUDM60560.jpg

然后使用浏览器访问:

6

可以看到文件被正确读取出来了。

查看nginx的access.log日志,可以看到访问返回200成功。

[root@tracker nginx-1.4.7]# tail -n 10 -f /usr/local/nginx/logs/access.log
172.16.1.201 - - [26/Mar/2014:13:15:00 +0800] "GET /group3/M00/00/00/rBABz1MzKG6Ad_hBAACKLsUDM60560.jpg HTTP/1.1" 200 35374 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"

查看nginx的cache目录,可以看到已经生成了缓存文件。

[root@tracker nginx-1.4.7]# ll /var/cache/nginx/proxy_cache/ -R
/var/cache/nginx/proxy_cache/:
总用量 8
drwx------. 3 nobody nobody 4096 3月  26 12:57 6
drwxr-xr-x. 2 nobody root   4096 3月  26 13:14 tmp

/var/cache/nginx/proxy_cache/6:
总用量 4
drwx------. 2 nobody nobody 4096 3月  26 13:08 11

/var/cache/nginx/proxy_cache/6/11:
总用量 36
-rw-------. 1 nobody nobody 35686 3月  26 13:08 b1f6fb1f7266f796765b6d6965021116

如果要手动清除缓存,可以在文件URL之前加上purge:

7

设置开机自动启动。

[root@tracker nginx-1.4.7]# vim /etc/rc.d/rc.local

将运行命令行添加进文件:/usr/local/nginx/sbin/nginx

至此,tracker服务器上的http反向代理+负载均衡+缓存已经安装完成。

测试及使用

上传文件

8

FastDFS提供了多种方式上传文件:
使用fdfs_upload_file上传(之前已经有演示)
使用C语言客户端接口上传
使用PHP客户端接口上传
使用Java客户端接口上传
使用Python客户端接口上传
使用.NET客户端接口上传

根据具体情况选择使用。上传均支持断点续传。

下载文件

9

使用fdfs_download_file通过tracker服务器下载
使用nginx通过http方式直接从storage下载(支持断点续传,作者推荐)

删除文件

监视服务器资源

使用fdfs_monitor查看tracker和所有group的运行情况。

[root@tracker tmp]# /usr/local/bin/fdfs_monitor /etc/fdfs/client.conf
2014-03-26 15:01:07] DEBUG - base_path=/fdfs/tracker, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 172.16.1.202:22122

group count: 3

Group 1:
group name = group1
disk total space = 27789 MB
disk free space = 23825 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0

	Storage 1:
		id = 172.16.1.203
		ip_addr = 172.16.1.203  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 01:48:19
		up time = 
		total storage = 27789 MB
		free storage = 23844 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 
		if_trunk_server= 0
		total_upload_count = 1
		success_upload_count = 1
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 24
		success_upload_bytes = 24
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 24
		success_sync_out_bytes = 24
		total_file_open_count = 1
		success_file_open_count = 1
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 1
		success_file_write_count = 1
		last_heart_beat_time = 2014-03-26 15:00:47
		last_source_update = 2014-03-26 13:13:55
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 172.16.1.204
		ip_addr = 172.16.1.204  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 18:53:19
		up time = 2014-03-26 18:53:19
		total storage = 27789 MB
		free storage = 23825 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 172.16.1.203
		if_trunk_server= 0
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 24
		success_sync_in_bytes = 24
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 1
		success_file_open_count = 1
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 1
		success_file_write_count = 1
		last_heart_beat_time = 2014-03-26 15:00:40
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 2014-03-27 03:38:23
		last_synced_timestamp = 2014-03-26 13:13:56 (-1s delay)

Group 2:
group name = group2
disk total space = 27789 MB
disk free space = 23825 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

	Storage 1:
		id = 172.16.1.205
		ip_addr = 172.16.1.205  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 19:37:36
		up time = 2014-03-26 19:37:36
		total storage = 27789 MB
		free storage = 23825 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 
		if_trunk_server= 0
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2014-03-26 15:01:02
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 172.16.1.206
		ip_addr = 172.16.1.206  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 22:38:04
		up time = 2014-03-26 22:38:04
		total storage = 27789 MB
		free storage = 23825 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 172.16.1.205
		if_trunk_server= 0
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 0
		success_file_open_count = 0
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 0
		success_file_write_count = 0
		last_heart_beat_time = 2014-03-26 15:01:02
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 

Group 3:
group name = group3
disk total space = 27789 MB
disk free space = 23825 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0

	Storage 1:
		id = 172.16.1.207
		ip_addr = 172.16.1.207  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 22:50:28
		up time = 2014-03-26 22:50:28
		total storage = 27789 MB
		free storage = 23825 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 172.16.1.208
		if_trunk_server= 0
		total_upload_count = 1
		success_upload_count = 1
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 35374
		success_upload_bytes = 35374
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 0
		success_sync_in_bytes = 0
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 1
		success_file_open_count = 1
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 1
		success_file_write_count = 1
		last_heart_beat_time = 2014-03-26 15:01:20
		last_source_update = 2014-03-27 03:20:13
		last_sync_update = 1970-01-01 08:00:00
		last_synced_timestamp = 1970-01-01 08:00:00 
	Storage 2:
		id = 172.16.1.208
		ip_addr = 172.16.1.208  ACTIVE
		http domain = 
		version = 5.01
		join time = 2014-03-26 22:49:37
		up time = 2014-03-26 22:49:37
		total storage = 27789 MB
		free storage = 23825 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8080
		current_write_path = 0
		source storage id= 
		if_trunk_server= 0
		total_upload_count = 0
		success_upload_count = 0
		total_append_count = 0
		success_append_count = 0
		total_modify_count = 0
		success_modify_count = 0
		total_truncate_count = 0
		success_truncate_count = 0
		total_set_meta_count = 0
		success_set_meta_count = 0
		total_delete_count = 0
		success_delete_count = 0
		total_download_count = 0
		success_download_count = 0
		total_get_meta_count = 0
		success_get_meta_count = 0
		total_create_link_count = 0
		success_create_link_count = 0
		total_delete_link_count = 0
		success_delete_link_count = 0
		total_upload_bytes = 0
		success_upload_bytes = 0
		total_append_bytes = 0
		success_append_bytes = 0
		total_modify_bytes = 0
		success_modify_bytes = 0
		stotal_download_bytes = 0
		success_download_bytes = 0
		total_sync_in_bytes = 35374
		success_sync_in_bytes = 35374
		total_sync_out_bytes = 0
		success_sync_out_bytes = 0
		total_file_open_count = 1
		success_file_open_count = 1
		total_file_read_count = 0
		success_file_read_count = 0
		total_file_write_count = 1
		success_file_write_count = 1
		last_heart_beat_time = 2014-03-26 15:01:32
		last_source_update = 1970-01-01 08:00:00
		last_sync_update = 2014-03-27 03:20:19
		last_synced_timestamp = 2014-03-27 03:20:14 (-1s delay)

其他功能

使用fdfs_crc32获取文件CRC
使用fdfs_delete_file删除文件
使用fdfs_file_info查看文件属性信息

以上所有功能均可使用接口在代码中调用。

CentOS下安装PHP扩展

因为php的客户端安装也会依赖FastDFS本身的一些库文件,所以请保证已经安装了FastDFS
安装过程和上边一样,这里参考上边安装过程!
这个php客户端所在的测试机本来我就在上边搭建好了LNMP,所以只需参考上述过程安装FastDFS即可!

安装PHP扩展

[root@web1 ~]# cd /root/FastDFS/php_client 
[root@web1 php_client]# /usr/local/php/bin/phpize 
Configuring for: 
PHP Api Version:         20090626 
Zend Module Api No:      20090626 
Zend Extension Api No:   220090626 
[root@web1 php_client]# ./configure --with-php-config=/usr/local/php/bin/php-config 
[root@web1 php_client]# make && make install 
[root@web1 php_client]# cat fastdfs_client.ini >> /usr/local/php/etc/php.ini
验证模块是否被正常加载
[root@web1 php_client]# /usr/local/php/bin/php -m | grep fastdfs_client 
fastdfs_client              #说明模块已加载 www.it165.net

重新加载php 
[root@web1 php_client]# service php-fpm reload

配置PHP扩展

[root@web1 php_client]# vi fastdfs_client.ini

测试

我们写个小页面,写个php脚本上传

[root@web1 ~]# echo "php client is here." > /opt/upload.txt 
[root@web1 ~]# vim test.php 
<?php
	var_dump(function_exists('fastdfs_storage_upload_by_filename')); 
	$ret = fastdfs_storage_upload_by_filename('/opt/upload.txt'); 
	var_dump($ret); 
?>
执行php脚本
[root@web1 ~]# /usr/local/php/bin/php test.php

结果如下图所示:
11

我们来访问下试试;

12

到此为止我们关于FastDFS的完整应用就介绍到此了,希望大家都动手做一下!还有关于FastDFS PHP插件的使用方法和函数调用方法,可以看/root/FastDFS/php_client/README,里面有详细介绍!此篇博文过程中如果有什么不对的地方,还请朋友们们指正,大家一起进步!

参考资料:
http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=4133106&page=2#pid24185011
http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=4134654
FastDFS分布式文件服务器安装,及配置,测试:http://www.linuxidc.com/Linux/2012-09/71458.htm
CentOS fastDFS结合nginx的配置以及php端的调用方法:http://blog.163.com/a12333a_li/blog/static/8759428520133189414521/
FastDFS分布式文件客户端安装,以及fastdfsapi:http://blog.51yip.com/server/219.html
FastDFS安装扩展篇——安装PHP、Apache及Nginx的FastDFS扩展:http://www.icultivator.com/p/4862.html

FastDFS集群扩展

1、扩展到多台tracker和storage的配置方法
2、按照以上安装tracker的方法配置所需数量的tracker,只需要ip地址不重复就可以。
3、按照以上安装storage的方法配置所需数量的storage,注意Group_name项,如果是同一组,那么group_name相同,端口号也相同,如果group_name不同,那么端口号也不同。
4、在storage的配置文件中修改tracker_server的地址和端口,可以指定多个,将新增的tracker也加进去就可以了。
原始文件名和FastDFS上的文件名对应关系都是保存在DB, 源文件只有client知道.
参考资料:
FastDFS部署总结:http://250688049.blog.51cto.com/643101/1067985
关于多个tracker的负载均衡问题:http://bbs.chinaunix.net/thread-2289651-1-1.html
多个Tracker之间有链接,相互交互吗?:http://bbs.chinaunix.net/thread-2287513-1-1.html
多个tracker的配置问题:http://bbs.chinaunix.net/thread-1929814-1-1.html

FastDFS配置文件

# cd /etc/fdfs/
下面一共有5个配置文件,分别为:client.conf、http.conf、mime.types、storage.conf、tracker.conf,fastdfs进程的启动是以加载的配置文件区分的

client.conf: 客户端上传配置文件
http.conf: http服务器配置文件
storage.conf: 文件存储服务器配置文件
tracker.conf: 负责均衡调度服务器配置文件
mime.types: 文件类型配置文件

FastDFS默认端口

FastDFS默认使用了以下端口,如果开启了防火墙,请保证以下端口可访问:

FastDFS Tracker端口:22122
FastDFS Storage端口:23000
FastDHT端口:11411

常用命令

#启动tracker、storaged
/usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf
/usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf

#查看trackerd状态:
/etc/init.d/fdfs_trackerd status

#查看trackerd监听的端口:
netstat -plantu | grep tracker

#查看Storage进程:
ps -ef | grep storaged | grep -v gre

#重启tracker、storaged
/usr/local/bin/restart.sh  /usr/local/bin/fdfs_trackerd  /etc/fdfs/tracker.conf
/usr/local/bin/restart.sh  /usr/local/bin/fdfs_storaged  /etc/fdfs/storage.conf
#重启tracker 
可以kill掉server进程后,执行启动命令行。如:
killall fdfs_trackerd
fdfs_trackerd /etc/fdfs/tracker.conf 

#停止tracker、storaged
/usr/local/bin/stop.sh  /usr/local/bin/fdfs_trackerd  /etc/fdfs/tracker.conf
/usr/local/bin/stop.sh  /usr/local/bin/fdfs_storaged  /etc/fdfs/storage.conf

#监控storage
/usr/local/bin/fdfs_monitor /etc/fdfs/storage.conf
#如果存在多个多个组,只需要监控其中一个组,就能调出所有组的状态

#删除组内服务器storage和查看各个组内服务器状态
/usr/local/bin/fdfs_monitor /etc/fdfs/client.conf delete group2 192.168.127.13
/usr/local/bin/fdfs_monitor /etc/fdfs/client.conf
storage server有7个状态,如下(数值从1~7):
# FDFS_STORAGE_STATUS:INIT      :初始化,尚未得到同步已有数据的源服务器
# FDFS_STORAGE_STATUS:WAIT_SYNC :等待同步,已得到同步已有数据的源服务器
# FDFS_STORAGE_STATUS:SYNCING   :同步中
# FDFS_STORAGE_STATUS:DELETED   :已删除,该服务器从本组中摘除
# FDFS_STORAGE_STATUS:OFFLINE   :离线
# FDFS_STORAGE_STATUS:ONLINE    :在线,尚不能提供服务
# FDFS_STORAGE_STATUS:ACTIVE    :在线,可以提供服务

# 启动和停止Nginx
/usr/local/nginx/sbin/nginx # 启动
/usr/local/nginx/sbin/nginx -s stop # 停止

# 调整防火墙的规则,开放端口:22122
iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 22122 -j ACCEPT
/etc/init.d/iptables save

#调整防火墙的规则,开放端口:23000
iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 23000 -j ACCEPT
/etc/init.d/iptables save

相关阅读

官方论坛:http://bbs.chinaunix.net/forum-240-1.html
FastDfs google Code:http://code.google.com/p/fastdfs/
分布式文件系统FastDFS架构剖析:http://www.programmer.com.cn/4380/
FastDFS 配置文件详解(修订版1):http://bbs.chinaunix.net/thread-1941456-1-1.html
分布式文件系统FastDFS配置优化:http://www.docin.com/p-454099549.html
FastDFS FAQ(很好):http://bbs.chinaunix.net/thread-1920470-1-1.html
关于使用FastDFS做图片及其缩略图存储 (主从文件):http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=3771347
lighttpd + fastdfs + imagemagick实现的图片缩放图的分布式存储:http://blog.csdn.net/saint1126/article/details/6317012
FastDFS v1.29和V2.0对比测试结果:http://bbs.chinaunix.net/forum.php?mod=viewthread&tid=1924272
FastDHT-高效分布式Hash系统:http://bbs.chinaunix.net/thread-1999971-1-1.html
分布式文件系统FastDFS(一~七):http://www.vktone.com/tags/fastdfs.html

参考资料:
官方安装文档:https://code.google.com/p/fastdfs/wiki/Setup
FastDFS 5.01 + nginx + cache 集群安装配置手册:http://bbs.chinaunix.net/thread-4133106-1-1.html
FastDFS+nginx+php的完整应用:http://www.it165.net/admin/html/201308/1628.html
分​布​式​文​件​系​统​F​a​s​t​D​F​S:http://wenku.baidu.com/view/145b4d6ab84ae45c3b358c57
CentOS 6.3(64) —> fastdfs_v4.07 / 实现多服务器:http://my.oschina.net/shking/blog/165326
CentOS 6.4 i386 版本安装 FastDFS、使用Nginx作为文件访问WEB服务器:http://www.cnblogs.com/zengen/p/3425886.html
FastDFS介绍和配置过程:http://ylw6006.blog.51cto.com/470441/948729
CentOS6.2下fastDFS的完整安装和配置步骤:http://www.cnblogs.com/ylh1223/archive/2012/02/08/2342224.html
FastDFS–安装篇:http://my.oschina.net/guol/blog/72845
CentOS FastDFS结合Nginx及PHP安装配置篇:http://blog.rekfan.com/articles/427.html
centos6.5下搭建fastdfs分布式存储:http://www.rootop.org/pages/2431.html

发表评论?

0 条评论。

发表评论

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据