fluentdのおためし（前編）

はじめに

ログを収集するツールfluentdをインストールして試してみます。

環境

ログ収集対象サーバ構築

Apacheサーバ2台をVagrantで作成します。
http://ikechampion.hatenablog.com/entry/2013/07/11/143120

http01

c:\ikechampion01>vagrant box add httpd01 http://developer.nrel.gov/downloads/vagrant-boxes/CentOS-6.5-x86_64-v20140504.box
==> box: Adding box 'httpd01' (v0) for provider:
    box: Downloading: http://developer.nrel.gov/downloads/vagrant-boxes/CentOS-6.5-x86_64-v20140504.box
    box: Progress: 100% (Rate: 6408k/s, Estimated time remaining: --:--:--)
==> box: Successfully added box 'httpd01' (v0) for 'virtualbox'!

c:\ikechampion01>vagrant init httpd01
A `Vagrantfile` has been placed in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as ocumentation on
`vagrantup.com` for more information on using Vagrant.

c:\ikechampion01にできたVagrantfileに下記を追加

  config.vm.network "private_network", ip: "192.168.33.10"

起動できた

c:\ikechampion01>vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'httpd01'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: ikechampion01_default_1407196355610_71647
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => C:/ikechampion01

サーバにhttpdインストール、起動（確認のためiptables止める）

# yum -y install httpd
# service httpd start
# service iptables stop

fluentdインストール、起動

# curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh 
# service td-agent start

http02

http01と同様の手順で作成する。

ログ収集サーバ構築

c:\ikechampion03>vagrant box add log01 http://developer.nrel.gov/downloads/vagrant-boxes/CentOS-6.5-x86_64-v20140504.box
c:\ikechampion03>vagrant init log01

c:\ikechampion03にできたVagrantfileに下記を追加

  config.vm.network "private_network", ip: "192.168.33.20"

起動。

c:\ikechampion03>vagrant up

fluentdインストール、起動

# curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh 
# service td-agent start

設定

httpd01,02からlog01への転送設定

# vi /etc/td-agent/td-agent.conf
<source>
  type tail
  path /var/log/httpd/access_log 
  format /^(?<date>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \w{3}) (?<processing_time>[^ ]*) (?<remote>[^ ]*) (?<user>[^ ]*) \[(?<method>.*)\] (?<status>[^ ]*) (?<size>[^ ]*) \[(?<referer>[^ ]*)\] \[(?<agent>.*)\]/
  pos_file /var/log/td-agent/tmp/apache.access.log.pos
  tag apache.access
</source>
<match apache.access>
  type copy
  <store>
    type file
    path /var/log/td-agent/apache.access
    time_slice_format %Y%m%d
    time_format %Y%m%dT%H%M%S%z 
  </store>
  <store>
    type forward
    send_timeout 60s
    recover_wait 10s
    heartbeat_interval 1s
    <server>
      host 192.168.56.20
      port 24224
    </server>
  </store>
</match>
# mkdir /var/log/td-agent/tmp
# chown td-agent.td-agent /var/log/td-agent/tmp
# service td-agent restart

log01の設定

# vi /etc/td-agent/td-agent.conf
<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>

<match apache.access>
  type file
  path /var/log/fluentd/all_access.log
  time_slice_format %Y%m%d
  time_format %Y%m%dT%H%M%S%z
</match>

# mkdir /var/log/fluentd
# chown td-agent.td-agent /var/log/fluentd/
# service td-agent restart

まとめ

その1、3台かそうマシンを起動するとPC重い。

その2、「ミッチェル橋本の次回作にご期待ください」のミッチェル橋本ってこの本の著者のことだった！

実践 Vagrant

実践 Vagrant