Large collection of log data applications Scribe installation and configuration instructions

Large collection of log data applications Scribe  installation and configuration instructions

1 Overview

Scribe is a Facebook open source log collection system, within Facebook has been a large number of applications. It logs collected from various sources logs stored on a central storage system, facilitates centralized statistical analysis. It is a log of "distributed collection, unified process" provides a scalable, highly fault-tolerant solution. scribe code is very simple, but very complex installation configuration, the paper chronicles the actual author once installed, I feel really unusual trivial, another version of the open source community has Scribe version a few years ago, consistent with the latest maintenance information I did not see, and HDFS version of Thrift use are older versions, consider an alternative to open a branch, look to upgrade Scribe, see when there is no time. If you are interested in big data development, want the system to learn big data , you can join the big data exchange technology to learn buttoned Junyang: 522 189 307 , welcome additions, understand Courses

2.Scribe installation

Scribe dependent on a plurality of environment, pthyon, boost, thirft, libevent, openssl like. At compile time and some compilation errors need to be corrected, installation and preparation are as follows.

2.1 Environment Preparation

　　1. Install the build environment

#yum install gcc-c++ libevent libevent-devel automake autoconf m4 bison zlib zlib-devel bzip2 bzip2-devel flex pkgconfig python python-devel ruby ruby-devel mono-devel libxml2 libxml2-devel ant openssl-devel

　　2. Install boost environment: minimum version 1.36

        #wget

        #tar zxvf boost_1_45_0.tar.gz

        #cd boost_1_45_0

        #./bootstrap.sh 

        # /. Bjam

     　#./bjam --s HAVE_ICU=1 --prefix=/usr/local/boost --includedir=/usr/local/boost/include  --libdir=/usr/local/boost/lib

       #./bjam install –prefix=/usr/local/boost

       #echo "/usr/local/boost/lib" >> /etc/ld.so.conf

       #echo /usr/local/boost/include >> /etc/ld.so.conf

     #ldconfig

　　3.  Installation thrift environment

　　#wget https://dist.apache.org/repos/dist/release/thrift/0.9.0/thrift-0.9.0.tar.gz

      #tar xzvf thrift-0.9.0.tar.gz

      #./configure --prefix=/usr/local/thrift --with-boost=/usr/local/boost --with-java=no  --with-erlang=no  --with-perl=no  --with-php=no  --with-ruby=no

　NOTE: If prompted to find libcrpty increase openssl path

　./configure --prefix=/usr/local/thrift --with-boost=/usr/local/boost --with-java=no  --with-erlang=no --with-perl=no --with-php=no  --with-　　ruby=no CPPFLAGS="-I/usr/local/openssl/include" LDFLAGS="-ldl -L/usr/local/openssl/lib"

　Modify the file:

　　Thrift / lib / src / thrift / Thrift.cpp increase headers: #include <stdlib.h>     

Meanwhile modify config.h file, commented:

/* Define to rpl_malloc if the replacement function should be used. */

// modby zhangzl ++:

//#define malloc rpl_malloc

//--

…

/* Define to rpl_realloc if the replacement function should be used. */

// modby zhangzl ++:

//#define realloc rpl_realloc

//--

#make

#make install

# echo "/usr/local/thrift/lib" >> /etc/ld.so.conf

#ldconfig

#cd contrib.

　　4. Installation fb303

#cd fb303

#./bootstrap.sh

#./configure --prefix=/usr/local/thrift/fb303 --with-boost=/usr/local/boost/ --with-thriftpath=/usr/local/thrift

#make

// Note to modify the generated Makefile observe information on header and library files path is correct, if correct, can be directly compiled.

  #make install  

4. Installation scribe

#unzip scribe-master.zip

#cd scribe-master

# export BOOST_ROOT=/usr/local/boost

#export LD_LIBRARY_PATH=/usr/local/thrift/lib:/usr/lib:/usr/local/lib:/usr/local/boost/lib:/usr/local/thrift/fb303/lib

#./bootstrap.sh  --with-boost=/usr/local/boost --with-boost-filesystem=boost_filesystem

Encounter problems:

checking whether the Boost::System library is available… yes

checking whether the Boost::Filesystem library is available… yes

configure: error: Could not link against  !

Solution, add a parameter in the configure --with-boost-filesystem = boost_filesystem

 

#./configure --prefix=/usr/local/scribe --with-boost=/usr/local/boost --with-thriftpath=/usr/local/thrift  --with-fb303path=/usr/local/thrift/fb303 --with-boost-filesystem=boost_filesystem

#make

Encounter problems:

1. thrift/server/TNonblockingServer.h:40:19: error: event.h: No such file or directory
2. Installation Libevent

#tar zxvf libevent-2.0.18-stable.tar.gz
#cd libevent-2.0.18-stable
#./configure –prefix=/usr/local/libevent
#make
#make install

　　3. Installation libevent-devel

yum install libevent-devel

error: there are no arguments to 'htons' that depend on a template parameter, so a declaration of 'htons' must be available

Need to modify their own files:

vim /usr/local/thrift/include/thrift/protocol/TBinaryProtocol.tcc

Adding header files referenced in the first line: #include <arpa / inet.h>

#echo /usr/local/libevent/include >> /etc/ld.so.conf

#ldconfig

#make install

2.2 Test

#cp example/example1.conf  /tmp

#/usr/local/scribe/bin/scribed /tmp/example1.conf

carry out testing.

If it fails, according to the message, make the appropriate changes.

I have the following error when starting:

./scribed: error while loading shared libraries: libevent-2.0.so.5: cannot open shared object file: No such file or directory

The solution is to find the library libevent-2.0.so.5, copy it to / usr / lib64 directory.

After a successful start, following screen appears:

    

3.Scribe Configuration   

Testing by the server-side and client-side configuration file as follows:

1. Server.conf

port=1463

max_msg_per_second=100000000

max_queue_size=10000000

check_interval=1

new_thread_per_category=true      

<store>

category=squid

type=file

file_path=/tmp/test/logcollect/

base_filename=21vianet__squid_access_all_

rotate_period=daily

rotate_hour=0

rotate_minute=0

max_size=1610612736

add_newlines=0

</store>

1. client.conf

port=1464

max_msg_per_second=2000000

max_queue_size=1000000

check_interval=1

new_thread_per_category=true      

 

<store>

category=default

type=buffer

target_write_size=20480

max_write_interval=1

buffer_send_rate=2

retry_interval=30

retry_interval_range=10

 

<primary>

type=network

#remote_host=218.93.205.106

remote_host=192.168.11.95

remote_port=1463

</primary>

 

<secondary>

type=file

fs_type=std

write_meta=no

file_path=/tmp/test/scribed

base_filename=accesslog

max_size=100000000

add_newlines=0

</secondary>

</store>

note:

 In the way server receives the log file will be recorded log, the file can not be deleted, it is a soft link. You need to copy the past, then call the touch command to clear the file.