Screen-2BShot-2B2015-07-15-2Bat-2B10.52.48-2BPM

How to use exec source type with HDFS sink in apache flume ng 1.6.0

Exec source type is used to run a command outside Flume environment. For using Exec type, just need to set the type property to exec.
agent.sources.s1.type=exec

The another property that need to specified here is the command property which need to be executed.

agent.sources.s1.command=tail -f $USER_DIRECTORY/app.log

where log file is the location of your log file.Here I have used HDFS sink for storing the log files.
conf file example for reading tomcat log is shown below. I have named the file as basicconfighdfs.conf.

#Name the components on this agent
a1.sources=r1
a1.sinks=k1
a1.channels=c1
#Configure the source
a1.sources.r1.type=exec
a1.sources.r1.command = tail -f /Users/fondofit/Documents/tomcat/apache-tomcat-6.0.44/logs/catalina.out
#Configure the HDFS sink
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=/Users/chikki
a1.sinks.k1.hdfs.fileSuffix=.log
#Configure the channels
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.channels.c1.keep-alive=2
#Bind the source and the sink to the channel
a1.sources.r1.channels=c1

 

a1.sinks.k1.channel=c1
Now use the following command to run the flume agent after navigating to the $APACHE_FLUME_HOME_DIRECTORY
bin/ng agent --conf conf --conf-file conf/basicconfighdfs.conf --name a1 -Dflume.root.logger=DEBUG,INFO,console
You will get the catalina file contents on HDFS directory browser as the output.
Output:
Exec source type is used to run a command outside Flume environment. For using Exec type, just need to set the type property to exec.

agent.sources.s1.type=execThe another property that need to specified here is the command property which need to be executed.

agent.sources.s1.command=tail -f $USER_DIRECTORY/app.log

where log file is the location of your log file.Here I have used HDFS sink for storing the log files.

conf file example for reading tomcat log is shown below. I have named the file as basicconfighdfs.conf.

#Name the components on this agent
a1.sources=r1
a1.sinks=k1
a1.channels=c1
#Configure the source
a1.sources.r1.type=exec
a1.sources.r1.command = tail -f /Users/fondofit/Documents/tomcat/apache-tomcat-6.0.44/logs/catalina.out
#Configure the HDFS sink
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=/Users/chikki
a1.sinks.k1.hdfs.fileSuffix=.log
#Configure the channels
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.channels.c1.keep-alive=2
#Bind the source and the sink to the channel
a1.sources.r1.channels=c1

 

a1.sinks.k1.channel=c1
Now use the following command to run the flume agent after navigating to the $APACHE_FLUME_HOME_DIRECTORY
bin/ng agent --conf conf --conf-file conf/basicconfighdfs.conf --name a1 -Dflume.root.logger=DEBUG,INFO,console
You will get the catalina file contents on HDFS directory browser as the output.
Output:
Screen Shot 2015-07-15 at 10.52.48 PM
Here you can see that FlumeData.1436980623799.log.tmp file has been added to the HDFS directory location specified in the basicconfighdfs.conf file.
Screen Shot 2015-07-15 at 10.52.48 PM
Here you can see that FlumeData.1436980623799.log.tmp file has been added to the HDFS directory location specified in the basicconfighdfs.conf file.

Leave a Reply

Your email address will not be published. Required fields are marked *