How to use exec source type with logger sink in apache flume ng 1.6.0

Exec source type is used to run a command outside Flume environment. For using Exec type, just need to set the type property to exec.

agent.sources.s1.type=exec

The another property that need to specified here is the command property which need to be executed.

agent.sources.s1.command=tail -f $USER_DIRECTORY/app.log

where log file is the location of your log file.

conf file example for reading tomcat log is shown below. I have named the file as basicconfig.conf.

#Name the components on this agent
a1.sources=r1
a1.sinks=k1
a1.channels=c1
#Configure the source
a1.sources.r1.type=exec
a1.sources.r1.command = tail -f /Users/fondofit/Documents/tomcat/apache-tomcat-6.0.44/logs/catalina.out
#Configure the sink
a1.sinks.k1.type=logger
#Configure the channels
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.channels.c1.keep-alive=2
#Bind the source and the sink to the channel
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1
Now use the following command to run the flume agent:
ng agent --conf conf --conf-file conf/basicconfig.conf --name a1 -Dflume.root.logger=DEBUG,INFO,console
You will get the catlalina file contents on console as the output. Just make sure that the correct log location file should be provided.
Output:
2015-06-29 21:51:47,492 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.source.ExecSource.start(ExecSource.java:187)] Exec source started
2015-06-29 21:51:50,506 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 49 4E 46 4F 3A 20 4A 6B 20 72 75 6E 6E 69 6E 67 INFO: Jk running }
2015-06-29 21:51:50,507 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 4A 75 6E 20 32 38 2C 20 32 30 31 35 20 31 30 3A Jun 28, 2015 10: }
2015-06-29 21:51:50,507 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 49 4E 46 4F 3A 20 53 65 72 76 65 72 20 73 74 61 INFO: Server sta }
2015-06-29 21:51:50,507 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }
2015-06-29 21:51:50,507 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }
2015-06-29 21:51:50,508 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }
2015-06-29 21:51:50,508 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }
2015-06-29 21:51:50,508 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }
2015-06-29 21:51:50,508 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }
2015-06-29 21:51:50,509 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 68 6F 77 20 72 20 75 hi how r u }

Leave a Reply

Your email address will not be published. Required fields are marked *