How to install Apache Flume 1.6.0 on MacOS

Here we will see in simple steps how to install Apache Flume 1.6.0 version on MacOS. The installation of Apache Flume is very easy. Below are the steps defined:

1) Download apache flume from the site.

https://flume.apache.org/download.html

2) Extract the tar file by using command

tar -xvf Documents/flume/apache-flume-1.6.0-bin

where Documents/flume is the directory where I have downloaded the Flume.

3) Create a conf file say HelloWorld.conf in the bin directory of Apache Flume. HelloWorld is taken since no example is complete without HelloWorld :)

The content of the file would be:

#Name the components on this agent
a1.sources=r1
a1.sinks=k1
a1.channels=c1
#Configure the source
a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=44444
#Configure the sink
a1.sinks.k1.type=logger
#Configure the channels
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
#Bind the source and the sink to the channel
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

Here, a1 is the Flume agent name. The source of even is netcat binding on the localhost for port 44444.

44444 is the default port on which Flume Agent run.

4) Save the file in $APACHE_FLUME_HOME_DIRECTORY/conf folder.

5) Now run command after navigating to the $APACHE_FLUME_HOME_DIRECTORY

bin/flume-ng agent --conf conf --conf-file conf/HelloWorld.conf --name a1 -Dflume.root.logger=INFO,console

where a1 is the Flume Agent name.

6) This will start the Flume Agent. Once its started, you can see at the console:

2015-06-25 21:22:52,992 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

7) Open a new terminal by Command+T. Use command:

telnet localhost 44444

Output of the command will be like:

Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

8) Type anything on this terminal and you will get the same message in the Flume Agent terminal as well:

Escape character is '^]'.
hello world
OK
Flume Agent terminal window output:
2015-06-25 21:23:31,779 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 0D hello world. }

This is all about the basic installation of Flume. In the next tutorial we will see how to configure Apache log files to set as Apache Flume source for aggregating the logs.

Leave a Reply

Your email address will not be published. Required fields are marked *