Word count example in Apache Spark

No BigData example is complete without WordCount example :)
Here in this example we will learn how to setup spark in standalone mode using Java API with word count example.
1) Create a new maven project in Eclipse.

Screen Shot 2015-09-18 at 10.37.34 PM

 

2) Add Spark dependency in the pom.xml file.

3) Get the SparkConf object in the java class.

where ‘local’ is the cluster address and ‘Search’ is the application name residing on the cluster.  For running on standalone mode you can give it any value.
4) Get the SparkContext object from SparkConf object created earlier.

5) Load the text file by using SparkContext object.  The text file object will be loaded as JavaRDD object.

6) Use the RDD count() method to count all the words

Full example code:

One thought on “Word count example in Apache Spark

  1. A simple MySQL table “people” is used in the example and this table has two columns, “name” and “age”. These algorithms cover tasks such as feature extraction, classification, regression, clustering, recommendation, and more.

Leave a Reply

Your email address will not be published. Required fields are marked *