head first elastic search on java with spring boot and data features

In this article I'll try to give you an easy introduction on how to use Elastic Search in a Java project. As Spring Boot is the easiest and fastest way to begin our project I choose to use it. Futhermore, we will heavily use Repository goodies of beloved Spring Data.

Let's begin by installing Elastic Search on our machine and run our elastic server for the first time.

I go to elastic-folder\bin and run elasticsearch.bat (yeah I'm using Windows) but no luck. I get this:

"Error occurred during initialization of VM

Could not reserve enough space for object heap

Error: Could not create the Java Virtual Machine.

Error: A fatal exception has occurred. Program will exit."

What a great start!

In my bin folder there's a "elasticsearch.in.bat" file. I set ES_MAX_MEM=1g to 

ES_MAX_MEM=512mb and voila it is fixed.

I start a new server without problem after that.

Now it is time to define the document we will index in elastic search. Assume we have movie information to index. Our model is quite straightforward. Movie has a name, rating and a genre in it. I chose "elastic_sample" as index name which sounds good as a database name and "movie" as type which is good for a table name if we think in relational database terms. Nothing fancy in the model as you can see.

@Document(indexName = "elastic_sample", type = "movie")
public class Movie {

    @Id
    private String id;

    private String name;

    @Field(type = FieldType.Nested)
    private List &lt Genre &gt  genre;

    private Double rating;

    public Double getRating() {
        return rating;
    }

    public void setRating(Double rating) {
        this.rating = rating;
    }

    public void setId(String id) {
        this.id = id;
    }

    public List &lt Genre &gt  getGenre() {
        return genre;
    }

    public void setGenre(List &lt Genre &gt  genre) {
        this.genre = genre;
    }

    public String getId() {
        return id;
    }

    public String getName() {
        return name;

    }

    public void setName(String name) {
        this.name = name;
    }

    @Override
    public String toString() {
        return "Movie{" +
                "id=" + id +
                ", name='" + name + '\'' +
                ", genre=" + genre +
                ", rating=" + rating +
                '}';
    }
}

For those who wonder what Genre is here is it. Just a POJO.

public class Genre {
    private String name;

    public Genre() {
    }

    public Genre(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    @Override
    public String toString() {
        return "Genre{" +
                "name='" + name + '\'' +
                '}';
    }

    public void setName(String name) {
        this.name = name;
    }
}

Not it is time to create DAO layer so we can save and load our document to/from our elastic search server. Our Repository extends the classic ElasticserchRepository (no idea why it is search and not Search). As you probably know Spring Data can query one or more fields with these predefined methods where we use our field names. findByName will search in the name field, findByRating will search in the rating field so on so forth. Furthermore thanks to Spring Data we don't need to write implementation for it, we just put method names in the interface and that's finished.

public interface MovieRepository extends ElasticsearchRepository &lt Movie, Long &gt {
    public List &lt Movie &gt  findByName(String name);

    public List &lt Movie&gt  findByRatingBetween(Double beginning, Double end);
}

Our DAO layer will be called by a Service layer:

@Service
public class MovieService {

    @Autowired
    private MovieRepository repository;

    public List &lt Movie &gt  getByName(String name) {
        return repository.findByName(name);
    }

    public List &lt Movie &gt  getByRatingInterval(Double beginning, Double end) {
        return repository.findByRatingBetween(beginning, end);
    }

    public void addMovie(Movie movie) {
        repository.save(movie);
    }
}

Here is the main Class we will use to run our application. EnableAutoConfiguration will auto-configure everything it recognizes under our classpath. ComponentScan will scan for Spring annotations under the main Class' directory.

@Configuration
@EnableAutoConfiguration
@ComponentScan
public class BootElastic implements CommandLineRunner {

    @Autowired
    private MovieService movieService;

    private static final Logger logger = LoggerFactory.getLogger(BootElastic.class);

// add star wars and
// princess bride as a movie
// to elastic search
    private void addSomeMovies() {
        Movie starWars = getFirstMovie();
        movieService.addMovie(starWars);

        Movie princessBride = getSecondMovie();
        movieService.addMovie(princessBride);
    }

    private Movie getSecondMovie() {
        Movie secondMovie = new Movie();
        secondMovie.setId("2");
        secondMovie.setRating(8.4d);
        secondMovie.setName("The Princess Bride");

        List &lt Genre &gt  princessPrideGenre = new ArrayList &lt Genre &gt();
        princessPrideGenre.add(new Genre("ACTION"));
        princessPrideGenre.add(new Genre("ROMANCE"));
        secondMovie.setGenre(princessPrideGenre);

        return secondMovie;
    }


    private Movie getFirstMovie() {
        Movie firstMovie = new Movie();
        firstMovie.setId("1");
        firstMovie.setRating(9.6d);
        firstMovie.setName("Star Wars");

        List &lt Genre &gt  starWarsGenre = new ArrayList &lt Genre &gt();
        starWarsGenre.add(new Genre("ACTION"));
        starWarsGenre.add(new Genre("SCI_FI"));
        firstMovie.setGenre(starWarsGenre);

        return firstMovie;
    }

    public void run(String... args) throws Exception {
        addSomeMovies();
        // We indexed star wars and pricess bride to our movie
        // listing in elastic search

        //Lets query if we have a movie with Star Wars as name
        List &lt Movie &gt starWarsNameQuery = movieService.getByName("Star Wars");
        logger.info("Content of star wars name query is {}", starWarsNameQuery);

        //Lets query if we have a movie with The Princess Bride as name
        List &lt Movie &gt  brideQuery = movieService.getByName("The Princess Bride");
        logger.info("Content of princess bride name query is {}", brideQuery);


        //Lets query if we have a movie with rating between 6 and 9
        List &lt Movie &gt  byRatingInterval = movieService.getByRatingInterval(6d, 9d);
        logger.info("Content of Rating Interval query is {}", byRatingInterval);
    }

    public static void main(String[] args) throws Exception {
        SpringApplication.run(BootElastic.class, args);
    }
}

If we run it the result is:

015-02-28 18:26:12.368  INFO 3616 --- [           main] main.BootElastic: Content of star wars name query is [Movie{id=1, name='Star Wars', genre=[Genre{name='ACTION'}, Genre{name='SCI_FI'}], rating=9.6}]
2015-02-28 18:26:12.373  INFO 3616 --- [           main] main.BootElastic: Content of princess bride name query is [Movie{id=2, name='The Princess Bride', genre=[Genre{name='ACTION'}, Genre{name='ROMANCE'}], rating=8.4}]
2015-02-28 18:26:12.384  INFO 3616 --- [           main] main.BootElastic: Content of Rating Interval query is [Movie{id=2, name='The Princess Bride', genre=[Genre{name='ACTION'}, Genre{name='ROMANCE'}], rating=8.4}]

As you can see the interval query only retrieved Princess Bride. We did not do any configuration right? It is unusual. I have to share the huge configuration file with you:

spring.data.elasticsearch.cluster-nodes=localhost:9300
 # if spring data repository support is enabled
spring.data.elasticsearch.repositories.enabled=true

Normally you would use port 9200 when you query your elastic server. But when we programmatically reach it we are using 9300. If you have more than one node you would separate them with a comma and use 9301, 9302 etc as port numbers. Our pom file is no surprise either. Just elastic starter pom and we are set to go.

    4.0.0

    caught.co.nr
    boot-elastic-sample
    1.0-SNAPSHOT
    war

    
    
        org.springframework.boot
        spring-boot-starter-parent
        1.2.2.RELEASE
    

    
        
            org.springframework.boot
            spring-boot-starter-data-elasticsearch
        

    

    
    
        
            
                org.springframework.boot
                spring-boot-maven-plugin
            
        
    

As you can see thanks to Spring Boot and Data it is quite easy to work with elastic search. Lets check what we indexed from the server api as well. I'll use Sense -a chrome plug-in for elastic commands-.

Here's the result json:

{
   "took": 2,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 1,
      "hits": [
         {
            "_index": "elastic_sample",
            "_type": "movie",
            "_id": "1",
            "_score": 1,
            "_source": {
               "id": 1,
               "name": "Star Wars",
               "genre": [
                  {
                     "name": "ACTION"
                  },
                  {
                     "name": "SCI_FI"
                  }
               ]
            }
         },
         {
            "_index": "elastic_sample",
            "_type": "movie",
            "_id": "2",
            "_score": 1,
            "_source": {
               "id": 2,
               "name": "The Princess Bride",
               "genre": [
                  {
                     "name": "ACTION"
                  },
                  {
                     "name": "ROMANCE"
                  }
               ]
            }
         }
      ]
   }
}

You can check out the whole project in the github.

rocking with mongodb on spring boot

I'm a fan of Spring Boot and here's my mongodb example project on Spring Boot. Most of the mongodb example projects are so basic that you won't go far with them. You can search for plain Spring Data examples but they can get much complex than you'd like. So here's mine.

Here's the pom I'll use.

    4.0.0

    caught.co.nr
    boottoymongodb
    1.0-SNAPSHOT
    war


    
    
        org.springframework.boot
        spring-boot-starter-parent
        1.0.0.BUILD-SNAPSHOT
    

    
        
            org.springframework.boot
            spring-boot-starter-data-mongodb
        

    

    
    
        
            
                org.springframework.boot
                spring-boot-maven-plugin
            
        
    

    
        
            spring-snapshots
            Spring Snapshots
            http://repo.spring.io/snapshot
            
                true
            
        
    

    
        
            spring-snapshots
            http://repo.spring.io/snapshot
        
    

The only dependency I need is "spring-boot-starter-data-mongodb" which contains all necessary dependencies for a spring boot mongodb project. Next is the model for my collection. Document annotation points to my collection named "products". It is need only if your model name does not match your collection name. You can see a field annotation which maps the field name in the collection to the model's field name.

@Document(collection = "products")
public class Product {
    @Id
    private String id;
    private String sku;

    @Field(value = "material_name")
    private String materialName;

    private Double price;
    private Integer availability;


    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getSku() {
        return sku;
    }

    public void setSku(String sku) {
        this.sku = sku;
    }

    public String getMaterialName() {
        return materialName;
    }

    public void setMaterialName(String materialName) {
        this.materialName = materialName;
    }

    public Double getPrice() {
        return price;
    }

    public void setPrice(Double price) {
        this.price = price;
    }

    public Integer getAvailability() {
        return availability;
    }

    public void setAvailability(Integer availability) {
        this.availability = availability;
    }

    @Override
    public String toString() {
        return "Product{" +
                "id='" + id + '\'' +
                ", sku='" + sku + '\'' +
                ", materialName='" + materialName + '\'' +
                ", price=" + price +
                ", availability=" + availability +
                '}';
    }
}

Not we will need a DAO layer to manipulate my data. MongoRepository is the interface I should implement if I want to use autogenerated find methods in my DAO layer and I want that. Every field of my model can be queried with these autogenerated methods. For a complete list of method name syntax check here. My query below will take a sku name and search my collection for this name and return the matching ones.

public interface ProductRepository extends MongoRepository < Product, String >{
    public List < Product > findBySku(String sku);
}

Now I'll introduce a Service which will call my DAO interface. But wait a minute, I didn't implement this interface and wrote necessary code for fetching the models right? Yep, these methods are autogenerated and I don't need an implementation for this interface.

@Service
public class ProductService {
    @Autowired
    private ProductRepository repository;

    public List < Product > getSku(String sku){
        return repository.findBySku(sku);
    }
}

Next, lets launch our Boot example. Here's our main class:

@Configuration
@EnableAutoConfiguration
@ComponentScan
public class BootMongoDB implements CommandLineRunner {

    @Autowired
    private ProductService productService;

    private static final Logger logger = LoggerFactory.getLogger(BootMongoDB.class);

    public void run(String... args) throws Exception {
        List < Product > sku = productService.getSku("NEX.6");
        logger.info("result of getSku is {}", sku);
    }

    public static void main(String[] args) throws Exception {
        SpringApplication.run(BootMongoDB.class, args);
    }
}

If you have a connection to a mongodb instance and a sku matching to the name you searched than you should see one or more Products as a result. What we did was quite basic. What if I want more complex queries? For instance if I want a specific sku with an availability equal to "1"? I can't do it without using some @Query magic. So I'm updating my DAO class.

public interface ProductRepository extends MongoRepository < Product, String >{
    public List < Product > findBySku(String sku);

    @Query(value = "{sku: ?0, availability : 1}")
    public List < Product > findBySkuOnlyAvailables(String sku);
}

I provided a direct query for mongodb where sku in the signature of my method will be inserted to "?0" in the query and will be sent to mongodb. You can update your Service and then your main method to see if it works. You may not like writing queries which are not much readable if you're not very familiar with mongodb's syntax. Then this is the time for adding custom DAO classes. It's not possible to add and use methods other than the autogenerated ones to ProductRepository. So we will add few classes and have a nice featured methods. Our repository class was named "ProductRepository". We will add a new interface named "ProductRepositoryCustom" and a new method which will find available skus for the given name (twin of findBySkuOnlyAvailables method).

public interface ProductRepositoryCustom {
    public List < Product > findBySkuOnlyAvailablesCustom(String sku);
}
 

Then provide an implementation for this. Below you see that we inject ProductRepositoryCustom's mongotemplate and do stuff with it. We create two criteria. First one is for the sku name and the second one is for availability.

public class ProductRepositoryImpl implements ProductRepositoryCustom {
    @Autowired
    private MongoTemplate mongoTemplate;

    public List < Product > findBySkuOnlyAvailablesCustom(String sku) {
        Criteria criteria = Criteria.where("sku").is(sku).
andOperator(Criteria.where("availability").is(1));
        return mongoTemplate.find(Query.query(criteria), Product.class);
    }
}

The last step for custom implemetation is the update of ProductRepository class. As you can see below the only update I need is the addition of my ProductRepositoryCustom so we can link both of them together. All this naming can sound a little stupid. But notice that although the name of your custom interface is not important, a change in the name of the implementation will result in the throw of an exception:

Invocation of init method failed; nested exception is org.springframework.data.mapping.PropertyReferenceException: No property only found for type String! Traversed path: Product.sku.

To fix this make sure that the name of your implementation class is "ProductRepositoryImpl" which is the concatenation of the name of the interface that extends MongoRepository and "Impl".

public interface ProductRepository extends MongoRepository < Product, String>, ProductRepositoryCustom

If we add our new method to our Service layer:

@Service
public class ProductService {
    @Autowired
    private ProductRepository repository;

    public List < Product > getSku(String sku){
        return repository.findBySku(sku);
    }

    public List < Product > getAvailableSkuCustom(String sku){
        return repository.findBySkuOnlyAvailablesCustom(sku);
    }
}

Then update our main class' run method:

   public void run(String... args) throws Exception {
        List < Product > sku = productService.getSku("NEX.6");
        logger.info("result of getSku is {}", sku);

        List < Product > availableSkuCustom = productService.getAvailableSkuCustom("NEX.6");
        logger.info("result of availableSkuCustom is {}", availableSkuCustom);
    }

Again you must see something in the log :). You can check the whole project on github.

mongodb default configuration in spring boot

I'm trying to integrate mongodb to my spring boot application. I create a db aptly named "mydb" and was wondering how mongodb classes were able to connect to my database without any configuration and why they were not retrieving my data. Answer to both is of course "default configuration" for mongodb.

You can see mongodb defaults that spring boot uses:

host: localhost
port: 27017
db: test

My mongodb is at localhost:27017, but as I'm using "mydb" instead of "test" no data is retrieved. So I have to override the default configuration by adding

spring.data.mongodb.uri=mongodb://localhost:27017/mydb

to my application.properties file.