simple introduction to CAP theorem

Yesterday I discovered a beautifully written introduction to CAP theorem. Thanks to Kaushik Sathupadi for this.

machine learning ass-kicking in java part 2

Welcome to the second part of the tutorial for scoring your PMML files using  LightningScorer, which is a side project of mine.

Let's find out how additional parameters work.
The initial steps are similar to the first part of the tutorial.

Get your local copy first

git clone https://github.com/sezinkarli/lightningscorer.git

and build it with maven

mvn clean install

and start it by going to your target folder

java -jar lightningscorer-uberjar-1.0.jar

Now lets make sure our server is up and running by going to

http://localhost:8080/

.
Server returns

{
"data": "I have come here to chew bubblegum and kick ass...",
"success": true
}

Ok then we are now ready to kick ass, again.

I'll use apache commons' http get/post methods. First, we'll deploy our machine learning model with an additional parameter. Then we will check if it's working and then use our input values and score it.  After the scoring we will use our additional parameter.

    
        final String url = "http://localhost:8080/model/";
        final String modelId = "test2";

        //http://dmg.org/pmml/pmml_examples/knime_pmml_examples/ElNinoPolReg.xml
        File pmmlFile = new File("/tmp/ElNinoPolReg.xml");

        CloseableHttpClient client = HttpClients.createDefault();

        // deployment
        // notice that I give a variance value as an additional parameter that I will use later
        HttpPost deployPost = new HttpPost(url + modelId + "?variance=3.25");
        MultipartEntityBuilder builder = MultipartEntityBuilder.create();
        builder.addBinaryBody("model", new File(pmmlFile.getAbsolutePath()), ContentType.APPLICATION_OCTET_STREAM, "model");
        HttpEntity multipart = builder.build();
        deployPost.setEntity(multipart);

        CloseableHttpResponse response = client.execute(deployPost);
        String deployResponse = IOUtils.toString(response.getEntity().getContent(), Charset.forName("UTF-8"));
        System.out.println(deployResponse);
        // {"data":true,"success":true}
        deployPost.releaseConnection();

        // check deployed model
        HttpGet httpGet = new HttpGet(url + "ids");

        response = client.execute(httpGet);
        String getAllModelsResponse = IOUtils.toString(response.getEntity().getContent(), Charset.forName("UTF-8"));
        System.out.println(getAllModelsResponse);
        // {"data":["test1"],"success":true}
        httpGet.releaseConnection();

        //score deployed model
        HttpPost scorePost = new HttpPost(url + modelId + "/score");
        StringEntity params = new StringEntity("{" +
                "\"fields\":" +
                "{\"latitude\":2.5," +
                "\"longitude\":11.4," +
                "\"zon_winds\":3.5," +
                "\"mer_winds\":3," +
                "\"humidity\":31.2," +
                "\"s_s_temp\":25.21" +
                "}" +
                "} ");
        scorePost.addHeader("content-type", "application/json");
        scorePost.setEntity(params);

        CloseableHttpResponse response2 = client.execute(scorePost);
        String scoreResponse = IOUtils.toString(response2.getEntity().getContent(), Charset.forName("UTF-8"));
        System.out.println(scoreResponse);
        // {"data":{"result":{"airtemp":29.788226026392735}},"success":true}
        scorePost.releaseConnection();


        HttpGet additionalParamGet = new HttpGet(url + modelId + "/additional");
        CloseableHttpResponse response3 = client.execute(additionalParamGet);
        String additionalParamResponse = IOUtils.toString(response3.getEntity().getContent(), Charset.forName("UTF-8"));
        System.out.println(additionalParamResponse);
        // {"data":{"variance":"3.25"},"success":true}
        additionalParamGet.releaseConnection();


        // Then you can use the variance value with your result in airtemp to calculate an interval for your score


        client.close();

machine learning ass-kicking in java part 1

You searched for some way to export your machine learning models so you can use them for evaluating your data and you see that you can export them in PMML format. You actually work in Java ecosystem but not motivated to write neither your PMML library nor an rest api for it. Then I will recommend you LightningScorer, which is a side project of mine.

Let's take you a tour for deploying, and scoring your machine learning models.


Get your local copy first

git clone https://github.com/sezinkarli/lightningscorer.git

and build it with maven

mvn clean install

and start it by going to your target folder

java -jar lightningscorer-uberjar-1.0.jar

Now lets make sure our server is up and running by going to 

http://localhost:8080/

.
Server returns

{
"data": "I have come here to chew bubblegum and kick ass...",
"success": true
}

Ok then we are now ready to kick ass.

I'll use apache commons' http get/post methods. First, we'll deploy our machine learning model. Then we will check if it's safe and sound and then use our input values and score it. We will use a decision tree trained with iris data set from UCI machine learning repository. We will send 4 parameters ( sepal length and width and petal length and width) and the model will classify it for us into one of 3 values.

    
    final String url = "http://localhost:8080/model/";
    final String modelId = "test1";

    //http://dmg.org/pmml/pmml_examples/KNIME_PMML_4.1_Examples/single_iris_dectree.xml
    File pmmlFile = new File("/tmp/single_iris_dectree.xml");

    CloseableHttpClient client = HttpClients.createDefault();

    //first we will deploy our pmml file
    HttpPost deployPost = new HttpPost(url + modelId);
    MultipartEntityBuilder builder = MultipartEntityBuilder.create();
    builder.addBinaryBody("model", new File(pmmlFile.getAbsolutePath()), ContentType.APPLICATION_OCTET_STREAM, "model");
    HttpEntity multipart = builder.build();
    deployPost.setEntity(multipart);

    CloseableHttpResponse response = client.execute(deployPost);
    String deployResponse = IOUtils.toString(response.getEntity().getContent(), Charset.forName("UTF-8"));
    System.out.println(deployResponse);
    // response is {"data":true,"success":true}
    deployPost.releaseConnection();

     //now we check the model 
    HttpGet httpGet = new HttpGet(url + "ids");
    response = client.execute(httpGet);
    String getAllModelsResponse = IOUtils.toString(response.getEntity().getContent(), Charset.forName("UTF-8"));
    System.out.println(getAllModelsResponse);
    // response is {"data":["test1"],"success":true}  
    httpGet.releaseConnection();

    // lets score our deployed mode with parameters below 
    HttpPost scorePost = new HttpPost(url + modelId + "/score");
    StringEntity params = new StringEntity("{" +
            "\"fields\":" +
                "{\"sepal_length\":4.5," +
            "\"sepal_width\":3.5," +
            "\"petal_length\":3.5," +
            "\"petal_width\":1" +
            "}" +
            "} ");
    scorePost.addHeader("content-type", "application/json");
    scorePost.setEntity(params);

    CloseableHttpResponse response2 = client.execute(scorePost);
    String scoreResponse = IOUtils.toString(response2.getEntity().getContent(), Charset.forName("UTF-8"));
    System.out.println(scoreResponse);
    //response is{"data":{"result":{"class":"Iris-versicolor"}},"success":true}
    scorePost.releaseConnection();

    client.close();

lightningscorer - blazing fast pmml scoring web service

I've been working on a side project lately dubbed "LightningScorer". You can basically deploy your PMML files (which is the xml language for machine learning models) and score them with your inputs. My main inspiration was Openscoring project and I'm quite happy to build something faster than that. LightningScorer is lightweight and easy to use.

custom deserialize your field in jackson

Currently I'm doing an integration to a third party service api. I had a trouble while trying to deserialize a field as a Date. Their json contains a date field with an odd formatting. e.g. /Date(1494579066000)/
So I have to deserialize it into a date by taking the number between paranthesis, then casting it to a Date object.
Here is my model for json

public class Result implements Serializable
{
    @JsonProperty("InspectionDate")
    @JsonDeserialize(using = MyCustomDeserializer.class, as = Date.class)
    private Date inspectionDate;
...
}

As you can see I tell Jackson to use my class for deserialization.
Now it is time to write the custom deserializer.
I'm extending JsonDeserializer and overriding deserialize method.
jsonParser will give me the field value and I'm going to parse millisecond part of the text then cast it to a Date object.

public class MyCustomDeserializer extends JsonDeserializer
{
    @Override
    public Date deserialize(JsonParser jsonParser, DeserializationContext deserializationContext) throws IOException
    {
        String timestampAsString = jsonParser.getText();

        if (StringUtils.isEmpty(timestampAsString))
        {
            return null;
        }

        Matcher matcher = Pattern.compile("/Date\\(([0-9]+)\\)/").matcher(timestampAsString);

        if (!matcher.find())
        {
            return null;
        }

        String millisecondAsString = matcher.group(1);

        if (StringUtils.isEmpty(millisecondAsString))
        {
            return null;
        }

        return new Date(Long.parseLong(millisecondAsString));
    }
}