Tuesday, October 3, 2023

SQS listener in your Spring Boot project

I was adding SQS listener support to our spring boot based microservice and I realized that most of the examples online are for old versions of spring boot (2.x) or aws starter and I had few problems with dependencies etc so I wanted to prepare this small tutorial for people having similar experiences. The most complete one I could find was this one but lack of local running SQS is a downside.

In this post, I'm going to use localstack to emulate SQS locally. Localstack will be handy with integration tests as well but in this post I'll skip that part.

For dependencies we are going to add 1 bom and 2 usual dependencies:
    implementation(platform("io.awspring.cloud:spring-cloud-aws-dependencies:3.0.2"))
    implementation("io.awspring.cloud:spring-cloud-aws-starter")
    implementation("io.awspring.cloud:spring-cloud-aws-starter-sqs")

BOM (bill of materials) is to keep working dependency versions together.

The next thing is docker compose we will use so that we would have a local SQS. OFC you can choose to use a real instance as well but being able to run it locally is much more straightforward.

 version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - SERVICES="sqs"
      - DEBUG=${DEBUG-}
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
This is directly copy pasted from localstack website and there are many ways you can do the same thing. As many projects make use of Docker, I thought this version might become handy.

Next thing we will do will be to run this docker compose with:
docker-compose up

This should run localstack sqs on port 4566.

Next thing we would need is to define local sqs values in our application. Our application.yml will contain

 spring:
  cloud:
    aws:
      credentials:
        access-key: local
        secret-key: local
      region:
        static: 'eu-west-1'
      endpoint: 'http://localhost:4566'

In case you would need to setup this for your servers, you probably won't be using StaticCredentialsProvider that's been used there but some other AwsCredentialsProvider such as WebIdentityTokenFileCredentialsProvider. This would require you to define your own bean for this but there's no need for more override of autoconfiguration.

Next thing we will do is to define the service/component to listen to our sqs queue:

@Component
class MessageListener {

    @SqsListener("my-message-queue-name")
    fun receiveMessage(
        message: Message<CustomMessage>,
    ) {
        println("Message received from SQS listener. msg=${message.payload.msg}, code=${message.payload.msgCode}, headers=${message.headers}")
    }

    data class CustomMessage @JsonCreator constructor(
        @JsonProperty("msg") val msg: String,
        @JsonProperty("msgCode") val msgCode: Int,
    )
}

Usually we would expect that CustomMessage can be handled easily with object mapper we would have in the app context but for some reason it does not. This is the reason I specified Json creator and json properties. This post has some brief explanation regarding this issue. I have not tried to specified an object mapper and register with kotlin module, this can be tried to fix the issue as well. For the sake completeness, here's the exception I was getting and I fixed with json annotations:
Caused by: org.springframework.messaging.converter.MessageConversionException: Could not read JSON: Cannot construct instance of `com.sezin.sqsdemo.listener.MessageListener$CustomMessage` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (String)"{"msg":"my message to you","msgCode":12345}"; line: 1, column: 2]



Now we have a running local sqs instance and a running app that listens to "my-message-queue-name" queue. So two more things are left. We would need to create our msg queue and send the message to test our implementation.

aws --endpoint-url=http://127.0.0.1:4566 sqs create-queue --queue-name my-message-queue-name
can be used to create our queue. OFC, another option would be to add this to an init script and include it into our docker compose but for the sake of simplicity, I did not include it.


Next thing we will do is to send a message to our local queue:
aws --endpoint-url=http://127.0.0.1:4566 sqs send-message --queue-url http://127.0.0.1:4566/000000000000/my-message-queue-name --message-body '{"msg":"my message to you","msgCode":12345}'
If everything run fine, then you should see your SQS listener method picking the message up and printing this:

 "Message received from SQS listener. msg=my message to you, code=12345, headers=..."

You can also prefer to use CustomMessage dto instead of Message or you can change the sqs listener behavior to batch message retrieval by expecting List<CustomMessage> as well.

For the code you can check my github repo.

Saturday, October 24, 2020

docker tutorial part 2 - the good the bad and the ip

In this second part of docker tutorial, I'll explain basics of networking in docker with stand-alone containers. Remember from the previous part of the tutorial where we mapped port of our host machine and the container? In this part, we will setup our containers so they can communicate with each other and for that we don't need to do an explicit mapping like we previously did with -p subcommand.

Containers have their own virtual network when created. Let's begin by checking the ip of our host machine and then container ip.

I'm on a mac so I'll do ifconfig:
ifconfig |grep "inet "
gives me
inet 192.168.178.164 netmask 0xffffff00 broadcast 192.168.178.255
Ok so my host machine ip is 192.168.178.164. Now let's see the docker container one.
docker container run -t -d --name alpine1 alpine
Notice that I used --name parameter so I can call my container with this name instead of its alphanumeric id.
docker container ls
will return us our container. Yep it's running. Now let's check container's ip.
docker container inspect alpine1
will return a huge list of properties related to container. If you feel on the adventurous side, you can also try format and get less data.
docker container inspect --format "{{.NetworkSettings.IPAddress}}" alpine1
In my case I got "172.17.0.2" as ip. As you can see it's different than my host machine's ip. The reason is that containers use their own virtual network. Can we check all these virtual networks? We are talking about them but I'm not really convinced if I don't see them.
docker network ls
This command will return virtual networks of docker containers. "bridge" is the default one. Which containers is using this virtual network then?
docker network inspect bridge
Under containers part you'll see our alpine1.
 "Containers": {
            "2b8b07d5062743fc4ffe6dc8557adb25948ae3385d66052b9f254fa60ba52c6d": {
                "Name": "alpine1",
                "EndpointID": "5ce2ea615f9022ac6f1cae1af4fa44d1058f79dd73994a96a2d15305b3539f7f",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        }
What about the promise of containers that can communicate with each other? Let's create a new custom virtual network and connect several alpine containers together.
docker network create my_custom_network
Let's check if my_customer_network is there:
NETWORK ID          NAME                     DRIVER              SCOPE
3ad04560bc37        bridge                   bridge              local
905cd858e5a6        host                     host                local
cba2a990c957        my_custom_network        bridge              local
Yep now let's attach the existing container to this virtual network.
docker network connect my_custom_network alpine1
If you want to check what's bound to this network:
docker network inspect my_custom_network
You'll see that it's successful. 
Now let's get crazy and run a container with a customer virtual network.
docker container run -t -d --name alpine2 --network my_custom_network alpine
If you do network inspect you'll see alpine1 and alpine2 as containers. 
Now it's time for communication between containers. Let's execute bash on them (or rather ash) and make them ping each other.
docker container exec -it alpine1 ash
/ # ping -c 3 alpine2
PING alpine2 (172.22.0.3): 56 data bytes
64 bytes from 172.22.0.3: seq=0 ttl=64 time=0.117 ms
64 bytes from 172.22.0.3: seq=1 ttl=64 time=0.153 ms
64 bytes from 172.22.0.3: seq=2 ttl=64 time=0.155 ms
Nice! alpine1 was able to ping alpine2! But how? Yes same virtual network and stuff but how do they know each others ip? How the container name gets resolved to an ip? It's thanks to automatic dns resolution. Containers can resolve each other based on their container name regardless of their ip address on the same virtual network. 

To recap, in this part of the course we learnt how to create a virtual network and attach our containers to it so they can communicate with each other. Also we learnt how to check container ip and why it is different than host ip.

Tuesday, October 13, 2020

docker tutorial part 1 - fistfull of container

In this docker tutorial series, my aim is to explain basics of docker and to do that with min number of words but maximum number of examples. Before doing anything I want you to install docker on your machine and I hope you have an idea what docker does. If not please check docker.com.

Now let me explain two basic things related to docker:

image: File to be used in docker container. You may have a docker image of mysql for instance. It will contain an operating system, and whatever is needed to make mysql run.

container: A running instance of an image. So if you have an image of mysql, you can use it and run mysql on docker.

Now let's see why we need these two.

First open a command prompt or terminal and write

docker version


This should give you details of your docker server and client similar to


Client: Docker Engine - Community

....

Server: Docker Engine - Community

...


This means we installed docker successfully and good to go.

Now let's download an image and start a container using it. I'll use nginx which's a web server + load balancer and perfect for my simple examples.

docker container run nginx

which gives me this

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration

/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/

/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh

10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf

10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf

/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh

/docker-entrypoint.sh: Configuration complete; ready for start up


Seems to be working right? Your docker first checks your local repo for images, if you don't have this image it checks docker hub and download the latest version of this image because you did not specify it.

Let's open our browser and type localhost. It's a web server so there should be something in localhost:80 right?

Hmm that does not seem right. Maybe we did something wrong. Did we bind container's port to our machine? I think we missed that.

docker container run --publish 1234:80 nginx


But which one's which? Left is your host machine, right is the container port.

Now let's check our browser by going to http://localhost:1234/.

Nice! I don't want to have a terminal window for every container I run so I hope engineers in docker find a solution to that.

docker container run --publish 1234:80 --detach nginx

66f1accc69854a254da58c009aa7cdbd3808036533847ece598190683a677035


Cool! I even have a huge alphanumeric thing. Also nginx is working on port 1234!

Let's check which containers are running right now:

docker container ls


Wait a minute that container id is similar to the one I got in the previous step! Also I can see my port binding under ports. Awesome!

What if I want to check some logs? 

docker container logs 66f

will return nginx logs of my container. I did not write the whole id but docker's super-smart as you can see. I have logs but this does not work like tail -f. Then try

docker container logs -f 66f


Then go to localhost:1234 for a few times and see the access log scrolling.

Let's stop the container before saying goodbye.

docker container stop 66f


Ok. It's stopped I guess? Should we check it?

docker container ls


Nothing's there, that's good news. You can also see your container cemetery:

docker container ls -a


In this part of my docker tutorial, you learnt two basics concepts related to docker: container and image.

Also you learnt how to run containers, to check running container logs, to stop containers. 


Tuesday, September 8, 2020

best intellij idea plug-ins

I've been using Intellij Idea as IDE for some time and wanted to share my favorite plug-ins as I think they can be quite useful for every developer out there. Without a specific order here they are:

Key Promoter X: Although its name's like cheap midi controller, it is a really useful plug-in for those who want to master intellij idea shortcuts. It shows notifications for almost each click you do and tells you its shortcuts. Also it holds statistics for you, so you have a list of most popular clicks and their corresponding keyboard shortcuts.

Lombok: This is the only plug-in in my selection that's specific to a programming language. Lombok is a java library that lets you boilerplate code with annotations. For instance, you can add @getter and have a getter for every field in your pojo. For a flawless experience with this java library, you will need the lombok plug-in. If you don't have a special need for upgrading to latest jdk with every version, I'll highly recommend lombok. If you always update to the latest, then lombok can be problematic as they don't support the new versions right away and there's no way to upgrade lombok if its version does not support your java version.

Sonarlint: You may already know sonarqube. It's a statistical analysis tool which finds code smells, potential bugs and so on in your codebase. Sonarlint is a plug-in that acts as a sonarqube server that runs inside your IDE. It's great for finding potential problems in your code without any additional step.

Rainbow Brackets: The idea is quite straightforward. This plug-in colors your parentheses for better readability.


Extra Icons: This plug-in provides additional icons for your project files. After this plug-in is installed, your gitignore file gets a git icon. Your mvnw and mvnw.cmd files have maven run icons and so on.

Sunday, July 19, 2020

how to publish jar to maven central

You created your brand new project and you want to release it to maven central so people can use in their maven based project. It is not a really straightforward thing to accomplish so I wanted to write a step by step guide on it (based on this stackoverflow post). 

-Login to your jira account
-Create a ticket for your project: For this step you will need a group id, a project website and a link to your source control. 

I used "com.sezinkarli" for group id because I own this domain. If you don't have a domain and you use github, you can easily use "io.github.yourusername"

Project website can be your github link for the project and you can link your github .git link as source control. So as you can see, github will be really useful here.

-Create a PGP key. You can download it here. Then open a command prompt and do

gpg --gen-key


It will generate our public and secret keys and sign them.
In the meantime it will print something like this:

gpg: key [Your key] marked as ultimately trusted

We will use this key for the following command:

gpg --keyserver hkp://pool.sks-keyservers.net --send-keys  [Your key] 

-Now we will update our user's maven settings. Go to your .m2 folder and edit/add settings.xml .

<settings>
  <servers>
    <server>
      <id>ossrh</id>
      <username>your jira username for sonatype</username>
      <password>your jira passwordfor sonatype</password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>ossrh</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <gpg.executable>gpg</gpg.executable>
        <gpg.passphrase>passphrase you used for gpg key</gpg.passphrase>
      </properties>
    </profile>
  </profiles>
</settings>
-Now we will update our project's pom.xml .

Add parent:

<parent>
    <groupId>org.sonatype.oss</groupId>
    <artifactId>oss-parent</artifactId>
    <version>9</version>
</parent>

Add distribution management:

<distributionManagement>
        <snapshotRepository>
            <id>ossrh</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </snapshotRepository>
        <repository>
            <id>ossrh</id>
            <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
        </repository>
</distributionManagement> 
 
Add build plugins:

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar-no-fork</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <version>3.2.0</version>
                <configuration>
                    <source>11</source>
                </configuration>
                <executions>
                    <execution>
                        <id>attach-javadocs</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-gpg-plugin</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <id>sign-artifacts</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>sign</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.sonatype.plugins</groupId>
                <artifactId>nexus-staging-maven-plugin</artifactId>
                <version>1.6.8</version>
                <extensions>true</extensions>
                <configuration>
                    <serverId>ossrh</serverId>
                    <nexusUrl>https://oss.sonatype.org/</nexusUrl>
                    <autoReleaseAfterClose>true</autoReleaseAfterClose>
                </configuration>
            </plugin>
        </plugins>
    </build>

-Now we have everything in place. After we made sure from sonatype jira ticket that we are good to go (they do a check for group id), we can deploy our project to maven central:

mvn clean deploy

And that's it!
I had a problem while deploying because I did not have any javadoc. After I added it, everything worked like a charm.