MicroService – Ktz's Concurrency

Few tips using Zipkin

Zipkin is a very useful tracing solution while building microservice. (It is also useful at any service because it measure latency between two services). It becomes more and more popular but there are little documents. This post is few tips while adapting Zipkin in production.

Zipkin with Finatra

Zipkin is firstly made by Twitter and it supports Finatra (Base on Twitter Server and Finagle) well. Integrating Zipkin with Finatra is very simple.

1. Just add a dependency to project.(libraryDependencies in build.sbt)


com.twitter" %% "finagle-zipkin" %% finagle-version

2. And after that, just add Twitter flag while running Finatra Server. You can see Twitter flags when you add ‘-help’ end of jar or SBT. And all you need to know about flags related to Zipkin is

-thrift.name='thrift': Thrift server name // When Thrift Server
or
-http.name='http': Http server name // When Http Server
-com.twitter.finagle.zipkin.host='localhost:1463': Host to scribe traces to
-com.twitter.finagle.zipkin.initialSampleRate='0.001': Initial sample rate

Yes! when you add Twitter flags end of your application, it works! But sometimes, I have to check whether it is applied. Then you can check in Twitter Server admin. Its URL is ‘admin/clients/zipkin-tracer’.

If you set Zipkin Server properly, you can see your server address in ‘Set(Inet({Server Address}, Map()))’.

Add Twitter flag in SBT

When you want to add Twitter flags while in sbt run, you can add twitter flags like this

$ sbt 'run -help'

Existing instrumentations

Zipkin support many framework and language. If you want to find a supported framework, You can find it here.

Scribe

Above page, there are main 2 kinds of protocol, HTTP, and Scribe. As you know, Zipkin Server supports Http and Scribe. I was very curious what a Scribe is. You can see description in here. Simply to say, Scribe is logging protocol base on thrift.

com.twitter:finagle-zipkin

finagle-zipkin has two version. One is openzipkin/zipkin-finagle and another is finagle-zipkin in twitter/finagle. What is a difference between two things? openzipkin/zipkin-finagle is maintained by the openzipkin group. You can select protocol HTTP and Scribe. But, finagle-zipkin in twitter/finagle is little different. It supports only Scribe. so, if you want to connect with Finatra, you have to load Zipkin Server which supports Scribe.

Ports

Zipkin needs two port. One is 9411 which get HTTP Request. And 9410 which support Scribe. 9410 port is only opened when you enable Scribe support.

Play with Lagom (0) – What is (Not Completed)

As you can see obove, it’s not completed. As I study Lagom, I can say more about these topic. And fill it more!

These days, Internet Technology is growing up and there occur many buzzwords around. ‘Reactive’, ‘Micro Service’, ‘Event Sourcing’, ‘CQRS’… Many so called ‘Progressive Company’ cry out these buzzwords. And I also want to know about this. After moving to Kakao, I usually use Play, Akka, Finatra.(Personally, Finatra is a very good framework to make a productive micro server.) And looking forward to finding a very good solution to manage micro services in our service. And these series is a very short journey about Lagom.(Because I just want to taste Lagom. Of course, if I decide that it is the real solution to our service, it will be long journey 🙂 )

What is Reactive?

One of the hard waves around here is ‘Reactive’. You can read and sign to ‘Reactive Manifesto’ in here. Ok, after sign to Manifesto, Let’s summary what is Reactive System.

Reactive is Responsive. Service has to be responsive at all time.
Reactive is Resilience. Service has to be responsive at to time of failure occur.
- It means that, if an error occurs, the error is isolated and not affect to other component or service.
Reactive is Elastic. Service has to be scaled elastically when system load.
- It means that, if the service loaded, for example, Akka system assign more thread to the loaded actor.
Reactive is Message Driven. Reactive call remote system by targeting message.

Reactive Programming vs Reactive System.

This topic is not related to the main subject. But it is very important to distinguish between Reactive Programming and Reactive System. It is referenced from this article. It’s just summary about this article.

The reactive system is the way to build service. Above description about reactive is about Reactive system. Reactive system passes messages to the target. By message passing and mail box, we can assign more resources to some component(or micro server). We can scale horizontally or vertically.

The Reactive Programming is the way to handle events. We catch events from event stream and handle by chaining functions. We catch event compare to Reactive System which operated by message passing. Reactive programming is usually async,(In fact, have to be async) we can assign to another job which needs computing resource.

Reactive Programming is fit to Reactive System. Words are very awkward, But it means very natural. Reactive Programming is tightly coupled between Event and computing by chaining. And this makes less resilience. And Reactive system is key to resolve this problem.

What is CQRS?

CQRS is very simple. In service, we separate Command (Write) and Query. (Read) Martin Fowler describes it very well in this article.

it’s like read replica in SQL, Application request writes and read in different API or server.

As you can see, Service Interface(API Gateway) request read to Query Model, and write to Command Model.

What is Event Sourcing?

Event Sourcing is making event as a sequence of data. Let’s think like this. We make some data ‘a’ to 3 by adding 1 and 2. In common architecture, We only store that a is 3. But in Event Sourcing, we store that

‘add 1’ event occur,
‘add 2’ event occur.

So, we can get that a is 3. It means that we can get system service when a is 1 or a is 0. This can restore the service any time, any status.

Monitoring Micro Service With Zipkin

This post is recommended to read with an example. You can clone example in my Github – https://github.com/ktz-boilerplate/finatra-zipkin-example.

Micro Service is becoming trendy technology. But there are many things to consider during migrate our service to micro service. One of them is latency. We have to monitor each micro service and find out which service is a bottleneck and why does that happen. And Zipkin can be a reasonable answer.

Zipkin

Zipkin is monitoring solution about latency. It traces each micro service and calculates latencies so that we can trace which micro take lots of time and handle it. This solution is base on Google Dapper.

Example

Start Zipkin on Docker

First, we will test Zipkin solution on Docker. So, if you don’t have Docker in local, please install it. You can see how to install in Docker Documentation.

Now, let’s start Zipkin on Docker.

$ docker run -d -p 9411:9411 -p 9410:9410 -e SCRIBE_ENABLED=true openzipkin/zipkin

9411 port is Zipkin web UI and 9410 port is scribe port. And you have to enable scribe port to accept Thrift by SCRIBE_ENABLED=true. If you up Zipkin on Docker, then you can see web UI in here.

Zipkin Twitter Flag

Before start servers, Let’s look around for twitter flags related to Zipkin. If your server is not compatible to this flags, you have to version up Finatra.

-http.name: HTTP Server name which displayed in Zipkin.

-thrift.name: Thrift Server name which displayed in Zipkin.

-com.twitter.finagle.zipkin.host: Zipkin host and port number to pass trace span(data).

-com.twitter.finagle.zipkin.initialSampleRate: Sample rate. from 0 to 1.0. 1.0 is that trace all request and send to Zipkin.

Start Finatra HTTP Server

Now, go to project directory. You can see 5 sub-projects. One of them is Finatra HTTP-Server. Let’s start HTTP Server first.

$ sbt 'project http-server' "run -http.name=http-server -com.twitter.finagle.zipkin.host=localhost:9410 -com.twitter.finagle.zipkin.initialSampleRate=1"

Start Finatra Thrift Server

Now, let’s start another server. It’s Thrift Server

$ sbt 'project thrift-server' "-thrift.name=thrift-server -com.twitter.finagle.zipkin.host=localhost:9410 -com.twitter.finagle.zipkin.initialSampleRate=1"

Query Something.

Now connect to HTTP-Server and Query Something. There are some URL you can test.

get

/users
/user/:userId
/user/car/
/user/:userId/car

post

/user

For example, if you want to get all users. You can get by call http://localhost:8080/users

After some queries, you can see tracing status in Zipkin Web UI like this.

The default setting is sort by longest latency first.

You can also see dependency like this.