Scala flatmap and Parallel programming

We usually use flatmap and map in scala. Of course, if we use future we chaining by using flatmap or map… etc. But is this run parallel? In fact (as you know) it’s not parallel. If you chaining some monads, it runs sequentially.

Future chaining is not parallel

Let’s see some example. Guess that, there is some db dependent function like beneath

Each takes 2 seconds, 2 seconds, 3 seconds. And we want to make userInfo by calling functions. You can call like this.

As you can see, It takes 7 seconds (2 sec + 2 sec + 3 sec). It is not parallel.

How to be parallel?

by function

There easy 2 ways to be parallel. Solve this by function. You can make like this.

It takes 5 seconds(2 seconds + max(2 seconds, 3 seconds)). It’s because, before chaining, Future already started. And after getting an email address, 2 seconds, we only need to wait for 1 second to get accountInfo because it runs parallel.

by using the Cartesian product and applicative

But it has to make function. It’s annoying. We can simply make this by using cats.

It also takes 5 seconds, But it is simpler than by using a function.

Few tips using Zipkin

Zipkin is a very useful tracing solution while building microservice. (It is also useful at any service because it measure latency between two services). It becomes more and more popular but there are little documents. This post is few tips while adapting Zipkin in production.

Zipkin with Finatra

Zipkin is firstly made by Twitter and it supports Finatra (Base on Twitter Server and Finagle) well. Integrating Zipkin with Finatra is very simple.

1. Just add a dependency to project.(libraryDependencies in build.sbt)


com.twitter" %% "finagle-zipkin" %% finagle-version

2. And after that, just add Twitter flag while running Finatra Server. You can see Twitter flags when you add ‘-help’ end of jar or SBT. And all you need to know about flags related to Zipkin is

-thrift.name='thrift': Thrift server name // When Thrift Server
or
-http.name='http': Http server name // When Http Server
-com.twitter.finagle.zipkin.host='localhost:1463': Host to scribe traces to
-com.twitter.finagle.zipkin.initialSampleRate='0.001': Initial sample rate

Yes! when you add Twitter flags end of your application, it works! But sometimes, I have to check whether it is applied. Then you can check in Twitter Server admin. Its URL is ‘admin/clients/zipkin-tracer’.

스크린샷 2017-10-17 오후 10.27.42

If you set Zipkin Server properly, you can see your server address in ‘Set(Inet({Server Address}, Map()))’.

Add Twitter flag in SBT

When you want to add Twitter flags while in sbt run, you can add twitter flags like this

$ sbt 'run -help'

Existing instrumentations

Zipkin support many framework and language. If you want to find a supported framework, You can find it here.

Scribe

Above page, there are main 2 kinds of protocol, HTTP, and Scribe. As you know, Zipkin Server supports Http and Scribe. I was very curious what a Scribe is. You can see description in here. Simply to say, Scribe is logging protocol base on thrift.

com.twitter:finagle-zipkin

finagle-zipkin has two version. One is openzipkin/zipkin-finagle and another is finagle-zipkin in twitter/finagle. What is a difference between two things? openzipkin/zipkin-finagle is maintained by the openzipkin group. You can select protocol HTTP and Scribe. But, finagle-zipkin in twitter/finagle is little different. It supports only Scribe. so, if you want to connect with Finatra, you have to load Zipkin Server which supports Scribe.

Ports

Zipkin needs two port. One is 9411 which get HTTP Request. And 9410 which support Scribe. 9410 port is only opened when you enable Scribe support.

Implicit in Scala

In Scala, Implicit is very important. But it is very ambiguous and hard to understand. And it is learning curve to enter Scala world. This post is basic about Scala Implicit.

Implicit is operated by the compiler. If there is some error, for example, Instance type is not compatible or need implicit parameter Compiler is looking for Implicit conversion or Implicit Instance. And this post is about making implicit Conversion or Instance.

Implicit Conversion

implicit def

We can convert some instance to other instance automatically by using implicit. First, we can use ‘Implicit Conversion’. Let’s say that, there are some instances like that.

If we want to convert ‘ValueCaseClass’ to ‘PrintableClass’, you can use by using implicit def like this.

In this code,


val printableClass1: PrintableClass = ValueCaseClass("Implicit Def")

printableClass1’s type is PrintableClass but we set ValueCaseClass. It is compile error. But, Compile is looking for implicit conversion and finally find implicit definition ‘convertValueCaseClassToPrintable’ and get PrintableClass.

implicit class

You can also convert by using implicit class.

In this code,


val printableClass2: PrintableClass = ValueCaseClass("Implicit class").toPrintable

ValueCaseClass don’t has toPrintable. Compiler looking for implicit class and find ‘ConvertableValueCaseClass’ and there is ‘toPrintable’.

Implicit Instance

In Scala, def can get implicit parameter.

In code above, ‘print’ need ‘printableClass’ parameter. Compiler automatically pass implicit instance ‘printableClass’.

Implicit Order

Above, there we can get the implicit instance. And I think it is very useful. We can get Implicit Instance in many ways. Let see an example.

Can you guess which string is printed in the console? The answer is ‘Implicit Instance in Same Conjure’ Implicit order is

  1. In same conjure
  2. Imported instance
  3. companion object
  4. default imported

So, if we delete

def getString(implicit implicitInstance: Instance[String]): String = implicitInstance.value

, “Implicit Instance in imported” will be printed. And if we delete

import Imported.implicitIntInstance1

, “Implicit Instance in Companion Object” will be printed

Default implicit instance by companion object.

Let’s pay attention to the companion object implicit instance. We can make default instance by implicit instance in companion object.

Above code


print(int) // Type: Int - 1
print(hello) // Type: Hello - Hello(ktz)

Function ‘print’ is passed default implicit instance in companion object


implicit val IntPrinter: Printer[Int] = new Printer[Int] {
  override def print(value: Int): String = s"Type: Int - $value"
}

implicit val PersonPrinter: Printer[Hello] = new Printer[Hello] {
  override def print(value: Hello): String = s"Type: Hello - $value"
}

But next ‘print(int)’ is passed ‘injectedIntPrinter’. Like this, we can inject any instance by importing some instance.

Tagged Type in Scala

Human always makes mistake. Also, software-engineer makes mistake. Guess there are some codes like this.

Two function calls


getHashCode(userId, deviceUuid, carSerialId) // Right Answer = res0: String = 13-1236-12348

getHashCode(deviceUuid, userId, carSerialId) // Wrong Answer = res1: String = 1235-14-12348

both compile. But the first one is a right answer, but secondly is wrong. Then What is a good way to help myself not make a mistake? At first thinking, It is very good way to make case class.

This way, we can reduce mistakes. But we lost many things. First, we have to write many boilerplate. We have to make case class every Id and other Long types. And we lost that Id is the Long type. So, if we want to use userId or carSerialId or DeviceUuid, we have to call member. And cannot assign to Long type variable! To solve these problems, we can use Tagged Type.

Tagged Type

Tagged Type is tagging to some type A and Define as a subtype of A. For example, if we want to define userId as a subtype of Long, we can declare UserId as a Tagged Type of Long. Here is following an example.

Now, UserId, DeviceUuid, CarSerialId is Subtype of Long. We can use this type as Long type. For example, add Number to UserId by just using ‘+’ operator. or can assign to Long. But, It is UserId Type, DeviceUuid cannot be passed as a parameter in UserId Position.

Tagged Type Eraser

Sometimes you want to override function by tagged type like this.

But if you override like this, Compiler says that ‘Not Compiled because tagged type erased after compile’. Yes, Tagged Type is erased after compile time. Then how can we solve the problem? You can solve this by using Either.

Of course, you can use Coproduct in shapeless

By this way, You can keep self from make mistake.

Play with Lagom (0) – What is (Not Completed)

As you can see obove, it’s not completed. As I study Lagom, I can say more about these topic. And fill it more!

These days, Internet Technology is growing up and there occur many buzzwords around. ‘Reactive’, ‘Micro Service’, ‘Event Sourcing’, ‘CQRS’… Many so called ‘Progressive Company’ cry out these buzzwords. And I also want to know about this. After moving to Kakao, I usually use Play, Akka, Finatra.(Personally, Finatra is a very good framework to make a productive micro server.) And looking forward to finding a very good solution to manage micro services in our service. And these series is a very short journey about Lagom.(Because I just want to taste Lagom. Of course, if I decide that it is the real solution to our service, it will be long journey 🙂 )

What is Reactive?

One of the hard waves around here is ‘Reactive’. You can read and sign to ‘Reactive Manifesto’ in here. Ok, after sign to Manifesto, Let’s summary what is Reactive System.

  • Reactive is Responsive. Service has to be responsive at all time.
  • Reactive is Resilience. Service has to be responsive at to time of failure occur.
    • It means that, if an error occurs, the error is isolated and not affect to other component or service.
  • Reactive is Elastic. Service has to be scaled elastically when system load.
    • It means that, if the service loaded, for example, Akka system assign more thread to the loaded actor.
  • Reactive is Message Driven. Reactive call remote system by targeting message.

Reactive Programming vs Reactive System.

This topic is not related to the main subject. But it is very important to distinguish between Reactive Programming and Reactive System. It is referenced from this article. It’s just summary about this article.

The reactive system is the way to build service. Above description about reactive is about Reactive system. Reactive system passes messages to the target. By message passing and mail box, we can assign more resources to some component(or micro server). We can scale horizontally or vertically.

The Reactive Programming is the way to handle events. We catch events from event stream and handle by chaining functions. We catch event compare to Reactive System which operated by message passing. Reactive programming is usually async,(In fact, have to be async) we can assign to another job which needs computing resource.

Reactive Programming is fit to Reactive System. Words are very awkward, But it means very natural. Reactive Programming is tightly coupled between Event and computing by chaining. And this makes less resilience. And Reactive system is key to resolve this problem.

What is CQRS?

CQRS is very simple. In service, we separate Command (Write) and Query. (Read) Martin Fowler describes it very well in this article.

it’s like read replica in SQL, Application request writes and read in different API or server.

cqrs
Martin Fowler – CQRS

As you can see, Service Interface(API Gateway) request read to Query Model, and write to Command Model.

What is Event Sourcing?

Event Sourcing is making event as a sequence of data. Let’s think like this. We make some data ‘a’ to 3 by adding 1 and 2. In common architecture, We only store that a is 3. But in Event Sourcing, we store that

  1. ‘add 1’ event occur,
  2. ‘add 2’ event occur.

So, we can get that a is 3. It means that we can get system service when a is 1 or a is 0. This can restore the service any time, any status.

Scala Implicitly

As I study about Scala, I open see implicitly. But, every time I see this, I just ignore just thinking like “It’s syntactic sugar of implicit”. This time, I write this post to really under stand this.

In Scala, implicitly is just this.


def implicitly[T](implicit e: T) = e

That’s all! implicitly is a function just get implicit e instance and return e. Then, what is an advantage of using implicitly?

Example 1

You can use implicit like this.

implicit val optionInt: Option[Int] = Some(1)
implicit val optionBoolean: Option[Boolean] = Some(true)

def getImplicitInt(implicit oInt: Option[Int]): Int = oInt.get
def getImplicitBoolean(implicit oBoolean: Option[Boolean]): Boolean = oBoolean.get

getImplicitInt
// res0: Int = 1
getImplicitBoolean
// res1: Boolean = true

Now, you can use just same by using implicitly


implicit val optionInt: Option[Int] = Some(1)
implicit val optionBoolean: Option[Boolean] = Some(true)

def getImplicitlyA[A: Option]: A = implicitly[Option[A]].get

getImplicitlyA[Int]
// res2: Int = 1
getImplicitlyA[Boolean]
// res3: Boolean = true

Implicitly can use in such situation.

Implicitly can use when you want to get implicit instance which have 1 type parameter.

As you can see above, by using ‘Implicitly[Option[Int]]’, you can get implicit instance ‘Option[Int]’. It has 1 type parameter Int.

Example 2

Then how about implicitly for the type which has no type parameter? For example, implicit can do like this.


implicit val string: String = "Hello"
implicit val boolean: Boolean = true
implicit val int: Int = 0

def getImplicitT[T](implicit t: T): T = t

getImplicitT[Int]
// res0: Int = 0

getImplicitT[String]
// res1: String = Hello

getImplicitT[Boolean]
// res2: Boolean = true

But, if you do the same with implicitly, errors occur.


def getImplicitlyT[T]: T = implicitly[T]

getImplicitlyT[Int]
getImplicitlyT[String]
getImplicitlyT[Boolean]

// compile error: not enough arguments for method implicitly: (implicit e: T)T. Unspecified value parameter e. def getImplicitlyT[T]: T = implicitly[T] ^

Yes, because I mentioned above, implicitly can be used the type which has a type parameter. Then how can we use implicitly in this situation? The answer is ‘make Id type’.


type Id[A] = A

def getImplicitlyT[T : Id]: T = implicitly[Id[T]]

getImplicitlyT[Int]
// res3: Int = 0

getImplicitlyT[String]
// res4: String = Hello

getImplicitlyT[Boolean]
// res5: Boolean = true

I make new type Id which has one type parameter. And Id[A] is same as A. So, you can use just like implicit

Context bound

Right side of function ‘getImplicitlyAt’ has type parameter(A : Option). It looks like type bound. But it is called Context bound. Type bound is ‘<:’.You can see type bound in Scala School. In context bound, it doesn’t mean that A is Option. Instead, A will be a type passing to Option.

Monitoring Micro Service With Zipkin

This post is recommended to read with an example. You can clone example in my Github – https://github.com/ktz-boilerplate/finatra-zipkin-example.

 

Micro Service is becoming trendy technology. But there are many things to consider during migrate our service to micro service. One of them is latency. We have to monitor each micro service and find out which service is a bottleneck and why does that happen. And Zipkin can be a reasonable answer.

Zipkin

Zipkin is monitoring solution about latency. It traces each micro service and calculates latencies so that we can trace which micro take lots of time and handle it. This solution is base on Google Dapper.

Example

Start Zipkin on Docker

First, we will test Zipkin solution on Docker. So, if you don’t have Docker in local, please install it. You can see how to install in Docker Documentation.

Now, let’s start Zipkin on Docker.

$ docker run -d -p 9411:9411 -p 9410:9410 -e SCRIBE_ENABLED=true openzipkin/zipkin

9411 port is Zipkin web UI and 9410 port is scribe port. And you have to enable scribe port to accept Thrift by SCRIBE_ENABLED=true. If you up Zipkin on Docker, then you can see web UI in here.

Zipkin Twitter Flag

Before start servers, Let’s look around for twitter flags related to Zipkin. If your server is not compatible to this flags, you have to version up Finatra.

-http.name: HTTP Server name which displayed in Zipkin.

-thrift.name: Thrift Server name which displayed in Zipkin.

-com.twitter.finagle.zipkin.host: Zipkin host and port number to pass trace span(data).

-com.twitter.finagle.zipkin.initialSampleRate: Sample rate. from 0 to 1.0. 1.0 is that trace all request and send to Zipkin.

Start Finatra HTTP Server

Now, go to project directory. You can see 5 sub-projects. One of them is Finatra HTTP-Server. Let’s start HTTP Server first.

$ sbt 'project http-server' "run -http.name=http-server -com.twitter.finagle.zipkin.host=localhost:9410 -com.twitter.finagle.zipkin.initialSampleRate=1"

Start Finatra Thrift Server

Now, let’s start another server. It’s Thrift Server

$ sbt 'project thrift-server' "-thrift.name=thrift-server -com.twitter.finagle.zipkin.host=localhost:9410 -com.twitter.finagle.zipkin.initialSampleRate=1"

Query Something.

Now connect to HTTP-Server and Query Something. There are some URL you can test.

get

  • /users
  • /user/:userId
  • /user/car/
  • /user/:userId/car

post

  • /user

For example, if you want to get all users. You can get by call http://localhost:8080/users

After some queries, you can see tracing status in Zipkin Web UI like this.

zipkinui1.png

The default setting is sort by longest latency first.

 

You can also see dependency like this.

dependency.png