Speedup tests execution in ScalaTest

Sometimes the duration of automated tests written with ScalaTests library can be reduced in times easly.

Here are the few steps I found to make it fast and easier.


  1. Maven: use the latest version of maven itself 3.5.x and scala-maven-plugin 3.4.x Be careful with other plugins aren't supporting parallel execution, if you see this message in build log, carefully review the plugins list

[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project *
[WARNING] * contains the following plugin(s) that have goals not marked *
[WARNING] * as @threadSafe to support parallel building. *

  2. Switch on parallel mode and disable JVM forking:

  Maven and sbt:

ScalaTest's normal approach for running suites of tests in parallel is to run different suites in parallel, but the tests of any one suite sequentially. This approach should provide sufficient distribution of the work load in most cases, but some suites may encapsulate multiple long-running tests. Such suites may dominate the execution time of the run. If so, mixing in this trait into just those suites will allow their long-running tests to run in parallel with each other, thereby helping to reduce the total time required to run an entire run.

  3. Analyse the shared state. Switching the tests into parallel mode sometimes introduces race conditions. If test methods are sharing mutable state in class fields. For example:
The best solution is usage of shared fixtures or test scopes. If you don't want to refactor your test in BigBang approach it's easier to extend the Suits/Specs with org.scalatest.OneInstancePerTest.

Trait that facilitates a style of testing in which each test is run in its own instance of the suite class to isolate each test from the side effects of the other tests in the suite. If you mix this trait into a Suite, you can initialize shared reassignable fixture variables as well as shared mutable fixture objects in the constructor of the class. Because each test will run in its own instance of the class, each test will get a fresh copy of the instance variables. This is the approach to test isolation taken, for example, by the JUnit framework.

  4. Reduce the number of heavy resources via re-sharing them. Sometimes there could be great performance boost when some state of tests is shared between the suits. For example embedded db instance - if it's started before each test and switched off after - it could require more time than test execution itself.
If you applied the forking policy from second recommendation to at most once - than all your tests are running in the same JVM and it's possible to share some class instances.
There are some cons with following to this approach - tests aren't independent anymore. Every test should be implemented with keeping in mind that resources are mutated in parallel.
For example Unit tests that counts the final number of rows in table - should be updated to check by the number of rows by condition. In meantime in before[Each/All] should not clear the whole state of shared resource but the parts are related to current test case only. Here is example how to share the embedded cassandra instance between the Specs:
  5. Migrate to async. In case of testing the async services/functionality instead of blocking the test for checking the results use *Async implementations for the Specs: AsyncFlatSpec, AsyncFunSpec etc. More details are here.

  6. Review your before/after[Each/All] methods. Sometimes the execution of those makes major influence into whole test run. If your close the resources - it could be not mandatory to do it - because of short running JVM process is spawned only for test run. If your delete all the data to clean up the state before the tests then think on how it's possible to isolate the tests - via UUID or unique table names etc, this is mandatory step for recommendation #3

  7. Tune the JVM params, test goal in maven is short duration Java process and these JVM args are representing this fact:

-XX:+TieredCompilation -XX:TieredStopAtLevel=1

Type level programming for Streams

Programming at this level is often considered an obscure art with little practical value, but it need not be so.

Conrad Parker (about Type-Level programming in Haskell)

 

There are a lot of domains where events are coming in stream-like way. The best one example are services based on http protocol.

The languages like Java with imperative programming legacy background are providing a sort of abstraction over stream nature of Http services - where stream is converted to branch-like structure and each leaf is handling in-variants for possible inputs. It's became popular to design systems in RMI way.

Regular handler for HTTP or RPC service is looking like:

It means that all the magic with routing HTTP requests to methods are provided by the frameworks.

Of course there is a hidden complexity behind this - like Filter, Handlers etc.

While Reactive and Functional Programming gain some popularity, streaming DSL's gain popularity in solving Flow handling complexity.

Those flows are well self explaining and easier to maintain/change in future.

The most of primitive domain problems can be easy solved via stream of events from Input to to Sink, like:
  In ~> Step1 ~> Step2 ~> StepN ~> Sink. 

Even if your DSL isn't sexy enough to look like math diagram, you can use some tools to visualise flow in nice ways (for example: travesty)

Of course there is always extra complexity like filtering, conditioning, throttling, backpressure etc. Stream like data handling is looking natural and mostly befit the functional programming, where every flow step is a function that accept In type and returns Out type:

  type Step1 = In1 => Out1

Piping the step functions is looking natural solution. Lest assume concrete input types: ShipmentsRequest, TrackRequest, PricingRequest and Ping:

The simplest way to handle those types is operate with Any Type:

Using Any makes possible to pipe (compose) the functions. Unfortunately compiler doesn't help us with any extra checks to make piping safe.
We can use best practices and operated with the marker trait:


Now compiler helps us to find if we have forgotten to handle Ping message:

It's still just a compiler warning - and it's not always possible to escalate it to the error - if you are not allowed to change compiler arguments. It's not always the case that all incoming message could extend some base trait.

There is other way to present input types - via union type. For example Haskell support declarations like this:


In Haskell it's evolved into Data Kinds language extension. Scala is planning to support this on language level as well in Dotty compiler as Union Types.
There is a fork of Scala - Typelevel Scala that aims to bring it early than Dotty. We will use Shapeless library for type level programming solution.
In Scala we can define our input type as a chain of embedded Either types:


Shapeless library implements some syntax sugar for that solution, here is the same code expressed via the Shapeless:

Than Step 1 can be defined as a Poly1 function determined for all unions (coproducts) types from In1:


As a result type of that step is ShipmentsResponse or TrackResponse or PricingResponse or Pong. The handling for all the cases is checked during the compilation time.

The full example for akka streams can look like:

scala.concurrent.Future leaks on timeouts.

Almost any business operation should have a reasonable deadline - if not completed before - it loses it value.
Imagine situation when we are building the service that should give a response in 500 milliseconds - if it's not ready we should return error.

Just to reduce the number of source code lines in example we will use Await.result - that allows to set the timeout for waiting.
If we define our heavy operation as:
We find out that despite service produces error on timeout - and we notify the users on that - we actually haven't canceled the heavy task itself. It keeps running despite the result isn't needed. Another simple example with akka-streams:
It's output is:
error: java.util.concurrent.TimeoutException: The stream has not been completed in 100 milliseconds.
I'm executed anyway

There is no way to get ride off resources leak except the changing the way how heavyOperation is working. For example if it's query to Cassandra - it should be regulated via timeouts, retry policy and other setting to limit the maximum execution time to expected duration.
Sometimes it's not that easy to predict and limit execution time as a result to avoid resource's leak. Akka streams proposes Back-pressure solution that actually isn't easy to apply for all the cases.
There is another way to cancel the jobs explicitly, for example Monix library implements Cancelable Of course cancelable is something that you have to manage explicitly - the way you would like the job to be cancelled. But the handling of timeouts is easy to use operation:


If we want to stop thread execution from our synthetic example we can use Task with cancelable behaviour:
- of course the real cancelation is sometimes hard to implement it correctly.
How do you handle the timeouts for long running tasks?

Testing Promise with Jasmine

There is a good library that provides syntax sugar for Promise validations via Jasmine jasmine-promise-matchers.  It comes hard to test complex result objects, example proposes to use jasmine.objectContaining from jasmine-matchers, but the easier way for the same is just to map promise into tested value and use toBeResolvedWith on a projection from pormise.

Ceremony Δ

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

The Zen of Python

...Scala's syntax is a bit more flexible than many people are used to..

underscore.io blog

When programming language gets the new features - it has to pay some price in complexity - but extra in ceremonies aspect. I call this Ceremony Delta, the learning curve for the language has some dependency on it - because of developer should study the implicit conventions.

Dynamically growing ecosystem isn't able to follow the bottom row closely for a long distance. Naturally it tends to jump away. It takes a lot of power to keep it on "The shortest way is the right one". Of course the language authors are allowed to review concepts and release the "brand new" language version.

Scala as a language with FP and OOP paradigms support was always hard to keep balance.
Unfortunately it's bring the requirement to invest into best practices.

It already has a big scope of implicit ceremonies that is growing.

Sometimes compiler helps us to find the problems:

eitherResult match { // [warn] match may not be exhaustive.  
    case Right(value) => assert(value == expected)
}

The example with matching only some part actually is the shortest one - especially with some context that it's a part of Unit test - and having failures for Left cases  is acceptable

This code still compiles but doesn't make sense:

val intList = List(1, 2)
intList.filter(_ == "4") 
// comparing values of types Int and String using `==' will always yield false

Closure is a shortest way to filter the list. As a solution there are different libraries implement === method that is type safe.

Here are example of "smell code" that is easy to detect for experienced developers - but compiler stays silent

// Declaring public variable that is expected to be injected // that is why it's initialized by null
@Inject var service: Service = _

Some(null) // ridiculous declaration 

Future.failed[T](new Exception) foreach doSideEffects // using foreach for side effects - but it's ignoring errors

Thread.sleep(1000) // Let the word wait for 1 second

val result = for {
  part1 <-callService1  // Future[T]  
  part2 <- callService2 // Future[T]
} yield (part1, part2)  // part2 starts execution after part1 has been completed// Tuple is used to combine the result

Using null from JVM sometimes is the shortest way, making side effects on Future in foreach callback is the easiest approach - but not the right one.

Sometimes it's not that easy to find the problem even for experienced developers and it's definitely of of the compiler's responsibility:

// despite method returns a Future - it hides the calling // of blocking code
def asyncCallService(params: Parameter): Future[T] = {
  val extraParam = callSomeBlockingCode() // usingBlockingApi
  callService3(extraParam, params) // returns Future[T] 

To reduce the Ceremony Δ there are many domain specific libraries addressing boilerplate places. The most of the examples for the bad usages of scala.concurrent.Future are solved in ScalaZ Task, cats-effect, Monix etc libs, that still is increasing the learning curve for language but keeps Ceremony delta low. The example with making side-effects in map-flatmap can't be solved with libraries and probably requires language support. It leaves as best practices recommendation on use a pure functions in map/flatmap in cats-effect library - that is ceremony we have to follow.

triple equals conflict scalactic and cats

ScalaTest library depends on org.scalactic.Equalizer.
When cats library is used via import cats.implicits._ - the both are providing function ===.

Of course compiler won't be happy with multi implicit conversions:

Note that implicit conversions are not applicable because they are ambiguous:
 both method catsSyntaxEq in trait EqSyntax of type [A](a: A)(implicit evidence$1: cats.Eq[A])cats.syntax.EqOps[A]
 and method convertToEqualizer in trait TripleEquals of type [T](left: T)SomeUnitTest.this.Equalizer[T]
 are possible conversion functions from Int to ?

As a workaround I recommend to disable cats implementation - it's a bit worse than scalactic:

import cats.implicits.{ catsSyntaxEq => _, _ } // import all implicits except triple '=' 

akka-remote and akka-cluster 2.4.x "network" issues

We've faced interesting issues with cross node delivery time in akka-cluster environment.

Out network is easy support 1 Gbit/s traffic, when it reaches 200+ Mbit, the

actoreRef ! Message

to another cluster node took 30+ seconds.

Unfortunately akka doesn't support any instruments for latency monitoring and we started from network blaming.

Using the Wireshark we found that in some moments of time our system was producing up to 30 messages a ms. On TCP level delivery time was <=1 ms, but on akka level it sometimes jumped to > 30 seconds.

After some investigation we found the root case:

akka does serialisation in Single Thread, that means that it could be easy become bottleneck, for multi core servers, when you can produce more messages - then 1 Thread could serialise.

1. Workaround: try to migrate to akka 2.5.0 and use inbound and outbound lanes feature. Be aware that documentation in akka starts from # WARNING: This feature is not supported yet. Don't use other value than 1. Use it on your own risk.
2. How to diagnose the issue - we are using small hack to measure the trip time. In our serializator used for akka-remote we inject serialised time and de-serialised time - the difference is travel time between nodes.