Packages

final class TaskCircuitBreaker extends AnyRef

The TaskCircuitBreaker is used to provide stability and prevent cascading failures in distributed systems.

Purpose

As an example, we have a web application interacting with a remote third party web service. Let's say the third party has oversold their capacity and their database melts down under load. Assume that the database fails in such a way that it takes a very long time to hand back an error to the third party web service. This in turn makes calls fail after a long period of time. Back to our web application, the users have noticed that their form submissions take much longer seeming to hang. Well the users do what they know to do which is use the refresh button, adding more requests to their already running requests. This eventually causes the failure of the web application due to resource exhaustion. This will affect all users, even those who are not using functionality dependent on this third party web service.

Introducing circuit breakers on the web service call would cause the requests to begin to fail-fast, letting the user know that something is wrong and that they need not refresh their request. This also confines the failure behavior to only those users that are using functionality dependent on the third party, other users are no longer affected as there is no resource exhaustion. Circuit breakers can also allow savvy developers to mark portions of the site that use the functionality unavailable, or perhaps show some cached content as appropriate while the breaker is open.

How It Works

The circuit breaker models a concurrent state machine that can be in any of these 3 states:

  1. Closed: During normal operations or when the TaskCircuitBreaker starts
    • Exceptions increment the failures counter
    • Successes reset the failure count to zero
    • When the failures counter reaches the maxFailures count, the breaker is tripped into Open state
  2. Open: The circuit breaker rejects all tasks with an ExecutionRejectedException
    • all tasks fail fast with ExecutionRejectedException
    • after the configured resetTimeout, the circuit breaker enters a HalfOpen state, allowing one task to go through for testing the connection
  3. HalfOpen: The circuit breaker has already allowed a task to go through, as a reset attempt, in order to test the connection
    • The first task when Open has expired is allowed through without failing fast, just before the circuit breaker is evolved into the HalfOpen state
    • All tasks attempted in HalfOpen fail-fast with an exception just as in Open state
    • If that task attempt succeeds, the breaker is reset back to the Closed state, with the resetTimeout and the failures count also reset to initial values
    • If the first call fails, the breaker is tripped again into the Open state (the resetTimeout is multiplied by the exponential backoff factor)

Usage

import monix.eval._
import scala.concurrent.duration._

val circuitBreaker = TaskCircuitBreaker(
  maxFailures = 5,
  resetTimeout = 10.seconds
)

//...
val problematic = Task {
  val nr = util.Random.nextInt()
  if (nr % 2 == 0) nr else
    throw new RuntimeException("dummy")
}

val task = circuitBreaker.protect(problematic)

When attempting to close the circuit breaker and resume normal operations, we can also apply an exponential backoff for repeated failed attempts, like so:

val circuitBreaker = TaskCircuitBreaker(
  maxFailures = 5,
  resetTimeout = 10.seconds,
  exponentialBackoffFactor = 2,
  maxResetTimeout = 10.minutes
)

In this sample we attempt to reconnect after 10 seconds, then after 20, 40 and so on, a delay that keeps increasing up to a configurable maximum of 10 minutes.

Credits

This Monix data type was inspired by the availability of Akka's Circuit Breaker.

Source
TaskCircuitBreaker.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TaskCircuitBreaker
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. def doOnClosed(callback: Task[Unit]): TaskCircuitBreaker

    Returns a new circuit breaker that wraps the state of the source and that will fire the given callback upon the circuit breaker transitioning to the Closed state.

    Returns a new circuit breaker that wraps the state of the source and that will fire the given callback upon the circuit breaker transitioning to the Closed state.

    Useful for gathering stats.

    NOTE: calling this method multiple times will create a circuit breaker that will call multiple callbacks, thus the callback given is cumulative with other specified callbacks.

    callback

    is to be executed when the state evolves into Closed

    returns

    a new circuit breaker wrapping the state of the source

  7. def doOnHalfOpen(callback: Task[Unit]): TaskCircuitBreaker

    Returns a new circuit breaker that wraps the state of the source and that will fire the given callback upon the circuit breaker transitioning to the HalfOpen state.

    Returns a new circuit breaker that wraps the state of the source and that will fire the given callback upon the circuit breaker transitioning to the HalfOpen state.

    Useful for gathering stats.

    NOTE: calling this method multiple times will create a circuit breaker that will call multiple callbacks, thus the callback given is cumulative with other specified callbacks.

    callback

    is to be executed when the state evolves into HalfOpen

    returns

    a new circuit breaker wrapping the state of the source

  8. def doOnOpen(callback: Task[Unit]): TaskCircuitBreaker

    Returns a new circuit breaker that wraps the state of the source and that will fire the given callback upon the circuit breaker transitioning to the Open state.

    Returns a new circuit breaker that wraps the state of the source and that will fire the given callback upon the circuit breaker transitioning to the Open state.

    Useful for gathering stats.

    NOTE: calling this method multiple times will create a circuit breaker that will call multiple callbacks, thus the callback given is cumulative with other specified callbacks.

    callback

    is to be executed when the state evolves into Open

    returns

    a new circuit breaker wrapping the state of the source

  9. def doOnRejectedTask(callback: Task[Unit]): TaskCircuitBreaker

    Returns a new circuit breaker that wraps the state of the source and that upon a task being rejected will execute the given callback.

    Returns a new circuit breaker that wraps the state of the source and that upon a task being rejected will execute the given callback.

    Useful for gathering stats.

    NOTE: calling this method multiple times will create a circuit breaker that will call multiple callbacks, thus the callback given is cumulative with other specified callbacks.

    callback

    is to be executed when tasks get rejected

    returns

    a new circuit breaker wrapping the state of the source

  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  12. val exponentialBackoffFactor: Double

    A factor to use for resetting the resetTimeout when in the HalfOpen state, in case the attempt for Close fails.

  13. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. val maxFailures: Int

    The maximum count for allowed failures before opening the circuit breaker.

  18. val maxResetTimeout: Duration

    The maximum timespan the circuit breaker is allowed to use as a resetTimeout when applying the exponentialBackoffFactor.

  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
  22. def protect[A](task: Task[A]): Task[A]

    Returns a new Task that upon execution will execute the given task, but with the protection of this circuit breaker.

  23. val resetTimeout: FiniteDuration

    The timespan to wait in the Open state before attempting a close of the circuit breaker (but without the backoff factor applied).

    The timespan to wait in the Open state before attempting a close of the circuit breaker (but without the backoff factor applied).

    If we have a specified exponentialBackoffFactor then the actual reset timeout applied will be this value multiplied repeatedly with that factor, a value that can be found by querying the state.

  24. def state: State

    Returns the current TaskCircuitBreaker.State, meant for debugging purposes.

  25. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  26. def toString(): String
    Definition Classes
    AnyRef → Any
  27. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped