In this article we will look at very simple basic example of Resilience4j circuit breaker & look at runtime behavior & all possible state changes of circuit breaker. Here is the maven dependency for resilience4j-circuitbreaker required for this example.
Circuit Breaker Concept
If service call is giving errors more than expected, stop calling that service for certain wait time & take some alternate recovery path during that time. After wait is over, check if service stopped giving error with limited calls & then resume calling service else wait again. This avoids unnecessary network traffic & also unwanted load on service when its already known to throw errors that time.
Example in this article
Circuit Breaker Configurations: (Default) Check status of last 100 calls. If 50% of those 100 calls ended up in exception, then open circuit i.e. block any further calls for next 60 seconds. After 60 seconds over, allow 10 calls to check if calls succeeding. If success then close circuit back i.e. allow calls or again open it & wait 60 seconds.
- Create mock service which returns success for first 100 calls & then throws BadProcessingException for rest of the calls.
- We will call this service with circuit breaker default configurations as give above.
- For test purpose we will make 1 call per second.
- We will observe how circuit breaker behaves in success & failures. As calls are happening per second, we can also see how wait duration works after open state.
Mock service
This is a mock service as explained above.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
package com.itsallbinary.resilience4j.tutorial; /* * Test service */ class ExternalService { private int counter = 0; public String callService() { counter++; // First 100 calls will be success & then everything failure. if (counter < 100) { return "SUCCESS"; } else { throw new BadProcessingException("Something bad happened."); } } } class BadProcessingException extends RuntimeException { public BadProcessingException(String message) { super(message); } } |
Circuit Breaker in action
- External service call is wrapped / decorated with circuit breaker.
- Circuit breaker has default configurations as documented in code.
- Code will call external service 1 per second.
- Code will print circuit breaker metrics after each call so that we can observer behavior in output.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
package com.itsallbinary.resilience4j.tutorial; import io.github.resilience4j.circuitbreaker.CircuitBreaker; public class CircuitBreakerBasics { public static void main(String[] args) { // Service client object to call service. ExternalService externalService = new ExternalService(); /* * Circuit breaker with default configurations. Below are defaults. * * DEFAULT_SLIDING_WINDOW_TYPE = SlidingWindowType.COUNT_BASED; --> State change * will happen based on count of failed calls. * * DEFAULT_RECORD_EXCEPTION_PREDICATE --> All exception recorded as failures * * DEFAULT_MINIMUM_NUMBER_OF_CALLS = 100 DEFAULT_SLIDING_WINDOW_SIZE = 100; --> * Minimum number of calls to record before decision to open. * * DEFAULT_FAILURE_RATE_THRESHOLD = 50% --> Out of recorded calls, if 50% calls * are failure, then open circuit breaker * * DEFAULT_WAIT_DURATION_IN_OPEN_STATE = 60 Seconds --> Wait this much time in * open state before moving to half open. * * DEFAULT_PERMITTED_CALLS_IN_HALF_OPEN_STATE = 10 --> COunt these many calls in * half open to decide if circuit breaker can be completely open. * * * */ CircuitBreaker circuitBreaker = CircuitBreaker.ofDefaults("externalService"); for (int i = 0; i < 500; i++) { System.out.println("counter = " + (i + 1)); try { // Call service every 1 second. Thread.sleep(1000); // Decorate service call in circuit breaker String status = circuitBreaker.executeSupplier(() -> externalService.callService()); System.out.println(status); } catch (Exception e) { e.printStackTrace(); } finally { // Print important metric stats to observe behavior of circuit breaker. System.out.println("Successful call count: " + circuitBreaker.getMetrics().getNumberOfSuccessfulCalls() + " | Failed call count: " + circuitBreaker.getMetrics().getNumberOfFailedCalls() + " | Failure rate %:" + circuitBreaker.getMetrics().getFailureRate() + " | State: " + circuitBreaker.getState()); } } } } |
Output (Circuit breaker state changes in action)
We can see behavior of circuit breaker in output. Several logs in output have been skipped to give better visual understanding.
Key behavior to look for in output
- Exception from service (Call counter = 100)
- External service starts throwing BadProcessingException as per mock service.
- Failed call count in circuit breaker will start increasing.
- Ring buffer (size 100) is also full, so failure rate will start calculating.
- Ring buffer slides (Call counter = 101)
- Since success/failure count is stored in ring/circular buffer of size 100, from 101 onwards, success count starts going down & failure count goes up i.e. total counted calls (success + failure) will always be 100 i.e. same as “sliding window size” or “ring buffer size”
- Failure rate threshold cross, hence CLOSED to OPEN (Call counter = 149)
- Current failure rate reaches 50%. “Default failure rate threshold” configured is also 50% so circuit breaker opens.
- Calls to service stops (Call counter = 150)
- Here onwards all calls to external service will receive io.github.resilience4j.circuitbreaker.CallNotPermittedException
- Actual call to external service won’t happen.
- Wait time in open is over, hence OPEN to HALF_OPEN (Call counter = 209)
- 60 seconds are over since circuit breaker is open.
- “Wait time in open state” configured is also 60 seconds, so circuit breaker goes to HALF_OPEN state. Calls to external service start happening.
- Still failing, hence HALF_OPEN to OPEN (Call counter = 218)
- 10 calls made since HALF_OPEN state & “Permitted calls in half open state” are configured to 10.
- So now again failure rate is calculated which comes to 100% which is more than failure rate threshold i.e. 50%. So circuit breaker opens again.
- Wait time in open is over, hence OPEN to HALF_OPEN (Call counter = 278)
- 60 seconds are over since last circuit breaker is open.
- “Wait time in open state” is also 60 seconds, so circuit breaker goes to HALF_OPEN state. Calls to external service start happening.
- Now calls successful, hence HALF_OPEN to CLOSED (Call counter = 287)
- 10 calls made since HALF_OPEN state & “Permitted calls in half open state” are configured to 10.
- So now again failure rate is calculated which comes to 0% since all calls are successful now. So circuit breaker closes & calls to external service starts happening.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 |
>> Call counter = 1 >> ExternalService received a call. Received call counter = 1 >> Call Status: SUCCESS >> Successful call count: 1 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 2 >> ExternalService received a call. Received call counter = 2 >> Call Status: SUCCESS >> Successful call count: 2 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 3 >> ExternalService received a call. Received call counter = 3 >> Call Status: SUCCESS >> Successful call count: 3 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 4 >> ExternalService received a call. Received call counter = 4 >> Call Status: SUCCESS >> Successful call count: 4 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED . <skipped similar trend logs> . >> Call counter = 99 >> ExternalService received a call. Received call counter = 99 >> Call Status: SUCCESS >> Successful call count: 99 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 100 >> ExternalService received a call. Received call counter = 100 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. >> Successful call count: 99 | Failed call count: 1 | Failure rate %:1.0 | State: CLOSED >> Call counter = 101 at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> ExternalService received a call. Received call counter = 101 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. >> Successful call count: 98 | Failed call count: 2 | Failure rate %:2.0 | State: CLOSED >> Call counter = 102 at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> ExternalService received a call. Received call counter = 102 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. >> Successful call count: 97 | Failed call count: 3 | Failure rate %:3.0 | State: CLOSED . <skipped similar trend logs> . >> Call counter = 148 >> ExternalService received a call. Received call counter = 148 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 51 | Failed call count: 49 | Failure rate %:49.0 | State: CLOSED >> Call counter = 149 >> ExternalService received a call. Received call counter = 149 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 50 | Failed call count: 50 | Failure rate %:50.0 | State: OPEN >> Call counter = 150 io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'externalService' is OPEN and does not permit further calls >> Successful call count: 50 | Failed call count: 50 | Failure rate %:50.0 | State: OPEN >> Call counter = 151 at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:37) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$OpenState.acquirePermission(CircuitBreakerStateMachine.java:488) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:146) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:724) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'externalService' is OPEN and does not permit further calls at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:37) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$OpenState.acquirePermission(CircuitBreakerStateMachine.java:488) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:146) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:724) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 50 | Failed call count: 50 | Failure rate %:50.0 | State: OPEN . <skipped similar trend logs> . >> Call counter = 208 io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'externalService' is OPEN and does not permit further calls at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:37) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$OpenState.acquirePermission(CircuitBreakerStateMachine.java:488) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:146) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:724) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 50 | Failed call count: 50 | Failure rate %:50.0 | State: OPEN >> Call counter = 209 >> ExternalService received a call. Received call counter = 150 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 0 | Failed call count: 1 | Failure rate %:-1.0 | State: HALF_OPEN >> Call counter = 210 >> ExternalService received a call. Received call counter = 151 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 0 | Failed call count: 2 | Failure rate %:-1.0 | State: HALF_OPEN . <skipped similar trend logs> . >> Call counter = 217 >> ExternalService received a call. Received call counter = 158 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. >> Successful call count: 0 | Failed call count: 9 | Failure rate %:-1.0 | State: HALF_OPEN >> Call counter = 218 at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> ExternalService received a call. Received call counter = 159 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 0 | Failed call count: 10 | Failure rate %:100.0 | State: OPEN >> Call counter = 219 io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'externalService' is OPEN and does not permit further calls at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:37) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$OpenState.acquirePermission(CircuitBreakerStateMachine.java:488) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:146) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:724) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 0 | Failed call count: 10 | Failure rate %:100.0 | State: OPEN . <skipped similar trend logs> . >> Call counter = 277 io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'externalService' is OPEN and does not permit further calls at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:37) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$OpenState.acquirePermission(CircuitBreakerStateMachine.java:488) at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:146) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:724) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 0 | Failed call count: 10 | Failure rate %:100.0 | State: OPEN >> Call counter = 278 >> ExternalService received a call. Received call counter = 160 com.itsallbinary.resilience4j.tutorial.BadProcessingException: Something bad happened. at com.itsallbinary.resilience4j.tutorial.ExternalService.callService(ExternalService.java:18) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.lambda$0(CircuitBreakerBasics.java:42) at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateSupplier$4(CircuitBreaker.java:727) at io.github.resilience4j.circuitbreaker.CircuitBreaker.executeSupplier(CircuitBreaker.java:203) at com.itsallbinary.resilience4j.tutorial.CircuitBreakerBasics.main(CircuitBreakerBasics.java:42) >> Successful call count: 0 | Failed call count: 1 | Failure rate %:-1.0 | State: HALF_OPEN . <skipped similar trend logs> . >> Call counter = 286 >> ExternalService received a call. Received call counter = 168 >> Call Status: SUCCESS >> Successful call count: 8 | Failed call count: 1 | Failure rate %:-1.0 | State: HALF_OPEN >> Call counter = 287 >> ExternalService received a call. Received call counter = 169 >> Call Status: SUCCESS >> Successful call count: 0 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 288 >> ExternalService received a call. Received call counter = 170 >> Call Status: SUCCESS >> Successful call count: 1 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 289 >> ExternalService received a call. Received call counter = 171 >> Call Status: SUCCESS >> Successful call count: 2 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 290 >> ExternalService received a call. Received call counter = 172 >> Call Status: SUCCESS >> Successful call count: 3 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED >> Call counter = 291 >> ExternalService received a call. Received call counter = 173 >> Call Status: SUCCESS >> Successful call count: 4 | Failed call count: 0 | Failure rate %:-1.0 | State: CLOSED |
Further reading
Resilience4j | Expose Circuit Breaker Metrics to JMX using Micrometer | Simple example
Resilience4j Complete Tutorial | Basics with runtime behavior | Simple examples for beginners