OSB performance issue :

Issue details :

During a high load test all OSB instances become out of order.

Diagnostic :

Take several thread dump during the benchmark, and analyze what the most threads are doing.

In our case : all Thread dump showed a lot of thread waiting for response on ServiceCallout action. And generally many STUCK threads are detected.
OSB Threading model :
 
The threading model is different between the service callout action and the route action. 

Route action use the benefits of the WLS AsyncResponseHandler callback and release the thread when after sending the request to the endpoint service. When the response is back a new thread is created. This behavior optimise the performance during high load.

Unfortunately the service callout action works differently. As we can see in the screenshot, the threads are blocked waiting for the response.
Because of this behavior, during the high load test a lot of threads (several hundreds) were waiting on service callout response.
At this step, the question was why weblogic is out of order after the bench and the threads were not released ?

Weblogic Workmanager :
By default all the OSB ressources (Proxy Services and Business Services) are instantiated with the default workmanager of Weblogic. This default workmanager is also used to instantiate kernel thread like for example socket muxer threads.

Because Weblogic have some threads number restrictions using the Execute Queue weblogic.kernel.Default (more or less 400 threads per instance) the default workmanager won’t instantiate more threads and it block the Weblogic kernel to work properly.
 
Only a reboot of all the instances can make the service working again.
In order to avoid threads starvation, we need to create a new custom workmanager and assign all of the OSB ressources (proxy services and business services) to this new custom workmanager using the dispatch policy parameter.
After applying this, the load won’t impact the weblogic kernel threads.

Correction steps :

1 – Create a custom workmanager in the weblogic domain
2 – Assign this custom workmanager to all the proxy and business service with the dispatch policy parameter


In some cases it would be preferable to create different custom workmanager to take all the workmanager benefits like min and max threads constraint and some priority parameters.

2 Comments on “OSB performance issue :

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *