.Net Core 3.0 Kestrel Poor Throughput with Newtonsoft.Json

TL;DR

Using newtonsoft.Json as output serializer API with .Net 3.0 may hurt application throughput.

Tell Me More

We recently upgraded all applications running on .net core to .net core 3.0. The applications are containerized and some of them are running in the cloud. There are couple of reasons for upgrading:

1. Better support for containerization (CPU/memory limit)
2. Thread starvation with .net 2.2

All the applications were running smoothly since the upgrade. Those thread starvation alerts that we saw with .net 2.2 (Heartbeat took longer than “00:00:01") just disappeared from the log. All was well except one application which started to show signs of deteriorating performance. The app was deployed in the cloud and was handling high volume of traffic. We were getting pretty consistent response time with 2.2 and almost zero timeouts. But since upgrade to 3.0, the response started to get very spiky with decent amount of timeouts. Things got even worse during peak traffic when some of the host node started to reboot. After some diagnosis, we found that the app was doing large volume disk writes that saturated I/O. It resulted in kernel panic.

Why Is That ?

Thanks to the amazingly helpful .net developer forum on github, we were able to get to the bottom of the issue pretty quickly.

.net 2.2 or prior used newtonsoft.json for json serialization which would write synchronously to output buffer. This approach involves blocking IO that could cause thread starvation for larger payload and poor throughput overall.

.net 3.0 and later went fully asynchronous to mitigate aforementioned limitations. But since newtonsoft.json doesn’t support async execution, output buffering is used as a workaround. For payload size 32kb or less, memory is used for buffering. for larger payload, it uses disk for buffering.

Is There Any Workaround?

There are couple of workarounds suggested by dotnet core development team.

Replace Newtonsoft.Json

Preferred solution to the problem is to replace newtonsoft.json with an async serializer such as System.Text.Json(STJ).

Disable Output Buffering

Another option is to enable synchronous IO and disable output buffering. This is pretty much same as .NET core 2.2 and like .NET Core 2.2, this may lead to potential thread starvation. To do this:

1. Disable output buffering in start.cs
services.AddControllers(options =>{
options.SuppressOutputFormatterBuffering = true;
});

2. Enable synchronous IO
WebHost.CreateDefaultBuilder(args)
.ConfigureKestrel(o > {
o.AllowSynchronousIO = true;
})

Use Async Serializer for Response Objects

If replacing newtonsoft.json seems too overwhelming and enabling synchronous IO not plausible, you can keep using newtonsoft.json everywhere else except output serialization.

You can read the full discussion on github

High disk IO volume after upgrading to 3.0 · Issue #17843 · aspnet/AspNetCore