Grpc transfer big data, one unary call is slower than streaming.

[LiBingtao]

I'm trying to transfer a big data between two services using grpc.

The data size is about 23M and is composed by 42 big List.

Then I test the performance using one unary call vs server side streaming(stream one list at a time).

The unary call takes 276.59 ms.

The streaming call takes 126.64 ms.

But if I change the data to contains 1000 small list, each list just have one number, the streaming call is much more slower than unary call.

Is the result normal? And Why?

Here is the server side code:

public override Task<MemDtoToWbs> GetLargeMEM(Empty request, ServerCallContext context)
{
    return Task.FromResult(MemData.GrpcLargeMem);
}
public override async Task StreamLargeMem(Empty request, IServerStreamWriter<LogDtoToWbs> responseStream, ServerCallContext context)
{
    foreach (var log in MemData.GrpcLargeMem.Logs)
    {
         await responseStream.WriteAsync(log);
    }
}

I use the .net core 3.1 and grpc nuget package 2.32.0.
Run test in aks cluster

Thanks.

[JamesNK]

Breaking up a large 23M array into smaller arrays is bringing a performance benefit but it is hard to know why. It could be on the client or the server, it could be in Protobuf or in HTTP/2.

I haven't investigated payload size impact in detail. You could use a .NET CPU or memory profiler to find out more if you're curious.

[LiBingtao]

I used .NET CPU and memory profiler to inspect what could caused the performance difference.
But found nothing.

Then I test in localhost, use difference array item size. The TCP segment size is 64K.
I found that when small messages is too small, like 32K, it only have 6.5K payload length in each TCP segment. But if message is much bigger, it can use all the 64K. So the overhead for sending each small message can add up and things can end up slower.

So it's reasonable to me that streaming is faster if message is big.
Because sending data in server side and process data in client side is running in parallel.