logtail: always zstd compress with FastestCompression and LowMemory (#11583)

This is based on empirical testing using actual logs data.

FastestCompression only incurs a marginal <1% compression ratio hit
for a 2.25x reduction in memory use for small payloads
(which are common if log uploads happen at a decently high frequency).
The memory savings for large payloads is much lower
(less than 1.1x reduction).

LowMemory only incurs a marginal <5% hit on performance
for a 1.6-2.0x reduction in memory use for small or large payloads.

The memory gains for both settings justifies the loss of benefits,
which are arguably minimal.

tailscale/corp#18514

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This commit is contained in:
Joe Tsai 2024-04-01 18:12:09 -07:00 committed by GitHub
parent 7558a1d594
commit e2586bc674
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 3 deletions

View File

@ -384,11 +384,9 @@ func (l *Logger) uploading(ctx context.Context) {
switch {
case l.zstdEncoder != nil:
zbody = l.zstdEncoder.EncodeAll(body, nil)
case l.lowMem:
default:
zbody = zstdframe.AppendEncode(nil, body,
zstdframe.FastestCompression, zstdframe.LowMemory(true))
default:
zbody = zstdframe.AppendEncode(nil, body)
}
// Only send it compressed if the bandwidth savings are sufficient.