SpringCloudGateway 3.1.4版本 Netty内存泄漏问题解决

2023-12-20 01:23:23

一、 产生的异常?

当时是服务器访问不到服务了，上去一看，无法申请资源OutOfDirectMemoryError了，内存级别的东西让人一阵头大，赶紧在线下模拟，

1. 减少分配的堆外内存，打开Netty的监测工具等有助于复现的配置和检查的信息

-XX:MaxDirectMemorySize=2M
-Dio.netty.leakDetection.acquireAndReleaseOnly=true
-Dio.netty.leakDetectionLevel=paranoid
-Dio.netty.leakDetection.targetRecords=50
-XX:-OmitStackTraceInFastThrow
-XX:NativeMemoryTracking=detail

Netty的内存泄漏检测分为四个级别：

  • DISABLED - 完成禁止检测内存泄漏，这个是不推荐。
  • SIMPLE - 如果buffer中出现1%的内存泄漏，打印错误日志，就是上面那样。但是日志是看不到详细在什么位置出现了内存泄漏。鉴于性能考虑，这个是在生产环境中默认使用。
  • ADVANCED - 如果buffer的1%出现内存泄漏，打印错误，并且输出详细的内存泄漏信息。
  • PARANOID - 这个和ADVANCED输出的内容是一样的，但它会对每一次请求的buffer做检查。很适用于调试和单元测试。

建议

1、为了今早的发现问题，强烈建议在开发调试过程中，把检测级别设为PARANOID，设置：

io.netty.leakDetection.level=PARANOID
2023-12-19 17:33:47.568 ERROR 19740 --- [tor-http-nio-54] reactor.netty.transport.ServerTransport  : [158851ed, L:/127.0.0.1:10101 - R:/127.0.0.1:58747] onUncaughtException(SimpleConnection{channel=[id: 0x158851ed, L:/127.0.0.1:10101 - R:/127.0.0.1:58747]})

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2048 byte(s) of direct memory (used: 2095623, max: 2097152)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:845) ~[netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:774) ~[netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:632) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:399) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140) ~[netty-buffer-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150) ~[netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.84.Final.jar:4.1.84.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.84.Final.jar:4.1.84.Final]
	at java.lang.Thread.run(Thread.java:750) [na:1.8.0_351]

二、 检查内存泄漏点位

由于有点多，还特意把检测的详细条数改到了50。

挨个查看是不是有自己写的代码或者熟悉的代码，看了一遍没认识的，完蛋这会麻烦了，各种找资料（百度），上官网（Netty的Issues · netty/netty · GitHub）（gateway的LEAK: ByteBuf.release() · Issue #26937 · spring-projects/spring-framework · GitHub）各种查，大部分都是版本问题但是都是老版本，我看我的都是新版的了咋还有，DEBUG后发现总是在请求进来时申请一次2048的，在后续的处理中涨涨降降最后就是这点没释放

2023-12-19 09:33:12.933 [reactor-http-nio-35] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
	io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:116)
	io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:271)
	io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
	io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:421)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931)
	io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895)
	io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:921)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:907)
	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:893)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531)
	io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
	io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:923)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:907)
	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:893)
	reactor.netty.channel.MonoSendMany$SendManyInner$AsyncFlush.run(MonoSendMany.java:789)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#2:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:962)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:526)
	io.netty.handler.codec.http.HttpObjectEncoder.writeOutList(HttpObjectEncoder.java:118)
	io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:110)
	io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#3:
	io.netty.handler.codec.http.HttpObjectEncoder.encodeByteBufContent(HttpObjectEncoder.java:405)
	io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:285)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:181)
	io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:203)
	io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:97)
	io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#4:
	io.netty.handler.codec.http.HttpObjectEncoder.encodeByteBufAndTrailers(HttpObjectEncoder.java:453)
	io.netty.handler.codec.http.HttpObjectEncoder.encodeByteBufContent(HttpObjectEncoder.java:403)
	io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:285)
	io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:181)
	io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:203)
	io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:97)
	io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#5:
	Hint: 'reactor.left.httpCodec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:962)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:423)
	reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#6:
	org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:93)
	org.springframework.cloud.gateway.support.ServerWebExchangeUtils$1.lambda$getBody$0(ServerWebExchangeUtils.java:375)
	reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:86)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:44)
	reactor.core.publisher.Flux.subscribe(Flux.java:8526)
	reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	reactor.core.publisher.Operators.complete(Operators.java:137)
	reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336)
	reactor.core.publisher.MonoSource.subscribe(MonoSource.java:69)
	reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)
	reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:677)
	reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:274)
	io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#7:
	org.springframework.core.io.buffer.NettyDataBuffer.release(NettyDataBuffer.java:326)
	org.springframework.core.io.buffer.DataBufferUtils.release(DataBufferUtils.java:521)
	org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:192)
	org.springframework.core.codec.StringDecoder.decode(StringDecoder.java:61)
	org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:109)
	org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:96)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1817)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
	reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:603)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#8:
	org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:93)
	org.springframework.cloud.gateway.support.ServerWebExchangeUtils$1.lambda$getBody$0(ServerWebExchangeUtils.java:375)
	reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:86)
	reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:44)
	reactor.core.publisher.Mono.subscribe(Mono.java:4455)
	reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
	reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
	reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
	reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:603)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#9:
	io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)
	io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:367)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#10:
	org.springframework.core.io.buffer.NettyDataBufferFactory.wrap(NettyDataBufferFactory.java:93)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
	reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#11:
	reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:185)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#12:
	Hint: [a2330415-3, L:/127.0.0.1:10101 - R:/127.0.0.1:52736] Receiver reactor.core.publisher.FluxMap$MapSubscriber will handle the message from this point
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:96)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
	reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:360)
	reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
	reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:598)
	reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#13:
	Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:96)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#14:
	Hint: 'reactor.left.httpTrafficHandler' will handle the message from this point.
	io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:96)
	io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:30)
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#15:
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#16:
	Hint: 'reactor.left.httpCodec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
#17:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
	io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:750)

三、 修复

别看上面一堆很长，关键的就是几个点：

Hint： 开头的

Spring的一些

如果有自己的代码记得释放就好，我的没有就需要对比试用版本和泄漏点是不是有使用没有释放，或者释放未成功。

第一次试着升级了gateway的版本但是无效，后来想单独升级会不会有冲突，回滚后一起升级的Springboot+SpringCloud，试运行后还是不行，查看依赖时发现Libraries中旧版Jar包引入在未删除，造成了冲突，删除后，Maven刷新，在运行一切就正常了。

原版本：

<spring-boot.version>2.6.13</spring-boot.version>
<spring-cloud-alibaba.version>2021.0.5.0</spring-cloud-alibaba.version>
<spring-cloud.version>2021.0.5</spring-cloud.version>

升级后版本：

<spring-boot.version>2.6.15</spring-boot.version>
<spring-cloud-alibaba.version>2021.0.5.0</spring-cloud-alibaba.version>
<spring-cloud.version>2021.0.8</spring-cloud.version>

推荐堆外内存监控程序和检查的一些资料可以参考：

spring gateway LEAK: ByteBuf.release() was not called before it's garbage-collected - 望)=天涯 - 博客园 (cnblogs.com)

SpringCloud-Gateway OutOfDirectMemoryError排查-CSDN博客

Spring Boot各版本与Java版本的对应兼容关系，与构建工具（Maven、Gradle）版本的对应兼容关系，对servlet 容器的支持-CSDN博客

还有一些找不到了，每个里面找个有用的汇总起来解决；希望可以帮到有需要的朋友

文章来源:https://blog.csdn.net/qq_38709953/article/details/135090316
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。