[Kafka] max.request.size configuration 오류

org.apache.kafka.common.errors.RecordTooLargeException: The message is 1826601 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

2021-02-02 15:44:23.387 ERROR 87540 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 1826601 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.] with root cause

org.apache.kafka.common.errors.RecordTooLargeException: The message is 1826601 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

 

 

List Object로 카프카에 메세지를 보내는데 해당 에러가 발생했다.

카프카에 어떠한 설정도 하지 않았는데 디폴트 값이 한 개의 메세지를 보내는데 1048576 bytes(약 1Mb) 정도인가 보다.

 

아래 링크들을 참고하여 해결해보았다.

stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb

How can I send large messages with Kafka (over 15MB)?

docs.cloudera.com/documentation/kafka/2-1-x/topics/kafka_performance.html

Configuring Kafka for Performance and Resource Management | 2.1.x | Cloudera Documentation

 

- MAX_REQUEST_SIZE_CONFIG

Producer에서 props 옵션에 추가

 

- message.max.bytes

server.properties

3072000 으로 설정

 

- fetch.max.bytes

consumer.properties

3072000 으로 설정