kafka CommitFailedException: The coordinator is not aware of this member. Though poll on every 100 millis and single consumer
I am using kafka 1.1.0, single consumer and I get this error though my poll is very quick, one poll on every 100 milli seconds elapsed. I use manual commit and duration between two commit request may change (can be even hours). But not poll.
I still get this error "The coordinator is not aware of this member." with message
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot
be completed since the group has already rebalanced and assigned the
partitions to another member. This means that the time between
subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is
spending too much time message processing. You can address this either
by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.
What could be the reason for this error. I have searched all over the place but I couldn't get any information in my context of problem.
Both "max.poll.interval.ms" and session timeout are default.
From logs I can see last successful commit was at "11:20:11.677" and failed commit was at "11:20:35.742" just 24 seconds apart.
Its application process with its own group id. I can confirm there was no other process running with same group id.
This is from debug log:
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Attempt to heartbeat failed for since member id
consumer-1-e237c8ed-fed7-4687-be4d-99e333e48af3 is not valid.
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Disabling heartbeat thread 2018-11-17 17:08:09.241 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:08:09.241 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
505 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.220:9092 (id: 1 rack: null),
10.0.10.221:9092 (id: 2 rack: null)], partitions = )
2018-11-17 17:12:09.472 - DEBUG o.apache.kafka.clients.NetworkClient Node 3
disconnected.
2018-11-17 17:12:09.472 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:12:09.472 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
506 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.220:9092 (id: 1 rack: null)], partitions = )
2018-11-17 17:17:09.502 - DEBUG o.apache.kafka.clients.NetworkClient Initialize
connection to node 10.0.10.222:9092 (id: 3 rack: null) for sending
metadata request
2018-11-17 17:17:09.503 - DEBUG
o.apache.kafka.clients.NetworkClient Initiating connection to node
10.0.10.221:9092 (id: 2 rack: null)
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:17:09.503 - DEBUG
o.a.kafka.common.network.Selector Created socket with SO_RCVBUF =
32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0]) 2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 2: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
I see member id not unique message in debug and kafka consumer client keeps retrying for connection.
apache-kafka
asked Nov 15 '18 at 4:56
manumanu
466
add a comment |
I am using kafka 1.1.0, single consumer and I get this error though my poll is very quick, one poll on every 100 milli seconds elapsed. I use manual commit and duration between two commit request may change (can be even hours). But not poll.
I still get this error "The coordinator is not aware of this member." with message
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot
be completed since the group has already rebalanced and assigned the
partitions to another member. This means that the time between
subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is
spending too much time message processing. You can address this either
by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.
What could be the reason for this error. I have searched all over the place but I couldn't get any information in my context of problem.
Both "max.poll.interval.ms" and session timeout are default.
From logs I can see last successful commit was at "11:20:11.677" and failed commit was at "11:20:35.742" just 24 seconds apart.
Its application process with its own group id. I can confirm there was no other process running with same group id.
This is from debug log:
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Attempt to heartbeat failed for since member id
consumer-1-e237c8ed-fed7-4687-be4d-99e333e48af3 is not valid.
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Disabling heartbeat thread 2018-11-17 17:08:09.241 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:08:09.241 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
505 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.220:9092 (id: 1 rack: null),
10.0.10.221:9092 (id: 2 rack: null)], partitions = )
2018-11-17 17:12:09.472 - DEBUG o.apache.kafka.clients.NetworkClient Node 3
disconnected.
2018-11-17 17:12:09.472 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:12:09.472 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
506 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.220:9092 (id: 1 rack: null)], partitions = )
2018-11-17 17:17:09.502 - DEBUG o.apache.kafka.clients.NetworkClient Initialize
connection to node 10.0.10.222:9092 (id: 3 rack: null) for sending
metadata request
2018-11-17 17:17:09.503 - DEBUG
o.apache.kafka.clients.NetworkClient Initiating connection to node
10.0.10.221:9092 (id: 2 rack: null)
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:17:09.503 - DEBUG
o.a.kafka.common.network.Selector Created socket with SO_RCVBUF =
32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0]) 2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 2: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
I see member id not unique message in debug and kafka consumer client keeps retrying for connection.
apache-kafka
asked Nov 15 '18 at 4:56
manumanu
466
Did you use the standalone consumer with agroup.idhappened to be the same with some other consumer group?
– amethystic
Nov 15 '18 at 6:15
@amethystic nope. Its application process with its own group id. I can confirm there was no other process running with same group id.
– manu
Nov 16 '18 at 1:45
@amethystic I have updated with debug log of kafka client. Please see if that helps. I am not understanding as to why this happened.
– manu
Nov 18 '18 at 3:24
add a comment |
I am using kafka 1.1.0, single consumer and I get this error though my poll is very quick, one poll on every 100 milli seconds elapsed. I use manual commit and duration between two commit request may change (can be even hours). But not poll.
I still get this error "The coordinator is not aware of this member." with message
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot
be completed since the group has already rebalanced and assigned the
partitions to another member. This means that the time between
subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is
spending too much time message processing. You can address this either
by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.
What could be the reason for this error. I have searched all over the place but I couldn't get any information in my context of problem.
Both "max.poll.interval.ms" and session timeout are default.
From logs I can see last successful commit was at "11:20:11.677" and failed commit was at "11:20:35.742" just 24 seconds apart.
Its application process with its own group id. I can confirm there was no other process running with same group id.
This is from debug log:
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Attempt to heartbeat failed for since member id
consumer-1-e237c8ed-fed7-4687-be4d-99e333e48af3 is not valid.
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Disabling heartbeat thread 2018-11-17 17:08:09.241 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:08:09.241 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
505 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.220:9092 (id: 1 rack: null),
10.0.10.221:9092 (id: 2 rack: null)], partitions = )
2018-11-17 17:12:09.472 - DEBUG o.apache.kafka.clients.NetworkClient Node 3
disconnected.
2018-11-17 17:12:09.472 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:12:09.472 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
506 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.220:9092 (id: 1 rack: null)], partitions = )
2018-11-17 17:17:09.502 - DEBUG o.apache.kafka.clients.NetworkClient Initialize
connection to node 10.0.10.222:9092 (id: 3 rack: null) for sending
metadata request
2018-11-17 17:17:09.503 - DEBUG
o.apache.kafka.clients.NetworkClient Initiating connection to node
10.0.10.221:9092 (id: 2 rack: null)
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:17:09.503 - DEBUG
o.a.kafka.common.network.Selector Created socket with SO_RCVBUF =
32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0]) 2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 2: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
I see member id not unique message in debug and kafka consumer client keeps retrying for connection.
apache-kafka
asked Nov 15 '18 at 4:56
manumanu
466
I am using kafka 1.1.0, single consumer and I get this error though my poll is very quick, one poll on every 100 milli seconds elapsed. I use manual commit and duration between two commit request may change (can be even hours). But not poll.
I still get this error "The coordinator is not aware of this member." with message
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot
be completed since the group has already rebalanced and assigned the
partitions to another member. This means that the time between
subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is
spending too much time message processing. You can address this either
by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.
What could be the reason for this error. I have searched all over the place but I couldn't get any information in my context of problem.
Both "max.poll.interval.ms" and session timeout are default.
From logs I can see last successful commit was at "11:20:11.677" and failed commit was at "11:20:35.742" just 24 seconds apart.
Its application process with its own group id. I can confirm there was no other process running with same group id.
This is from debug log:
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Attempt to heartbeat failed for since member id
consumer-1-e237c8ed-fed7-4687-be4d-99e333e48af3 is not valid.
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator
Disabling heartbeat thread 2018-11-17 17:08:09.241 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:08:09.241 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
505 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.220:9092 (id: 1 rack: null),
10.0.10.221:9092 (id: 2 rack: null)], partitions = )
2018-11-17 17:12:09.472 - DEBUG o.apache.kafka.clients.NetworkClient Node 3
disconnected.
2018-11-17 17:12:09.472 - DEBUG
o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:12:09.472 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
506 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092
(id: 3 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.220:9092 (id: 1 rack: null)], partitions = )
2018-11-17 17:17:09.502 - DEBUG o.apache.kafka.clients.NetworkClient Initialize
connection to node 10.0.10.222:9092 (id: 3 rack: null) for sending
metadata request
2018-11-17 17:17:09.503 - DEBUG
o.apache.kafka.clients.NetworkClient Initiating connection to node
10.0.10.221:9092 (id: 2 rack: null)
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request
(type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack:
null)
2018-11-17 17:17:09.503 - DEBUG
o.a.kafka.common.network.Selector Created socket with SO_RCVBUF =
32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed
connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 -
DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions
fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG
org.apache.kafka.clients.Metadata - Updated cluster metadata version
507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092
(id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null),
10.0.10.222:9092 (id: 3 rack: null)], partitions = )
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0]) 2018-11-17
17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded
API versions for node 2: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0
to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0
to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5):
0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4],
ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3
[usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10):
0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12):
0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14):
0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1],
ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable:
1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2
[usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21):
0 [usable: 0], InitProducerId(22): 0 [usable: 0],
OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0
[usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0
[usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28):
0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0
[usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to
1 [usable: 1], AlterConfigs(33): 0 [usable: 0],
AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0
[usable: 0], SaslAuthenticate(36): 0 [usable: 0],
CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0
[usable: 0], RenewDelegationToken(39): 0 [usable: 0],
ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41):
0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
I see member id not unique message in debug and kafka consumer client keeps retrying for connection.
apache-kafka
apache-kafka
asked Nov 15 '18 at 4:56
manumanu
466
asked Nov 15 '18 at 4:56
manumanu
466
edited Nov 18 '18 at 6:33
manu
asked Nov 15 '18 at 4:56
manumanu
466
asked Nov 15 '18 at 4:56
manumanu
466
asked Nov 15 '18 at 4:56
manumanu
466
466
Did you use the standalone consumer with agroup.idhappened to be the same with some other consumer group?
– amethystic
Nov 15 '18 at 6:15
@amethystic nope. Its application process with its own group id. I can confirm there was no other process running with same group id.
– manu
Nov 16 '18 at 1:45
@amethystic I have updated with debug log of kafka client. Please see if that helps. I am not understanding as to why this happened.
– manu
Nov 18 '18 at 3:24
add a comment |
Did you use the standalone consumer with agroup.idhappened to be the same with some other consumer group?
– amethystic
Nov 15 '18 at 6:15
@amethystic nope. Its application process with its own group id. I can confirm there was no other process running with same group id.
– manu
Nov 16 '18 at 1:45
@amethystic I have updated with debug log of kafka client. Please see if that helps. I am not understanding as to why this happened.
– manu
Nov 18 '18 at 3:24
Did you use the standalone consumer with a
group.id happened to be the same with some other consumer group?– amethystic
Nov 15 '18 at 6:15
Did you use the standalone consumer with a
group.id happened to be the same with some other consumer group?– amethystic
Nov 15 '18 at 6:15
@amethystic nope. Its application process with its own group id. I can confirm there was no other process running with same group id.
– manu
Nov 16 '18 at 1:45
@amethystic nope. Its application process with its own group id. I can confirm there was no other process running with same group id.
– manu
Nov 16 '18 at 1:45
@amethystic I have updated with debug log of kafka client. Please see if that helps. I am not understanding as to why this happened.
– manu
Nov 18 '18 at 3:24
@amethystic I have updated with debug log of kafka client. Please see if that helps. I am not understanding as to why this happened.
– manu
Nov 18 '18 at 3:24
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312664%2fkafka-commitfailedexception-the-coordinator-is-not-aware-of-this-member-though%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312664%2fkafka-commitfailedexception-the-coordinator-is-not-aware-of-this-member-though%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Did you use the standalone consumer with a
group.idhappened to be the same with some other consumer group?– amethystic
Nov 15 '18 at 6:15
@amethystic nope. Its application process with its own group id. I can confirm there was no other process running with same group id.
– manu
Nov 16 '18 at 1:45
@amethystic I have updated with debug log of kafka client. Please see if that helps. I am not understanding as to why this happened.
– manu
Nov 18 '18 at 3:24