A Dive into Multi-Topic Subscriptions with Apache Kafka

Apache Kafka allows a single consumer to subscribe to multiple topics at the same time and process both in a combined stream of records. But we also know that we only get strict ordering guarantees within a single partition, so what kind of message ordering do we get when we subscribe to two topics? That is what we are going to look at in this post.

Terminology

Before we start looking at the tests, we need to describe some terms used:

• Tailing the topic vs Time-travel: A tailing consumer is aiming to consume messages in real-time. Time-travel refers to starting at an offset in the past, such as the earliest offset.
• Equal vs unequal message arrival rate: Equals is when two topics receive messages at the same rate and unequal is when one topic is receiving messages faster than the other.
• Zero or short constant time vs jitter: How long a consumer takes to process each message.
• Fast producer: The producer is sending messages faster than the consumer is able to process them..

The Test Cases

We will run the following tests:

1. Tailing the topic, equal message arrival rate, zero or short constant processing time
2. Tailing the topic, equal message arrival rate, short processing time with jitter
3. Tailing the topic, equal message arrival rate, fast producer
4. Tailing the topic,unequal message arrival rate, zero or short constant processing time
5. Tailing the topic, unequal message arrival rate, short processing time with jitter
6. Tailing the topic, unequal message arrival rate, fast producer
7. Time-travel, equal message arrival rate

Determining ordering

We’ll have a single producer sending messages to two topics with two types of incrementing counter:

• Global Counter that we’ll use to test the ordering across both topics, creates a global monotonic sequence
• Topic Counter that we’ll use to ensure that within a single topic that we get messages in the right order

If the producer sends:

• Topic 1 { “GC”: 1, “TC”: 1 }
• Topic 2 { “GC”: 2, “TC”: 1 }
• Topic 1 { “GC”: 3, “TC”: 2 }
• Topic 2 { “GC”: 4, “TC”: 2 }
• Topic 1 { “GC”: 5, “TC”: 3 }
• Topic 2 { “GC”: 6, “TC”: 3 }
• Topic 1 { “GC”: 7, “TC”: 4 }
• Topic 2 { “GC”: 8, “TC”: 4 }
• Topic 1 { “GC”: 9, “TC”: 5 }
• Topic 2 { “GC”: 10, “TC”: 5 }

The final result in the topics will be:

Spoiler alert, no test had a violation of topic ordering that the topic counter would detect. This is because we use single partition topics in these tests and Kafka guarantees ordering at the partition level. However, we’ll see that the global ordering does not hold up, which we expect.

Case 1 - Tailing the topic, equal message arrival rate, constant processing time

In this test our producer sends messages in a round robin manner to both topics and the consumer has essentially zero processing time as it only verifies the topic and global ordering based on the two counters.

2019-09-05T17:39:45.966Z : Consumer : Subscribing
2019-09-05T17:39:46.020Z : Consumer : Partitions revoked at 2019-09-05T17:39:46.020Z
2019-09-05T17:39:49.046Z : Consumer : Topic: topic1 Partition: 0 assigned at 2019-09-05T17:39:49.046Z
2019-09-05T17:39:49.046Z : Consumer : Topic: topic2 Partition: 0 assigned at 2019-09-05T17:39:49.046Z
2019-09-05T17:39:49.099Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T17:39:49.105Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T17:39:49.105Z : Producer : Topic: topic2, TC: 1, GC: 2
2019-09-05T17:39:49.107Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 OK
2019-09-05T17:39:49.108Z : Producer : Topic: topic1, TC: 2, GC: 3
2019-09-05T17:39:49.109Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 OK
2019-09-05T17:39:49.110Z : Producer : Topic: topic2, TC: 2, GC: 4
2019-09-05T17:39:49.110Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 OK
2019-09-05T17:39:49.112Z : Producer : Topic: topic1, TC: 3, GC: 5
2019-09-05T17:39:49.112Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 OK
2019-09-05T17:39:49.114Z : Producer : Topic: topic2, TC: 3, GC: 6
2019-09-05T17:39:49.114Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 OK
2019-09-05T17:39:49.115Z : Producer : Topic: topic1, TC: 4, GC: 7
2019-09-05T17:39:49.116Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 OK
2019-09-05T17:39:49.117Z : Producer : Topic: topic2, TC: 4, GC: 8
2019-09-05T17:39:49.118Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 OK
2019-09-05T17:39:49.119Z : Producer : Topic: topic1, TC: 5, GC: 9
2019-09-05T17:39:49.119Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 OK
2019-09-05T17:39:49.120Z : Producer : Topic: topic2, TC: 5, GC: 10
2019-09-05T17:39:49.121Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 OK
2019-09-05T17:39:49.122Z : Producer : Topic: topic1, TC: 6, GC: 11
2019-09-05T17:39:49.123Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 OK
2019-09-05T17:39:49.125Z : Producer : Topic: topic2, TC: 6, GC: 12
2019-09-05T17:39:49.125Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 OK
2019-09-05T17:39:49.127Z : Producer : Topic: topic1, TC: 7, GC: 13
2019-09-05T17:39:49.128Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 OK
2019-09-05T17:39:49.128Z : Producer : Topic: topic2, TC: 7, GC: 14
2019-09-05T17:39:49.129Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 OK
2019-09-05T17:39:49.130Z : Producer : Topic: topic1, TC: 8, GC: 15
2019-09-05T17:39:49.131Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 OK
2019-09-05T17:39:49.132Z : Producer : Topic: topic2, TC: 8, GC: 16
2019-09-05T17:39:49.133Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 OK
2019-09-05T17:39:49.133Z : Producer : Topic: topic1, TC: 9, GC: 17
2019-09-05T17:39:49.134Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 OK
2019-09-05T17:39:49.135Z : Producer : Topic: topic2, TC: 9, GC: 18
2019-09-05T17:39:49.136Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 OK
2019-09-05T17:39:49.137Z : Producer : Topic: topic1, TC: 10, GC: 19
2019-09-05T17:39:49.138Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 OK
2019-09-05T17:39:49.139Z : Producer : Topic: topic2, TC: 10, GC: 20
2019-09-05T17:39:49.140Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 OK
2019-09-05T17:39:49.141Z : Producer : Topic: topic1, TC: 11, GC: 21
2019-09-05T17:39:49.143Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 OK
2019-09-05T17:39:49.143Z : Producer : Topic: topic2, TC: 11, GC: 22
2019-09-05T17:39:49.145Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 OK
2019-09-05T17:39:49.146Z : Producer : Topic: topic1, TC: 12, GC: 23
2019-09-05T17:39:49.146Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 OK
2019-09-05T17:39:49.148Z : Producer : Topic: topic2, TC: 12, GC: 24
2019-09-05T17:39:49.149Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 OK
2019-09-05T17:39:49.149Z : Producer : Topic: topic1, TC: 13, GC: 25
2019-09-05T17:39:49.150Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 OK
2019-09-05T17:39:49.151Z : Producer : Topic: topic2, TC: 13, GC: 26
2019-09-05T17:39:49.152Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 OK
2019-09-05T17:39:49.153Z : Producer : Topic: topic1, TC: 14, GC: 27
2019-09-05T17:39:49.154Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 OK
2019-09-05T17:39:49.157Z : Producer : Topic: topic2, TC: 14, GC: 28
2019-09-05T17:39:49.157Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 OK
2019-09-05T17:39:49.159Z : Producer : Topic: topic1, TC: 15, GC: 29
2019-09-05T17:39:49.160Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 OK
2019-09-05T17:39:49.161Z : Producer : Topic: topic2, TC: 15, GC: 30
2019-09-05T17:39:49.162Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 OK
2019-09-05T17:39:49.163Z : Producer : Topic: topic1, TC: 16, GC: 31
2019-09-05T17:39:49.164Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 OK
2019-09-05T17:39:49.165Z : Producer : Topic: topic2, TC: 16, GC: 32
2019-09-05T17:39:49.166Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 OK
2019-09-05T17:39:49.166Z : Producer : Topic: topic1, TC: 17, GC: 33
2019-09-05T17:39:49.167Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 OK
2019-09-05T17:39:49.168Z : Producer : Topic: topic2, TC: 17, GC: 34
2019-09-05T17:39:49.169Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 OK
2019-09-05T17:39:49.170Z : Producer : Topic: topic1, TC: 18, GC: 35
2019-09-05T17:39:49.171Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 OK
2019-09-05T17:39:49.171Z : Producer : Topic: topic2, TC: 18, GC: 36
2019-09-05T17:39:49.172Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 OK
2019-09-05T17:39:49.173Z : Producer : Topic: topic1, TC: 19, GC: 37
2019-09-05T17:39:49.174Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 OK
2019-09-05T17:39:49.175Z : Producer : Topic: topic2, TC: 19, GC: 38
2019-09-05T17:39:49.176Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 OK
2019-09-05T17:39:49.177Z : Producer : Topic: topic1, TC: 20, GC: 39
2019-09-05T17:39:49.177Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 OK
2019-09-05T17:39:49.178Z : Producer : Topic: topic2, TC: 20, GC: 40
2019-09-05T17:39:49.179Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 OK
2019-09-05T17:39:49.180Z : Producer : Topic: topic1, TC: 21, GC: 41
2019-09-05T17:39:49.181Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 OK
2019-09-05T17:39:49.182Z : Producer : Topic: topic2, TC: 21, GC: 42
2019-09-05T17:39:49.182Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 OK
2019-09-05T17:39:49.184Z : Producer : Topic: topic1, TC: 22, GC: 43
2019-09-05T17:39:49.185Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 OK
2019-09-05T17:39:49.186Z : Producer : Topic: topic2, TC: 22, GC: 44
2019-09-05T17:39:49.186Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 OK
2019-09-05T17:39:49.187Z : Producer : Topic: topic1, TC: 23, GC: 45
2019-09-05T17:39:49.189Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 OK
2019-09-05T17:39:49.189Z : Producer : Topic: topic2, TC: 23, GC: 46
2019-09-05T17:39:49.190Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 OK
2019-09-05T17:39:49.191Z : Producer : Topic: topic1, TC: 24, GC: 47
2019-09-05T17:39:49.191Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 OK
2019-09-05T17:39:49.192Z : Producer : Topic: topic2, TC: 24, GC: 48
2019-09-05T17:39:49.193Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 OK
2019-09-05T17:39:49.194Z : Producer : Topic: topic1, TC: 25, GC: 49
2019-09-05T17:39:49.195Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 OK
2019-09-05T17:39:49.196Z : Producer : Topic: topic2, TC: 25, GC: 50
2019-09-05T17:39:49.197Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 OK
2019-09-05T17:39:49.199Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 OK
2019-09-05T17:39:49.199Z : Producer : Topic: topic1, TC: 26, GC: 51
2019-09-05T17:39:49.201Z : Producer : Topic: topic2, TC: 26, GC: 52
2019-09-05T17:39:49.202Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 OK
2019-09-05T17:39:49.203Z : Producer : Topic: topic1, TC: 27, GC: 53
2019-09-05T17:39:49.204Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 OK
2019-09-05T17:39:49.205Z : Producer : Topic: topic2, TC: 27, GC: 54
2019-09-05T17:39:49.205Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 OK
2019-09-05T17:39:49.206Z : Producer : Topic: topic1, TC: 28, GC: 55
2019-09-05T17:39:49.207Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 OK
2019-09-05T17:39:49.208Z : Producer : Topic: topic2, TC: 28, GC: 56
2019-09-05T17:39:49.209Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 OK
2019-09-05T17:39:49.210Z : Producer : Topic: topic1, TC: 29, GC: 57
2019-09-05T17:39:49.210Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 OK
2019-09-05T17:39:49.212Z : Producer : Topic: topic2, TC: 29, GC: 58
2019-09-05T17:39:49.213Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 OK
2019-09-05T17:39:49.213Z : Producer : Topic: topic1, TC: 30, GC: 59
2019-09-05T17:39:49.214Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 OK
2019-09-05T17:39:49.215Z : Producer : Topic: topic2, TC: 30, GC: 60
2019-09-05T17:39:49.216Z : Producer : Topic: topic1, TC: 31, GC: 61
2019-09-05T17:39:49.217Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 OK
2019-09-05T17:39:49.218Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 OK
2019-09-05T17:39:49.219Z : Producer : Topic: topic2, TC: 31, GC: 62
2019-09-05T17:39:49.220Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 OK
2019-09-05T17:39:49.221Z : Producer : Topic: topic1, TC: 32, GC: 63
2019-09-05T17:39:49.222Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 OK
2019-09-05T17:39:49.223Z : Producer : Topic: topic2, TC: 32, GC: 64
2019-09-05T17:39:49.224Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 OK
2019-09-05T17:39:49.225Z : Producer : Topic: topic1, TC: 33, GC: 65
2019-09-05T17:39:49.226Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 OK
2019-09-05T17:39:49.227Z : Producer : Topic: topic2, TC: 33, GC: 66
2019-09-05T17:39:49.228Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 OK
2019-09-05T17:39:49.229Z : Producer : Topic: topic1, TC: 34, GC: 67
2019-09-05T17:39:49.229Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 OK
2019-09-05T17:39:49.230Z : Producer : Topic: topic2, TC: 34, GC: 68
2019-09-05T17:39:49.231Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 OK
2019-09-05T17:39:49.232Z : Producer : Topic: topic1, TC: 35, GC: 69
2019-09-05T17:39:49.232Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 OK
2019-09-05T17:39:49.234Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 OK
2019-09-05T17:39:49.234Z : Producer : Topic: topic2, TC: 35, GC: 70
2019-09-05T17:39:49.236Z : Producer : Topic: topic1, TC: 36, GC: 71
2019-09-05T17:39:49.237Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 OK
2019-09-05T17:39:49.238Z : Producer : Topic: topic2, TC: 36, GC: 72
2019-09-05T17:39:49.239Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 OK
2019-09-05T17:39:49.239Z : Producer : Topic: topic1, TC: 37, GC: 73
2019-09-05T17:39:49.240Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 OK
2019-09-05T17:39:49.240Z : Producer : Topic: topic2, TC: 37, GC: 74
2019-09-05T17:39:49.241Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 OK
2019-09-05T17:39:49.242Z : Producer : Topic: topic1, TC: 38, GC: 75
2019-09-05T17:39:49.242Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 OK
2019-09-05T17:39:49.244Z : Producer : Topic: topic2, TC: 38, GC: 76
2019-09-05T17:39:49.244Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 OK
2019-09-05T17:39:49.246Z : Producer : Topic: topic1, TC: 39, GC: 77
2019-09-05T17:39:49.247Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 OK
2019-09-05T17:39:49.248Z : Producer : Topic: topic2, TC: 39, GC: 78
2019-09-05T17:39:49.248Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 OK
2019-09-05T17:39:49.249Z : Producer : Topic: topic1, TC: 40, GC: 79
2019-09-05T17:39:49.250Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 OK
2019-09-05T17:39:49.251Z : Producer : Topic: topic2, TC: 40, GC: 80
2019-09-05T17:39:49.251Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 OK
2019-09-05T17:39:49.253Z : Producer : Topic: topic1, TC: 41, GC: 81
2019-09-05T17:39:49.254Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 OK
2019-09-05T17:39:49.254Z : Producer : Topic: topic2, TC: 41, GC: 82
2019-09-05T17:39:49.255Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 OK
2019-09-05T17:39:49.257Z : Producer : Topic: topic1, TC: 42, GC: 83
2019-09-05T17:39:49.257Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 OK
2019-09-05T17:39:49.259Z : Producer : Topic: topic2, TC: 42, GC: 84
2019-09-05T17:39:49.259Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 OK
2019-09-05T17:39:49.260Z : Producer : Topic: topic1, TC: 43, GC: 85
2019-09-05T17:39:49.261Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 OK
2019-09-05T17:39:49.262Z : Producer : Topic: topic2, TC: 43, GC: 86
2019-09-05T17:39:49.263Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 OK
2019-09-05T17:39:49.264Z : Producer : Topic: topic1, TC: 44, GC: 87
2019-09-05T17:39:49.265Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 OK
2019-09-05T17:39:49.265Z : Producer : Topic: topic2, TC: 44, GC: 88
2019-09-05T17:39:49.266Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 OK
2019-09-05T17:39:49.267Z : Producer : Topic: topic1, TC: 45, GC: 89
2019-09-05T17:39:49.268Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 OK
2019-09-05T17:39:49.269Z : Producer : Topic: topic2, TC: 45, GC: 90
2019-09-05T17:39:49.270Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 OK
2019-09-05T17:39:49.271Z : Producer : Topic: topic1, TC: 46, GC: 91
2019-09-05T17:39:49.271Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 OK
2019-09-05T17:39:49.272Z : Producer : Topic: topic2, TC: 46, GC: 92
2019-09-05T17:39:49.273Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 OK
2019-09-05T17:39:49.274Z : Producer : Topic: topic1, TC: 47, GC: 93
2019-09-05T17:39:49.274Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 OK
2019-09-05T17:39:49.276Z : Producer : Topic: topic2, TC: 47, GC: 94
2019-09-05T17:39:49.277Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 OK
2019-09-05T17:39:49.278Z : Producer : Topic: topic1, TC: 48, GC: 95
2019-09-05T17:39:49.279Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 OK
2019-09-05T17:39:49.280Z : Producer : Topic: topic2, TC: 48, GC: 96
2019-09-05T17:39:49.280Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 OK
2019-09-05T17:39:49.281Z : Producer : Topic: topic1, TC: 49, GC: 97
2019-09-05T17:39:49.282Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 OK
2019-09-05T17:39:49.283Z : Producer : Topic: topic2, TC: 49, GC: 98
2019-09-05T17:39:49.283Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 OK
2019-09-05T17:39:49.285Z : Producer : Topic: topic1, TC: 50, GC: 99
2019-09-05T17:39:49.285Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 OK
2019-09-05T17:39:49.286Z : Producer : Topic: topic2, TC: 50, GC: 100
2019-09-05T17:39:49.286Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 OK

We got perfect message ordering, so we know that the Kafka client is capable of receiving messages in across topics in the right order, at least when consumed in real-time.

Case 2 - Tailing the topic, equal message arrival rate, processing time with jitter

If we add a bit of random jitter consumer processing times then we see a different picture. With the publish interval at 25ms and consumption rate between 1ms and 50ms with peaks of 200ms we see:

2019-09-05T17:43:02.562Z : Consumer : Subscribing
2019-09-05T17:43:02.621Z : Consumer : Partitions revoked at 2019-09-05T17:43:02.620Z
2019-09-05T17:43:05.645Z : Consumer : Topic: topic1 Partition: 0 assigned at 2019-09-05T17:43:05.645Z
2019-09-05T17:43:05.645Z : Consumer : Topic: topic2 Partition: 0 assigned at 2019-09-05T17:43:05.645Z
2019-09-05T17:43:05.695Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T17:43:05.701Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T17:43:05.726Z : Producer : Topic: topic2, TC: 1, GC: 2
2019-09-05T17:43:05.748Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 OK
2019-09-05T17:43:05.754Z : Producer : Topic: topic1, TC: 2, GC: 3
2019-09-05T17:43:05.783Z : Producer : Topic: topic2, TC: 2, GC: 4
2019-09-05T17:43:05.797Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 OK
2019-09-05T17:43:05.811Z : Producer : Topic: topic1, TC: 3, GC: 5
2019-09-05T17:43:05.838Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 OK
2019-09-05T17:43:05.839Z : Producer : Topic: topic2, TC: 3, GC: 6
2019-09-05T17:43:05.842Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 OK
2019-09-05T17:43:05.867Z : Producer : Topic: topic1, TC: 4, GC: 7
2019-09-05T17:43:05.871Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 OK
2019-09-05T17:43:05.877Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 OK
2019-09-05T17:43:05.896Z : Producer : Topic: topic2, TC: 4, GC: 8
2019-09-05T17:43:05.909Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 OK
2019-09-05T17:43:05.925Z : Producer : Topic: topic1, TC: 5, GC: 9
2019-09-05T17:43:05.925Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 OK
2019-09-05T17:43:05.953Z : Producer : Topic: topic2, TC: 5, GC: 10
2019-09-05T17:43:05.968Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 OK
2019-09-05T17:43:05.982Z : Producer : Topic: topic1, TC: 6, GC: 11
2019-09-05T17:43:06.007Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 OK
2019-09-05T17:43:06.010Z : Producer : Topic: topic2, TC: 6, GC: 12
2019-09-05T17:43:06.038Z : Producer : Topic: topic1, TC: 7, GC: 13
2019-09-05T17:43:06.058Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 JUMP FORWARD 1
2019-09-05T17:43:06.066Z : Producer : Topic: topic2, TC: 7, GC: 14
2019-09-05T17:43:06.094Z : Producer : Topic: topic1, TC: 8, GC: 15
2019-09-05T17:43:06.121Z : Producer : Topic: topic2, TC: 8, GC: 16
2019-09-05T17:43:06.149Z : Producer : Topic: topic1, TC: 9, GC: 17
2019-09-05T17:43:06.177Z : Producer : Topic: topic2, TC: 9, GC: 18
2019-09-05T17:43:06.205Z : Producer : Topic: topic1, TC: 10, GC: 19
2019-09-05T17:43:06.233Z : Producer : Topic: topic2, TC: 10, GC: 20
2019-09-05T17:43:06.260Z : Producer : Topic: topic1, TC: 11, GC: 21
2019-09-05T17:43:06.279Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 JUMP BACKWARDS 1
2019-09-05T17:43:06.289Z : Producer : Topic: topic2, TC: 11, GC: 22
2019-09-05T17:43:06.289Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 JUMP FORWARD 2
2019-09-05T17:43:06.317Z : Producer : Topic: topic1, TC: 12, GC: 23
2019-09-05T17:43:06.321Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 JUMP BACKWARDS 1
2019-09-05T17:43:06.345Z : Producer : Topic: topic2, TC: 12, GC: 24
2019-09-05T17:43:06.347Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 JUMP FORWARD 2
2019-09-05T17:43:06.373Z : Producer : Topic: topic1, TC: 13, GC: 25
2019-09-05T17:43:06.401Z : Producer : Topic: topic2, TC: 13, GC: 26
2019-09-05T17:43:06.429Z : Producer : Topic: topic1, TC: 14, GC: 27
2019-09-05T17:43:06.457Z : Producer : Topic: topic2, TC: 14, GC: 28
2019-09-05T17:43:06.485Z : Producer : Topic: topic1, TC: 15, GC: 29
2019-09-05T17:43:06.513Z : Producer : Topic: topic2, TC: 15, GC: 30
2019-09-05T17:43:06.540Z : Producer : Topic: topic1, TC: 16, GC: 31
2019-09-05T17:43:06.568Z : Producer : Topic: topic2, TC: 16, GC: 32
2019-09-05T17:43:06.584Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 JUMP FORWARD 1
2019-09-05T17:43:06.594Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 JUMP FORWARD 1
2019-09-05T17:43:06.596Z : Producer : Topic: topic1, TC: 17, GC: 33
2019-09-05T17:43:06.620Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 JUMP BACKWARDS 5
2019-09-05T17:43:06.624Z : Producer : Topic: topic2, TC: 17, GC: 34
2019-09-05T17:43:06.652Z : Producer : Topic: topic1, TC: 18, GC: 35
2019-09-05T17:43:06.656Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 JUMP FORWARD 1
2019-09-05T17:43:06.680Z : Producer : Topic: topic2, TC: 18, GC: 36
2019-09-05T17:43:06.702Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 JUMP FORWARD 1
2019-09-05T17:43:06.708Z : Producer : Topic: topic1, TC: 19, GC: 37
2019-09-05T17:43:06.735Z : Producer : Topic: topic2, TC: 19, GC: 38
2019-09-05T17:43:06.764Z : Producer : Topic: topic1, TC: 20, GC: 39
2019-09-05T17:43:06.792Z : Producer : Topic: topic2, TC: 20, GC: 40
2019-09-05T17:43:06.820Z : Producer : Topic: topic1, TC: 21, GC: 41
2019-09-05T17:43:06.847Z : Producer : Topic: topic2, TC: 21, GC: 42
2019-09-05T17:43:06.875Z : Producer : Topic: topic1, TC: 22, GC: 43
2019-09-05T17:43:06.904Z : Producer : Topic: topic2, TC: 22, GC: 44
2019-09-05T17:43:06.932Z : Producer : Topic: topic1, TC: 23, GC: 45
2019-09-05T17:43:06.952Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 JUMP FORWARD 1
2019-09-05T17:43:06.960Z : Producer : Topic: topic2, TC: 23, GC: 46
2019-09-05T17:43:06.988Z : Producer : Topic: topic1, TC: 24, GC: 47
2019-09-05T17:43:06.997Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 OK
2019-09-05T17:43:07.016Z : Producer : Topic: topic2, TC: 24, GC: 48
2019-09-05T17:43:07.042Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 OK
2019-09-05T17:43:07.044Z : Producer : Topic: topic1, TC: 25, GC: 49
2019-09-05T17:43:07.072Z : Producer : Topic: topic2, TC: 25, GC: 50
2019-09-05T17:43:07.087Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 OK
2019-09-05T17:43:07.100Z : Producer : Topic: topic1, TC: 26, GC: 51
2019-09-05T17:43:07.128Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 JUMP FORWARD 1
2019-09-05T17:43:07.128Z : Producer : Topic: topic2, TC: 26, GC: 52
2019-09-05T17:43:07.138Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 JUMP FORWARD 1
2019-09-05T17:43:07.156Z : Producer : Topic: topic1, TC: 27, GC: 53
2019-09-05T17:43:07.184Z : Producer : Topic: topic2, TC: 27, GC: 54
2019-09-05T17:43:07.186Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 JUMP FORWARD 1
2019-09-05T17:43:07.213Z : Producer : Topic: topic1, TC: 28, GC: 55
2019-09-05T17:43:07.233Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 JUMP FORWARD 1
2019-09-05T17:43:07.240Z : Producer : Topic: topic2, TC: 28, GC: 56
2019-09-05T17:43:07.268Z : Producer : Topic: topic1, TC: 29, GC: 57
2019-09-05T17:43:07.273Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 JUMP FORWARD 1
2019-09-05T17:43:07.286Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 JUMP FORWARD 1
2019-09-05T17:43:07.295Z : Producer : Topic: topic2, TC: 29, GC: 58
2019-09-05T17:43:07.313Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 JUMP FORWARD 1
2019-09-05T17:43:07.323Z : Producer : Topic: topic1, TC: 30, GC: 59
2019-09-05T17:43:07.350Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 JUMP FORWARD 1
2019-09-05T17:43:07.351Z : Producer : Topic: topic2, TC: 30, GC: 60
2019-09-05T17:43:07.379Z : Producer : Topic: topic1, TC: 31, GC: 61
2019-09-05T17:43:07.384Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 JUMP FORWARD 1
2019-09-05T17:43:07.401Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 JUMP FORWARD 1
2019-09-05T17:43:07.407Z : Producer : Topic: topic2, TC: 31, GC: 62
2019-09-05T17:43:07.435Z : Producer : Topic: topic1, TC: 32, GC: 63
2019-09-05T17:43:07.451Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 JUMP FORWARD 1
2019-09-05T17:43:07.462Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 JUMP BACKWARDS 21
2019-09-05T17:43:07.463Z : Producer : Topic: topic2, TC: 32, GC: 64
2019-09-05T17:43:07.467Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 JUMP FORWARD 1
2019-09-05T17:43:07.491Z : Producer : Topic: topic1, TC: 33, GC: 65
2019-09-05T17:43:07.515Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 JUMP FORWARD 1
2019-09-05T17:43:07.519Z : Producer : Topic: topic2, TC: 33, GC: 66
2019-09-05T17:43:07.538Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 JUMP FORWARD 1
2019-09-05T17:43:07.547Z : Producer : Topic: topic1, TC: 34, GC: 67
2019-09-05T17:43:07.575Z : Producer : Topic: topic2, TC: 34, GC: 68
2019-09-05T17:43:07.578Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 JUMP FORWARD 1
2019-09-05T17:43:07.594Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 JUMP FORWARD 1
2019-09-05T17:43:07.603Z : Producer : Topic: topic1, TC: 35, GC: 69
2019-09-05T17:43:07.630Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 JUMP FORWARD 1
2019-09-05T17:43:07.630Z : Producer : Topic: topic2, TC: 35, GC: 70
2019-09-05T17:43:07.635Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 JUMP FORWARD 1
2019-09-05T17:43:07.653Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 JUMP FORWARD 1
2019-09-05T17:43:07.656Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 JUMP FORWARD 1
2019-09-05T17:43:07.658Z : Producer : Topic: topic1, TC: 36, GC: 71
2019-09-05T17:43:07.658Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 JUMP FORWARD 1
2019-09-05T17:43:07.685Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 JUMP FORWARD 2
2019-09-05T17:43:07.686Z : Producer : Topic: topic2, TC: 36, GC: 72
2019-09-05T17:43:07.714Z : Producer : Topic: topic1, TC: 37, GC: 73
2019-09-05T17:43:07.725Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 JUMP BACKWARDS 1
2019-09-05T17:43:07.742Z : Producer : Topic: topic2, TC: 37, GC: 74
2019-09-05T17:43:07.771Z : Producer : Topic: topic1, TC: 38, GC: 75
2019-09-05T17:43:07.776Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 JUMP FORWARD 1
2019-09-05T17:43:07.783Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 OK
2019-09-05T17:43:07.798Z : Producer : Topic: topic2, TC: 38, GC: 76
2019-09-05T17:43:07.813Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 JUMP FORWARD 1
2019-09-05T17:43:07.822Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 JUMP FORWARD 1
2019-09-05T17:43:07.826Z : Producer : Topic: topic1, TC: 39, GC: 77
2019-09-05T17:43:07.849Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 JUMP FORWARD 1
2019-09-05T17:43:07.853Z : Producer : Topic: topic2, TC: 39, GC: 78
2019-09-05T17:43:07.858Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 JUMP FORWARD 1
2019-09-05T17:43:07.881Z : Producer : Topic: topic1, TC: 40, GC: 79
2019-09-05T17:43:07.906Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 JUMP FORWARD 1
2019-09-05T17:43:07.908Z : Producer : Topic: topic2, TC: 40, GC: 80
2019-09-05T17:43:07.928Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 JUMP FORWARD 1
2019-09-05T17:43:07.936Z : Producer : Topic: topic1, TC: 41, GC: 81
2019-09-05T17:43:07.964Z : Producer : Topic: topic2, TC: 41, GC: 82
2019-09-05T17:43:07.969Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 JUMP FORWARD 1
2019-09-05T17:43:07.975Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 JUMP FORWARD 1
2019-09-05T17:43:07.978Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 JUMP FORWARD 1
2019-09-05T17:43:07.992Z : Producer : Topic: topic1, TC: 42, GC: 83
2019-09-05T17:43:08.019Z : Producer : Topic: topic2, TC: 42, GC: 84
2019-09-05T17:43:08.027Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 JUMP FORWARD 1
2019-09-05T17:43:08.030Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 JUMP BACKWARDS 19
2019-09-05T17:43:08.042Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 JUMP FORWARD 1
2019-09-05T17:43:08.047Z : Producer : Topic: topic1, TC: 43, GC: 85
2019-09-05T17:43:08.076Z : Producer : Topic: topic2, TC: 43, GC: 86
2019-09-05T17:43:08.087Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 JUMP FORWARD 1
2019-09-05T17:43:08.104Z : Producer : Topic: topic1, TC: 44, GC: 87
2019-09-05T17:43:08.107Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 JUMP FORWARD 1
2019-09-05T17:43:08.121Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 JUMP FORWARD 1
2019-09-05T17:43:08.132Z : Producer : Topic: topic2, TC: 44, GC: 88
2019-09-05T17:43:08.160Z : Producer : Topic: topic1, TC: 45, GC: 89
2019-09-05T17:43:08.163Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 JUMP FORWARD 1
2019-09-05T17:43:08.181Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 JUMP FORWARD 1
2019-09-05T17:43:08.187Z : Producer : Topic: topic2, TC: 45, GC: 90
2019-09-05T17:43:08.198Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 JUMP FORWARD 1
2019-09-05T17:43:08.215Z : Producer : Topic: topic1, TC: 46, GC: 91
2019-09-05T17:43:08.234Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 JUMP FORWARD 1
2019-09-05T17:43:08.243Z : Producer : Topic: topic2, TC: 46, GC: 92
2019-09-05T17:43:08.257Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 JUMP FORWARD 1
2019-09-05T17:43:08.271Z : Producer : Topic: topic1, TC: 47, GC: 93
2019-09-05T17:43:08.284Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 JUMP FORWARD 1
2019-09-05T17:43:08.298Z : Producer : Topic: topic2, TC: 47, GC: 94
2019-09-05T17:43:08.324Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 OK
2019-09-05T17:43:08.325Z : Producer : Topic: topic1, TC: 48, GC: 95
2019-09-05T17:43:08.333Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 JUMP FORWARD 1
2019-09-05T17:43:08.351Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 JUMP BACKWARDS 1
2019-09-05T17:43:08.352Z : Producer : Topic: topic2, TC: 48, GC: 96
2019-09-05T17:43:08.379Z : Producer : Topic: topic1, TC: 49, GC: 97
2019-09-05T17:43:08.407Z : Producer : Topic: topic2, TC: 49, GC: 98
2019-09-05T17:43:08.435Z : Producer : Topic: topic1, TC: 50, GC: 99
2019-09-05T17:43:08.462Z : Producer : Topic: topic2, TC: 50, GC: 100
2019-09-05T17:43:08.598Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 JUMP FORWARD 2
2019-09-05T17:43:08.606Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 JUMP FORWARD 1
2019-09-05T17:43:08.652Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 JUMP FORWARD 1
2019-09-05T17:43:08.656Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 JUMP FORWARD 1
2019-09-05T17:43:08.684Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 JUMP FORWARD 1
2019-09-05T17:43:08.687Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 JUMP FORWARD 1
2019-09-05T17:43:08.724Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 JUMP FORWARD 1
2019-09-05T17:43:08.761Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 JUMP FORWARD 1
2019-09-05T17:43:08.767Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 JUMP FORWARD 1
2019-09-05T17:43:09.012Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 JUMP BACKWARDS 17
2019-09-05T17:43:09.033Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 JUMP FORWARD 1
2019-09-05T17:43:09.052Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 JUMP FORWARD 1
2019-09-05T17:43:09.074Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 JUMP FORWARD 1
2019-09-05T17:43:09.111Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 JUMP FORWARD 1
2019-09-05T17:43:09.143Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 JUMP FORWARD 1
2019-09-05T17:43:09.162Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 JUMP FORWARD 1
2019-09-05T17:43:09.172Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 JUMP FORWARD 1
2019-09-05T17:43:09.173Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 JUMP FORWARD 1
2019-09-05T17:43:09.386Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 JUMP FORWARD 1
2019-09-05T17:43:09.435Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 OK
2019-09-05T17:43:09.481Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 JUMP FORWARD 1
2019-09-05T17:43:09.500Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 JUMP FORWARD 1
2019-09-05T17:43:09.543Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 JUMP BACKWARDS 3
2019-09-05T17:43:09.558Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 JUMP FORWARD 1
2019-09-05T17:43:09.577Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 JUMP FORWARD 1

So we see that as soon as we get a bit of variation in our processing time, we lose our ordering. The Global Counter shows a lot of jumping forwards and backwards indicating that this time we didn’t get ordering across the topics.

The Topic Counter is still perfect and that will always be perfect as that is a guarantee that Kafka provides given our topics have a single partition.

Case 3 - Fast Producers

We can also have periods of time where the message arrival rate at a topic is higher than a consumer can cope with. How does that affect our ordering?

When we have a period that covers 100 messages where the produce rate to both topics far outweighs the consumer processing rate, we see that the consumer ends up only reading from a single topic for long stretches at a time.

2019-09-05T18:06:03.021Z : Consumer : Subscribing
2019-09-05T18:06:03.074Z : Consumer : Partitions revoked at 2019-09-05T18:06:03.074Z
2019-09-05T18:06:06.095Z : Consumer : Topic: topic2_1646 Partition: 0 assigned at 2019-09-05T18:06:06.095Z
2019-09-05T18:06:06.095Z : Consumer : Topic: topic1_1646 Partition: 0 assigned at 2019-09-05T18:06:06.095Z
2019-09-05T18:06:06.154Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T18:06:06.160Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T18:06:06.181Z : Producer : Topic: topic2, TC: 1, GC: 2
2019-09-05T18:06:06.204Z : Producer : Topic: topic1, TC: 2, GC: 3
2019-09-05T18:06:06.227Z : Producer : Topic: topic2, TC: 2, GC: 4
2019-09-05T18:06:06.250Z : Producer : Topic: topic1, TC: 3, GC: 5
2019-09-05T18:06:06.263Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 OK
2019-09-05T18:06:06.272Z : Producer : Topic: topic2, TC: 3, GC: 6
2019-09-05T18:06:06.294Z : Producer : Topic: topic1, TC: 4, GC: 7
2019-09-05T18:06:06.317Z : Producer : Topic: topic2, TC: 4, GC: 8
2019-09-05T18:06:06.340Z : Producer : Topic: topic1, TC: 5, GC: 9
2019-09-05T18:06:06.363Z : Producer : Topic: topic2, TC: 5, GC: 10
2019-09-05T18:06:06.365Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 JUMP FORWARD 1
2019-09-05T18:06:06.386Z : Producer : Topic: topic1, TC: 6, GC: 11
2019-09-05T18:06:06.409Z : Producer : Topic: topic2, TC: 6, GC: 12
2019-09-05T18:06:06.432Z : Producer : Topic: topic1, TC: 7, GC: 13
2019-09-05T18:06:06.455Z : Producer : Topic: topic2, TC: 7, GC: 14
2019-09-05T18:06:06.466Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 JUMP BACKWARDS 1
2019-09-05T18:06:06.477Z : Producer : Topic: topic1, TC: 8, GC: 15
2019-09-05T18:06:06.500Z : Producer : Topic: topic2, TC: 8, GC: 16
2019-09-05T18:06:06.524Z : Producer : Topic: topic1, TC: 9, GC: 17
2019-09-05T18:06:06.546Z : Producer : Topic: topic2, TC: 9, GC: 18
2019-09-05T18:06:06.568Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 JUMP FORWARD 2
2019-09-05T18:06:06.569Z : Producer : Topic: topic1, TC: 10, GC: 19
2019-09-05T18:06:06.592Z : Producer : Topic: topic2, TC: 10, GC: 20
2019-09-05T18:06:06.615Z : Producer : Topic: topic1, TC: 11, GC: 21
2019-09-05T18:06:06.638Z : Producer : Topic: topic2, TC: 11, GC: 22
2019-09-05T18:06:06.661Z : Producer : Topic: topic1, TC: 12, GC: 23
2019-09-05T18:06:06.669Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 JUMP FORWARD 1
2019-09-05T18:06:06.684Z : Producer : Topic: topic2, TC: 12, GC: 24
2019-09-05T18:06:06.707Z : Producer : Topic: topic1, TC: 13, GC: 25
2019-09-05T18:06:06.729Z : Producer : Topic: topic2, TC: 13, GC: 26
2019-09-05T18:06:06.752Z : Producer : Topic: topic1, TC: 14, GC: 27
2019-09-05T18:06:06.770Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 JUMP FORWARD 1
2019-09-05T18:06:06.775Z : Producer : Topic: topic2, TC: 14, GC: 28
2019-09-05T18:06:06.799Z : Producer : Topic: topic1, TC: 15, GC: 29
2019-09-05T18:06:06.822Z : Producer : Topic: topic2, TC: 15, GC: 30
2019-09-05T18:06:06.844Z : Producer : Topic: topic1, TC: 16, GC: 31
2019-09-05T18:06:06.867Z : Producer : Topic: topic2, TC: 16, GC: 32
2019-09-05T18:06:06.872Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 JUMP BACKWARDS 5
2019-09-05T18:06:06.889Z : Producer : Topic: topic1, TC: 17, GC: 33
2019-09-05T18:06:06.913Z : Producer : Topic: topic2, TC: 17, GC: 34
2019-09-05T18:06:06.936Z : Producer : Topic: topic1, TC: 18, GC: 35
2019-09-05T18:06:06.959Z : Producer : Topic: topic2, TC: 18, GC: 36
2019-09-05T18:06:06.973Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 JUMP FORWARD 1
2019-09-05T18:06:06.982Z : Producer : Topic: topic1, TC: 19, GC: 37
2019-09-05T18:06:07.005Z : Producer : Topic: topic2, TC: 19, GC: 38
2019-09-05T18:06:07.028Z : Producer : Topic: topic1, TC: 20, GC: 39
2019-09-05T18:06:07.052Z : Producer : Topic: topic2, TC: 20, GC: 40
2019-09-05T18:06:07.074Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 JUMP FORWARD 1
2019-09-05T18:06:07.075Z : Producer : Topic: topic1, TC: 21, GC: 41
2019-09-05T18:06:07.098Z : Producer : Topic: topic2, TC: 21, GC: 42
2019-09-05T18:06:07.121Z : Producer : Topic: topic1, TC: 22, GC: 43
2019-09-05T18:06:07.144Z : Producer : Topic: topic2, TC: 22, GC: 44
2019-09-05T18:06:07.168Z : Producer : Topic: topic1, TC: 23, GC: 45
2019-09-05T18:06:07.176Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 JUMP FORWARD 2
2019-09-05T18:06:07.191Z : Producer : Topic: topic2, TC: 23, GC: 46
2019-09-05T18:06:07.215Z : Producer : Topic: topic1, TC: 24, GC: 47
2019-09-05T18:06:07.238Z : Producer : Topic: topic2, TC: 24, GC: 48
2019-09-05T18:06:07.261Z : Producer : Topic: topic1, TC: 25, GC: 49
2019-09-05T18:06:07.276Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 JUMP FORWARD 1
2019-09-05T18:06:07.283Z : Producer : Topic: topic2, TC: 25, GC: 50
2019-09-05T18:06:07.306Z : Producer : Topic: topic1, TC: 26, GC: 51
2019-09-05T18:06:07.329Z : Producer : Topic: topic2, TC: 26, GC: 52
2019-09-05T18:06:07.353Z : Producer : Topic: topic1, TC: 27, GC: 53
2019-09-05T18:06:07.375Z : Producer : Topic: topic2, TC: 27, GC: 54
2019-09-05T18:06:07.378Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 JUMP FORWARD 1
2019-09-05T18:06:07.398Z : Producer : Topic: topic1, TC: 28, GC: 55
2019-09-05T18:06:07.421Z : Producer : Topic: topic2, TC: 28, GC: 56
2019-09-05T18:06:07.444Z : Producer : Topic: topic1, TC: 29, GC: 57
2019-09-05T18:06:07.467Z : Producer : Topic: topic2, TC: 29, GC: 58
2019-09-05T18:06:07.479Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 JUMP FORWARD 1
2019-09-05T18:06:07.489Z : Producer : Topic: topic1, TC: 30, GC: 59
2019-09-05T18:06:07.512Z : Producer : Topic: topic2, TC: 30, GC: 60
2019-09-05T18:06:07.534Z : Producer : Topic: topic1, TC: 31, GC: 61
2019-09-05T18:06:07.558Z : Producer : Topic: topic2, TC: 31, GC: 62
2019-09-05T18:06:07.580Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 JUMP BACKWARDS 7
2019-09-05T18:06:07.581Z : Producer : Topic: topic1, TC: 32, GC: 63
2019-09-05T18:06:07.603Z : Producer : Topic: topic2, TC: 32, GC: 64
2019-09-05T18:06:07.626Z : Producer : Topic: topic1, TC: 33, GC: 65
2019-09-05T18:06:07.650Z : Producer : Topic: topic2, TC: 33, GC: 66
2019-09-05T18:06:07.673Z : Producer : Topic: topic1, TC: 34, GC: 67
2019-09-05T18:06:07.680Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 JUMP FORWARD 1
2019-09-05T18:06:07.695Z : Producer : Topic: topic2, TC: 34, GC: 68
2019-09-05T18:06:07.718Z : Producer : Topic: topic1, TC: 35, GC: 69
2019-09-05T18:06:07.742Z : Producer : Topic: topic2, TC: 35, GC: 70
2019-09-05T18:06:07.765Z : Producer : Topic: topic1, TC: 36, GC: 71
2019-09-05T18:06:07.781Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 JUMP FORWARD 1
2019-09-05T18:06:07.788Z : Producer : Topic: topic2, TC: 36, GC: 72
2019-09-05T18:06:07.811Z : Producer : Topic: topic1, TC: 37, GC: 73
2019-09-05T18:06:07.834Z : Producer : Topic: topic2, TC: 37, GC: 74
2019-09-05T18:06:07.857Z : Producer : Topic: topic1, TC: 38, GC: 75
2019-09-05T18:06:07.879Z : Producer : Topic: topic2, TC: 38, GC: 76
2019-09-05T18:06:07.883Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 JUMP FORWARD 1
2019-09-05T18:06:07.903Z : Producer : Topic: topic1, TC: 39, GC: 77
2019-09-05T18:06:07.926Z : Producer : Topic: topic2, TC: 39, GC: 78
2019-09-05T18:06:07.948Z : Producer : Topic: topic1, TC: 40, GC: 79
2019-09-05T18:06:07.971Z : Producer : Topic: topic2, TC: 40, GC: 80
2019-09-05T18:06:07.987Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 JUMP FORWARD 2
2019-09-05T18:06:07.994Z : Producer : Topic: topic1, TC: 41, GC: 81
2019-09-05T18:06:08.016Z : Producer : Topic: topic2, TC: 41, GC: 82
2019-09-05T18:06:08.040Z : Producer : Topic: topic1, TC: 42, GC: 83
2019-09-05T18:06:08.062Z : Producer : Topic: topic2, TC: 42, GC: 84
2019-09-05T18:06:08.085Z : Producer : Topic: topic1, TC: 43, GC: 85
2019-09-05T18:06:08.088Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 JUMP FORWARD 1
2019-09-05T18:06:08.108Z : Producer : Topic: topic2, TC: 43, GC: 86
2019-09-05T18:06:08.132Z : Producer : Topic: topic1, TC: 44, GC: 87
2019-09-05T18:06:08.154Z : Producer : Topic: topic2, TC: 44, GC: 88
2019-09-05T18:06:08.177Z : Producer : Topic: topic1, TC: 45, GC: 89
2019-09-05T18:06:08.188Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 JUMP FORWARD 1
2019-09-05T18:06:08.200Z : Producer : Topic: topic2, TC: 45, GC: 90
2019-09-05T18:06:08.224Z : Producer : Topic: topic1, TC: 46, GC: 91
2019-09-05T18:06:08.247Z : Producer : Topic: topic2, TC: 46, GC: 92
2019-09-05T18:06:08.269Z : Producer : Topic: topic1, TC: 47, GC: 93
2019-09-05T18:06:08.289Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 JUMP FORWARD 1
2019-09-05T18:06:08.292Z : Producer : Topic: topic2, TC: 47, GC: 94
2019-09-05T18:06:08.315Z : Producer : Topic: topic1, TC: 48, GC: 95
2019-09-05T18:06:08.337Z : Producer : Topic: topic2, TC: 48, GC: 96
2019-09-05T18:06:08.360Z : Producer : Topic: topic1, TC: 49, GC: 97
2019-09-05T18:06:08.383Z : Producer : Topic: topic2, TC: 49, GC: 98
2019-09-05T18:06:08.390Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 JUMP FORWARD 1
2019-09-05T18:06:08.406Z : Producer : Topic: topic1, TC: 50, GC: 99
2019-09-05T18:06:08.429Z : Producer : Topic: topic2, TC: 50, GC: 100
2019-09-05T18:06:08.491Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 JUMP FORWARD 1
2019-09-05T18:06:08.592Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 JUMP FORWARD 1
2019-09-05T18:06:08.693Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 JUMP FORWARD 1
2019-09-05T18:06:08.794Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 JUMP FORWARD 1
2019-09-05T18:06:08.896Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 JUMP FORWARD 1
2019-09-05T18:06:08.997Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 JUMP FORWARD 1
2019-09-05T18:06:09.098Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 JUMP FORWARD 1
2019-09-05T18:06:09.199Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 JUMP FORWARD 1
2019-09-05T18:06:09.300Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 JUMP BACKWARDS 25
2019-09-05T18:06:09.401Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 JUMP FORWARD 1
2019-09-05T18:06:09.502Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 JUMP FORWARD 1
2019-09-05T18:06:09.603Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 JUMP FORWARD 1
2019-09-05T18:06:09.705Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 JUMP FORWARD 1
2019-09-05T18:06:09.806Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 JUMP FORWARD 1
2019-09-05T18:06:09.907Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 JUMP FORWARD 1
2019-09-05T18:06:10.008Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 JUMP FORWARD 1
2019-09-05T18:06:10.109Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 JUMP FORWARD 1
2019-09-05T18:06:10.209Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 JUMP FORWARD 1
2019-09-05T18:06:10.310Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 JUMP FORWARD 1
2019-09-05T18:06:10.412Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 JUMP FORWARD 1
2019-09-05T18:06:10.512Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 JUMP FORWARD 1
2019-09-05T18:06:10.614Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 JUMP FORWARD 1
2019-09-05T18:06:10.716Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 OK
2019-09-05T18:06:10.817Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 JUMP FORWARD 1
2019-09-05T18:06:10.918Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 JUMP FORWARD 1
2019-09-05T18:06:11.019Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 JUMP FORWARD 1
2019-09-05T18:06:11.120Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 JUMP FORWARD 1
2019-09-05T18:06:11.221Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 JUMP FORWARD 1
2019-09-05T18:06:11.322Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 JUMP FORWARD 1
2019-09-05T18:06:11.423Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 JUMP FORWARD 1
2019-09-05T18:06:11.524Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 JUMP FORWARD 1
2019-09-05T18:06:11.626Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 JUMP FORWARD 1
2019-09-05T18:06:11.726Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 JUMP FORWARD 1
2019-09-05T18:06:11.827Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 JUMP FORWARD 1
2019-09-05T18:06:11.928Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 JUMP FORWARD 1
2019-09-05T18:06:12.030Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 JUMP FORWARD 1
2019-09-05T18:06:12.131Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 JUMP FORWARD 1
2019-09-05T18:06:12.232Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 JUMP FORWARD 1
2019-09-05T18:06:12.332Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 JUMP FORWARD 1
2019-09-05T18:06:12.433Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 JUMP FORWARD 1
2019-09-05T18:06:12.534Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 JUMP BACKWARDS 33
2019-09-05T18:06:12.634Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 JUMP FORWARD 1
2019-09-05T18:06:12.736Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 JUMP FORWARD 1
2019-09-05T18:06:12.836Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 JUMP FORWARD 1
2019-09-05T18:06:12.937Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 JUMP FORWARD 1
2019-09-05T18:06:13.038Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 JUMP FORWARD 1
2019-09-05T18:06:13.139Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 JUMP FORWARD 1
2019-09-05T18:06:13.240Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 JUMP FORWARD 1
2019-09-05T18:06:13.340Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 JUMP FORWARD 1
2019-09-05T18:06:13.441Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 JUMP FORWARD 1
2019-09-05T18:06:13.542Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 JUMP FORWARD 1
2019-09-05T18:06:13.643Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 JUMP FORWARD 1
2019-09-05T18:06:13.744Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 JUMP FORWARD 1
2019-09-05T18:06:13.845Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 JUMP FORWARD 1
2019-09-05T18:06:13.946Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 JUMP FORWARD 1
2019-09-05T18:06:14.047Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 JUMP FORWARD 1
2019-09-05T18:06:14.148Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 JUMP FORWARD 1
2019-09-05T18:06:14.253Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 JUMP FORWARD 2
2019-09-05T18:06:14.354Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 JUMP FORWARD 1
2019-09-05T18:06:14.455Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 JUMP FORWARD 1
2019-09-05T18:06:14.556Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 JUMP FORWARD 1
2019-09-05T18:06:14.657Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 JUMP FORWARD 1
2019-09-05T18:06:14.758Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 JUMP FORWARD 1
2019-09-05T18:06:14.859Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 JUMP FORWARD 1
2019-09-05T18:06:14.960Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 JUMP FORWARD 1
2019-09-05T18:06:15.061Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 JUMP FORWARD 1
2019-09-05T18:06:15.162Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 JUMP FORWARD 1
2019-09-05T18:06:15.263Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 JUMP BACKWARDS 19
2019-09-05T18:06:15.364Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 JUMP FORWARD 1
2019-09-05T18:06:15.465Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 JUMP FORWARD 1
2019-09-05T18:06:15.566Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 JUMP FORWARD 1
2019-09-05T18:06:15.667Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 JUMP FORWARD 1
2019-09-05T18:06:15.767Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 JUMP FORWARD 1
2019-09-05T18:06:15.868Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 JUMP FORWARD 1
2019-09-05T18:06:15.969Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 JUMP FORWARD 1
2019-09-05T18:06:16.070Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 JUMP FORWARD 1
2019-09-05T18:06:16.171Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 JUMP FORWARD 1

It seems that as soon as the consumer cannot keep up with the production rate, we get poor ordering across topics.

Case 4 - Tailing the topic,unequal message arrival rate, constant processing time

So far we’ve seen two topics with identical message arrival rates with messages arriving in lock step, which is not how messages arrive in the real world. Usually topics will have different arrival rates. The below result shows zero processing time, like case 1, but with topic 1 getting ten times the messages of topic 2.

2019-09-05T17:48:08.082Z : Consumer : Subscribing
2019-09-05T17:48:08.134Z : Consumer : Partitions revoked at 2019-09-05T17:48:08.134Z
2019-09-05T17:48:11.156Z : Consumer : Topic: topic1 Partition: 0 assigned at 2019-09-05T17:48:11.156Z
2019-09-05T17:48:11.156Z : Consumer : Topic: topic2 Partition: 0 assigned at 2019-09-05T17:48:11.156Z
2019-09-05T17:48:11.215Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T17:48:11.217Z : Producer : Topic: topic1, TC: 2, GC: 2
2019-09-05T17:48:11.219Z : Producer : Topic: topic1, TC: 3, GC: 3
2019-09-05T17:48:11.220Z : Producer : Topic: topic1, TC: 4, GC: 4
2019-09-05T17:48:11.222Z : Producer : Topic: topic1, TC: 5, GC: 5
2019-09-05T17:48:11.222Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T17:48:11.223Z : Producer : Topic: topic1, TC: 6, GC: 6
2019-09-05T17:48:11.224Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
2019-09-05T17:48:11.224Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
2019-09-05T17:48:11.224Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
2019-09-05T17:48:11.224Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
2019-09-05T17:48:11.225Z : Producer : Topic: topic1, TC: 7, GC: 7
2019-09-05T17:48:11.226Z : Producer : Topic: topic1, TC: 8, GC: 8
2019-09-05T17:48:11.227Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
2019-09-05T17:48:11.227Z : Producer : Topic: topic1, TC: 9, GC: 9
2019-09-05T17:48:11.229Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
2019-09-05T17:48:11.229Z : Producer : Topic: topic1, TC: 10, GC: 10
2019-09-05T17:48:11.229Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
2019-09-05T17:48:11.229Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
2019-09-05T17:48:11.230Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
2019-09-05T17:48:11.235Z : Producer : Topic: topic2, TC: 1, GC: 11
2019-09-05T17:48:11.237Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 OK
2019-09-05T17:48:11.237Z : Producer : Topic: topic1, TC: 11, GC: 12
2019-09-05T17:48:11.238Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 OK
2019-09-05T17:48:11.239Z : Producer : Topic: topic1, TC: 12, GC: 13
2019-09-05T17:48:11.240Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
2019-09-05T17:48:11.241Z : Producer : Topic: topic1, TC: 13, GC: 14
2019-09-05T17:48:11.242Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
2019-09-05T17:48:11.243Z : Producer : Topic: topic1, TC: 14, GC: 15
2019-09-05T17:48:11.243Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
2019-09-05T17:48:11.244Z : Producer : Topic: topic1, TC: 15, GC: 16
2019-09-05T17:48:11.245Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
2019-09-05T17:48:11.246Z : Producer : Topic: topic1, TC: 16, GC: 17
2019-09-05T17:48:11.247Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
2019-09-05T17:48:11.248Z : Producer : Topic: topic1, TC: 17, GC: 18
2019-09-05T17:48:11.249Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 OK
2019-09-05T17:48:11.249Z : Producer : Topic: topic1, TC: 18, GC: 19
2019-09-05T17:48:11.250Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
2019-09-05T17:48:11.251Z : Producer : Topic: topic1, TC: 19, GC: 20
2019-09-05T17:48:11.252Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
2019-09-05T17:48:11.253Z : Producer : Topic: topic1, TC: 20, GC: 21
2019-09-05T17:48:11.254Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
2019-09-05T17:48:11.256Z : Producer : Topic: topic2, TC: 2, GC: 22
2019-09-05T17:48:11.257Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 OK
2019-09-05T17:48:11.257Z : Producer : Topic: topic1, TC: 21, GC: 23
2019-09-05T17:48:11.258Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 OK
2019-09-05T17:48:11.259Z : Producer : Topic: topic1, TC: 22, GC: 24
2019-09-05T17:48:11.260Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
2019-09-05T17:48:11.261Z : Producer : Topic: topic1, TC: 23, GC: 25
2019-09-05T17:48:11.262Z : Producer : Topic: topic1, TC: 24, GC: 26
2019-09-05T17:48:11.262Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
2019-09-05T17:48:11.264Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
2019-09-05T17:48:11.264Z : Producer : Topic: topic1, TC: 25, GC: 27
2019-09-05T17:48:11.265Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
2019-09-05T17:48:11.265Z : Producer : Topic: topic1, TC: 26, GC: 28
2019-09-05T17:48:11.266Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
2019-09-05T17:48:11.267Z : Producer : Topic: topic1, TC: 27, GC: 29
2019-09-05T17:48:11.268Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
2019-09-05T17:48:11.269Z : Producer : Topic: topic1, TC: 28, GC: 30
2019-09-05T17:48:11.269Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
2019-09-05T17:48:11.271Z : Producer : Topic: topic1, TC: 29, GC: 31
2019-09-05T17:48:11.271Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
2019-09-05T17:48:11.272Z : Producer : Topic: topic1, TC: 30, GC: 32
2019-09-05T17:48:11.273Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
2019-09-05T17:48:11.274Z : Producer : Topic: topic2, TC: 3, GC: 33
2019-09-05T17:48:11.275Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 OK
2019-09-05T17:48:11.276Z : Producer : Topic: topic1, TC: 31, GC: 34
2019-09-05T17:48:11.277Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 OK
2019-09-05T17:48:11.278Z : Producer : Topic: topic1, TC: 32, GC: 35
2019-09-05T17:48:11.278Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
2019-09-05T17:48:11.279Z : Producer : Topic: topic1, TC: 33, GC: 36
2019-09-05T17:48:11.280Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
2019-09-05T17:48:11.280Z : Producer : Topic: topic1, TC: 34, GC: 37
2019-09-05T17:48:11.281Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
2019-09-05T17:48:11.282Z : Producer : Topic: topic1, TC: 35, GC: 38
2019-09-05T17:48:11.282Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
2019-09-05T17:48:11.284Z : Producer : Topic: topic1, TC: 36, GC: 39
2019-09-05T17:48:11.284Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
2019-09-05T17:48:11.285Z : Producer : Topic: topic1, TC: 37, GC: 40
2019-09-05T17:48:11.286Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
2019-09-05T17:48:11.287Z : Producer : Topic: topic1, TC: 38, GC: 41
2019-09-05T17:48:11.287Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
2019-09-05T17:48:11.289Z : Producer : Topic: topic1, TC: 39, GC: 42
2019-09-05T17:48:11.290Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
2019-09-05T17:48:11.291Z : Producer : Topic: topic1, TC: 40, GC: 43
2019-09-05T17:48:11.291Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
2019-09-05T17:48:11.293Z : Producer : Topic: topic2, TC: 4, GC: 44
2019-09-05T17:48:11.293Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 OK
2019-09-05T17:48:11.294Z : Producer : Topic: topic1, TC: 41, GC: 45
2019-09-05T17:48:11.295Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 OK
2019-09-05T17:48:11.296Z : Producer : Topic: topic1, TC: 42, GC: 46
2019-09-05T17:48:11.297Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
2019-09-05T17:48:11.297Z : Producer : Topic: topic1, TC: 43, GC: 47
2019-09-05T17:48:11.298Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
2019-09-05T17:48:11.299Z : Producer : Topic: topic1, TC: 44, GC: 48
2019-09-05T17:48:11.300Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
2019-09-05T17:48:11.301Z : Producer : Topic: topic1, TC: 45, GC: 49
2019-09-05T17:48:11.301Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
2019-09-05T17:48:11.302Z : Producer : Topic: topic1, TC: 46, GC: 50
2019-09-05T17:48:11.303Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
2019-09-05T17:48:11.304Z : Producer : Topic: topic1, TC: 47, GC: 51
2019-09-05T17:48:11.304Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
2019-09-05T17:48:11.305Z : Producer : Topic: topic1, TC: 48, GC: 52
2019-09-05T17:48:11.306Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
2019-09-05T17:48:11.307Z : Producer : Topic: topic1, TC: 49, GC: 53
2019-09-05T17:48:11.307Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
2019-09-05T17:48:11.308Z : Producer : Topic: topic1, TC: 50, GC: 54
2019-09-05T17:48:11.309Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
2019-09-05T17:48:11.310Z : Producer : Topic: topic2, TC: 5, GC: 55
2019-09-05T17:48:11.311Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 OK
2019-09-05T17:48:11.312Z : Producer : Topic: topic1, TC: 51, GC: 56
2019-09-05T17:48:11.313Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 OK
2019-09-05T17:48:11.314Z : Producer : Topic: topic1, TC: 52, GC: 57
2019-09-05T17:48:11.314Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
2019-09-05T17:48:11.315Z : Producer : Topic: topic1, TC: 53, GC: 58
2019-09-05T17:48:11.316Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
2019-09-05T17:48:11.317Z : Producer : Topic: topic1, TC: 54, GC: 59
2019-09-05T17:48:11.317Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
2019-09-05T17:48:11.318Z : Producer : Topic: topic1, TC: 55, GC: 60
2019-09-05T17:48:11.319Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
2019-09-05T17:48:11.320Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
2019-09-05T17:48:11.320Z : Producer : Topic: topic1, TC: 56, GC: 61
2019-09-05T17:48:11.323Z : Producer : Topic: topic1, TC: 57, GC: 62
2019-09-05T17:48:11.323Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
2019-09-05T17:48:11.324Z : Producer : Topic: topic1, TC: 58, GC: 63
2019-09-05T17:48:11.325Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
2019-09-05T17:48:11.325Z : Producer : Topic: topic1, TC: 59, GC: 64
2019-09-05T17:48:11.327Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
2019-09-05T17:48:11.327Z : Producer : Topic: topic1, TC: 60, GC: 65
2019-09-05T17:48:11.328Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
2019-09-05T17:48:11.329Z : Producer : Topic: topic2, TC: 6, GC: 66
2019-09-05T17:48:11.329Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 OK
2019-09-05T17:48:11.331Z : Producer : Topic: topic1, TC: 61, GC: 67
2019-09-05T17:48:11.331Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 OK
2019-09-05T17:48:11.332Z : Producer : Topic: topic1, TC: 62, GC: 68
2019-09-05T17:48:11.333Z : Producer : Topic: topic1, TC: 63, GC: 69
2019-09-05T17:48:11.334Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
2019-09-05T17:48:11.335Z : Producer : Topic: topic1, TC: 64, GC: 70
2019-09-05T17:48:11.335Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
2019-09-05T17:48:11.336Z : Producer : Topic: topic1, TC: 65, GC: 71
2019-09-05T17:48:11.336Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
2019-09-05T17:48:11.337Z : Producer : Topic: topic1, TC: 66, GC: 72
2019-09-05T17:48:11.337Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
2019-09-05T17:48:11.338Z : Producer : Topic: topic1, TC: 67, GC: 73
2019-09-05T17:48:11.338Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
2019-09-05T17:48:11.339Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
2019-09-05T17:48:11.339Z : Producer : Topic: topic1, TC: 68, GC: 74
2019-09-05T17:48:11.340Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
2019-09-05T17:48:11.341Z : Producer : Topic: topic1, TC: 69, GC: 75
2019-09-05T17:48:11.341Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
2019-09-05T17:48:11.342Z : Producer : Topic: topic1, TC: 70, GC: 76
2019-09-05T17:48:11.343Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
2019-09-05T17:48:11.344Z : Producer : Topic: topic2, TC: 7, GC: 77
2019-09-05T17:48:11.344Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 OK
2019-09-05T17:48:11.345Z : Producer : Topic: topic1, TC: 71, GC: 78
2019-09-05T17:48:11.346Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 OK
2019-09-05T17:48:11.347Z : Producer : Topic: topic1, TC: 72, GC: 79
2019-09-05T17:48:11.347Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
2019-09-05T17:48:11.348Z : Producer : Topic: topic1, TC: 73, GC: 80
2019-09-05T17:48:11.349Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
2019-09-05T17:48:11.349Z : Producer : Topic: topic1, TC: 74, GC: 81
2019-09-05T17:48:11.350Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
2019-09-05T17:48:11.351Z : Producer : Topic: topic1, TC: 75, GC: 82
2019-09-05T17:48:11.351Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
2019-09-05T17:48:11.352Z : Producer : Topic: topic1, TC: 76, GC: 83
2019-09-05T17:48:11.352Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
2019-09-05T17:48:11.353Z : Producer : Topic: topic1, TC: 77, GC: 84
2019-09-05T17:48:11.354Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
2019-09-05T17:48:11.354Z : Producer : Topic: topic1, TC: 78, GC: 85
2019-09-05T17:48:11.355Z : Producer : Topic: topic1, TC: 79, GC: 86
2019-09-05T17:48:11.356Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
2019-09-05T17:48:11.357Z : Producer : Topic: topic1, TC: 80, GC: 87
2019-09-05T17:48:11.357Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
2019-09-05T17:48:11.358Z : Producer : Topic: topic2, TC: 8, GC: 88
2019-09-05T17:48:11.359Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
2019-09-05T17:48:11.359Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 OK
2019-09-05T17:48:11.360Z : Producer : Topic: topic1, TC: 81, GC: 89
2019-09-05T17:48:11.361Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 OK
2019-09-05T17:48:11.362Z : Producer : Topic: topic1, TC: 82, GC: 90
2019-09-05T17:48:11.362Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
2019-09-05T17:48:11.363Z : Producer : Topic: topic1, TC: 83, GC: 91
2019-09-05T17:48:11.364Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
2019-09-05T17:48:11.364Z : Producer : Topic: topic1, TC: 84, GC: 92
2019-09-05T17:48:11.365Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
2019-09-05T17:48:11.366Z : Producer : Topic: topic1, TC: 85, GC: 93
2019-09-05T17:48:11.366Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
2019-09-05T17:48:11.368Z : Producer : Topic: topic1, TC: 86, GC: 94
2019-09-05T17:48:11.368Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
2019-09-05T17:48:11.369Z : Producer : Topic: topic1, TC: 87, GC: 95
2019-09-05T17:48:11.370Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
2019-09-05T17:48:11.371Z : Producer : Topic: topic1, TC: 88, GC: 96
2019-09-05T17:48:11.371Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
2019-09-05T17:48:11.372Z : Producer : Topic: topic1, TC: 89, GC: 97
2019-09-05T17:48:11.373Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
2019-09-05T17:48:11.374Z : Producer : Topic: topic1, TC: 90, GC: 98
2019-09-05T17:48:11.374Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
2019-09-05T17:48:11.375Z : Producer : Topic: topic2, TC: 9, GC: 99
2019-09-05T17:48:11.376Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 OK
2019-09-05T17:48:11.377Z : Producer : Topic: topic1, TC: 91, GC: 100
2019-09-05T17:48:11.377Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 OK

Looks like message arrival rates do not affect our ordering when the consumer is able to keep up.

Case 5 - Tailing the topic, unequal message arrival rate, processing time with jitter

2019-09-05T17:54:28.743Z : Consumer : Subscribing
2019-09-05T17:54:28.795Z : Consumer : Partitions revoked at 2019-09-05T17:54:28.795Z
2019-09-05T17:54:31.816Z : Consumer : Topic: topic2_7900 Partition: 0 assigned at 2019-09-05T17:54:31.816Z
2019-09-05T17:54:31.816Z : Consumer : Topic: topic1_7900 Partition: 0 assigned at 2019-09-05T17:54:31.816Z
2019-09-05T17:54:31.882Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T17:54:31.888Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T17:54:31.911Z : Producer : Topic: topic1, TC: 2, GC: 2
2019-09-05T17:54:31.939Z : Producer : Topic: topic1, TC: 3, GC: 3
2019-09-05T17:54:31.966Z : Producer : Topic: topic1, TC: 4, GC: 4
2019-09-05T17:54:31.994Z : Producer : Topic: topic1, TC: 5, GC: 5
2019-09-05T17:54:32.021Z : Producer : Topic: topic1, TC: 6, GC: 6
2019-09-05T17:54:32.049Z : Producer : Topic: topic1, TC: 7, GC: 7
2019-09-05T17:54:32.077Z : Producer : Topic: topic1, TC: 8, GC: 8
2019-09-05T17:54:32.093Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
2019-09-05T17:54:32.105Z : Producer : Topic: topic1, TC: 9, GC: 9
2019-09-05T17:54:32.133Z : Producer : Topic: topic1, TC: 10, GC: 10
2019-09-05T17:54:32.166Z : Producer : Topic: topic2, TC: 1, GC: 11
2019-09-05T17:54:32.193Z : Producer : Topic: topic1, TC: 11, GC: 12
2019-09-05T17:54:32.221Z : Producer : Topic: topic1, TC: 12, GC: 13
2019-09-05T17:54:32.249Z : Producer : Topic: topic1, TC: 13, GC: 14
2019-09-05T17:54:32.277Z : Producer : Topic: topic1, TC: 14, GC: 15
2019-09-05T17:54:32.305Z : Producer : Topic: topic1, TC: 15, GC: 16
2019-09-05T17:54:32.333Z : Producer : Topic: topic1, TC: 16, GC: 17
2019-09-05T17:54:32.343Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 JUMP FORWARD 8
2019-09-05T17:54:32.361Z : Producer : Topic: topic1, TC: 17, GC: 18
2019-09-05T17:54:32.365Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 JUMP BACKWARDS 8
2019-09-05T17:54:32.389Z : Producer : Topic: topic1, TC: 18, GC: 19
2019-09-05T17:54:32.415Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
2019-09-05T17:54:32.417Z : Producer : Topic: topic1, TC: 19, GC: 20
2019-09-05T17:54:32.446Z : Producer : Topic: topic1, TC: 20, GC: 21
2019-09-05T17:54:32.465Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
2019-09-05T17:54:32.473Z : Producer : Topic: topic2, TC: 2, GC: 22
2019-09-05T17:54:32.484Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
2019-09-05T17:54:32.500Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
2019-09-05T17:54:32.501Z : Producer : Topic: topic1, TC: 21, GC: 23
2019-09-05T17:54:32.529Z : Producer : Topic: topic1, TC: 22, GC: 24
2019-09-05T17:54:32.557Z : Producer : Topic: topic1, TC: 23, GC: 25
2019-09-05T17:54:32.584Z : Producer : Topic: topic1, TC: 24, GC: 26
2019-09-05T17:54:32.612Z : Producer : Topic: topic1, TC: 25, GC: 27
2019-09-05T17:54:32.641Z : Producer : Topic: topic1, TC: 26, GC: 28
2019-09-05T17:54:32.668Z : Producer : Topic: topic1, TC: 27, GC: 29
2019-09-05T17:54:32.696Z : Producer : Topic: topic1, TC: 28, GC: 30
2019-09-05T17:54:32.719Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
2019-09-05T17:54:32.724Z : Producer : Topic: topic1, TC: 29, GC: 31
2019-09-05T17:54:32.749Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 JUMP FORWARD 13
2019-09-05T17:54:32.751Z : Producer : Topic: topic1, TC: 30, GC: 32
2019-09-05T17:54:32.758Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 JUMP BACKWARDS 13
2019-09-05T17:54:32.779Z : Producer : Topic: topic2, TC: 3, GC: 33
2019-09-05T17:54:32.800Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
2019-09-05T17:54:32.807Z : Producer : Topic: topic1, TC: 31, GC: 34
2019-09-05T17:54:32.812Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 JUMP FORWARD 1
2019-09-05T17:54:32.816Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
2019-09-05T17:54:32.835Z : Producer : Topic: topic1, TC: 32, GC: 35
2019-09-05T17:54:32.841Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
2019-09-05T17:54:32.849Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
2019-09-05T17:54:32.863Z : Producer : Topic: topic1, TC: 33, GC: 36
2019-09-05T17:54:32.865Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
2019-09-05T17:54:32.891Z : Producer : Topic: topic1, TC: 34, GC: 37
2019-09-05T17:54:32.907Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
2019-09-05T17:54:32.919Z : Producer : Topic: topic1, TC: 35, GC: 38
2019-09-05T17:54:32.926Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 JUMP FORWARD 15
2019-09-05T17:54:32.945Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 JUMP BACKWARDS 15
2019-09-05T17:54:32.948Z : Producer : Topic: topic1, TC: 36, GC: 39
2019-09-05T17:54:32.954Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
2019-09-05T17:54:32.976Z : Producer : Topic: topic1, TC: 37, GC: 40
2019-09-05T17:54:32.994Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
2019-09-05T17:54:33.003Z : Producer : Topic: topic1, TC: 38, GC: 41
2019-09-05T17:54:33.030Z : Producer : Topic: topic1, TC: 39, GC: 42
2019-09-05T17:54:33.033Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
2019-09-05T17:54:33.050Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 JUMP FORWARD 1
2019-09-05T17:54:33.053Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
2019-09-05T17:54:33.059Z : Producer : Topic: topic1, TC: 40, GC: 43
2019-09-05T17:54:33.084Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
2019-09-05T17:54:33.088Z : Producer : Topic: topic2, TC: 4, GC: 44
2019-09-05T17:54:33.116Z : Producer : Topic: topic1, TC: 41, GC: 45
2019-09-05T17:54:33.144Z : Producer : Topic: topic1, TC: 42, GC: 46
2019-09-05T17:54:33.172Z : Producer : Topic: topic1, TC: 43, GC: 47
2019-09-05T17:54:33.200Z : Producer : Topic: topic1, TC: 44, GC: 48
2019-09-05T17:54:33.228Z : Producer : Topic: topic1, TC: 45, GC: 49
2019-09-05T17:54:33.255Z : Producer : Topic: topic1, TC: 46, GC: 50
2019-09-05T17:54:33.283Z : Producer : Topic: topic1, TC: 47, GC: 51
2019-09-05T17:54:33.287Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
2019-09-05T17:54:33.311Z : Producer : Topic: topic1, TC: 48, GC: 52
2019-09-05T17:54:33.329Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
2019-09-05T17:54:33.338Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
2019-09-05T17:54:33.338Z : Producer : Topic: topic1, TC: 49, GC: 53
2019-09-05T17:54:33.364Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
2019-09-05T17:54:33.367Z : Producer : Topic: topic1, TC: 50, GC: 54
2019-09-05T17:54:33.368Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
2019-09-05T17:54:33.394Z : Producer : Topic: topic2, TC: 5, GC: 55
2019-09-05T17:54:33.406Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
2019-09-05T17:54:33.422Z : Producer : Topic: topic1, TC: 51, GC: 56
2019-09-05T17:54:33.443Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 JUMP FORWARD 12
2019-09-05T17:54:33.450Z : Producer : Topic: topic1, TC: 52, GC: 57
2019-09-05T17:54:33.454Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 JUMP BACKWARDS 12
2019-09-05T17:54:33.461Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 JUMP FORWARD 1
2019-09-05T17:54:33.469Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
2019-09-05T17:54:33.478Z : Producer : Topic: topic1, TC: 53, GC: 58
2019-09-05T17:54:33.491Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
2019-09-05T17:54:33.506Z : Producer : Topic: topic1, TC: 54, GC: 59
2019-09-05T17:54:33.534Z : Producer : Topic: topic1, TC: 55, GC: 60
2019-09-05T17:54:33.540Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
2019-09-05T17:54:33.562Z : Producer : Topic: topic1, TC: 56, GC: 61
2019-09-05T17:54:33.588Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
2019-09-05T17:54:33.590Z : Producer : Topic: topic1, TC: 57, GC: 62
2019-09-05T17:54:33.604Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 JUMP FORWARD 16
2019-09-05T17:54:33.617Z : Producer : Topic: topic1, TC: 58, GC: 63
2019-09-05T17:54:33.645Z : Producer : Topic: topic1, TC: 59, GC: 64
2019-09-05T17:54:33.648Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 JUMP BACKWARDS 16
2019-09-05T17:54:33.672Z : Producer : Topic: topic1, TC: 60, GC: 65
2019-09-05T17:54:33.683Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
2019-09-05T17:54:33.700Z : Producer : Topic: topic2, TC: 6, GC: 66
2019-09-05T17:54:33.727Z : Producer : Topic: topic1, TC: 61, GC: 67
2019-09-05T17:54:33.733Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
2019-09-05T17:54:33.743Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
2019-09-05T17:54:33.756Z : Producer : Topic: topic1, TC: 62, GC: 68
2019-09-05T17:54:33.781Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
2019-09-05T17:54:33.784Z : Producer : Topic: topic1, TC: 63, GC: 69
2019-09-05T17:54:33.812Z : Producer : Topic: topic1, TC: 64, GC: 70
2019-09-05T17:54:33.814Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 JUMP FORWARD 1
2019-09-05T17:54:33.840Z : Producer : Topic: topic1, TC: 65, GC: 71
2019-09-05T17:54:33.842Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
2019-09-05T17:54:33.868Z : Producer : Topic: topic1, TC: 66, GC: 72
2019-09-05T17:54:33.879Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
2019-09-05T17:54:33.895Z : Producer : Topic: topic1, TC: 67, GC: 73
2019-09-05T17:54:33.923Z : Producer : Topic: topic1, TC: 68, GC: 74
2019-09-05T17:54:33.925Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
2019-09-05T17:54:33.951Z : Producer : Topic: topic1, TC: 69, GC: 75
2019-09-05T17:54:33.975Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
2019-09-05T17:54:33.978Z : Producer : Topic: topic1, TC: 70, GC: 76
2019-09-05T17:54:34.007Z : Producer : Topic: topic2, TC: 7, GC: 77
2019-09-05T17:54:34.014Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
2019-09-05T17:54:34.035Z : Producer : Topic: topic1, TC: 71, GC: 78
2019-09-05T17:54:34.039Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
2019-09-05T17:54:34.063Z : Producer : Topic: topic1, TC: 72, GC: 79
2019-09-05T17:54:34.089Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
2019-09-05T17:54:34.091Z : Producer : Topic: topic1, TC: 73, GC: 80
2019-09-05T17:54:34.119Z : Producer : Topic: topic1, TC: 74, GC: 81
2019-09-05T17:54:34.121Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
2019-09-05T17:54:34.147Z : Producer : Topic: topic1, TC: 75, GC: 82
2019-09-05T17:54:34.156Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
2019-09-05T17:54:34.175Z : Producer : Topic: topic1, TC: 76, GC: 83
2019-09-05T17:54:34.202Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 JUMP FORWARD 1
2019-09-05T17:54:34.203Z : Producer : Topic: topic1, TC: 77, GC: 84
2019-09-05T17:54:34.232Z : Producer : Topic: topic1, TC: 78, GC: 85
2019-09-05T17:54:34.247Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 JUMP FORWARD 9
2019-09-05T17:54:34.257Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 JUMP BACKWARDS 9
2019-09-05T17:54:34.259Z : Producer : Topic: topic1, TC: 79, GC: 86
2019-09-05T17:54:34.288Z : Producer : Topic: topic1, TC: 80, GC: 87
2019-09-05T17:54:34.297Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
2019-09-05T17:54:34.312Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
2019-09-05T17:54:34.316Z : Producer : Topic: topic2, TC: 8, GC: 88
2019-09-05T17:54:34.343Z : Producer : Topic: topic1, TC: 81, GC: 89
2019-09-05T17:54:34.350Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
2019-09-05T17:54:34.367Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
2019-09-05T17:54:34.371Z : Producer : Topic: topic1, TC: 82, GC: 90
2019-09-05T17:54:34.398Z : Producer : Topic: topic1, TC: 83, GC: 91
2019-09-05T17:54:34.406Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
2019-09-05T17:54:34.427Z : Producer : Topic: topic1, TC: 84, GC: 92
2019-09-05T17:54:34.455Z : Producer : Topic: topic1, TC: 85, GC: 93
2019-09-05T17:54:34.457Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 JUMP FORWARD 14
2019-09-05T17:54:34.463Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 JUMP BACKWARDS 14
2019-09-05T17:54:34.471Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
2019-09-05T17:54:34.477Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
2019-09-05T17:54:34.483Z : Producer : Topic: topic1, TC: 86, GC: 94
2019-09-05T17:54:34.493Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 JUMP FORWARD 1
2019-09-05T17:54:34.498Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
2019-09-05T17:54:34.511Z : Producer : Topic: topic1, TC: 87, GC: 95
2019-09-05T17:54:34.538Z : Producer : Topic: topic1, TC: 88, GC: 96
2019-09-05T17:54:34.539Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
2019-09-05T17:54:34.562Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
2019-09-05T17:54:34.566Z : Producer : Topic: topic1, TC: 89, GC: 97
2019-09-05T17:54:34.587Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
2019-09-05T17:54:34.594Z : Producer : Topic: topic1, TC: 90, GC: 98
2019-09-05T17:54:34.609Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
2019-09-05T17:54:34.622Z : Producer : Topic: topic2, TC: 9, GC: 99
2019-09-05T17:54:34.649Z : Producer : Topic: topic1, TC: 91, GC: 100
2019-09-05T17:54:34.656Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
2019-09-05T17:54:34.674Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
2019-09-05T17:54:34.721Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
2019-09-05T17:54:34.749Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
2019-09-05T17:54:34.785Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 JUMP FORWARD 1
2019-09-05T17:54:34.827Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
2019-09-05T17:54:34.836Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
2019-09-05T17:54:34.842Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
2019-09-05T17:54:34.884Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
2019-09-05T17:54:34.906Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
2019-09-05T17:54:34.943Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
2019-09-05T17:54:34.961Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
2019-09-05T17:54:34.993Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 JUMP FORWARD 2
2019-09-05T17:54:35.034Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 JUMP BACKWARDS 2
2019-09-05T17:54:35.050Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
2019-09-05T17:54:35.076Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 JUMP FORWARD 1
2019-09-05T17:54:35.122Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
2019-09-05T17:54:35.144Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
2019-09-05T17:54:35.175Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
2019-09-05T17:54:35.203Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
2019-09-05T17:54:35.218Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 JUMP FORWARD 5
2019-09-05T17:54:35.264Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 JUMP BACKWARDS 5
2019-09-05T17:54:35.294Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
2019-09-05T17:54:35.516Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
2019-09-05T17:54:35.551Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
2019-09-05T17:54:35.553Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
2019-09-05T17:54:35.585Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 JUMP FORWARD 1

We see few out-of-order messages compared to Case 2, but the jumps forwards and backwards in the global sequence are larger.

Case 6 - Tailing the topic, unequal message arrival rate, fast producer

Same as Case 3 but topic 1 receives messages at 10 times the rate of topic 2, and the publisher sends messages 5 times faster than the consumer can process them.

2019-09-05T18:03:00.416Z : Consumer : Subscribing
2019-09-05T18:03:00.472Z : Consumer : Partitions revoked at 2019-09-05T18:03:00.471Z
2019-09-05T18:03:03.494Z : Consumer : Topic: topic2 Partition: 0 assigned at 2019-09-05T18:03:03.494Z
2019-09-05T18:03:03.494Z : Consumer : Topic: topic1 Partition: 0 assigned at 2019-09-05T18:03:03.494Z
2019-09-05T18:03:03.554Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T18:03:03.561Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T18:03:03.578Z : Producer : Topic: topic1, TC: 2, GC: 2
2019-09-05T18:03:03.601Z : Producer : Topic: topic1, TC: 3, GC: 3
2019-09-05T18:03:03.623Z : Producer : Topic: topic1, TC: 4, GC: 4
2019-09-05T18:03:03.646Z : Producer : Topic: topic1, TC: 5, GC: 5
2019-09-05T18:03:03.663Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
2019-09-05T18:03:03.668Z : Producer : Topic: topic1, TC: 6, GC: 6
2019-09-05T18:03:03.691Z : Producer : Topic: topic1, TC: 7, GC: 7
2019-09-05T18:03:03.714Z : Producer : Topic: topic1, TC: 8, GC: 8
2019-09-05T18:03:03.736Z : Producer : Topic: topic1, TC: 9, GC: 9
2019-09-05T18:03:03.759Z : Producer : Topic: topic1, TC: 10, GC: 10
2019-09-05T18:03:03.764Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
2019-09-05T18:03:03.785Z : Producer : Topic: topic2, TC: 1, GC: 11
2019-09-05T18:03:03.808Z : Producer : Topic: topic1, TC: 11, GC: 12
2019-09-05T18:03:03.830Z : Producer : Topic: topic1, TC: 12, GC: 13
2019-09-05T18:03:03.853Z : Producer : Topic: topic1, TC: 13, GC: 14
2019-09-05T18:03:03.866Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
2019-09-05T18:03:03.876Z : Producer : Topic: topic1, TC: 14, GC: 15
2019-09-05T18:03:03.899Z : Producer : Topic: topic1, TC: 15, GC: 16
2019-09-05T18:03:03.922Z : Producer : Topic: topic1, TC: 16, GC: 17
2019-09-05T18:03:03.945Z : Producer : Topic: topic1, TC: 17, GC: 18
2019-09-05T18:03:03.967Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
2019-09-05T18:03:03.968Z : Producer : Topic: topic1, TC: 18, GC: 19
2019-09-05T18:03:03.991Z : Producer : Topic: topic1, TC: 19, GC: 20
2019-09-05T18:03:04.014Z : Producer : Topic: topic1, TC: 20, GC: 21
2019-09-05T18:03:04.037Z : Producer : Topic: topic2, TC: 2, GC: 22
2019-09-05T18:03:04.060Z : Producer : Topic: topic1, TC: 21, GC: 23
2019-09-05T18:03:04.069Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
2019-09-05T18:03:04.082Z : Producer : Topic: topic1, TC: 22, GC: 24
2019-09-05T18:03:04.106Z : Producer : Topic: topic1, TC: 23, GC: 25
2019-09-05T18:03:04.129Z : Producer : Topic: topic1, TC: 24, GC: 26
2019-09-05T18:03:04.152Z : Producer : Topic: topic1, TC: 25, GC: 27
2019-09-05T18:03:04.170Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
2019-09-05T18:03:04.175Z : Producer : Topic: topic1, TC: 26, GC: 28
2019-09-05T18:03:04.197Z : Producer : Topic: topic1, TC: 27, GC: 29
2019-09-05T18:03:04.220Z : Producer : Topic: topic1, TC: 28, GC: 30
2019-09-05T18:03:04.244Z : Producer : Topic: topic1, TC: 29, GC: 31
2019-09-05T18:03:04.267Z : Producer : Topic: topic1, TC: 30, GC: 32
2019-09-05T18:03:04.270Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
2019-09-05T18:03:04.289Z : Producer : Topic: topic2, TC: 3, GC: 33
2019-09-05T18:03:04.312Z : Producer : Topic: topic1, TC: 31, GC: 34
2019-09-05T18:03:04.335Z : Producer : Topic: topic1, TC: 32, GC: 35
2019-09-05T18:03:04.357Z : Producer : Topic: topic1, TC: 33, GC: 36
2019-09-05T18:03:04.371Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
2019-09-05T18:03:04.380Z : Producer : Topic: topic1, TC: 34, GC: 37
2019-09-05T18:03:04.404Z : Producer : Topic: topic1, TC: 35, GC: 38
2019-09-05T18:03:04.428Z : Producer : Topic: topic1, TC: 36, GC: 39
2019-09-05T18:03:04.451Z : Producer : Topic: topic1, TC: 37, GC: 40
2019-09-05T18:03:04.472Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
2019-09-05T18:03:04.473Z : Producer : Topic: topic1, TC: 38, GC: 41
2019-09-05T18:03:04.496Z : Producer : Topic: topic1, TC: 39, GC: 42
2019-09-05T18:03:04.518Z : Producer : Topic: topic1, TC: 40, GC: 43
2019-09-05T18:03:04.541Z : Producer : Topic: topic2, TC: 4, GC: 44
2019-09-05T18:03:04.564Z : Producer : Topic: topic1, TC: 41, GC: 45
2019-09-05T18:03:04.574Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 OK
2019-09-05T18:03:04.587Z : Producer : Topic: topic1, TC: 42, GC: 46
2019-09-05T18:03:04.610Z : Producer : Topic: topic1, TC: 43, GC: 47
2019-09-05T18:03:04.632Z : Producer : Topic: topic1, TC: 44, GC: 48
2019-09-05T18:03:04.655Z : Producer : Topic: topic1, TC: 45, GC: 49
2019-09-05T18:03:04.676Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 JUMP FORWARD 10
2019-09-05T18:03:04.677Z : Producer : Topic: topic1, TC: 46, GC: 50
2019-09-05T18:03:04.700Z : Producer : Topic: topic1, TC: 47, GC: 51
2019-09-05T18:03:04.722Z : Producer : Topic: topic1, TC: 48, GC: 52
2019-09-05T18:03:04.746Z : Producer : Topic: topic1, TC: 49, GC: 53
2019-09-05T18:03:04.768Z : Producer : Topic: topic1, TC: 50, GC: 54
2019-09-05T18:03:04.777Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 JUMP BACKWARDS 10
2019-09-05T18:03:04.791Z : Producer : Topic: topic2, TC: 5, GC: 55
2019-09-05T18:03:04.814Z : Producer : Topic: topic1, TC: 51, GC: 56
2019-09-05T18:03:04.837Z : Producer : Topic: topic1, TC: 52, GC: 57
2019-09-05T18:03:04.860Z : Producer : Topic: topic1, TC: 53, GC: 58
2019-09-05T18:03:04.878Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
2019-09-05T18:03:04.882Z : Producer : Topic: topic1, TC: 54, GC: 59
2019-09-05T18:03:04.905Z : Producer : Topic: topic1, TC: 55, GC: 60
2019-09-05T18:03:04.928Z : Producer : Topic: topic1, TC: 56, GC: 61
2019-09-05T18:03:04.951Z : Producer : Topic: topic1, TC: 57, GC: 62
2019-09-05T18:03:04.974Z : Producer : Topic: topic1, TC: 58, GC: 63
2019-09-05T18:03:04.979Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
2019-09-05T18:03:04.997Z : Producer : Topic: topic1, TC: 59, GC: 64
2019-09-05T18:03:05.019Z : Producer : Topic: topic1, TC: 60, GC: 65
2019-09-05T18:03:05.041Z : Producer : Topic: topic2, TC: 6, GC: 66
2019-09-05T18:03:05.064Z : Producer : Topic: topic1, TC: 61, GC: 67
2019-09-05T18:03:05.080Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
2019-09-05T18:03:05.087Z : Producer : Topic: topic1, TC: 62, GC: 68
2019-09-05T18:03:05.109Z : Producer : Topic: topic1, TC: 63, GC: 69
2019-09-05T18:03:05.132Z : Producer : Topic: topic1, TC: 64, GC: 70
2019-09-05T18:03:05.154Z : Producer : Topic: topic1, TC: 65, GC: 71
2019-09-05T18:03:05.176Z : Producer : Topic: topic1, TC: 66, GC: 72
2019-09-05T18:03:05.180Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
2019-09-05T18:03:05.199Z : Producer : Topic: topic1, TC: 67, GC: 73
2019-09-05T18:03:05.222Z : Producer : Topic: topic1, TC: 68, GC: 74
2019-09-05T18:03:05.245Z : Producer : Topic: topic1, TC: 69, GC: 75
2019-09-05T18:03:05.268Z : Producer : Topic: topic1, TC: 70, GC: 76
2019-09-05T18:03:05.281Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
2019-09-05T18:03:05.291Z : Producer : Topic: topic2, TC: 7, GC: 77
2019-09-05T18:03:05.314Z : Producer : Topic: topic1, TC: 71, GC: 78
2019-09-05T18:03:05.337Z : Producer : Topic: topic1, TC: 72, GC: 79
2019-09-05T18:03:05.360Z : Producer : Topic: topic1, TC: 73, GC: 80
2019-09-05T18:03:05.382Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 OK
2019-09-05T18:03:05.383Z : Producer : Topic: topic1, TC: 74, GC: 81
2019-09-05T18:03:05.406Z : Producer : Topic: topic1, TC: 75, GC: 82
2019-09-05T18:03:05.429Z : Producer : Topic: topic1, TC: 76, GC: 83
2019-09-05T18:03:05.452Z : Producer : Topic: topic1, TC: 77, GC: 84
2019-09-05T18:03:05.475Z : Producer : Topic: topic1, TC: 78, GC: 85
2019-09-05T18:03:05.483Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
2019-09-05T18:03:05.497Z : Producer : Topic: topic1, TC: 79, GC: 86
2019-09-05T18:03:05.519Z : Producer : Topic: topic1, TC: 80, GC: 87
2019-09-05T18:03:05.541Z : Producer : Topic: topic2, TC: 8, GC: 88
2019-09-05T18:03:05.564Z : Producer : Topic: topic1, TC: 81, GC: 89
2019-09-05T18:03:05.584Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
2019-09-05T18:03:05.586Z : Producer : Topic: topic1, TC: 82, GC: 90
2019-09-05T18:03:05.609Z : Producer : Topic: topic1, TC: 83, GC: 91
2019-09-05T18:03:05.632Z : Producer : Topic: topic1, TC: 84, GC: 92
2019-09-05T18:03:05.655Z : Producer : Topic: topic1, TC: 85, GC: 93
2019-09-05T18:03:05.678Z : Producer : Topic: topic1, TC: 86, GC: 94
2019-09-05T18:03:05.685Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
2019-09-05T18:03:05.700Z : Producer : Topic: topic1, TC: 87, GC: 95
2019-09-05T18:03:05.723Z : Producer : Topic: topic1, TC: 88, GC: 96
2019-09-05T18:03:05.746Z : Producer : Topic: topic1, TC: 89, GC: 97
2019-09-05T18:03:05.769Z : Producer : Topic: topic1, TC: 90, GC: 98
2019-09-05T18:03:05.786Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 JUMP FORWARD 1
2019-09-05T18:03:05.793Z : Producer : Topic: topic2, TC: 9, GC: 99
2019-09-05T18:03:05.816Z : Producer : Topic: topic1, TC: 91, GC: 100
2019-09-05T18:03:05.888Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 JUMP FORWARD 9
2019-09-05T18:03:05.989Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 JUMP FORWARD 10
2019-09-05T18:03:06.089Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 JUMP BACKWARDS 20
2019-09-05T18:03:06.190Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
2019-09-05T18:03:06.291Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
2019-09-05T18:03:06.392Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
2019-09-05T18:03:06.493Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
2019-09-05T18:03:06.593Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
2019-09-05T18:03:06.694Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
2019-09-05T18:03:06.795Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
2019-09-05T18:03:06.896Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
2019-09-05T18:03:06.997Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 JUMP FORWARD 1
2019-09-05T18:03:07.098Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
2019-09-05T18:03:07.198Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
2019-09-05T18:03:07.299Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
2019-09-05T18:03:07.400Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
2019-09-05T18:03:07.500Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
2019-09-05T18:03:07.602Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
2019-09-05T18:03:07.703Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
2019-09-05T18:03:07.804Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
2019-09-05T18:03:07.905Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
2019-09-05T18:03:08.005Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 JUMP FORWARD 1
2019-09-05T18:03:08.108Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 JUMP FORWARD 9
2019-09-05T18:03:08.209Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 JUMP FORWARD 10
2019-09-05T18:03:08.310Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 JUMP FORWARD 10
2019-09-05T18:03:08.411Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 JUMP FORWARD 10
2019-09-05T18:03:08.512Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 JUMP FORWARD 10
2019-09-05T18:03:08.613Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 JUMP BACKWARDS 53
2019-09-05T18:03:08.714Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
2019-09-05T18:03:08.815Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
2019-09-05T18:03:08.916Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
2019-09-05T18:03:09.016Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
2019-09-05T18:03:09.117Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
2019-09-05T18:03:09.218Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
2019-09-05T18:03:09.319Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
2019-09-05T18:03:09.420Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
2019-09-05T18:03:09.521Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 JUMP FORWARD 1
2019-09-05T18:03:09.622Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
2019-09-05T18:03:09.722Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
2019-09-05T18:03:09.823Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
2019-09-05T18:03:09.925Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
2019-09-05T18:03:10.026Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
2019-09-05T18:03:10.126Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
2019-09-05T18:03:10.227Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
2019-09-05T18:03:10.328Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
2019-09-05T18:03:10.429Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
2019-09-05T18:03:10.530Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 JUMP FORWARD 1
2019-09-05T18:03:10.631Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
2019-09-05T18:03:10.732Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
2019-09-05T18:03:10.833Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
2019-09-05T18:03:10.934Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
2019-09-05T18:03:11.035Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
2019-09-05T18:03:11.136Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
2019-09-05T18:03:11.237Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
2019-09-05T18:03:11.338Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
2019-09-05T18:03:11.438Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
2019-09-05T18:03:11.539Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 JUMP FORWARD 1
2019-09-05T18:03:11.640Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
2019-09-05T18:03:11.741Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
2019-09-05T18:03:11.842Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
2019-09-05T18:03:11.943Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
2019-09-05T18:03:12.043Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
2019-09-05T18:03:12.145Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
2019-09-05T18:03:12.246Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
2019-09-05T18:03:12.346Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
2019-09-05T18:03:12.448Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
2019-09-05T18:03:12.548Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 JUMP FORWARD 1
2019-09-05T18:03:12.650Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
2019-09-05T18:03:12.751Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
2019-09-05T18:03:12.852Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
2019-09-05T18:03:12.953Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
2019-09-05T18:03:13.054Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
2019-09-05T18:03:13.155Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
2019-09-05T18:03:13.256Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
2019-09-05T18:03:13.356Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
2019-09-05T18:03:13.457Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
2019-09-05T18:03:13.558Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 JUMP FORWARD 1

Very similar to Case 3. I think it is safe to say that have an equal arrival rate or not, doesn’t have a significant impact on tailing consumers.

So far we see that any jitter in processing time or a consumer cannot keep up with a producer results in poor cross topic ordering.

Case 7 - Time-travel, equal message arrival rate

That was all when consumers are more or less keeping up with producers. But what happens if you start to consume from the beginning of the topic, or want to move back your consumer group offset back a day or a week to reprocess messages. What happens to your ordering then?

In this example a producer will send 100 messages in round robin fashion to two topics as before. Then we’ll start up a consumer that will consume the 100 messages from the two topics.

2019-09-05T20:31:18.425Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T20:31:18.564Z : Producer : Topic: topic2, TC: 1, GC: 2
2019-09-05T20:31:18.578Z : Producer : Topic: topic1, TC: 2, GC: 3
2019-09-05T20:31:18.591Z : Producer : Topic: topic2, TC: 2, GC: 4
2019-09-05T20:31:18.604Z : Producer : Topic: topic1, TC: 3, GC: 5
2019-09-05T20:31:18.617Z : Producer : Topic: topic2, TC: 3, GC: 6
2019-09-05T20:31:18.630Z : Producer : Topic: topic1, TC: 4, GC: 7
2019-09-05T20:31:18.644Z : Producer : Topic: topic2, TC: 4, GC: 8
2019-09-05T20:31:18.657Z : Producer : Topic: topic1, TC: 5, GC: 9
2019-09-05T20:31:18.670Z : Producer : Topic: topic2, TC: 5, GC: 10
2019-09-05T20:31:18.683Z : Producer : Topic: topic1, TC: 6, GC: 11
2019-09-05T20:31:18.696Z : Producer : Topic: topic2, TC: 6, GC: 12
2019-09-05T20:31:18.709Z : Producer : Topic: topic1, TC: 7, GC: 13
2019-09-05T20:31:18.722Z : Producer : Topic: topic2, TC: 7, GC: 14
2019-09-05T20:31:18.735Z : Producer : Topic: topic1, TC: 8, GC: 15
2019-09-05T20:31:18.748Z : Producer : Topic: topic2, TC: 8, GC: 16
2019-09-05T20:31:18.761Z : Producer : Topic: topic1, TC: 9, GC: 17
2019-09-05T20:31:18.774Z : Producer : Topic: topic2, TC: 9, GC: 18
2019-09-05T20:31:18.787Z : Producer : Topic: topic1, TC: 10, GC: 19
2019-09-05T20:31:18.799Z : Producer : Topic: topic2, TC: 10, GC: 20
2019-09-05T20:31:18.813Z : Producer : Topic: topic1, TC: 11, GC: 21
2019-09-05T20:31:18.825Z : Producer : Topic: topic2, TC: 11, GC: 22
2019-09-05T20:31:18.839Z : Producer : Topic: topic1, TC: 12, GC: 23
2019-09-05T20:31:18.852Z : Producer : Topic: topic2, TC: 12, GC: 24
2019-09-05T20:31:18.865Z : Producer : Topic: topic1, TC: 13, GC: 25
2019-09-05T20:31:18.878Z : Producer : Topic: topic2, TC: 13, GC: 26
2019-09-05T20:31:18.891Z : Producer : Topic: topic1, TC: 14, GC: 27
2019-09-05T20:31:18.904Z : Producer : Topic: topic2, TC: 14, GC: 28
2019-09-05T20:31:18.917Z : Producer : Topic: topic1, TC: 15, GC: 29
2019-09-05T20:31:18.931Z : Producer : Topic: topic2, TC: 15, GC: 30
2019-09-05T20:31:18.944Z : Producer : Topic: topic1, TC: 16, GC: 31
2019-09-05T20:31:18.957Z : Producer : Topic: topic2, TC: 16, GC: 32
2019-09-05T20:31:18.970Z : Producer : Topic: topic1, TC: 17, GC: 33
2019-09-05T20:31:18.983Z : Producer : Topic: topic2, TC: 17, GC: 34
2019-09-05T20:31:18.995Z : Producer : Topic: topic1, TC: 18, GC: 35
2019-09-05T20:31:19.008Z : Producer : Topic: topic2, TC: 18, GC: 36
2019-09-05T20:31:19.021Z : Producer : Topic: topic1, TC: 19, GC: 37
2019-09-05T20:31:19.033Z : Producer : Topic: topic2, TC: 19, GC: 38
2019-09-05T20:31:19.045Z : Producer : Topic: topic1, TC: 20, GC: 39
2019-09-05T20:31:19.058Z : Producer : Topic: topic2, TC: 20, GC: 40
2019-09-05T20:31:19.071Z : Producer : Topic: topic1, TC: 21, GC: 41
2019-09-05T20:31:19.084Z : Producer : Topic: topic2, TC: 21, GC: 42
2019-09-05T20:31:19.097Z : Producer : Topic: topic1, TC: 22, GC: 43
2019-09-05T20:31:19.110Z : Producer : Topic: topic2, TC: 22, GC: 44
2019-09-05T20:31:19.122Z : Producer : Topic: topic1, TC: 23, GC: 45
2019-09-05T20:31:19.135Z : Producer : Topic: topic2, TC: 23, GC: 46
2019-09-05T20:31:19.148Z : Producer : Topic: topic1, TC: 24, GC: 47
2019-09-05T20:31:19.160Z : Producer : Topic: topic2, TC: 24, GC: 48
2019-09-05T20:31:19.174Z : Producer : Topic: topic1, TC: 25, GC: 49
2019-09-05T20:31:19.186Z : Producer : Topic: topic2, TC: 25, GC: 50
2019-09-05T20:31:19.197Z : Producer : Topic: topic1, TC: 26, GC: 51
2019-09-05T20:31:19.210Z : Producer : Topic: topic2, TC: 26, GC: 52
2019-09-05T20:31:19.223Z : Producer : Topic: topic1, TC: 27, GC: 53
2019-09-05T20:31:19.235Z : Producer : Topic: topic2, TC: 27, GC: 54
2019-09-05T20:31:19.248Z : Producer : Topic: topic1, TC: 28, GC: 55
2019-09-05T20:31:19.260Z : Producer : Topic: topic2, TC: 28, GC: 56
2019-09-05T20:31:19.273Z : Producer : Topic: topic1, TC: 29, GC: 57
2019-09-05T20:31:19.286Z : Producer : Topic: topic2, TC: 29, GC: 58
2019-09-05T20:31:19.299Z : Producer : Topic: topic1, TC: 30, GC: 59
2019-09-05T20:31:19.313Z : Producer : Topic: topic2, TC: 30, GC: 60
2019-09-05T20:31:19.326Z : Producer : Topic: topic1, TC: 31, GC: 61
2019-09-05T20:31:19.338Z : Producer : Topic: topic2, TC: 31, GC: 62
2019-09-05T20:31:19.352Z : Producer : Topic: topic1, TC: 32, GC: 63
2019-09-05T20:31:19.365Z : Producer : Topic: topic2, TC: 32, GC: 64
2019-09-05T20:31:19.377Z : Producer : Topic: topic1, TC: 33, GC: 65
2019-09-05T20:31:19.390Z : Producer : Topic: topic2, TC: 33, GC: 66
2019-09-05T20:31:19.403Z : Producer : Topic: topic1, TC: 34, GC: 67
2019-09-05T20:31:19.415Z : Producer : Topic: topic2, TC: 34, GC: 68
2019-09-05T20:31:19.428Z : Producer : Topic: topic1, TC: 35, GC: 69
2019-09-05T20:31:19.440Z : Producer : Topic: topic2, TC: 35, GC: 70
2019-09-05T20:31:19.452Z : Producer : Topic: topic1, TC: 36, GC: 71
2019-09-05T20:31:19.465Z : Producer : Topic: topic2, TC: 36, GC: 72
2019-09-05T20:31:19.477Z : Producer : Topic: topic1, TC: 37, GC: 73
2019-09-05T20:31:19.490Z : Producer : Topic: topic2, TC: 37, GC: 74
2019-09-05T20:31:19.503Z : Producer : Topic: topic1, TC: 38, GC: 75
2019-09-05T20:31:19.516Z : Producer : Topic: topic2, TC: 38, GC: 76
2019-09-05T20:31:19.528Z : Producer : Topic: topic1, TC: 39, GC: 77
2019-09-05T20:31:19.540Z : Producer : Topic: topic2, TC: 39, GC: 78
2019-09-05T20:31:19.553Z : Producer : Topic: topic1, TC: 40, GC: 79
2019-09-05T20:31:19.566Z : Producer : Topic: topic2, TC: 40, GC: 80
2019-09-05T20:31:19.578Z : Producer : Topic: topic1, TC: 41, GC: 81
2019-09-05T20:31:19.590Z : Producer : Topic: topic2, TC: 41, GC: 82
2019-09-05T20:31:19.602Z : Producer : Topic: topic1, TC: 42, GC: 83
2019-09-05T20:31:19.615Z : Producer : Topic: topic2, TC: 42, GC: 84
2019-09-05T20:31:19.628Z : Producer : Topic: topic1, TC: 43, GC: 85
2019-09-05T20:31:19.640Z : Producer : Topic: topic2, TC: 43, GC: 86
2019-09-05T20:31:19.653Z : Producer : Topic: topic1, TC: 44, GC: 87
2019-09-05T20:31:19.665Z : Producer : Topic: topic2, TC: 44, GC: 88
2019-09-05T20:31:19.677Z : Producer : Topic: topic1, TC: 45, GC: 89
2019-09-05T20:31:19.689Z : Producer : Topic: topic2, TC: 45, GC: 90
2019-09-05T20:31:19.702Z : Producer : Topic: topic1, TC: 46, GC: 91
2019-09-05T20:31:19.715Z : Producer : Topic: topic2, TC: 46, GC: 92
2019-09-05T20:31:19.727Z : Producer : Topic: topic1, TC: 47, GC: 93
2019-09-05T20:31:19.739Z : Producer : Topic: topic2, TC: 47, GC: 94
2019-09-05T20:31:19.752Z : Producer : Topic: topic1, TC: 48, GC: 95
2019-09-05T20:31:19.765Z : Producer : Topic: topic2, TC: 48, GC: 96
2019-09-05T20:31:19.778Z : Producer : Topic: topic1, TC: 49, GC: 97
2019-09-05T20:31:19.790Z : Producer : Topic: topic2, TC: 49, GC: 98
2019-09-05T20:31:19.802Z : Producer : Topic: topic1, TC: 50, GC: 99
2019-09-05T20:31:19.814Z : Producer : Topic: topic2, TC: 50, GC: 100
2019-09-05T20:31:19.840Z : Consumer : Subscribing
2019-09-05T20:31:19.883Z : Consumer : Partitions revoked at 2019-09-05T20:31:19.883Z
2019-09-05T20:31:22.903Z : Consumer : Topic: topic1_7756 Partition: 0 assigned at 2019-09-05T20:31:22.903Z
2019-09-05T20:31:22.903Z : Consumer : Topic: topic2_7756 Partition: 0 assigned at 2019-09-05T20:31:22.903Z
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 JUMP FORWARD 1
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 JUMP FORWARD 1
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 JUMP FORWARD 1
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 JUMP FORWARD 1
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 JUMP FORWARD 1
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 JUMP FORWARD 1
2019-09-05T20:31:22.938Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 JUMP FORWARD 1
2019-09-05T20:31:22.939Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 JUMP FORWARD 1
2019-09-05T20:31:22.940Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 JUMP FORWARD 1
2019-09-05T20:31:22.940Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 JUMP FORWARD 1
2019-09-05T20:31:22.940Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 JUMP FORWARD 1
2019-09-05T20:31:22.941Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 JUMP FORWARD 1
2019-09-05T20:31:22.941Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 JUMP FORWARD 1
2019-09-05T20:31:22.941Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 JUMP FORWARD 1
2019-09-05T20:31:22.941Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 JUMP FORWARD 1
2019-09-05T20:31:22.941Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 JUMP FORWARD 1
2019-09-05T20:31:22.941Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 JUMP FORWARD 1
2019-09-05T20:31:22.942Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 JUMP FORWARD 1
2019-09-05T20:31:22.942Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 JUMP FORWARD 1
2019-09-05T20:31:22.942Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 JUMP FORWARD 1
2019-09-05T20:31:22.942Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 JUMP FORWARD 1
2019-09-05T20:31:22.942Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 JUMP FORWARD 1
2019-09-05T20:31:22.942Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 JUMP FORWARD 1
2019-09-05T20:31:22.943Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 JUMP BACKWARDS 97
2019-09-05T20:31:22.944Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 JUMP FORWARD 1
2019-09-05T20:31:22.944Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 JUMP FORWARD 1
2019-09-05T20:31:22.945Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 JUMP FORWARD 1
2019-09-05T20:31:22.946Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 JUMP FORWARD 1
2019-09-05T20:31:22.947Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 JUMP FORWARD 1
2019-09-05T20:31:22.948Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 JUMP FORWARD 1

So we saw that the consumer consumed all of topic1 first, then all of topic2.

So What is Going On With Multi-Topic Subscriptions?

Basically, when you consume from two or more topics, the consumer will be served from the topics in a given order. This seems to hold for the duration of the consumer, for example, when it calls poll: it will be served from topic2, then from topic1.

So for example it chooses to fetch in the order:

1. topic2, partition 0
2. topic1, partition 0

This choice looks to be the same on all subsequent calls. So if there are 700 records in each topic and max.poll.records is 500, then it will fetch:

1. poll() - 500 from topic2, partition 0
2. poll() - 200 from topic2, partition 0 and 300 from topic1, partition 0
3. poll() - 400 from topic1, partition 0

This means if you want best effort ordering and:

• You can get periods where producers are much faster than consumers
• Or you want to consume from the start of the topics
• Or you want to reset your offset back a while
• Or you have a process that wakes up every once and a while consumes from two or more topics

then don’t count on any ordering between topics at all.

This is not a criticism of Kafka. The docs have always made it abundantly clear that there are no ordering guarantees between topics or partitions. But it’s nice to know what to expect.

A Note on Partitions

Everything we have seen here also applies to single topic subscriptions where multiple partitions get assigned to a single consumer, which is much more common. In the end, whether it is different topics or the same topic, we are ultimately talking about partitions. This may make all this a little more relevant to you.

A note on my use round-robin producing

So what are the chances that both of the topics your consumer consumes have the same message arrive rate?

What happens when we have a 5-to-1 ratio. For every message that arrives at topic2, five arrive at topic1. The result is the same. When the consumer keeps up and it processes the messages in a perfect or close to perfect global ordering. When a consumer falls behind, things go awry and when a consumer starts at the beginning of two topics, it processes all of one topic, then the next.

I have been asked many times if Kafka provides best effort ordering between topics and I always say no. But what if there was a way to achieve that? This is what we’ll look at next.

Achieving Best Effort Cross Topic Ordering

Apache Kafka provides a nice little feature, it adds a timestamp to each message. You can configure it to be set from the producer (called CreateTime) or by the broker when it appends it to the partition (LogAppendTime). The trouble with using timestamps for ordering in a distributed system is clock skew. Clocks do not stay synchronized, even when coordinated by NTP and you might not even have control over all the servers in the system.

For that reason we want to discard CreateTime, we might not control all the producers and their clocks. The LogAppendTime is more attractive as there are fewer servers and clock skew should be much less between brokers than between producers. Remember we’re just talking best effort here.

A solution for best-effort ordering is this:

• Replace a single consumer with multiple consumer instances, where each consumer subscribes to one topic (single partition)
• Each consumer pushes messages into a bounded buffer and keeps it full
• The main thread pulls one message at a time from these buffers, according to the timestamp of the message

This provides us the output:

2019-09-05T20:35:24.462Z : Producer : Topic: topic1, TC: 1, GC: 1
2019-09-05T20:35:24.593Z : Producer : Topic: topic2, TC: 1, GC: 2
2019-09-05T20:35:24.596Z : Producer : Topic: topic1, TC: 2, GC: 3
2019-09-05T20:35:24.601Z : Producer : Topic: topic2, TC: 2, GC: 4
2019-09-05T20:35:24.604Z : Producer : Topic: topic1, TC: 3, GC: 5
2019-09-05T20:35:24.609Z : Producer : Topic: topic2, TC: 3, GC: 6
2019-09-05T20:35:24.612Z : Producer : Topic: topic1, TC: 4, GC: 7
2019-09-05T20:35:24.615Z : Producer : Topic: topic2, TC: 4, GC: 8
2019-09-05T20:35:24.619Z : Producer : Topic: topic1, TC: 5, GC: 9
2019-09-05T20:35:24.622Z : Producer : Topic: topic2, TC: 5, GC: 10
2019-09-05T20:35:24.626Z : Producer : Topic: topic1, TC: 6, GC: 11
2019-09-05T20:35:24.630Z : Producer : Topic: topic2, TC: 6, GC: 12
2019-09-05T20:35:24.633Z : Producer : Topic: topic1, TC: 7, GC: 13
2019-09-05T20:35:24.637Z : Producer : Topic: topic2, TC: 7, GC: 14
2019-09-05T20:35:24.640Z : Producer : Topic: topic1, TC: 8, GC: 15
2019-09-05T20:35:24.643Z : Producer : Topic: topic2, TC: 8, GC: 16
2019-09-05T20:35:24.647Z : Producer : Topic: topic1, TC: 9, GC: 17
2019-09-05T20:35:24.650Z : Producer : Topic: topic2, TC: 9, GC: 18
2019-09-05T20:35:24.654Z : Producer : Topic: topic1, TC: 10, GC: 19
2019-09-05T20:35:24.658Z : Producer : Topic: topic2, TC: 10, GC: 20
2019-09-05T20:35:24.662Z : Producer : Topic: topic1, TC: 11, GC: 21
2019-09-05T20:35:24.666Z : Producer : Topic: topic2, TC: 11, GC: 22
2019-09-05T20:35:24.669Z : Producer : Topic: topic1, TC: 12, GC: 23
2019-09-05T20:35:24.673Z : Producer : Topic: topic2, TC: 12, GC: 24
2019-09-05T20:35:24.677Z : Producer : Topic: topic1, TC: 13, GC: 25
2019-09-05T20:35:24.681Z : Producer : Topic: topic2, TC: 13, GC: 26
2019-09-05T20:35:24.685Z : Producer : Topic: topic1, TC: 14, GC: 27
2019-09-05T20:35:24.689Z : Producer : Topic: topic2, TC: 14, GC: 28
2019-09-05T20:35:24.692Z : Producer : Topic: topic1, TC: 15, GC: 29
2019-09-05T20:35:24.696Z : Producer : Topic: topic2, TC: 15, GC: 30
2019-09-05T20:35:24.699Z : Producer : Topic: topic1, TC: 16, GC: 31
2019-09-05T20:35:24.703Z : Producer : Topic: topic2, TC: 16, GC: 32
2019-09-05T20:35:24.707Z : Producer : Topic: topic1, TC: 17, GC: 33
2019-09-05T20:35:24.711Z : Producer : Topic: topic2, TC: 17, GC: 34
2019-09-05T20:35:24.715Z : Producer : Topic: topic1, TC: 18, GC: 35
2019-09-05T20:35:24.719Z : Producer : Topic: topic2, TC: 18, GC: 36
2019-09-05T20:35:24.722Z : Producer : Topic: topic1, TC: 19, GC: 37
2019-09-05T20:35:24.725Z : Producer : Topic: topic2, TC: 19, GC: 38
2019-09-05T20:35:24.728Z : Producer : Topic: topic1, TC: 20, GC: 39
2019-09-05T20:35:24.731Z : Producer : Topic: topic2, TC: 20, GC: 40
2019-09-05T20:35:24.735Z : Producer : Topic: topic1, TC: 21, GC: 41
2019-09-05T20:35:24.738Z : Producer : Topic: topic2, TC: 21, GC: 42
2019-09-05T20:35:24.742Z : Producer : Topic: topic1, TC: 22, GC: 43
2019-09-05T20:35:24.745Z : Producer : Topic: topic2, TC: 22, GC: 44
2019-09-05T20:35:24.748Z : Producer : Topic: topic1, TC: 23, GC: 45
2019-09-05T20:35:24.751Z : Producer : Topic: topic2, TC: 23, GC: 46
2019-09-05T20:35:24.754Z : Producer : Topic: topic1, TC: 24, GC: 47
2019-09-05T20:35:24.757Z : Producer : Topic: topic2, TC: 24, GC: 48
2019-09-05T20:35:24.760Z : Producer : Topic: topic1, TC: 25, GC: 49
2019-09-05T20:35:24.763Z : Producer : Topic: topic2, TC: 25, GC: 50
2019-09-05T20:35:24.767Z : Producer : Topic: topic1, TC: 26, GC: 51
2019-09-05T20:35:24.770Z : Producer : Topic: topic2, TC: 26, GC: 52
2019-09-05T20:35:24.773Z : Producer : Topic: topic1, TC: 27, GC: 53
2019-09-05T20:35:24.777Z : Producer : Topic: topic2, TC: 27, GC: 54
2019-09-05T20:35:24.780Z : Producer : Topic: topic1, TC: 28, GC: 55
2019-09-05T20:35:24.784Z : Producer : Topic: topic2, TC: 28, GC: 56
2019-09-05T20:35:24.788Z : Producer : Topic: topic1, TC: 29, GC: 57
2019-09-05T20:35:24.792Z : Producer : Topic: topic2, TC: 29, GC: 58
2019-09-05T20:35:24.796Z : Producer : Topic: topic1, TC: 30, GC: 59
2019-09-05T20:35:24.800Z : Producer : Topic: topic2, TC: 30, GC: 60
2019-09-05T20:35:24.803Z : Producer : Topic: topic1, TC: 31, GC: 61
2019-09-05T20:35:24.806Z : Producer : Topic: topic2, TC: 31, GC: 62
2019-09-05T20:35:24.809Z : Producer : Topic: topic1, TC: 32, GC: 63
2019-09-05T20:35:24.812Z : Producer : Topic: topic2, TC: 32, GC: 64
2019-09-05T20:35:24.815Z : Producer : Topic: topic1, TC: 33, GC: 65
2019-09-05T20:35:24.818Z : Producer : Topic: topic2, TC: 33, GC: 66
2019-09-05T20:35:24.822Z : Producer : Topic: topic1, TC: 34, GC: 67
2019-09-05T20:35:24.824Z : Producer : Topic: topic2, TC: 34, GC: 68
2019-09-05T20:35:24.827Z : Producer : Topic: topic1, TC: 35, GC: 69
2019-09-05T20:35:24.831Z : Producer : Topic: topic2, TC: 35, GC: 70
2019-09-05T20:35:24.834Z : Producer : Topic: topic1, TC: 36, GC: 71
2019-09-05T20:35:24.837Z : Producer : Topic: topic2, TC: 36, GC: 72
2019-09-05T20:35:24.840Z : Producer : Topic: topic1, TC: 37, GC: 73
2019-09-05T20:35:24.843Z : Producer : Topic: topic2, TC: 37, GC: 74
2019-09-05T20:35:24.846Z : Producer : Topic: topic1, TC: 38, GC: 75
2019-09-05T20:35:24.849Z : Producer : Topic: topic2, TC: 38, GC: 76
2019-09-05T20:35:24.852Z : Producer : Topic: topic1, TC: 39, GC: 77
2019-09-05T20:35:24.856Z : Producer : Topic: topic2, TC: 39, GC: 78
2019-09-05T20:35:24.860Z : Producer : Topic: topic1, TC: 40, GC: 79
2019-09-05T20:35:24.864Z : Producer : Topic: topic2, TC: 40, GC: 80
2019-09-05T20:35:24.867Z : Producer : Topic: topic1, TC: 41, GC: 81
2019-09-05T20:35:24.871Z : Producer : Topic: topic2, TC: 41, GC: 82
2019-09-05T20:35:24.875Z : Producer : Topic: topic1, TC: 42, GC: 83
2019-09-05T20:35:24.879Z : Producer : Topic: topic2, TC: 42, GC: 84
2019-09-05T20:35:24.883Z : Producer : Topic: topic1, TC: 43, GC: 85
2019-09-05T20:35:24.888Z : Producer : Topic: topic2, TC: 43, GC: 86
2019-09-05T20:35:24.891Z : Producer : Topic: topic1, TC: 44, GC: 87
2019-09-05T20:35:24.895Z : Producer : Topic: topic2, TC: 44, GC: 88
2019-09-05T20:35:24.899Z : Producer : Topic: topic1, TC: 45, GC: 89
2019-09-05T20:35:24.903Z : Producer : Topic: topic2, TC: 45, GC: 90
2019-09-05T20:35:24.906Z : Producer : Topic: topic1, TC: 46, GC: 91
2019-09-05T20:35:24.910Z : Producer : Topic: topic2, TC: 46, GC: 92
2019-09-05T20:35:24.914Z : Producer : Topic: topic1, TC: 47, GC: 93
2019-09-05T20:35:24.918Z : Producer : Topic: topic2, TC: 47, GC: 94
2019-09-05T20:35:24.922Z : Producer : Topic: topic1, TC: 48, GC: 95
2019-09-05T20:35:24.926Z : Producer : Topic: topic2, TC: 48, GC: 96
2019-09-05T20:35:24.929Z : Producer : Topic: topic1, TC: 49, GC: 97
2019-09-05T20:35:24.933Z : Producer : Topic: topic2, TC: 49, GC: 98
2019-09-05T20:35:24.936Z : Producer : Topic: topic1, TC: 50, GC: 99
2019-09-05T20:35:24.939Z : Producer : Topic: topic2, TC: 50, GC: 100
2019-09-05T20:35:35.003Z : Consumer : Topic: topic1, TC: 1 TOPIC: OK, GC: 1 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic2, TC: 1 TOPIC: OK, GC: 2 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic1, TC: 2 TOPIC: OK, GC: 3 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic2, TC: 2 TOPIC: OK, GC: 4 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic1, TC: 3 TOPIC: OK, GC: 5 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic2, TC: 3 TOPIC: OK, GC: 6 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic1, TC: 4 TOPIC: OK, GC: 7 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic2, TC: 4 TOPIC: OK, GC: 8 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic1, TC: 5 TOPIC: OK, GC: 9 GLOBAL: OK
2019-09-05T20:35:35.004Z : Consumer : Topic: topic2, TC: 5 TOPIC: OK, GC: 10 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic1, TC: 6 TOPIC: OK, GC: 11 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic2, TC: 6 TOPIC: OK, GC: 12 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic1, TC: 7 TOPIC: OK, GC: 13 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic2, TC: 7 TOPIC: OK, GC: 14 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic1, TC: 8 TOPIC: OK, GC: 15 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic2, TC: 8 TOPIC: OK, GC: 16 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic1, TC: 9 TOPIC: OK, GC: 17 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic2, TC: 9 TOPIC: OK, GC: 18 GLOBAL: OK
2019-09-05T20:35:35.005Z : Consumer : Topic: topic1, TC: 10 TOPIC: OK, GC: 19 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic2, TC: 10 TOPIC: OK, GC: 20 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic1, TC: 11 TOPIC: OK, GC: 21 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic2, TC: 11 TOPIC: OK, GC: 22 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic1, TC: 12 TOPIC: OK, GC: 23 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic2, TC: 12 TOPIC: OK, GC: 24 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic1, TC: 13 TOPIC: OK, GC: 25 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic2, TC: 13 TOPIC: OK, GC: 26 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic1, TC: 14 TOPIC: OK, GC: 27 GLOBAL: OK
2019-09-05T20:35:35.006Z : Consumer : Topic: topic2, TC: 14 TOPIC: OK, GC: 28 GLOBAL: OK
2019-09-05T20:35:35.007Z : Consumer : Topic: topic1, TC: 15 TOPIC: OK, GC: 29 GLOBAL: OK
2019-09-05T20:35:35.007Z : Consumer : Topic: topic2, TC: 15 TOPIC: OK, GC: 30 GLOBAL: OK
2019-09-05T20:35:35.007Z : Consumer : Topic: topic1, TC: 16 TOPIC: OK, GC: 31 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic2, TC: 16 TOPIC: OK, GC: 32 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic1, TC: 17 TOPIC: OK, GC: 33 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic2, TC: 17 TOPIC: OK, GC: 34 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic1, TC: 18 TOPIC: OK, GC: 35 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic2, TC: 18 TOPIC: OK, GC: 36 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic1, TC: 19 TOPIC: OK, GC: 37 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic2, TC: 19 TOPIC: OK, GC: 38 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic1, TC: 20 TOPIC: OK, GC: 39 GLOBAL: OK
2019-09-05T20:35:35.008Z : Consumer : Topic: topic2, TC: 20 TOPIC: OK, GC: 40 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic1, TC: 21 TOPIC: OK, GC: 41 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic2, TC: 21 TOPIC: OK, GC: 42 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic1, TC: 22 TOPIC: OK, GC: 43 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic2, TC: 22 TOPIC: OK, GC: 44 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic1, TC: 23 TOPIC: OK, GC: 45 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic2, TC: 23 TOPIC: OK, GC: 46 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic1, TC: 24 TOPIC: OK, GC: 47 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic2, TC: 24 TOPIC: OK, GC: 48 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic1, TC: 25 TOPIC: OK, GC: 49 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic2, TC: 25 TOPIC: OK, GC: 50 GLOBAL: OK
2019-09-05T20:35:35.009Z : Consumer : Topic: topic1, TC: 26 TOPIC: OK, GC: 51 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic2, TC: 26 TOPIC: OK, GC: 52 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic1, TC: 27 TOPIC: OK, GC: 53 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic2, TC: 27 TOPIC: OK, GC: 54 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic1, TC: 28 TOPIC: OK, GC: 55 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic2, TC: 28 TOPIC: OK, GC: 56 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic1, TC: 29 TOPIC: OK, GC: 57 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic2, TC: 29 TOPIC: OK, GC: 58 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic1, TC: 30 TOPIC: OK, GC: 59 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic2, TC: 30 TOPIC: OK, GC: 60 GLOBAL: OK
2019-09-05T20:35:35.010Z : Consumer : Topic: topic1, TC: 31 TOPIC: OK, GC: 61 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic2, TC: 31 TOPIC: OK, GC: 62 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic1, TC: 32 TOPIC: OK, GC: 63 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic2, TC: 32 TOPIC: OK, GC: 64 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic1, TC: 33 TOPIC: OK, GC: 65 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic2, TC: 33 TOPIC: OK, GC: 66 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic1, TC: 34 TOPIC: OK, GC: 67 GLOBAL: OK
2019-09-05T20:35:35.011Z : Consumer : Topic: topic2, TC: 34 TOPIC: OK, GC: 68 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic1, TC: 35 TOPIC: OK, GC: 69 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic2, TC: 35 TOPIC: OK, GC: 70 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic1, TC: 36 TOPIC: OK, GC: 71 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic2, TC: 36 TOPIC: OK, GC: 72 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic1, TC: 37 TOPIC: OK, GC: 73 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic2, TC: 37 TOPIC: OK, GC: 74 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic1, TC: 38 TOPIC: OK, GC: 75 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic2, TC: 38 TOPIC: OK, GC: 76 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic1, TC: 39 TOPIC: OK, GC: 77 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic2, TC: 39 TOPIC: OK, GC: 78 GLOBAL: OK
2019-09-05T20:35:35.012Z : Consumer : Topic: topic1, TC: 40 TOPIC: OK, GC: 79 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic2, TC: 40 TOPIC: OK, GC: 80 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic1, TC: 41 TOPIC: OK, GC: 81 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic2, TC: 41 TOPIC: OK, GC: 82 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic1, TC: 42 TOPIC: OK, GC: 83 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic2, TC: 42 TOPIC: OK, GC: 84 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic1, TC: 43 TOPIC: OK, GC: 85 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic2, TC: 43 TOPIC: OK, GC: 86 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic1, TC: 44 TOPIC: OK, GC: 87 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic2, TC: 44 TOPIC: OK, GC: 88 GLOBAL: OK
2019-09-05T20:35:35.013Z : Consumer : Topic: topic1, TC: 45 TOPIC: OK, GC: 89 GLOBAL: OK
2019-09-05T20:35:35.014Z : Consumer : Topic: topic2, TC: 45 TOPIC: OK, GC: 90 GLOBAL: OK
2019-09-05T20:35:35.014Z : Consumer : Topic: topic1, TC: 46 TOPIC: OK, GC: 91 GLOBAL: OK
2019-09-05T20:35:35.014Z : Consumer : Topic: topic2, TC: 46 TOPIC: OK, GC: 92 GLOBAL: OK
2019-09-05T20:35:35.015Z : Consumer : Topic: topic1, TC: 47 TOPIC: OK, GC: 93 GLOBAL: OK
2019-09-05T20:35:35.015Z : Consumer : Topic: topic2, TC: 47 TOPIC: OK, GC: 94 GLOBAL: OK
2019-09-05T20:35:35.015Z : Consumer : Topic: topic1, TC: 48 TOPIC: OK, GC: 95 GLOBAL: OK
2019-09-05T20:35:35.016Z : Consumer : Topic: topic2, TC: 48 TOPIC: OK, GC: 96 GLOBAL: OK
2019-09-05T20:35:35.016Z : Consumer : Topic: topic1, TC: 49 TOPIC: OK, GC: 97 GLOBAL: OK
2019-09-05T20:35:35.016Z : Consumer : Topic: topic2, TC: 49 TOPIC: OK, GC: 98 GLOBAL: OK
2019-09-05T20:35:35.016Z : Consumer : Topic: topic1, TC: 50 TOPIC: OK, GC: 99 GLOBAL: OK
2019-09-05T20:35:35.016Z : Consumer : Topic: topic2, TC: 50 TOPIC: OK, GC: 100 GLOBAL: OK

I got perfect ordering because all three brokers had clocks that are synchronized (all on my workstation via Docker). In a real deployment synchronized clocks are not always to be guaranteed, so still no bullet proof guarantees, but if for whatever reason, you want best effort ordering, then perhaps look at this approach.

This technique requires a little more work for it to function with multi-partition, single topic subscriptions. You’ll need to either manually assign partitions, again replacing your single consumer with one per partition. You could do it with automatic assignment, configuring each consumer to only be assigned one partition at maximum, both have tricky parts to managing them correctly.

Alternatives

Remember, there are usually better alternatives!

1. If you have related events that you want ordering for, then send them into the same topic. If your topic has multiple partitions, then make sure the related events share the same message key. https://www.confluent.io/blog/put-several-event-types-kafka-topic/
2. Build your system in a way that is agnostic to the ordering of related events (could be hard)
   1. Remove referential integrity
   2. Try to use monotonicity https://blog.acolyer.org/2019/03/06/keeping-calm-when-distributed-consistency-is-easy/
   3. Please suggest patterns to me on Twitter!

The downside of mixing events in a single topic is that it is a decision that affects all consuming applications. A topic is shared infrastructure and mixing events becomes an architectural decision. This is not something that affects RabbitMQ as each application can bind a single queue to multiple exchanges and get custom mixing without impacting other applications, which is one killer feature of RabbitMQ.

Anyway, I hope that shines a light on multi-topic subscriptions and what kind of expectations you should have regarding ordering. I’m not going to give you any advice, just the information: forewarned is forearmed after all.