Limit retry backoff (and unlimited retries)

[morrone]

The existing retry options are a bit limiting:

max.retries
retry.backoff.ms

When the elasticsearch cluster is inaccessible (e.g. a network interruption), we want the connector to be resilient to that situation and resume operation when connectivity to elasticsearch is reestablished. And we want it to reconnect in a timely manner.

So first of all, it would be nice to allow a way to set max.retries to "unlimited". Granted, the max value of 2147483647 is pretty darn large, and probably going to be enough in practice.

But more important, we really need to put an upper limited on the retry backoff. It is nice that it is designed to "wait up to twice as long as the previous wait", but if elasticsearch is down for a few hours that would let the backoff time grow to an unacceptably long interval.

We could really use an option to cap the growth of the backoff to some maximum value.

[pricelessjunk]

I was looking into this a little. There does seem to be an upper limit to the back off time which is 24 hours. This value is also automatically applied after 32 attempts (due to overflow).

Did you want to make this configurable instead of it being a constant?

Reference -
https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/src/main/java/io/confluent/connect/elasticsearch/RetryUtil.java#L94

[morrone]

Yes, making it configurable would be great. We would want to configure the maximum to be something like 5-10 minutes.

[pricelessjunk]

Hey so I created a pull request for this change. #756.

Please verify if this is how you imagined it. If its good I can ask the maintainers to have a look.

[morrone]

Please verify if this is how you imagined it. If its good I can ask the maintainers to have a look.

Yes, that looks perfect. Thank you!