I did try the recommended workaround of specifying an explicit bind, but it fails too. It seems that it won't work without setting IP_TRANSPARENT option on the socket, which redis currently does not.

Related to #2335

Agreed, will write here a design spec in the next days. p.s. in the meanwhile you may want to prevent Docker from mapping ports, there is a specific feature for 1:1 mapping.

I'm having the same trouble - see #2621
copy/pasta from there:
[...] Master instances know the slaves running on the same machine by local (172.*) IPs and vice versa. This may possibly lead to getting the MOVED response to an IP that is not reachable by the client:

192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6379 cluster nodes
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352847839 5 connected
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352848039 6 connected
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352847638 2 connected 5461-10922
e1fc77808551cccf5efe6bcbb5816b624bff9633 192.168.1.245:6379 myself,master - 0 0 1 connected 0-5460
721e4986d1cc2dbf87ac11054242233ac3d67ede 172.17.42.1:6380 slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434352847139 4 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352847839 3 connected 10923-16383
192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6380 cluster nodes
e1fc77808551cccf5efe6bcbb5816b624bff9633 172.17.42.1:6379 master - 0 1434352872207 1 connected 0-5460
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352872509 5 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352872308 3 connected 10923-16383
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352872108 2 connected 5461-10922
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352872509 6 connected
721e4986d1cc2dbf87ac11054242233ac3d67ede 192.168.1.245:6380 myself,slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434101988241 4 connected

however i've noticed they all act the same -- the ones on the 192.168.1.246 IP behave exactly the same (for their own IP). Relevant bits as seen by 192.168.1.246:6379:

e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 myself,master - 0 1434101985463 2 connected 5461-10922
8bd0598380fc05aa374ef22a74da1a691963cfbb 172.17.42.1:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434375798687 5 connected

So they each have a (machine local) 172.17.* net that is not available from outside the machine

+1 please

If have the same problem. It isn't possible to create a redis cluster on different servers with docker.
Redis send the container IP instead of the host IP.
So for this Redis should add the support of announce-ip like the sentinel.

+1 please

+1
This would be amazing.

Prioritized in order to be shipped with the next release (3.2). Thanks for the feedbacks.

👍 i there a proposed work-around available for this issue?

+1

+1

As a workaround, if possible in your env, run the containers with the option --net=host (see https://docs.docker.com/articles/networking/#container-networking)

+1

Scheduled for Redis 3.2 that will go in RC in a matter of weeks (~4 hopefully).

p.s. this is going to be backported into 3.0.x anyway...

+1

Hi @antirez @harley84
Do you have some work-around solutions?
I think you can get the hostname instead of ip address. In docker container, I can pass hostname env to the container.

@fatduo the workaround is using --net=host as mentioned here: #2527 (comment)

Hi @antirez @harley84
I have put in a fix in a fork and opened a PR (as per the CONTRIBUTING document). I have tested them in kubernetes on openstack. Please review. Thanks!!

@mostolog Can you explain how elastic search does this and what is incompatible with swarm mode? Swarm Mode is smth. Kubernetes does for a long time :) As soon as ip announcement works everything is fine, cluster nodes can be found by contacting an infrastructure service which routes the request to the appropriate node.

Hi

Consider the following docker command:

docker service create --name my-redis --replicas 3 redis:3.2.1 redis-server /etc/redis/redis.conf

Instead of creating a 3 node redis cluster, it creates a 3 node redis array (3 standalone instances of redis).

It would be great to have an option to configure a cluster name, multicast group, SSDP...to have a "PnP" cluster, if they shared the same password..
In other words: https://en.wikipedia.org/wiki/Zero-configuration_networking

Elasticsearch had multicast support enabled by default on previous versions, but they just moved to unicast, and we are experiencing the same issues: "the need of address to be able to form a cluster"
IIRC, docker is considering something like apple bonjour mDNS-SD

Regards

PS: Even more: it would be great allowing redis listen on a ip/mask (eg: 172.16.0.2/16) instead of fixed IP. this would enable dynamic IP configuration while preserving the redis-network isolation

I don't think multicast is an option :( It'd mean a lot of redesign in Redis, afaik Redis is TCP/IP only, on top this answer might be interesting:

https://groups.google.com/forum/#!topic/coreos-user/tbh1op__qNM

Multicast is very hard to implement with cloud providers and from all available networking layers for docker only weave supports multicast.

Ok about multicast...but what about the other options?

Have you considered an easier way of deploying a cluster not involving redis-trib/utilities?
For example: a unicast list of hostname:port in the config file, sharing the same password and creating a cluster+slots...working just out of the box

If you consider other proposals (bonjour or listening on network/mask) are interesting I can create those issues

Regards

@vvanholl I didn't mention those, 'cause as a docker user, swarm mode is replacing external registry needs (IMHO: that's the way to take)

Docker Swarm is Good to me

An additional feedback based on my own try-and-fail attempt with Redis cluster and K8s:
We need to be able to distinguish between the internal Redis network and the external one. Specifically, we need to be able to have each Redis server identify by two IPs - one for it's fellow cluster members (local network) and that would be reported to clients using the "cluster nodes" and other cluster commands, as "moved" response and when using "asking".
Because some commands rely on IP rather than node IDs this is where it gets complicated. With the current implementation I see no easy way out for the general case. However, if we can relax the requirements and commit to no overlap in networks, we can simply use each additional IP as an alias to the original IP the cluster would use, then add alias resolution to commands and add all known IP aliases to each response. As this is breaking protocol, perhaps toggle this by flag.

Can someone summarize -- can redis be run under Docker Swarm in the master/slave/sentinel configuration? I have tried (redis 3.2) but the cluster seems to get befuddled as containers change IP addresses when they are restarted. It seems to get into a state where it will never elect a master even though there are two redis servers running + 3 sentinels.

@drnybble If I am not mistaken, this feature was first introduced in Redis 4.0.

Here is what I read from the release history log's section "Redis 4.0.0-RC1 Released Fri Dec 2 10:40:01 CEST 2016" located in (https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES):

• Redis Cluster support for NAT / Docker. There are new functionalities in order to force cluster instances to announce specific sets of IP address, client and bus ports, to the rest of the cluster, regardless of the auto detected IP. This required a bus protocol change that will force users to mass-restart all the nodes of a Redis 3.2 installation in order to upgrade to 4.0.

Thanks -- I am talking master/slave/sentinel configuration (replication) though, not cluster.

I seem to have some success specifying this for each master/slave:

slave-announce-ip [my-virtual-ip]

I specified the hostname of the service -- there still seems to be some confusion here since the sentinels seem to recognize both the hostname and the ip address as separate entities:

1:X 17 Aug 18:04:37.371 # -sdown slave redis1:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:04:37.372 # -sdown slave 10.0.0.3:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:05.395 * +reboot slave redis2:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:06.255 * +reboot slave 10.0.0.5:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379

I can always resolve the hostname before writing it to the .conf file if it is a problem.

I was able to force update services for masters and slaves (docker stack update --force redis_redis1 for instance) and so far it seems to be working as expected.

@drnybble I am a bit mislead: I think this is wrong issue to discuss about announce-ip in HA deployment, as this issue explicitly deals with Redis Cluster.

@ovanes You are correct. However, I think this information is also relevant for running Redis Cluster in Docker Swarm. The announce-ip should announce the Virtual IP of the service so it is stable over container updates.

@drnybble Yes, you are right. But than it should be tested with the version where it was introduced than, i.e. 4.0 and not 3.2. And redis release docs explicitly state, that there were bus protocol updates which are incompatible with version 3.2, etc. IMO, all the findings presented for 3.2 are pointless :( As one can't make any kind of conclusion to the version 4.0 because of severe changes.

So is this supposed to work in redis 4.0.2? There are cluster-announce directives now though I still can't get it to work.
I have a docker container which contains three redis cluster nodes. I've set up the container in bridge mode and exposed the redis server and bus ports. All ports can be reached from the host machine and from within the container, however if I try to set up the cluster with redis-trib.rb the cluster won't join.
I get this message in the redis.log:
Unable to connect to Cluster Node [1.2.3.4]:12345 -> Name or service not known
(dummy ip and port)
However, the address is reachable from host and container. I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

Cheers

I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

The square brackets are part of the error message formatting, so the address is fine.
See https://github.com/antirez/redis/blob/d9565379da0adf97caec12e9b88c17da51900e40/src/cluster.c#L3275 (I think this is the spot, but could be wrong).

As for the actual problem, sorry but don't have any idea about that.

@wizzu thank you for pointing this out. So if the ip address is correct I have no idea what else could be wrong at this point.

+1

Hey all. I ran into this as well.

I wrote a proxy that appears to work for simple set/gets on small clusters. This is a work in progress and was created only for testing a RedisCluster running in a docker-swarm, but it should work for K8s, too.

This proxy runs on the cluster-side and alters the IP addresses from the Redis Cluster to lie to clients so that they can route to a single proxy end-point. The proxy end-point listens on a single port for each node in the cluster and passes the traffic onto the actual nodes.

https://github.com/wojnosystems/redis_cluster_proxy

Please feel free to contribute for anything that's broken. This code has some limitations, but this enables me to test my cluster-integration tests.

Hope it helps you as well.

This has been implemented long ago in 5ac5e3e, closing. This issue has diverted to other topics, please feel free to re-open new issues if relevant.