In Coherence 12.2.1 a number of changes were made to make working with ports in Coherence easier. These changes allow multiple clusters (including different versions from 12.2.1 on), to share the same cluster port, and in most cases ports and addresses do not need to be explicitly set. This makes cluster discovery easier for both TCMP and Coherence*Extend clients.
We will first summarise these changes and then go through some practical examples.
- Coherence now includes a common cluster port that is used for multicast, WKA discovery as well as proxy service discovery via the names service.
- Unicast ports (for each member) use automatically assigned ephemeral ports by default.
- Address resolution has been simplified to help with cluster discovery. Coherence automatcially selects the IP which is routable to the IP’s on the WKA list. We also have the ability to store the full WKA list under a single DNS entry.
Coherence now has a registered port number, 7574, with the Internet Assigned Numbers Authority (IANA). (See here for the registration.) This means that other products/ processes “should” not use this port. Since multiple Coherence clusters can share this port then there should be no reason to ever change the value.
Cluster names are the preferred mechanism to distinguish clusters, and having multiple clusters share a port adds no overhead once you’ve discovered the cluster.
Lets see this in action. I have a 184.108.40.206.1 installation on my Mac. We do have OOTB scripts to start cache servers, etc. but i’m going to do this by interacting directly with the jar files so we can see it working.
Change to your lib directory in your Coherence install and run the following. (From 12.2.1 onwards when you using -jar coherence.jar you get a Cache Server rather than the console!)
$ cd /u01/oracle/product/coherence1221/coherence/lib $ java -jar coherence.jar
You should see the following in the logs indicating the (automatically generated) cluster name and the default cluster port. Also you can see the ephemeral port generated for the member is 56839 in this case.
... (thread=main, member=n/a): Started cluster Name=timmiddleton's cluster, ClusterPort=7574 ... ThisMember=Member(Id=1, Timestamp=2016-05-03 11:56:53.071, Address=X.X.X.X:56839, MachineId=20521, Location= ...)
Now that you can leave the cluster port as is, the cluster name becomes even more important in identifying a Coherence cluster. When I issue a netstat we can see that Coherence is listening on all interfaces on IPV4/6 and UDP as we have multicast enabled by default.
$ netstat -an | grep 7574 tcp6 0 0 xxx::aebc:32ff:.7574 *.* LISTEN tcp6 0 0 xxxx::4c2:20ff:f.7574 *.* LISTEN tcp6 0 0 xxxx::1%lo0.7574 *.* LISTEN tcp6 0 0 xxxx:b400:c70:54.7574 *.* LISTEN tcp4 0 0 127.0.0.1.7574 *.* LISTEN tcp4 0 0 X.X.132.149.7574 *.* LISTEN tcp4 0 0 X.0.0.55.7574 *.* LISTEN tcp6 0 0 ::1.7574 *.* LISTEN udp46 0 0 *.7574 *.*
I can now start a second cache server but specify the cluster name with
-Dcoherence.cluster=TimsCluster. We can see that the clusters are sharing the cluster port but each using a separate ephemeral member port.
$ java -Dcoherence.cluster=TimsCluster -jar coherence.jar ... Started cluster Name=TimsCluster, ClusterPort=7574 ThisMember=Member(Id=1, Timestamp=2016-05-03 12:40:47.186, Address=X.X.X.X:57943
Now if we run CohQL using the default cluster, you will see that the first terminal shows the cluster member has joined.
$ java -cp coherence.jar com.tangosol.coherence.dslquery.QueryPlus CohQL: insert into 'test' key(1) value('one'); CohQL: select count() from 'test'; Results 1
Now restart CohQL and specify a cluster name of TimsCluster.
$ java -Dcoherence.cluster=TimsCluster -cp coherence.jar com.tangosol.coherence.dslquery.QueryPlus CohQL: select count() from 'test'; Results 0
The result is zero as we have connected to a separate cluster by specifying the cluster name.
Extend clients can query the name service on the known cluster port and ask for the address of a proxy server. This has been possible since 12.1.2 but in 12.2.1 combined with the new example coherence-cache-config.xml file this has been made a lot easier.
Note: It is now recommended that if you require proxy clients, all cache servers also run the proxy service. This allows for better scalability of cluster data storage as well as aggregate client proxy bandwidth.
The new cache config has two things that help:
- A proxy-scheme with no acceptor config means it will listen on an ephemeral port like members do.
<proxy-scheme> <service-name>Proxy</service-name> <autostart>true</autostart> </proxy-scheme>
- A remote-cache-scheme, that has no initiator-config will lookup the required proxy service name by asking the name service. To reach the name service the proxy client must share the same operational configuration, and as a result the discovery information for the cluster. In our case it will connect to the name service on port 7574 and ask for the address of the proxy service.
<remote-cache-scheme> <scheme-name>thin-remote</scheme-name> <service-name>RemoteCache</service-name> <proxy-service-name>Proxy</proxy-service-name> </remote-cache-scheme>
Another advantage of the changes that were made to the cache config in 12.2.1 is that you can specify -Dcoherence.client option which will change the way the client connects to the cluster. You can choose either direct which means cluster client or remote which means extend client. Direct is the default. (You can also set -Dcoherence.profile to near for a near cache or thin for a normal client.)
The following, with using the default cache config, will ensure CohQL will connect as an extend client.
$ java -cp coherence.jar -Dcoherence.client=remote com.tangosol.coherence.dslquery.QueryPlus
Its worth extracting the cache config from coherence.jar and taking a look at how it works. Its a good model to utilize and extend. To do this issue:
jar xvf /path/to/coherence.jar coherence-cache-config.xml
Well Known Addresses (WKA)
Up to now i’ve been using the default discovery mechanism which is multicast. If this is not allowed in your environment you can utilize WKA which has also been made a lot easier in 12.2.1 in that it no longer contains ports, only IP addresses. The cluster port also applies to WKA as well as multicast cluster, therefore you can just specify a list of hosts and a cluster name to discover your cluster.
Lets change to using WKA by specifying -Dcoherence.wka=127.0.0.1.
$ java -Dcoherence.wka=127.0.0.1 -jar coherence.jar ... Started cluster Name=timmiddleton's cluster, ClusterPort=7574 ... WellKnownAddressList( 127.0.0.1 )
You can see the the cluster port is still used, but now we see the well known address list. This is not a good WKA address as such because its only on the localhost, but you could of course specify your hostname instead.
In this example I have used a system property to define the WKA, but it is best practice to put this in the operational configuration and ensure all members use the same configuration.
Where the real power is here is that say you have clusters running on host1, host2, host3 and host4 you could add these entries to a DNS entry called production.servers and then connect to the cluster using the following:
# Connect to HRCluster $ java -Dcoherence.wka=production.servers -Dcoherence.cluster=HR ... # Connect to GLCluster $ java -Dcoherence.wka=production.servers -Dcoherence.cluster=GL ...
The idea with all of these changes is to make configuring clusters simpler and to avoid hard coding ports which help with resolving port conflicts. Its also important to note that all existing “legacy” configurations will still work and there may be cases where you need to specify port ranges for firewall access. See Setting Up a Cluster in the Coherence Documentation.
The key here is that the cluster name is now very important and you should set it always!
A number of resources are available to help you understand the changes.
- Mark Falco’s screencast is a great place to start and he does a great job of explaining the detail – https://www.youtube.com/watch?v=Dm3jO2pAorc
- Coherence 12.2.1 Technical Changes in the Documentation.
- Setting up A Cluster in the Online Documentation.