How to Sync Updates between Multiple Controllers using High Availability Support
This tutorial assumes that you are familiar with How to Write a Module and How to Add Fault Tolerance to the Control Plane.
This tutorial describes how you can configure a multiple controller setup in order to enable them to communicate with each other and be able to synchronize updates with each other from the other modules. In addition, this module also performs a leader election process which returns a single Controller ID as the leader of the currently configured active controllers. We will look at how to assign tasks to this leader and follower controllers separately. In order to see how all of this works, we describe how to setup a four controller topology in order to demonstrate what this module can do. We will be learning how to build and work with the topology illustrated below.
Using the HA module
Initial Setup
Configuring four controllers
In order to start four instances of the controller, the following changes were made in the floodlightdefault.properties file under src/main/resources/
- We added three more copies of the floodlightdefault.properties file namely floodlightNodeBackup.properties, floodlightNodeBackup3.properties and floodlightNodeBackup4.properties respectively.
- The following changes were then incorporated into the four different properties files:
net.floodlightcontroller.hasupport.HAController.nodeid=1 net.floodlightcontroller.hasupport.HAController.serverPort=127.0.0.1:4242
net.floodlightcontroller.hasupport.HAController.nodeid=2 net.floodlightcontroller.hasupport.HAController.serverPort=127.0.0.1:4243
net.floodlightcontroller.hasupport.HAController.nodeid=3 net.floodlightcontroller.hasupport.HAController.serverPort=127.0.0.1:4244
net.floodlightcontroller.hasupport.HAController.nodeid=4 net.floodlightcontroller.hasupport.HAController.serverPort=127.0.0.1:4245
We also need a file called server.config which is located under src/main/resources/ as well. This file is read by all controllers in order for them to be able to figure out how to contact the other controllers in the network, the only information in this file is the IP address and the port numbers needed to contact controller 'x' where 'x' is the line number in the file. The file starts with controller 1 and lists in ascending order the IP:port numbers till controller n. This file can be extended when you want to add more controllers.
127.0.0.1:4242 127.0.0.1:4243 127.0.0.1:4244 127.0.0.1:4245
Running the four instances
From the folder where you cloned floodlight on your computer, run the following commands on multiple terminals:
java -jar target/floodlight.jar java -jar target/floodlight.jar -cf src/main/resources/floodlightNodeBackup.properties java -jar target/floodlight.jar -cf src/main/resources/floodlightNodeBackup3.properties java -jar target/floodlight.jar -cf src/main/resources/floodlightNodeBackup4.properties
Using the HAControllerService
This is the first service that the HA support module provides. This interface lets you access the current network wide leader that is active. You can either retrieve the value immediately, in which case you are not guaranteed to get a value (might be None), or you can poll till the algorithm finishes electing a leader (~500ms for four controllers). Additionally, you can use this class to communicate with the other controllers for your own purposes independent of the functioning of this module. We have exposed simple send() and receive() functions which enable you to perform synchronous communication between controllers in a cluster. This can also be re-purposed to listen for events or notifications from one controller to another.
Initialization
In order to get started, create a module which implements IFloodlightModule, as illustrated here:
After this has been done, we first need to create a variable of type IHAControllerService, in order to access the current instance of the HAController.
protected static IHAControllerService hacontroller;
In addition to the above, we need to get the dependencies for the module as follows:
@Override
public Collection<Class<? extends IFloodlightService>> getModuleDependencies() {
Collection<Class<? extends IFloodlightService>> l = new ArrayList<Class<? extends IFloodlightService>>();
l.add(IStorageSourceService.class);
l.add(IFloodlightProviderService.class);
l.add(ISyncService.class);
l.add(IHAControllerService.class);
return l;
}
Next, in order to get the service implementation of the HAController from the context, you need to use the getServiceImpl function.
@Override
public void init(FloodlightModuleContext context) throws FloodlightModuleException {
hacontroller = context.getServiceImpl(IHAControllerService.class);
}
Functions exposed by the IHAControllerService
The following functions are provided by the HA Controller Service.
Now, we can use the hacontroller variable in order to provide the following functions:
public String getLeaderNonBlocking();
This function lets you get the current leader seen by the system, if there is no current leader, this function will return “none”.
public String pollForLeader();
Poll for leader waits for the election to complete and returns the current leader of the system when the election is done.
public void setElectionPriorities(ArrayList<Integer> priorities);
This sets the election priorities for the current election that is happening, i.e. make a list in the specified order of preference in which you would like the controllers to be elected:
Example:
[ 2, 1, 4, 3] => means that controller 2 will be considered first for election, if it fails we fall back to controller 1 as leader, and then controller 4 and so on.
Establishing communication between two or more controllers
public boolean send(String to, String msg);
Example: Send HelloWorld to Controller1.
to: "127.0.0.1:4242"
msg: "mHelloWorld"
send("127.0.0.1:4242", "mHelloWorld");
Messages can be sent from one controller to another using the send function, and the messages have to be tagged with an ‘m’ “m<Actual message>”. You will send this message TO a particular controller so the TO address comprises of two parts IP:port. The IP is the machine IP address of the other controller (HAServer is listening on all ips), and the port is the corresponding listening port of HAServer on that machine.
By default, HAServer on controller 1 is listening on 4242, on controller 2 on 4243, on controller 3 on 4244 … and so on.
public String recv(String from);
Example: Receive from Controller 1.
recv("127.0.0.1:4242");
This function is similar to the send function and you will be giving the FROM address to hear back FROM a particular controller. The from address also comprises of two parts, IP:port. The IP is the machine IP address of the other controller (HAServer is listening on all ips), and the port is the corresponding listening port of HAServer on that machine.
By default, HAServer on controller 1 is listening on 4242, on controller 2 on 4243, on controller 3 on 4244 … and so on. Ideally, this function is called after calling a corresponding send() function, otherwise, a connection might not have been established, and it will just return an error.
Using the HAWorkerService
This is the second service that the HASupport module provides. This service can be used to obtain objects that contain the different HAWorker classes (currently LDHAWorker and TopoHAWorker) thereby enabling us to call the respective publish and subscribe hooks of these HAWorkers. It allows any floodlight module to leverage the HA Support module in order to obtain the updates/state information stored by these HAWorkers. This can be extended by adding more HAWorker classes similar to LDHAWorker.
Initialization
To use the HAWorkerService, we need to create a variable for IHAWorkerService. (this is the IFloodlightService we will leverage)
protected static IHAWorkerService haworker;
In addition to this, we need to get the dependencies for the module as follows:
@Override
public Collection<Class<? extends IFloodlightService>> getModuleDependencies() {
Collection<Class<? extends IFloodlightService>> l = new ArrayList<Class<? extends IFloodlightService>>();
l.add(IStorageSourceService.class);
l.add(IFloodlightProviderService.class);
l.add(ISyncService.class);
l.add(IHAWorkerService.class);
return l;
}
In order to get the service implementation of the HAWorkerService from the context, use the getServiceImpl function.
@Override
public void init(FloodlightModuleContext context) throws FloodlightModuleException {
haworker = context.getServiceImpl(IHAWorkerService.class);
}
Getting specified HAWorker objects
Retrieve a HAWorker class object that has been registered, using its serviceName.
IHAWorker ldhaworker = haworker.getService(“LDHAWorker”); ldhaworker.publishHook();
The publish hook will make the LDHAWorker publish current updates to the syncDB at this particular instant.
However, note that in ControllerLogic, we already publish LDUpdates to the SyncDB every 5 seconds like this:
To subscribe to updates from the SyncDB, for a particular HAWorker:
For example, if you need to subscribe to LDUpdates from controller 2, then use the LDHAWorker in order to do this action.
private String controllerID = “C2”; ldhaworker.subscribeHook(controllerID);
This will result in the HAWorker subscribing to the updates and it will be available in the subscribe hook function in the specified HAWorker, for example in LDHAWorker.java:
public List<String> subscribeHook(String controllerID) {
List<String> updates = new ArrayList<String>();
try {
myLDFilterQueue.subscribe(controllerID);
myLDFilterQueue.dequeueReverse();
return updates;
} catch (Exception e) {
e.printStackTrace();
}
return updates;
}
Getting the worker keys
public Set<String> getWorkerKeys();
The worker keys are a set of <String, Object> pairs, where the Object holds a Singleton instance of the particular HAWorker for a given module. LDHAWorker does HA for a single module which is the Link Discovery service. In order to access that object, we provide a key called "LDHAWorker" which then retrieves that object.
Retrieve a set of keys for all currently registered HAWorker class object serviceNames. Refer to ControllerLogic.java
Creating a HAWorker chain to publish updates from any module to the SyncDB
In order to create a chain of HAWorker classes in order to be able to Sync updates from your module to the Sync DB, you need to create a chain of classes using the interfaces IHAWorker, IFilterQueue, ISyncAdapter which are included in this module. The HAWorker class must include methods publishHook and subscribeHook that are used to call functions in the FilterQueues. publishHook must call enqueueForward and dequeueForward in order to load and unload the queue in the forward direction towards the SyncDB from the module. Similarly, the subscribe hook must call the enqueueReverse and dequeueReverse in order to load and unload the FilterQueue in the reverse direction i.e. from the SyncDB to the module. packJson and unpackJson are functions that are used to finally write the data to the SyncDB. This mechanism has been created for the Link Discovery service module as illustrated by these three classes:
After creating classes similar to these, now you need to hook them up to IHAWorkerService in the HAController.java class so that it gets loaded into the system. This can be done using a fully constructed HAWorker object, which needs a SyncDB datastore, and the controller ID of the current controller. The controller ID of the current controller can be obtained using the IFloodlightProviderService, and the haworker object needs to be initialized as well, both of which can be done in the init function as shown. A SyncDB datastore needs to be created for your module in order for it to be able to push and pull from the SyncDB as shown in this snippet. Refer to this tutorial to learn more about the SyncDB.
protected static IHAWorkerService haworker;
protected static ILinkDiscoveryService linkserv;
protected static ISyncService syncService;
protected static IStoreClient<String, String> storeLD;
private static String controllerID;
protected static LDHAWorker ldhaworker;
protected static IFloodlightProviderService floodlightProvider;
public void init(FloodlightModuleContext context) throws FloodlightModuleException {
haworker = this;
linkserv = context.getServiceImpl(ILinkDiscoveryService.class);
floodlightProvider = context.getServiceImpl(IFloodlightProviderService.class);
syncService = context.getServiceImpl(ISyncService.class);
controllerID = new String("C" + floodlightProvider.getControllerId());
}
logger.info("LDHAWorker is starting...");
try {
HAController.syncService.registerStore("LDUpdates", Scope.GLOBAL);
HAController.storeLD = HAController.syncService.getStoreClient("LDUpdates", String.class, String.class);
HAController.storeLD.addStoreListener(this);
} catch (SyncException e) {
throw new FloodlightModuleException("Error while setting up sync service", e);
}
HAController.ldhaworker = new LDHAWorker(HAController.storeLD, controllerID);
linkserv.addListener(HAController.ldhaworker);
haworker.registerService("LDHAWorker", HAController.ldhaworker); //The string "LDHAWorker" is the worker key which is obtained from the getWorkerKeys() function.
Extending the MAC Tracker example:
In this section of the tutorial we look at how to build an example which can explain how to use the various functions exposed by this module through a simple example. You may be familiar with the MACTracker example which is illustrated in How to Write a Module. We will be extending that example using several functions provided by IHAControllerService and the IHAWorkerService to demonstrate how they can be used.
Example: IHAControllerService
In this example, we look at how we can get the current leader of the multiple controller system and print it on the screen. This function can be called and used in any IFloodlightModule, so we will be using the MACTracker module in order to illustrate this. The leader can further be used for role based programming for tasks such as monitoring the system for events, assigning tasks to the other controllers etc. Here we use both the pollForLeader() and the getLeaderNonBlocking() functions to retrieve the current leader. The difference between the two functions is that pollForLeader waits for the election process to end and then immediately returns the current leader once it is decided upon by all the controllers in quorum. The getLeaderNonBlocking function returns the current local value of the leader variable in the AsyncElection.java class, hence there is a chance that it will return "None" as the value which means that the election process is still in progress.
package net.floodlightcontroller.mactracker;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentSkipListSet;
import net.floodlightcontroller.core.FloodlightContext;
import net.floodlightcontroller.core.IFloodlightProviderService;
import net.floodlightcontroller.core.IOFMessageListener;
import net.floodlightcontroller.core.IOFSwitch;
import net.floodlightcontroller.core.module.FloodlightModuleContext;
import net.floodlightcontroller.core.module.FloodlightModuleException;
import net.floodlightcontroller.core.module.IFloodlightModule;
import net.floodlightcontroller.core.module.IFloodlightService;
import net.floodlightcontroller.hasupport.IHAControllerService;
import net.floodlightcontroller.packet.Ethernet;
import org.projectfloodlight.openflow.protocol.OFMessage;
import org.projectfloodlight.openflow.protocol.OFType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MACTracker implements IFloodlightModule, IOFMessageListener {
protected static IHAControllerService hacontroller;
protected static Logger logger = LoggerFactory.getLogger(MACTracker.class);
protected IFloodlightProviderService floodlightProvider;
protected Set<Long> macAddresses;
@Override
public Collection<Class<? extends IFloodlightService>> getModuleServices() {
// TODO Auto-generated method stub
return null;
}
@Override
public Map<Class<? extends IFloodlightService>, IFloodlightService> getServiceImpls() {
// TODO Auto-generated method stubs
return null;
}
@Override
public Collection<Class<? extends IFloodlightService>> getModuleDependencies() {
// TODO Auto-generated method stub
Collection<Class<? extends IFloodlightService>> l =
new ArrayList<Class<? extends IFloodlightService>>();
l.add(IFloodlightProviderService.class);
l.add(IHAControllerService.class);
return l;
}
@Override
public void init(FloodlightModuleContext context) throws FloodlightModuleException {
// TODO Auto-generated method stub
hacontroller = context.getServiceImpl(IHAControllerService.class);
floodlightProvider = context.getServiceImpl(IFloodlightProviderService.class);
macAddresses = new ConcurrentSkipListSet<Long>();
}
@Override
public void startUp(FloodlightModuleContext context) throws FloodlightModuleException {
// TODO Auto-generated method stub
floodlightProvider.addOFMessageListener(OFType.PACKET_IN, this);
// After more than 51% of configured controllers are started, this function will return,
// or when a timeout of 60s is reached, whichever is earlier.
hacontroller.pollForLeader();
logger.info("Leader is: "+hacontroller.getLeaderNonBlocking());
}
@Override
public String getName() {
// TODO Auto-generated method stub
return MACTracker.class.getSimpleName();
}
@Override
public boolean isCallbackOrderingPrereq(OFType type, String name) {
// TODO Auto-generated method stub
return false;
}
@Override
public boolean isCallbackOrderingPostreq(OFType type, String name) {
// TODO Auto-generated method stub
return false;
}
@Override
public net.floodlightcontroller.core.IListener.Command receive(IOFSwitch sw, OFMessage msg,
FloodlightContext cntx) {
// TODO Auto-generated method stub
Ethernet eth =
IFloodlightProviderService.bcStore.get(cntx,
IFloodlightProviderService.CONTEXT_PI_PAYLOAD);
Long sourceMACHash = eth.getSourceMACAddress().getLong();
if (!macAddresses.contains(sourceMACHash)) {
macAddresses.add(sourceMACHash);
logger.info("MAC Address: {} seen on switch: {}",
eth.getSourceMACAddress().toString(),
sw.getId().toString());
}
return Command.CONTINUE;
}
}
Compile: ant Run: Start three controllers at the same time in three different terminal windows: java -jar target/floodlight.jar java -jar target/floodlight.jar -cf src/main/resources/floodlightNodeBackup.properties java -jar target/floodlight.jar -cf src/main/resources/floodlightNodeBackup3.properties This will print: 2017-03-14 23:04:42.100 INFO [n.f.m.MACTracker] Leader is: 3
Example: IHAWorkerService
In this example, we are going to see how to use the HAWorker Service in order to try publishing and subscribing Link Discovery updates of a particular controller using the SyncDB. We need to create a HAWorker chain as explained above, for your particular module and then you'll be able to retrieve the HAWorker object corresponding to your module using the getWorkerKeys function as shown above. This will allow you to retrieve updates that your module has published to the SyncDB across any one or all controllers.
package net.floodlightcontroller.mactracker;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentSkipListSet;
import net.floodlightcontroller.core.FloodlightContext;
import net.floodlightcontroller.core.IFloodlightProviderService;
import net.floodlightcontroller.core.IOFMessageListener;
import net.floodlightcontroller.core.IOFSwitch;
import net.floodlightcontroller.core.module.FloodlightModuleContext;
import net.floodlightcontroller.core.module.FloodlightModuleException;
import net.floodlightcontroller.core.module.IFloodlightModule;
import net.floodlightcontroller.core.module.IFloodlightService;
import net.floodlightcontroller.hasupport.IHAWorker;
import net.floodlightcontroller.hasupport.IHAWorkerService;
import net.floodlightcontroller.packet.Ethernet;
import org.projectfloodlight.openflow.protocol.OFMessage;
import org.projectfloodlight.openflow.protocol.OFType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MacTracker implements IFloodlightModule, IOFMessageListener {
protected IHAWorkerService haworker;
protected IHAWorker ldhaworker;
protected List<String> updates = new ArrayList<String>();
protected static Logger logger = LoggerFactory.getLogger(MacTracker.class);
protected IFloodlightProviderService floodlightProvider;
protected Set<Long> macAddresses;
@Override
public Collection<Class<? extends IFloodlightService>> getModuleServices() {
// TODO Auto-generated method stub
return null;
}
@Override
public Map<Class<? extends IFloodlightService>, IFloodlightService> getServiceImpls() {
// TODO Auto-generated method stubs
return null;
}
@Override
public Collection<Class<? extends IFloodlightService>> getModuleDependencies() {
// TODO Auto-generated method stub
Collection<Class<? extends IFloodlightService>> l =
new ArrayList<Class<? extends IFloodlightService>>();
l.add(IFloodlightProviderService.class);
l.add(IHAWorkerService.class);
return l;
}
@Override
public void init(FloodlightModuleContext context) throws FloodlightModuleException {
// TODO Auto-generated method stub
haworker = context.getServiceImpl(IHAWorkerService.class);
floodlightProvider = context.getServiceImpl(IFloodlightProviderService.class);
macAddresses = new ConcurrentSkipListSet<Long>();
}
@Override
public void startUp(FloodlightModuleContext context) throws FloodlightModuleException {
// TODO Auto-generated method stub
floodlightProvider.addOFMessageListener(OFType.PACKET_IN, this);
}
@Override
public String getName() {
// TODO Auto-generated method stub
return MacTracker.class.getSimpleName();
}
@Override
public boolean isCallbackOrderingPrereq(OFType type, String name) {
// TODO Auto-generated method stub
return false;
}
@Override
public boolean isCallbackOrderingPostreq(OFType type, String name) {
// TODO Auto-generated method stub
return false;
}
@Override
public net.floodlightcontroller.core.IListener.Command receive(IOFSwitch sw, OFMessage msg,
FloodlightContext cntx) {
// haworker.getService() => Get the various HAWorker objects of all modules which have HAWorker enabled.
// For Example, get the HAWorker associated with LinkDiscoveryManager
// Implemented so far { LinkDiscoveryManager => "LDHAWorker" , TopologyManagaer => "TopoHAWorker" }
ldhaworker = haworker.getService("LDHAWorker");
// Print LDUpdates, this is triggered by a PacketIn message.
// Publish and Subscribe to your own (Controller 1) LDUpdates.
try {
ldhaworker.publishHook();
updates = ldhaworker.subscribeHook("C1");
for (String update: updates){
logger.info("Subscribe: {}", new Object[] {update} );
}
} catch (Exception e) {
e.printStackTrace();
}
// TODO Auto-generated method stub
Ethernet eth =
IFloodlightProviderService.bcStore.get(cntx,
IFloodlightProviderService.CONTEXT_PI_PAYLOAD);
Long sourceMACHash = eth.getSourceMACAddress().getLong();
if (!macAddresses.contains(sourceMACHash)) {
macAddresses.add(sourceMACHash);
logger.info("MAC Address: {} seen on switch: {}",
eth.getSourceMACAddress().toString(),
sw.getId().toString());
}
return Command.CONTINUE;
}
}
Compile:
ant
Run:
java -jar target/floodlight.jar
Start a mininet topology:
mininet@mininet-vm:~$ sudo python IslandTest1.py
This will print:
2017-03-15 00:12:17.445 INFO [n.f.m.MacTracker] Subscribe: {"dstPort":"2","dst":"00:00:00:00:00:00:00:02","src":"00:00:00:00:00:00:00:01","latency":"0x0000000000000011","srcPort":"2","type":"internal","operation":"Link Updated","timestamp":"148955113153000000"}
2017-03-15 00:12:17.445 INFO [n.f.m.MacTracker] Subscribe: {"dstPort":"2","dst":"00:00:00:00:00:00:00:01","src":"00:00:00:00:00:00:00:02","latency":"0x0000000000000002, 0x0000000000000002","srcPort":"2","type":"internal","operation":"Link Updated, Port Up","timestamp":"148955113154000000, 148955113157000000"}
2017-03-15 00:12:17.445 INFO [n.f.m.MacTracker] Subscribe: {"dstPort":"2","dst":"00:00:00:00:00:00:00:01","src":"00:00:00:00:00:00:00:02","latency":"0x0000000000000002","srcPort":"1","type":"internal","operation":"Port Up","timestamp":"148955113155000000"}
...
#!/usr/bin/python
import time
from mininet.net import Mininet
from mininet.node import Controller, OVSKernelSwitch, RemoteController
from mininet.cli import CLI
from mininet.log import setLogLevel, info
setLogLevel('info')
def addHost(net, N):
name= 'h%d' % N
ip = '10.0.0.%d' % N
return net.addHost(name, ip=ip)
net = Mininet(controller=RemoteController, switch=OVSKernelSwitch)
c1 = net.addController('c1', controller=RemoteController, ip="192.168.56.1", port=6653)
c2 = net.addController('c2', controller=RemoteController, ip="192.168.56.1", port=7753)
print "*** Creating switches"
s1 = net.addSwitch( 's1' )
s2 = net.addSwitch( 's2' )
s3 = net.addSwitch( 's3' )
s4 = net.addSwitch( 's4' )
print "*** Creating hosts"
hosts1 = [ addHost( net, n ) for n in 3, 4 ]
hosts2 = [ addHost( net, n ) for n in 5, 6 ]
hosts3 = [ addHost( net, n ) for n in 7, 8 ]
hosts4 = [ addHost( net, n ) for n in 9, 10 ]
print "*** Creating links"
for h in hosts1:
s1.linkTo( h )
for h in hosts2:
s2.linkTo( h )
for h in hosts3:
s3.linkTo( h )
for h in hosts4:
s4.linkTo( h )
s1.linkTo( s2 )
s2.linkTo( s3 )
s4.linkTo( s2 )
print "*** Building network"
net.build()
# In theory this doesn't do anything
c1.start()
c2.start()
#print "*** Starting Switches"
s1.start( [c1] )
s2.start( [c2] )
s3.start( [c1] )
s4.start( [c1] )
net.start()
CLI( net )
net.stop()