This tutorial assumes an understanding of the OpenFlowJ-Loxi API and of Floodlight module writing.
This tutorial was largely contributed by Josh Boley – Thanks Josh!
Table of Contents
What are meters?
OpenFlow 1.3 introduces meters to the OpenFlow protocol. Meters complement the queue framework already in place in OpenFlow by allowing for the rate-monitoring of traffic prior to output. More specifically, with meters, we can monitor the ingress rate of traffic as defined by a flow. Flows can direct packets to a meter using the goto-meter OpenFlow instruction, where the meter can then perform some operation based on the rate it receives packets.
Contrast this to a queue, which accepts packets for output and processes them at a min/max specified rate. As such, note that meters and queues are complementary and are not different implementations of the same thing. A common misconception is that meters are a replacement for queues, which is not true. If you are interested in using queues with Floodlight, please check out this tutorial.
Unlike queues though, which are rather rigid and must be defined by the switch out of band (e.g. if using Open vSwitch (OVS) with OVS commands), meters can be installed, modified, and removed at runtime using OpenFlow. In fact, we can liken meters to flows themselves. OpenFlow defines an abstraction called a meter table, which simply contains rows of meters. These meters can be manipulated in a similar manner to flows. Also like flows, meters receive packets as input and (optionally) send packets as output – pretty cool!
What switches support meters?
OpenFlow 1.3 and up all define meters within the OpenFlow protocol; however, they are an optional feature. At present, only ofsoftswitch and various hardware switch vendors support meters. Note that OVS currently only the userspace datapath implements meters, as of v2.8.90.
ofsoftswitch is a software OpenFlow switch implementation, similar to OVS. It can be dowloaded and installed according to these instructions.
In addition to the ofsoftswitch instructions linked above, there's an undocumented bug in ofsoftswitch that causes it to choke on kernel buffering schemes like TCP Segment Offloading (TSO), which means we have to disable all of those features on every interface on the network – both switches and hosts.
If running tail /var/log/syslog shows a lot of error messages from the ofdatapath process along the lines of "Message size too large" or "Dropping packets due to excessive rates" then this is likely the problem you're experiencing. The easiest way to handle it in Mininet is with a startup script, the relevant part would be something like this:
nodes = net.hosts + net.switches
for node in nodes:
for port in node.ports:
if str.format('{}', port) != 'lo':
node.cmdPrint(str.format('ethtool --offload {} gro off gso off tso off', port))
Performance may still be a little on the flaky side but you should see some major improvements. Running Mininet natively on a RedHat server box we saw a consistent max total throughput increase from about 7-10 Mbps to 200 Mbps just from this. Metered flow rates are fairly close to the mark, though we have started to notice what looks like sampling rate errors cropping up in the iperf interval reporting.
Using meters in Floodlight
In order to use meters we first need to install one. This can be done using an ADD meter-mod message. Meters can then be modified or removed using the meter-mod MODIFY and DELETE messages, respectively. The following is an example:
public class DropMeter {
public static enum Cmd {
ADD(0),
MODIFY(1),
DELETE(2);
int cmd;
Cmd(int code) {
cmd = code;
}
public int getCode() {
return cmd;
}
}
protected DatapathId swId; /* Switch ID */
protected int flags, /* Meter flags */
rate, /* Meter band drop rate */
id, /* Meter ID */
burstSize; /* Burst control rate */
public DropMeter(DatapathId swId, int id, int flags, int rate, int burst) {
this.swId = swId;
this.flags = flags;
this.rate = rate;
this.id = id;
this.burstSize = burst;
}
// ...
public void write(Cmd cmd) {
OFFactory meterFactory = OFFactories.getFactory(OFVersion.OF_13);
OFMeterMod.Builder meterModBuilder = meterFactory.buildMeterMod()
.setMeterId(id)
.setCommand(cmd.getCode());
switch(cmd) {
case ADD:
case MODIFY:
/* Create and set meter band */
OFMeterBandDrop.Builder bandBuilder = meterFactory.meterBands().buildDrop()
.setRate(rate);
if (this.burstSize != 0) {
bandBuilder = bandBuilder.setBurstSize(this.burstSize);
}
OFMeterBand band = bandBuilder.build();
List<OFMeterBand> bands = new ArrayList<OFMeterBand>();
bands.add(band);
/* Create meter modification message */
meterModBuilder.setMeters(bands)
.setFlags(flags)
.build();
break;
case DELETE:;
}
/* Send meter modification message to switch */
IOFSwitch sw = switchService.getSwitch(swId); /* The IOFSwitchService */
sw.write(meterModBuilder.build());
}
}
After we install a meter, we can then tell packets to go to this meter by using the goto-meter instruction in a flow. Note that multiple flows can send packets to the same meter, or each flow can direct packets to individual meters – there is no prescribed mapping we need to use.
List<OFInstruction> instructions = new ArrayList<OFInstruction>();
/* Meters only supported in OpenFlow 1.3 and up --> need 1.3+ factory */
OFInstructionMeter meter = myOF13Factory.instructions().buildMeter()
.setMeterId(1)
.build();
OFInstructionApplyActions output = myOF13Factory.actions().buildOutput()
.setPort(OFPort.of(2))
.setMaxLen(0xffFFffFF)
.build();
/*
* Regardless of the instruction order in the flow, the switch is required
* to process the meter instruction prior to any apply actions instruction.
*/
instructions.add(meter);
instructions.add(Collections.singletonList((OFAction) output));
/* Flow will send matched packets to meter ID 1 and then possibly output on port 2 */
OFFlowAdd flowAdd = my13Factory.buildFlowAdd()
/* set anything else you need, e.g. match */
.setInstructions(instructions)
.build();
A final thing to note is that OpenFlow requires the switch evaluate the goto-meter instruction in a flow prior to any apply actions instruction. This ensures the meter can perform its prescribed task (e.g. drop packet or DSCP remark) prior to potentially sending the packet out. If a meter drops a packet, any further instructions in the flow will not be processed for that particular packet. (See pages 20 and 21 of the OpenFlow 1.3 spec for more information on instruction ordering/processing with regard to meters.)
Questions, comments? Send us an email at floodlight-dev@openflowhub.org.