Can JNetPcap be used to build packet bridge/forwarder?

10 replies [Last post]
abb
Offline
Joined: 01/10/2011

Hello,

I am trying to use JNetPcap to create a bridge-like tool. The problem is that I need to receive and send the packets on several interfaces more or less simultaneously. Could you please advise what is the best approach to do the same?

My attempts to use different threads to send and receive packets didn't work out so far, so I have resorted to dedicating a thread per interface. That thread does reading/sending in the loop.

I have written a proof-of-concept code, which is supposed to transparently bridge between two interfaces specified on the command line.

This code works fine when the load is very low. But under higher load (1kpps of ICMP, generated with hping) it breaks. It appears that it starts sending (or receiving) corrupted packets:

$ sudo java -Djava.library.path=lib -jar dist/JBridge.jar vmnet11 vmnet12
non-Ethernet frame, dropping
non-Ethernet frame, dropping
...
non-Ethernet frame, dropping
Exception in thread "Thread-1" java.nio.BufferUnderflowException
at org.jnetpcap.nio.JBuffer.check(Unknown Source)
at org.jnetpcap.nio.JBuffer.getByteArray(Unknown Source)
at org.jnetpcap.nio.JBuffer.getByteArray(Unknown Source)
at org.jnetpcap.protocol.lan.Ethernet.source(Unknown Source)
at jbridge.PcapForwarder.process(PcapForwarder.java:85)
at jbridge.PcapForwarder.access$000(PcapForwarder.java:19)
at jbridge.PcapForwarder$1.nextPacket(PcapForwarder.java:40)
at jbridge.PcapForwarder$1.nextPacket(PcapForwarder.java:37)
at org.jnetpcap.Pcap.dispatch(Native Method)
at org.jnetpcap.Pcap.dispatch(Unknown Source)
at jbridge.PcapForwarder.run(PcapForwarder.java:45)
at java.lang.Thread.run(Thread.java:662)

I am testing on Ubuntu 10, trying to bridge between two host-only virtual networks. The java code runs on host OS. I have tried both JNetPcap 1.3 and 1.4. There is a slight chance of glitches in VMWare code I suppose, but from the networking perspective the load is still extremely low (1-2Kpps), and if I replace my bridge with standard Linux bridge everything goes smoothly.

Here is the code, please let me know if I can do anything to make it more readable.


package jbridge;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import org.jnetpcap.Pcap;

/**
*
* @author Alexandre Bezroutchko abb@gremwell.com
*/
public class Main {

public static void main(String[] args) throws InterruptedException {
StringBuilder errbuf = new StringBuilder(); // For any error msgs

int snaplen = 64 * 1024; // Capture all packets, no trucation
int flags = Pcap.MODE_PROMISCUOUS; // capture all packets
int timeout = 100;

// open
Pcap pcap1 = Pcap.openLive(args[0], snaplen, flags, timeout, errbuf);
Pcap pcap2 = Pcap.openLive(args[1], snaplen, flags, timeout, errbuf);

// forward two-ways
Map mac2pcap = Collections.synchronizedMap(new HashMap());

PcapForwarder fwd1 = new PcapForwarder(pcap1, mac2pcap);
PcapForwarder fwd2 = new PcapForwarder(pcap2, mac2pcap);
fwd1.setForwarder(fwd2);
fwd2.setForwarder(fwd1);
Thread thread1 = new Thread(fwd1);
Thread thread2 = new Thread(fwd2);

thread1.start();
thread2.start();
thread1.join();
thread2.join();
}
}

package jbridge;

import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingDeque;
import java.util.concurrent.TimeUnit;
import org.jnetpcap.JBufferHandler;
import org.jnetpcap.Pcap;
import org.jnetpcap.PcapHeader;
import org.jnetpcap.nio.JBuffer;
import org.jnetpcap.packet.JMemoryPacket;
import org.jnetpcap.packet.JPacket;
import org.jnetpcap.protocol.lan.Ethernet;

/**
* This class reads and writes packets to the specified Pcap interface.
* It extends Runnable and is supposed to be run in its own thread.
*
* While running, it does the following:
* 1) receive a packet from JNetPcap,
* 2) make sure it is not a duplicate by consulting a mac table,
* 3) updates a mac table if necessary,
* 4) submits it to another PcapForwarder,
* 5) check if any packets were enqueued for transmission
* 6) sends enqueued packet, if any
*
* @author Alexandre Bezroutchko abb@gremwell.com
*/
public class PcapForwarder implements Runnable {

private final Pcap pcap;
private PcapForwarder fwd;
private final Map smac2pcap;
private final Ethernet ethP = new Ethernet();
private final BlockingQueue egressQueue = new LinkedBlockingDeque();

public PcapForwarder(Pcap pcap, Map mac2iface) {
this.pcap = pcap;
this.smac2pcap = mac2iface;
}

public void setForwarder(PcapForwarder fwd) {
this.fwd = fwd;
}

public void run() {
JBufferHandler handler = new JBufferHandler() {

public void nextPacket(PcapHeader header, JBuffer buffer, Pcap pcap) {
processIngressPacket(buffer);
}
};

for (Batting Eyelashes {
// try to receive a packet or timeout
int dispatchRes = pcap.dispatch(1, handler, pcap);
if (dispatchRes >= Innocent {
// success, the packet has been processed by the handler
} else if (dispatchRes == -2) {
System.err.println("dispatch() was interrupted");
break;
} else if (dispatchRes == -1) {
// error
System.err.printf("dispatch() error: " + pcap.getErr());
break;
} else {
// ?
System.err.printf("dispatch() returned " + dispatchRes);
break;
}

// take a packet form the egress queue (if any) and send it away
try {
JBuffer buf = egressQueue.poll(0, TimeUnit.MILLISECONDS);
if (buf != null) {
// there is a packet to send
if (pcap.sendPacket(buf) != Pcap.OK) {
System.err.println("send error: " + pcap.getErr());
break;
}
}
} catch (InterruptedException ex) {
System.err.println("interrupted");
break;
}
}
}

private void processIngressPacket(JBuffer buf) {
JPacket pkt = new JMemoryPacket(Ethernet.ID, buf);

if (!pkt.hasHeader(ethP)) {
System.err.println("non-Ethernet frame, dropping");
return;
}

synchronized (smac2pcap) {
HardwareAddress smac = new HardwareAddress(ethP.source());
Pcap iface = smac2pcap.get(smac);

if (iface == null) {
// first time we see this MAC
smac2pcap.put(smac, iface);
} else if (iface != pcap) {
// this MAC was seen on another interface,
// it must be a duplicate, drop it
return;
}
}

// the packet is not a duplicate, forward it
fwd.enqueue(buf);
}

private void enqueue(JBuffer buf) {
egressQueue.add(buf);
}
}

package jbridge;

import java.util.Arrays;

/**
* This class represents a MAC address. It implements equals() and hashCode()
* as necessary and can be used as a key in Maps.
*
* @author Alexandre Bezroutchko abb@gremwell.com
*/
public class HardwareAddress {

private byte[] mac;

public HardwareAddress(byte[] mac) {
assert (mac.length == 6);
this.mac = Arrays.copyOf(mac, mac.length);
}

@Override
public boolean equals(Object obj) {
if (obj == null) {
return false;
}

if (getClass() != obj.getClass()) {
return false;
}

final HardwareAddress other = (HardwareAddress) obj;
if (!Arrays.equals(this.mac, other.mac)) {
return false;
}

return true;
}

@Override
public int hashCode() {
return Arrays.hashCode(mac);
}
}

Best regards,
Alexandre Bezroutchko
www.gremwell.com

abb
Offline
Joined: 01/10/2011
I have just found another

I have just found another post on this forum describing problems with VMWare http://www.jnetpcap.org/node/482. Unfortunately the thread is locked, so I can't post there.

The tool I am developing must support VMWare, so I would be interested in workarounds. One thing which comes to my mind is to use a physical USB network adapter attached straight to VM, instead of host-only network driver. I wonder if anybody has tried it. Should work I suppose.

Anyway, the post I refer to talks about packet drops, not packet corruption I observe. I still wonder whether the approach I have described in the previous post is reliable or not. Or perhaps it is even ok send and receive to the same Pcap device simultaneously from different threads?

Mark Bednarczyk
Offline
Joined: 03/22/2008
JBufferHandler references

JBufferHandler references the libpcap ring buffer memory as it delivers packets without copies. You need to do a copy of the buffer before you enque it. It gets overriden with the following packets after it. That is why you are seeing corruption. Future packets are overriding old packets on the queue.

As to VMWare, those issues are still there with the default 'pcap' settings. They have been resolved by use of the newer libpcap 1.0.0 API (Pcap.create, Pcap.setBufferSize, Pcap.activate) and increasing the ring-buffer size. The latest build 1.4.r1300 provides those functions and is based on the same stable 1.3 code base. jNetPcap is used with VMWare quiet frequently, but does need a bit more attention. Native libpcap applications have the same issues.

I'll look at your code some more a bit later and provide you with more feedback. I can't do it right now.

Sly Technologies, Inc.
http://slytechs.com

abb
Offline
Joined: 01/10/2011
Thank you for feedback. I

Thank you for feedback. I have added deep copy to the code above, but this alone did not help, I get exactly the same exceptions. There seems to be more severe problem with my approach. I have a feeling that my problems are related not only to simultaneous send-receive, but to simultaneous receive's as well.

The code above uses two threads, each handling send/receive for one network interface. I suspect that the corruption occurs when two threads try to receive from pcap simultaneously. They do it for different interfaces, but it seems to cause problems anyway.

I have modified the code to use only one thread, it takes care of send/receive tasks for both interfaces sequentially, one after another. Now I don't get exceptions or corrupted packets at all, but I still see packet loss (0.5% or so) at higher loads. I have changed the code to support libpcap API 1.0.0, but it didn't seem to made any difference.

When you talk about increasing ring-buffer size, you refer to Pcap.setBufferSize() or there is something else to tweak?

Below is the modified code.

package jbridge;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import org.jnetpcap.Pcap;
import org.jnetpcap.Pcap.Direction;

/**
*
* @author Alexandre Bezroutchko abb@gremwell.com
*/
public class Main {

public static void main(String[] args) throws InterruptedException {
// capture settings
int snaplen = 64 * 1024;
int flags = Pcap.MODE_PROMISCUOUS;
int timeoutMs = 10;
int bufsizeBytes = 128 * 1024 * 1024;

// open pcap interfaces
Pcap pcap1, pcap2;
StringBuilder errbuf1 = new StringBuilder(), errbuf2 = new StringBuilder();
if (Pcap.isPcap100Loaded()) {
System.out.println("libpcap API 1.0.0 or above is loaded");
pcap1 = openLive100(args[0], snaplen, flags, timeoutMs, Pcap.Direction.IN, bufsizeBytes, errbuf1);
pcap2 = openLive100(args[1], snaplen, flags, timeoutMs, Pcap.Direction.IN, bufsizeBytes, errbuf2);
} else {
pcap1 = Pcap.openLive(args[0], snaplen, flags, timeoutMs, errbuf1);
pcap2 = Pcap.openLive(args[1], snaplen, flags, timeoutMs, errbuf2);
}

if (pcap1 == null) {
System.err.println("failed to open pcap1: " + errbuf1.toString());
}

if (pcap2 == null) {
System.err.println("failed to open pcap2: " + errbuf2.toString());
}

if (pcap1 == null || pcap2 == null) {
return;
}

// instantiate packet forwarders
Map mac2pcap = Collections.synchronizedMap(new HashMap());
PcapForwarder fwd1 = new PcapForwarder(pcap1, mac2pcap);
PcapForwarder fwd2 = new PcapForwarder(pcap2, mac2pcap);
fwd1.setForwarder(fwd2);
fwd2.setForwarder(fwd1);

//Thread thread1 = new Thread(fwd1);
//Thread thread2 = new Thread(fwd2);
//thread1.start();
//thread2.start();
//thread1.join();
//thread2.join();

// forward the packets around
while (fwd1.run1() && fwd2.run1()) {
}
}

private static Pcap openLive100(String name, int snaplen, int flags, int timeoutMs, Direction direction, int bufsizeBytes, StringBuilder errbuf) {
Pcap pcap = Pcap.create(name, errbuf);
if (pcap == null) {
return null;
}

// standard properties
pcap.setSnaplen(snaplen);
pcap.setPromisc(flags);
pcap.setTimeout(timeoutMs);

// specific to libpcap 1.0.0
pcap.setDirection(direction); // We now have IN, OUT or INOUT
pcap.setBufferSize(bufsizeBytes);

pcap.activate();

return pcap;
}
}

and the actual forwarders:


package jbridge;

import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.jnetpcap.Pcap;
import org.jnetpcap.PcapHeader;
import org.jnetpcap.nio.JBuffer;
import org.jnetpcap.nio.JMemory;
import org.jnetpcap.packet.JMemoryPacket;
import org.jnetpcap.packet.JPacket;
import org.jnetpcap.packet.PcapPacket;
import org.jnetpcap.protocol.lan.Ethernet;

/**
* This class reads and writes packets from/to the specified interface.
*
* Its run1() method does the following:
*
* 1) receives a packet from JNetPcap,
* 2) makes sure it is not a duplicate by consulting a MAC table,
* 3) updates a MAC table if necessary,
* 4) submits it to another PcapForwarder,
* 5) checks if any packets were enqueued for transmission
* 6) sends enqueued packet, if any
*
* @author Alexandre Bezroutchko abb@gremwell.com
*/
public class PcapForwarder /* implements Runnable */ {

//** the interface
private final Pcap pcap;
//** a reference to peer forwarder object
private PcapForwarder fwd;
//** a map to track on what interface MACs were seen, used to suppress duplicates
private final Map smac2pcap;
//** the following object will be used for parsing ingress packets
private final Ethernet ethP;
//** egress packets will be placed in this queue
private final BlockingQueue egressQueue = new LinkedBlockingQueue();

public PcapForwarder(Pcap pcap, Map smac2pcap) {
this.pcap = pcap;
this.smac2pcap = smac2pcap;

ethP = new Ethernet();
}

public void setForwarder(PcapForwarder fwd) {
this.fwd = fwd;
}

//** process one ingress and one egress packet, if any
public boolean run1() {
return processIngressPackets()
&& processEgressPacket();
}

private boolean processIngressPackets() {
PcapHeader pcapHeader = new PcapHeader(JMemory.POINTER);
JBuffer jBuf = new JBuffer(JMemory.POINTER);

int res = pcap.nextEx(pcapHeader, jBuf);
if (res == Pcap.NEXT_EX_OK) {
processIngressPacket(pcapHeader, jBuf);
return true; // ignore packet processing errors
} else if (res == Pcap.NEXT_EX_TIMEDOUT) {
// timeout, do nothing
return true;
} else if (res == Pcap.NEXT_EX_EOF) {
// it must be EOF
System.err.println("nextEx: EOF");
return false;
} else {
// error?
System.err.printf("nextEx error: " + pcap.getErr());
return false;
}
}

private boolean processIngressPacket(PcapHeader pcapHeader, JBuffer jBuf) {
// parse Ethernet headers
JPacket pkt = new JMemoryPacket(Ethernet.ID, jBuf);
if (!pkt.hasHeader(ethP)) {
System.err.println("non-Ethernet frame, dropping");
return false;
}

// based on SMAC, check if the packet is a duplicate
synchronized (smac2pcap) {
HardwareAddress smac = new HardwareAddress(ethP.source());
Pcap iface = smac2pcap.get(smac);

if (iface == null) {
// first time we see this MAC
smac2pcap.put(smac, iface);
} else if (iface != pcap) {
// this MAC was seen on another interface,
// it must be a duplicate, just drop it
return true;
}
}

// deep copy and place to the egress queue of another PcapForwarder
PcapPacket pktCopy = new PcapPacket(pcapHeader, jBuf);
fwd.egressQueue.add(pktCopy);
return true;
}

//** Take a packet from the egress queue (if any) and send it away
private boolean processEgressPacket() {
try {
JBuffer egBuf = egressQueue.poll(0, TimeUnit.MILLISECONDS);
if (egBuf != null) {
// there is a packet to send
if (pcap.sendPacket(egBuf) != Pcap.OK) {
System.err.println("send error: " + pcap.getErr());
return false;
}
}
return true;
} catch (InterruptedException ex) {
System.err.println("interrupted");
return false;
}
}
}

abb
Offline
Joined: 01/10/2011
the problem is solved

The problem with my code was unsafe use of scan() and hasHeader() functions. This topic http://www.jnetpcap.org/node/514 describes the intended behavior of these functions.

Bridging code I have now works pretty well. So far I tested it running on host OS (Ubuntu 10.04, VMWare Workstation 6.5.4) and bridges traffic between two VMs (based on BackTrack 4, also Debian/Ubuntu-based).

When tested with netperf (default TCP_STREAM mode, single TCP connection), it pumps 15Mbps/2kpps. It also handles 15kpps ICMP echo request/reply flood without loosing packets, sent by hping3.

I will try to clean up the bridging code and publish it somewhere. While browsing through the forum I have noticed that I am not the only one who tried make bridge-like tool.

Marc, thank you for sharing your library with us!

Best regards,
Alex
www.gremwell.com

abb
Offline
Joined: 01/10/2011
jnetbridge

I've published the code which can be used to build bridge/router-like applications based on jnetpcap. It is under LGPL, like jnetpcap itself. More details here: http://www.gremwell.com/jnetbridge-0.1-release

Mark Bednarczyk
Offline
Joined: 03/22/2008
That is great news! If you

That is great news! If you wouldn't mind blogging about it and I'll publish it to front page. (From you admin menu, create new content, blog and that it.)

I also looked at your example and wow, that is pretty neat and concise code that does so much.

Sly Technologies, Inc.
http://slytechs.com

Mark Bednarczyk
Offline
Joined: 03/22/2008
As to performance

abb wrote:

When tested with netperf (default TCP_STREAM mode, single TCP connection), it pumps 15Mbps/2kpps. It also handles 15kpps ICMP echo request/reply flood without loosing packets, sent by hping3.

As to performance. Scanning packets currently is terribly inefficient with the default 'general' scanner. The reason is that the scanner is so general, it allows java to native calls and then it also allows java based scanner and binding functions to be part of the header definition, so at times it goes back to java from native space. All this back and forth, creating JNI references etc, is terribly inefficient, but extremely flexible.

For more specialized purposes, such as scanning data-link layer, which is something a bridge would be interested in, will be part of a new set of scanners that are a bit more specialized. I have created a test TCP scanner that is about 10 times faster then the current general scanner. I've used it in some benchmarks and they are off the chart. I plan on adding these specialized scanners which you can easily substitute for the general one with JPacket.setDefaultScanner and probably some new API I will be adding in one this becomes more common.

For basic scanning, and your type of functionality, with some assistance, we can probably tweak the performance in the 100k pps and above range. Queing packets on a queue will also kill performance big time, there are better ways to synchronize and pass data around, without resorting to blocking queues at least on a per packet basis.

I think we are just getting warmed up here.

Best regards,
mark...

Sly Technologies, Inc.
http://slytechs.com

abb
Offline
Joined: 01/10/2011
Thanks for the feedback!

Thanks for the feedback! Actually, the benchmarks were related to the code I have posted above. JNetBridge uses slightly different approach: it uses one thread per interface, which polls the packet (with nextEx()) and sends out all packets from the egress queue. The ingress packets get scanned and placed into the ingress queue.

With two threads on my rather low-end testbed (FastEthernet links, single core 2.4MHz P4 processor) I get ~90Mbps TCP throughput, with 70% CPU utilization or so. The tool I'm making is a bit more complicated than a transparent bridge. It does address/port translation, keeps/expires connection states, etc which takes a lot of CPU cycles. So for my particular application packet receiving/sending is will not be a bottleneck. Nevertheless, if there is a way to improve its performance it would be useful.

I thought the queue-based API used by JNetBridge is reasonably efficient and easy to use. If you have better ideas, please tell about it, when you have a moment. I am willing to spend a bit more time to adjust the API to make sure it does not become a bottleneck when you publish the optimized scanners. There is one additional consideration: the API should not be tied to libpcap as a packet source/sink. My application will use TAP interfaces in some near future.

Best regards,
Alex

Mark Bednarczyk
Offline
Joined: 03/22/2008
One step ahead of you on this one

abb wrote:
There is one additional consideration: the API should not be tied to libpcap as a packet source/sink. My application will use TAP interfaces in some near future.

We are in agreement here. You will see the API becoming more modular overtime. The initial plan was to do a heavy handed split starting with a major 2.0 release, but that will not work any longer because of jnetpcap's general acceptance thus far.

The software will become modular such that jnetpcap will only contain the 'pcap' based code, while a new 'protocol' module will contain the packet decoding and protocol analysis code. The changes will be introduced gradually. The first step is to sever the hard API connection between scanners, packets and pcap library wrapper. I will be introducing a new class that will accomplish this and still allow the old API to exist. This will be a generic callback class that jnetpcap library can be coded for. The new "Callback" class will abstract away various handlers, and sinks for the packet. Its very elegant solution and allows very gradual migration away from the current hard-coded pcap class loop and dispatch methods that take a user "handler".

The main driving reason for the slit however, is that the 'pcap' wrapper part hardly ever changes and is pretty stable. While all the complexity and changes occur with the decoders. So 'pcap' wrapper is kind of held as a hostage to the 'decoder' part when they are released as one. There are many other benefits of modules so this is definitely the plan.

Sly Technologies, Inc.
http://slytechs.com

abb
Offline
Joined: 01/10/2011
I see. I have also had a

I see. I have also had a look at the roadmap you have published, looks interesting.

If you feel like merging jnetbridge into your source code tree and make it one of the modules -- please feel free doing so. I have also written some packet creation/validation methods, mainly for unit testing. For example, I have a unit test method like:


@Test
public void createAssertArpRequest() {
JPacket packet = PacketFactory.createArpRequestJPacket(testConfig.MACa, testConfig.IPa, testConfig.IPb);
PacketFactoryAsserts.assertArpRequestJPacketEquals(testConfig.MACa, testConfig.IPa, testConfig.IPb, packet);
}

I feel this kind of routines belong to JNetPcap, if you think the same I can contribute the code as well. Or perhaps there is already such methods in JNetPcap?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.