Tutorials

Here are some tutorials that demonstrate step by step usage of jNetPcap SDK.

Tutorial 1 - creating IP fragment reassembler

IP fragment reassembly is a bit more advanced topic. The tutorial will demonstrate more typical usage of Packet Decoding Framework, part of jNetPcap API.

In order to understand some of the networking concepts discussed in this tutorial you need to first study IP fragmentation and reassembly. Wikipedia has a nice short write up on Ip fragmentation and reassembly (link: Wikipedia). Only a cursory overview understanding is needed to follow the tutorial.

Then we will plan out our strategy for our little application and implement it.

Here are some key concepts about the API that are covered in this tutorial:

The complete application source code can be downloaded here: IpReassemblyExample.java

Step 1 - design our application

Our application is going to be very simple. It is going to read a pcap capture file that contains some Ip4 packets that have been fragmented. It is going to reassemble those fragments, create a new packet that is made up of just the reassembled data. We're going to drop the datalink headers and simply insert a new Ip header in front all those fragments, so that our packet's DLT will be Ip4 (DLT is the first header in the packet.)

Here is what our new packet will look like in memory:

+------------+--------+--------+
| Ip4 header | frag 1 | frag 2 |
+------------+--------+--------+

We need to handle the incoming stream of packets. So the first thing we need to setup is a packet handler that will receive packets from libpcap. We're not going to be concerned with multi-threading issues in this tutorial. So to receive packets our main application class will simply implement the PcapPacketHandler interface. Once we have the packets we will need to check if the packet is Ip4 packet and if its fragmented or not.

For all Ip4 packets, fragmented or not, we going to stuff them into a reassembly buffer that we are going to use for IP datagram, fragmented or not.

The Ip4 flag NO_MORE_FRAGMENTS is going to give us a clue about when the fragment is complete, but we can't always rely on that flag. Fragments can arrive out of sequence or even be dropped along the way and never arrive. So we are also going to keep track of how many bytes we have reassembled. When that total matches the length of the entire unfragmented datagram, then we know we have received all fragments and we are done.

For those cases where fragment is dropped and never arrives, we are also going to implement a simple timeout mechanism that will timeout each reassembly buffer past certain amount of time.

Here is some pseudo code that our application is going to implement:

loop {
  Receive packet from libpcap;
  if packet is Ip4 packet then
    get or create reassembly buffer and store in a map;

    calculate offset into the buffer and add fragment
  
    if the packet is complete then
      remove buffer from map;
      dispatch buffer to user's handler;
    endif

  endif

  timeout buffer entries;
}

User handler {
  receive reassembly buffers;
  create a new IP only packet;
  scan the packet;
  to packet.toString() to get pretty output;
}

Reassembly buffer

This is a very important piece of our application therefore we need to plan it out in detail. We're calling this buffer IpReassemblyBuffer and it extends a JBuffer.

We are going to allocate a large JBuffer which will hold our ip header and all the fragments combined. Like so:

+------------+--------+--------+
| Ip4 header | frag 1 | frag 2 |
+------------+--------+--------+

The buffer is also going to keep track of timeouts. We're going to set a time value at which time the buffer becomes officially timed out. We will implement a simple isTimedout():boolean method to check for that condition. The method simply compares the timeout timestamp with the current time and if its past its due date, return true.

The buffer needs to keep track of number of bytes already assembled and the total length of the IP data gram. When the 2 are equal, that means the buffer is complete and we can dispatch if to the user. We're also going to implement as boolean method that checks for this condition isComplete().

To keep track of all the buffers, we're going to use a JRE Map and use a 32 bit int hash we generate from ip fragments ip header using fields, Ip4.id(), Ip4.source(), Ip4.destination(), Ip4.protocol like so:
int hash = (id() << 16) ^ source() ^ destination() ^ type();

We're also going to use a PriorityQueue that will prioritize buffers for us based on the timeout timestamp value. Buffers will be ordered according to timeout value. The packets on top of the queue are going to be either timedout or closer to timeout than any other buffer on the queue. This is going to lets us efficiently check packets on the queue, until we reach a packet that is not timedout, at which time we can stop.

The first fragment that we see is the one that creates the buffer for that Ip datagram. At the time of the construction of the buffer, we're going to use the ip header of that fragment as a template for the IP header we need to insert infront off all the fragments in the buffer. We also need to reset a few fields in the header to match the new packet that we are creating out of the fragments. We need to either recalculate or reset to 0 the header crc, clear the MORE_FRAGMENTS flag, drop any optional headers by resetting the hlen field to 5 and also set the total length field to the new length of our IP datagram.

The buffer will never be complete unless we receive that last fragment. That last fragment is crucial since it tells us the length of the original IP datagram. If all the fragment arrive in sequence then the last fragment also means that reassembly is complete and we can dispatch to user. Although we could receive fragments out of sequence and still receive a fragment after the last one has been received. Another important thing we need to set, is to change the size of the buffer to match that of the entire datagram. The buffer's physical size is 8K, our datagrams are probably going to be smaller than that, so there will be some unused space at the end, but the buffer will be strictly bounded to datagram data.

So in summary. We have a buffer Map and a timeout Queue. The Map keeps track of reassembly buffers for us based on a special hashcode, while the timeout queue uses the priority queue mechanism to sort our buffers and keep buffers that have timed out at the top.

The user handler

The user handler is going to receive ip reassembly buffers. These buffers may or may not be complete, but they will always have atleast an ip header and 1 fragment.

We will check if the buffer is complete and report an error message if its not. Otherwise we will just create a packet out of it.

There is no need to copy the data out of the buffer, it already contains everything we need. It is freshly allocated so its our to do as we please. It has an Ip4 header at the start and then all of the reassembled fragments already copied into it.

We are simply going to peer the a JMemoryPacket with our buffer. Then we are going to run a scan on the packet to decode it, telling the scanner that the first header is Ip4.

And that's it.

Step 2 - setup the main class

We are going to setup the main class as the packet handler as well. All it needs to do is implement the PcapPacketHandler interface. We are also going to use a static main method that will setup the libpcap portion and register our application as listener to libpcap packets.

public class IpReassemblyExample implements PcapPacketHandler {

  public static void main(String[] args) {
    StringBuilder errbuf = new StringBuilder();
    Pcap pcap = Pcap.openOffline("tests/test-ipreassembly2.pcap", errbuf);
    if (pcap == null) {
      System.err.println(errbuf.toString());
      return;
    }

    pcap.loop(
      6, // Collect 6 packets
      new IpReassemblyExample(
        5 * 1000, // 5 second timeout for reassembly buffer
        new IpReassemblyBufferHandler() {/*Omitted for now */}),
      "");
  }
}

Here we are using the static main method as a start for our application. We open up a capture file and enter a dispatch loop. We're only collecting 6 packets and we are registering out application as the PcapPacketHandler. Our constructor takes a IpReassemblyBufferHandler that will be notified with reassembled buffers. We create an anonymous class for that since we only do very minor work in its handler callback method.

Step 3 - setup the packet handler

This is the meat of our application and the main loop for us. All the packets come here. We pick out the ip4 packets and we process each one of them.

public class IpReassemblyExample implements 
PcapPacketHandler {

  private Ip4 ip = new Ip4(); // Ip4 header

  public void nextPacket(PcapPacket packet, Object user) {

    if (packet.hasHeader(ip)) {
      final int flags = ip.flags();

      /*
       * Check if we have an IP fragment
       */
      if ((flags & Ip4.FLAG_MORE_FRAGEMNTS) != 0) {
        bufferFragment(packet, ip);

        /*
         * record the last fragment
         */
      } else {
        bufferLastFragment(packet, ip);
      }

      /*
       * Our crude timeout mechanism, should be implemented as a separate thread
       */
      timeoutBuffers();
    }
  }
}

We process each Ip4 packet a little differently depending if the the Ip4.FLAG_MORE_FRAGEMNTS is set. If it is not set that means it is the last fragment, otherwise we received a packet inside fragment. If a packet is not fragmented at all, it only contains a single fragment and is always the last fragment and we treat it as a last segment.

We use to methods, bufferFragment() and bufferLastFragment() to record the fragments in the reassembly buffer. The bufferLastFragment() is a little bit special in that it records the length of the entire ip datagram we are reassembling and if all the fragments arrived in sequence it also means we're done with this buffer.

At the end of this loop, we check the time out queue to see if there is anything ready to be timed out. A better approach would be put this check in a sub thread, but for our purpose this a simple check every time a packet arrives is sufficient.

Step 4 - reassemble buffers

public static class IpReassemblyBuffer
	    extends JBuffer implements Comparable<IpReassemblyBuffer> {

Our reassembly buffer simply overrides a JBuffer. Therefore we can copy fragment data directly into it. As a matter of fact we are setting up the buffer to be a complete ip-only packet with all the fragments in it.

+------------+--------+--------+
| Ip4 header | frag 1 | frag 2 |
+------------+--------+--------+

We just have to make sure that as fragments arrive we copy their data into the buffer at the right offset. Lucky for us Ip4.offset() field gives us an offset of every fragment we have to process. We know exactly where the fragment needs to be copied to. Its always going to be ip.offset * 8 + 20. We multiply by 8 because offset is in multiples of 8 per Ip4 specification. We add 20 because that is exactly how many bytes our Ip4 header upfront takes up.

Our buffer will also keep track of how many bytes were already copied into it. When the number of bytes copied equals the length of the original ip datagram we know we are done. In order to keep things simplified a bit, we are assuming that all the fragments arriving are not duplicates and they do not overlap. We would have to implement additional logic to also keep track of each individual fragment by the buffer.

We are going to use the first fragment that triggered the creation of this reassembly buffer as a template to write our ip header up front. We do need to reset certain values to make the header applicable to the new reassembled datagram.

All the fragment data will be written into the buffer starting at offset 20, past our ip header.

private final Queue<IpReassemblyBuffer> timeoutQueue = new PriorityQueue<IpReassemblyBuffer>();

private void timeoutBuffers() {
  while (timeoutQueue.isEmpty() == false) {

    if (timeoutQueue.peek().isTimedout()) {
      dispatch(timeoutQueue.poll());
    } else {
      break;
    }
  }
}

The buffer also maintains a timeout timestamp. This is a time when the buffer will been timedout. If that timestamp is still in the future, the buffer is still active, if its in the past, the buffer is timed out and can be cleared from our buffer map and timeout queues. Notice that the buffer implements a JRE Comparable interface. This is because we are using a JRE PriorityQueue class to manage and sort our timeout queue. We prioritise the buffers on the queue according to their timeout timestamp. Oldest up on top, youngest down below. The priority queue sorts that for us automatically. We can check starting at the top of the queue, expiring all the buffers until we reach a buffer that is not expired yet. We stop and break out of that time out loop since there can be no more expired buffer blow due to priority queue sorting.

Tutorial 2 - API usage


Note: There is a bug in 1.3.b4 and 1.4.r1300 with "flows". The example below fails when flows are encountered due to this bug. The bug has been fixed and will be released with next release of 1.3 and 1.4 which are currently being tested before a broader public release.

Download Source from SVN:

package org.jnetpcap.examples.packet;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.jnetpcap.Pcap;
import org.jnetpcap.nio.JMemory;
import org.jnetpcap.packet.JFlow;
import org.jnetpcap.packet.JFlowKey;
import org.jnetpcap.packet.JFlowMap;
import org.jnetpcap.packet.JPacket;
import org.jnetpcap.packet.JPacketHandler;
import org.jnetpcap.packet.JScanner;
import org.jnetpcap.packet.PcapPacket;
import org.jnetpcap.protocol.tcpip.Http;
import org.jnetpcap.protocol.tcpip.Tcp;

/**
 * This example demonstrates various usage scenerios for jNetPcap API. The test
 * file used in this example can be found under the "tests" directory located
 * under the root installation directory of the source package. The tests
 * directory is not normally provided with binary distribution of jnetpcap. The
 * test file contains 483 packets most of which are http or tcp segments.
 * 
 * @author Mark Bednarczyk
 * @author Sly Technologies, Inc.
 */
public class CommonUsageExamples {

	/**
	 * Various examples
	 * 
	 * @param args
	 *          none expected
	 */
	public static void main(String[] args) {

		/*
		 * Example #1 open offline capture file for reading packets.
		 */
		final String FILENAME = "tests/test-http-jpeg.pcap";
		final StringBuilder errbuf = new StringBuilder();

		final Pcap pcap = Pcap.openOffline(FILENAME, errbuf);
		if (pcap == null) {
			System.err.println(errbuf); // Error is stored in errbuf if any
			return;
		}

		/*
		 * We have an opened the capture file now time to read packets. We use a
		 * Pcap.loop function to retrieve 10 packets from the file. We supply an
		 * annonymous handler which will receive packets as they are read from the
		 * offline file by libpcap. We parameterize it with a StringBuilder class.
		 * This allows us to pass in any type of object we need inside the our
		 * dispatch handler. For this example we are passing in the errorbuf object
		 * so we can pass back a string, if we need to. Of course in our example
		 * this is not strictly needed since our anonymous class can access errbuf
		 * object directly from the enclosing main method as that local variable is
		 * marked final allowing anonymous classes access to it.
		 */
		pcap.loop(10, new JPacketHandler<StringBuilder>() {

			/**
			 * We purposely define and allocate our working tcp header (accessor)
			 * outside the dispatch function and thus the libpcap loop, as this type
			 * of object is reusable and it would be a very big waist of time and
			 * resources to allocate it per every dispatch of a packet. We mark it
			 * final since we do not plan on allocating any other instances of Tcp.
			 */
			final Tcp tcp = new Tcp();

			/*
			 * Same thing for our http header
			 */
			final Http http = new Http();

			/**
			 * Our custom handler that will receive all the packets libpcap will
			 * dispatch to us. This handler is inside a libpcap loop and will receive
			 * exactly 10 packets as we specified on the Pcap.loop(10, ...) line
			 * above.
			 * 
			 * @param packet
			 *          a packet from our capture file
			 * @param errbuf
			 *          our custom user parameter which we chose to be a StringBuilder
			 *          object, but could have chosen anything else we wanted passed
			 *          into our handler by libpcap
			 */
			public void nextPacket(JPacket packet, StringBuilder errbuf) {

				/*
				 * Here we receive 1 packet at a time from the capture file. We are
				 * going to check if we have a tcp packet and do something with tcp
				 * header. We are actually going to do this twice to show 2 different
				 * ways how we can check if a particular header exists in the packet and
				 * then get that header (peer header definition instance with memory in
				 * the packet) in 2 separate steps.
				 */
				if (packet.hasHeader(Tcp.ID)) {

					/*
					 * Now get our tcp header definition (accessor) peered with actual
					 * memory that holds the tcp header within the packet.
					 */
					packet.getHeader(tcp);

					System.out.printf("tcp.dst_port=%d%n", tcp.destination());
					System.out.printf("tcp.src_port=%d%n", tcp.source());
					System.out.printf("tcp.ack=%x%n", tcp.ack());

				}

				/*
				 * An easier way of checking if header exists and peering with memory
				 * can be done using a conveniece method JPacket.hasHeader(? extends
				 * JHeader). This method performs both operations at once returning a
				 * boolean true or false. True means that header exists in the packet
				 * and our tcp header difinition object is peered or false if the header
				 * doesn't exist and no peering was performed.
				 */
				if (packet.hasHeader(tcp)) {
					System.out.printf("tcp header::%s%n", tcp.toString());
				}

				/*
				 * A typical and common approach to getting headers from a packet is to
				 * chain them as a condition for the if statement. If we need to work
				 * with both tcp and http headers, for example, we place both of them on
				 * the command line.
				 */
				if (packet.hasHeader(tcp) && packet.hasHeader(http)) {
					/*
					 * Now we are guarranteed to have both tcp and http header peered. If
					 * the packet only contained tcp segment even though tcp may have http
					 * port number, it still won't show up here since headers appear right
					 * at the beginning of http session.
					 */

					System.out.printf("http header::%s%n", http);

					/*
					 * jNetPcap keeps track of frame numbers for us. The number is simply
					 * incremented with every packet scanned.
					 */

				}

				System.out.printf("frame #%d%n", packet.getFrameNumber());
			}

		}, errbuf);

		/*
		 * Now that we have captured our 10 packets, lets use Pcap.nextEx to get the
		 * next 5 packets. We will also reset the frame number back to 0 just so we
		 * can see how its done. Each scanner keeps track of its own frame numbers,
		 * so we want to get the default one, for this thread, and change it there.
		 */
		JScanner.getThreadLocal().setFrameNumber(0);

		final PcapPacket packet = new PcapPacket(JMemory.POINTER);
		final Tcp tcp = new Tcp();

		for (int i = 0; i < 5; i++) {
			pcap.nextEx(packet);

			if (packet.hasHeader(tcp)) {
				System.out.printf("#%d seq=%08X%n", packet.getFrameNumber(), tcp.seq());
			}
		}

		/*
		 * Each packet scanned, also has a flow key associated with it. The flow key
		 * is generated based on the headers in each packet and stored with packet
		 * state. We can use the flow key to uniquely identify packets belonging to
		 * the same stream of packets between end host systems. We will keep a map
		 * of various flows with packets in it.
		 */
		final Map<JFlowKey, JFlow> flows = new HashMap<JFlowKey, JFlow>();

		for (int i = 0; i < 50; i++) {
			pcap.nextEx(packet);
			final JFlowKey key = packet.getState().getFlowKey();

			/*
			 * A hashmap uses the equals method to determine if a key is already
			 * present in the map or not and to retrieve values. jNetPcap provides us
			 * with a special object called a JFlow which keeps a list of packets part
			 * of that flow. We can add new packets to a flow and later we can get a
			 * list of those packets. So first we check if a flow for a given key
			 * already exists. All packets part of the same flow will have the same
			 * key.
			 */
			JFlow flow = flows.get(key);
			if (flow == null) {
				flows.put(key, flow = new JFlow(key));
			}

			/*
			 * Now that we know for sure we have a flow this packet belongs to, we can
			 * add this packet to this flow. Before we can actuall add a packet to a
			 * queue for later processing, we must first make a copy of the packet to
			 * a new object. We can only process each libpcap packet immediately
			 * before any other calls or nextEx or another iteration of a loop. The
			 * packets are delivered to us without copies so what we are working with
			 * is the data within libpcap buffer. If we want to preserve a packet
			 * beyond this point, we have to make a copy of the packet and its decoded
			 * state and then we can keep the packet around for as long as its needed.
			 * There is a convenience PcapPacket constructor that does a copy of
			 * everything needed for us.
			 */
			flow.add(new PcapPacket(packet));
		}

		/*
		 * Now that we added 50 packets to various flows maintained by the flows
		 * Map, we can now access those flows and the packet within it. The packets
		 * are now grouped into flows.
		 */

		for (JFlow flow : flows.values()) {

			/*
			 * Flows can be bi-directional. That is packets going between host A and B
			 * would be considered in forward-direction, while packets between host B
			 * and A can be considered reserverse direction. Although both forward and
			 * reverse are going in the opposite directions, jnetpcap flows consider
			 * them the same flows. You have 3 types of accessors for retrieving
			 * packets from a flow. JFlow.getForward, JFlow.getReverse or
			 * JFlow.getAll. JFlow.getAll gets a list of packets, no matter which
			 * direction they are going, while the other 2 accessors only get the
			 * packets that are going in the specified direction.
			 */
			if (flow.isReversable()) {
				/*
				 * We can get directional flow packets, but only if the flow is
				 * reversable. Not all flows are reversable and this is determined by
				 * the header types. If a flow is not reversable, flow.getReverse will
				 * return empty list, which is something we don't want to have to
				 * process.
				 */

				List<JPacket> forward = flow.getForward();
				for (JPacket p : forward) {
					System.out.printf("%d, ", p.getFrameNumber());
				}
				System.out.println();

				List<JPacket> reverse = flow.getReverse();
				for (JPacket p : reverse) {
					System.out.printf("%d, ", p.getFrameNumber());
				}
			} else {

				/*
				 * Otherwise we have to get All the packets and there is no
				 * forward/reverse direction associated with the packets. Here is how we
				 * can do this a little more compactly.
				 */
				for (JPacket p : flow.getAll()) {
					System.out.printf("%d, ", p.getFrameNumber());
				}
			}
			System.out.println();
		}

		/*
		 * We still haven't read all the packets from our offline file. Here is an
		 * easier way to retrieve all the packets while grouping them into flows.
		 * jNetPcap provides a neat little class that does all of the above work for
		 * us. Its called JFlowMap, not only that it implements a JPacketHandler
		 * interface suitable for usage with Pcap.loop or Pcap.dispatch calls and it
		 * will add all packets received into appropriate flows.
		 */
		JFlowMap superFlowMap = new JFlowMap();

		/*
		 * So lets finish this file off, and read the remaining packets into our new
		 * superFlowMap and do a pretty print of all the flows it finds. The 3rd
		 * argument to Pcap.loop is unused so we just set it to null.
		 * Pcap.LOOP_INFINITE flag tells the Pcap.loop method to read all the
		 * packets until the end of file. Since we already read some packets, this
		 * will read remaining packets from the current position in the file until
		 * the end.
		 */
		pcap.loop(Pcap.LOOP_INFINITE, superFlowMap, null);

		System.out.printf("superFlowMap::%s%n", superFlowMap);

		/*
		 * Now we have read the remaining packets and we no longer need to keep the
		 * pcap file open.
		 */
		pcap.close();

	}
}

Lastly here is the output from the last printf statement that prints the contents of the JFlowmap:

superFlowMap::total packet count=418
total flow count=18
flow[0] 10.1.1.101:3198 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[9/10/19],
flow[1] 10.1.1.101:3199 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[9/11/20],
flow[2] 209.225.11.237:80 -> 10.1.1.101:3179 Tcp fw/rev/tot pkts=[1/0/1],
flow[3] 10.1.1.101:3196 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[6/6/12],
flow[4] 10.1.1.101:3197 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[6/6/12],
flow[5] 10.1.1.101:3195 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[5/5/10],
flow[6] 209.225.0.6:80 -> 10.1.1.101:3183 Tcp fw/rev/tot pkts=[6/6/12],
flow[7] 10.1.1.101:3190 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[5/5/10],
flow[8] 209.225.0.6 -> 10.1.1.101:6 Ip4 tot pkts=[18],
flow[9] 10.1.1.101:3189 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[1/1/2],
flow[10] 10.1.1.101:3200 -> 10.1.1.1:80 Tcp fw/rev/tot pkts=[74/135/209],
flow[11] 10.1.1.101:3193 -> 209.225.0.6:80 Tcp fw/rev/tot pkts=[9/6/15],
flow[12] 10.1.1.101:3192 -> 209.225.0.6:80 Tcp fw/rev/tot pkts=[8/6/14],
flow[13] 10.1.1.101:3194 -> 209.225.0.6:80 Tcp fw/rev/tot pkts=[8/6/14],
flow[14] 209.225.0.6:80 -> 10.1.1.101:3185 Tcp fw/rev/tot pkts=[6/6/12],
flow[15] 209.225.0.6:80 -> 10.1.1.101:3184 Tcp fw/rev/tot pkts=[6/6/12],
flow[16] 209.225.0.6:80 -> 10.1.1.101:3187 Tcp fw/rev/tot pkts=[6/6/12],
flow[17] 10.1.1.101:3191 -> 209.225.0.6:80 Tcp fw/rev/tot pkts=[8/6/14],

Its much easier to just read all the packets into a JFlowMap if no special processing is needed.

Other Tutorials

External libpcap tutorials

Although these are native C library tutorials, you will find that the procedures and sequences of calls are nearly identical. If you know the C and java languages, you shouldn't have much trouble translating these tutorials to java and jNetPcap API.

For example:

		char *dev, errbuf[PCAP_ERRBUF_SIZE];

		dev = pcap_lookupdev(errbuf);
		if (dev == NULL) {
			fprintf(stderr, "Couldn't find default device: %s\n", errbuf);
			return(2);
		}

translates to jNetPcap API as follows:

StringBuilder errbuf = new StringBuilder();
String dev = Pcap.lookupDev(errbuf);
if (dev == null) {
  System.err.printf("Couldn't find default device: %s\n", errbuf);
  return 2;
}

Here is a list of tutorials that you may find useful: