network.httppush.PackerExample Class Reference

Use a packer to send events in batch and reduce the number of HTTP requests. More...

Static Public Member Functions
static void	main (String[] args) throws ProcessorException, InterruptedException

Detailed Description

Use a packer to send events in batch and reduce the number of HTTP requests.

In this example, we send a relatively fast flow of events (about 1,000 per second) from one host to another using HTTP gateways (see PushLocalSerialize for an explanation of gateways). If we used the HttpUpstreamGateway directly, this would result in one HTTP request-response cycle for each event to be pushed over the network. However, each request has an associated overhead, which places an upper bound on the number of requests per second that can be sent.

The TimePack processor can be used to reduce the number of such requests. The packer accumulates events for a predetermined amount of time (say, one second), and outputs all accumulated events as a single List object when the time interval is expired. When coupled with a JSON serializer (to transform that list into a JSON string) and an HttpUpstreamGateway, this results in fewer HTTP requests, a smaller overhead, and hence an increased throughput.

Setting the packer to a relatively long interval (e.g. 1 second) will be such that a single HTTP request will contain a batch of about 1,000 events at a time. The output of the program should look like this:

Pushing 1000 events per second. Hit Ctrl+C to end.
1,717,1574,2432,3280,...

In contrast, setting the packer to a short time interval (e.g. 2 milliseconds) will result in the packer making much smaller bundles, which in turn will make the HTTP gateway send more HTTP requests. The output of the program should look like this:

Pushing 1000 events per second. Hit Ctrl+C to end.
1,393,830,1375,1966,...

Since every number corresponds to the cumulative number of events received at every passing second, one can see that sending too many HTTP requests results in a slower throughput than packing events and sending them in bulk periodically.

Author

Sylvain Hallé

Definition at line 86 of file PackerExample.java.

---

The documentation for this class was generated from the following file:

• Source/src/network/httppush/PackerExample.java