..
	Copyright (c) 2011-2015 Varnish Software AS
	SPDX-License-Identifier: BSD-2-Clause
	See LICENSE file for full text of license

.. _phk_gzip:

=======================================
How GZIP, and GZIP+ESI works in Varnish
=======================================

First of all, everything you read about GZIP here, is controlled by the
parameter:

	http_gzip_support

Which defaults to "on" if you do not want Varnish to try to be smart
about compression, set it to "off" instead.

What does http_gzip_support do
------------------------------

A request which is sent into 'pipe' or 'pass' mode from vcl_recv{}
will not experience any difference, this processing only affects
cache hit/miss requests.

Unless vcl_recv{} results in "pipe" or "pass", we determine if the
client is capable of receiving gzipped content.  The test amounts to:

	Is there a Accept-Encoding header that mentions gzip, and if
	is has a q=# number, is it larger than zero.

Clients which can do gzip, gets their header rewritten to:

	Accept-Encoding: gzip

And clients which do not support gzip gets their Accept-Encoding
header removed.  This ensures conformity with respect to creating
Vary: strings during object creation.

During lookup, we ignore any "Accept-encoding" in objects Vary: strings,
to avoid having a gzip and gunzip'ed version of the object, varnish
can gunzip on demand.  (We implement this bit of magic at lookup time,
so that any objects stored in persistent storage can be used with
or without gzip support enabled.)

Varnish will not do any other types of compressions than gzip, in particular
we will not do deflate, as there are browser bugs in that case.

Before vcl_miss{} is called, the backend requests Accept-Encoding is
always set to:

	Accept-Encoding: gzip

Even if this particular client does not support

To always entice the backend into sending us gzipped content.

Varnish will not gzip any content on its own (but see below), we trust
the backend to know what content can be sensibly gzipped (html) and what
cannot (jpeg)

If in vcl_backend_response{} we find out that we are trying to deliver a
gzipped object to a client that has not indicated willingness to receive
gzip, we will ungzip the object during deliver.

Tuning, tweaking and frobbing
-----------------------------

In vcl_recv{} you have a chance to modify the client's
Accept-Encoding: header before anything else happens.

In vcl_pass{} the client's Accept-Encoding header is copied to the
backend request unchanged.
Even if the client does not support gzip, you can force the A-E header
to "gzip" on bereq.http in vcl_backend_fetch{} to save bandwidth between
the backend and varnish. Varnish will gunzip the object before delivering
to the client.

In vcl_miss{} you can remove the "Accept-Encoding: gzip" header, if you
do not want the backend to gzip this object.

In vcl_backend_response{} two new variables allow you to modify the
gzip-ness of objects during fetch:

	set beresp.do_gunzip = true;

Will make varnish gunzip an already gzipped object from the backend during
fetch.  (I have no idea why/when you would use this...)

	set beresp.do_gzip = true;

Will make varnish gzip the object during fetch from the backend, provided
the backend didn't send us a gzipped object.

Remember that a lot of content types cannot sensibly be gzipped, most
notably compressed image formats like jpeg, png and similar, so a
typical use would be::

	sub vcl_backend_response {
		if (bereq.url ~ "html$") {
			set beresp.do_gzip = true;
		}
	}

GZIP and ESI
------------

First, note the new syntax for activating ESI::

	sub vcl_backend_response {
		set beresp.do_esi = true;
	}

In theory, and hopefully in practice, all you read above should apply also
when you enable ESI, if not it is a bug you should report.

But things are vastly more complicated now.  What happens for
instance, when the backend sends a gzipped object we ESI process
it and it includes another object which is not gzipped, and we want
to send the result gzipped to the client ?

Things can get really hairy here, so let me explain it in stages.

Assume we have a ungzipped object we want to ESI process.

The ESI parser will run through the object looking for the various
magic strings and produce a byte-stream we call the "VEC" for Varnish
ESI Codes.

The VEC contains instructions like "skip 234 bytes", "deliver 12919 bytes",
"include /foobar", "deliver 122 bytes" etc and it is stored with the
object.

When we deliver an object, and it has a VEC, special esi-delivery code
interprets the VEC string and sends the output to the client as ordered.

When the VEC says "include /foobar" we do what amounts to a restart with
the new URL and possibly Host: header, and call vcl_recv{} etc.  You
can tell that you are in an ESI include by examining the 'req.esi_level'
variable in VCL.

The ESI-parsed object is stored gzipped under the same conditions as
above:  If the backend sends gzipped and VCL did not ask for do_gunzip,
or if the backend sends ungzipped and VCL asked for do_gzip.

Please note that since we need to insert flush and reset points in
the gzip file, it will be slightly larger than a normal gzip file of
the same object.

When we encounter gzipped include objects which should not be, we
gunzip them, but when we encounter gunzip'ed objects which should
be, we gzip them, but only at compression level zero.

So in order to avoid unnecessary work, and in order to get maximum
compression efficiency, you should::

	sub vcl_miss {
		if (object needs ESI processing) {
			unset req.http.accept-encoding;
		}
	}

	sub vcl_backend_response {
		if (object needs ESI processing) {
			set beresp.do_esi = true;
			set beresp.do_gzip = true;
		}
	}

So that the backend sends these objects uncompressed to varnish.

You should also attempt to make sure that all objects which are
esi:included are gzipped, either by making the backend do it or
by making varnish do it.
