Packfile transfer protocols
===========================

Git supports transferring data in packfiles over the ssh://, git:// and
file:// transports.  There exist two sets of protocols, one for pushing
data from a client to a server and another for fetching data from a
server to a client.  All three transports (ssh, git, file) use the same
protocol to transfer data.

The processes invoked in the canonical Git implementation are 'upload-pack'
on the server side and 'fetch-pack' on the client side for fetching data;
then 'receive-pack' on the server and 'send-pack' on the client for pushing
data.  The protocol functions to have a server tell a client what is
currently on the server, then for the two to negotiate the smallest amount
of data to send in order to fully update one or the other.

Transports
----------
There are three transports over which the packfile protocol is
initiated.  The Git transport is a simple, unauthenticated server that
takes the command (almost always 'upload-pack', though Git
servers can be configured to be globally writable, in which 'receive-
pack' initiation is also allowed) with which the client wishes to
communicate and executes it and connects it to the requesting
process.

In the SSH transport, the client just runs the 'upload-pack'
or 'receive-pack' process on the server over the SSH protocol and then
communicates with that invoked process over the SSH connection.

The file:// transport runs the 'upload-pack' or 'receive-pack'
process locally and communicates with it over a pipe.

Git Transport
-------------

The Git transport starts off by sending the command and repository
on the wire using the pkt-line format, followed by a NUL byte and a
hostname parameter, terminated by a NUL byte.

   0032git-upload-pack /project.git\0host=myserver.com\0

--
   git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]
   request-command   = "git-upload-pack" / "git-receive-pack" /
		       "git-upload-archive"   ; case sensitive
   pathname          = *( %x01-ff ) ; exclude NUL
   host-parameter    = "host=" hostname [ ":" port ]
--

Only host-parameter is allowed in the git-proto-request. Clients
MUST NOT attempt to send additional parameters. It is used for the
git-daemon name based virtual hosting.  See --interpolated-path
option to git daemon, with the %H/%CH format characters.

Basically what the Git client is doing to connect to an 'upload-pack'
process on the server side over the Git protocol is this:

   $ echo -e -n \
     "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" |
     nc -v example.com 9418

If the server refuses the request for some reasons, it could abort
gracefully with an error message.

----
  error-line     =  PKT-LINE("ERR" SP explanation-text)
----


SSH Transport
-------------

Initiating the upload-pack or receive-pack processes over SSH is
executing the binary on the server via SSH remote execution.
It is basically equivalent to running this:

   $ ssh git.example.com "git-upload-pack '/project.git'"

For a server to support Git pushing and pulling for a given user over
SSH, that user needs to be able to execute one or both of those
commands via the SSH shell that they are provided on login.  On some
systems, that shell access is limited to only being able to run those
two commands, or even just one of them.

In an ssh:// format URI, it's absolute in the URI, so the '/' after
the host name (or port number) is sent as an argument, which is then
read by the remote git-upload-pack exactly as is, so it's effectively
an absolute path in the remote filesystem.

       git clone ssh://user@example.com/project.git
		    |
		    v
    ssh user@example.com "git-upload-pack '/project.git'"

In a "user@host:path" format URI, its relative to the user's home
directory, because the Git client will run:

     git clone user@example.com:project.git
		    |
		    v
  ssh user@example.com "git-upload-pack 'project.git'"

The exception is if a '~' is used, in which case
we execute it without the leading '/'.

      ssh://user@example.com/~alice/project.git,
		     |
		     v
   ssh user@example.com "git-upload-pack '~alice/project.git'"

A few things to remember here:

- The "command name" is spelled with dash (e.g. git-upload-pack), but
  this can be overridden by the client;

- The repository path is always quoted with single quotes.

Fetching Data From a Server
---------------------------

When one Git repository wants to get data that a second repository
has, the first can 'fetch' from the second.  This operation determines
what data the server has that the client does not then streams that
data down to the client in packfile format.


Reference Discovery
-------------------

When the client initially connects the server will immediately respond
with a listing of each reference it has (all branches and tags) along
with the object name that each reference currently points to.

   $ echo -e -n "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" |
      nc -v example.com 9418
   00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\0multi_ack thin-pack
		side-band side-band-64k ofs-delta shallow no-progress include-tag
   00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration
   003f7217a7c7e582c46cec22a130adf4b9d7d950fba0 refs/heads/master
   003cb88d2441cac0977faf98efc80305012112238d9d refs/tags/v0.9
   003c525128480b96c89e6418b1e40909bf6c5b2d580f refs/tags/v1.0
   003fe92df48743b7bc7d26bcaabfddde0a1e20cae47c refs/tags/v1.0^{}
   0000

Server SHOULD terminate each non-flush line using LF ("\n") terminator;
client MUST NOT complain if there is no terminator.

The returned response is a pkt-line stream describing each ref and
its current value.  The stream MUST be sorted by name according to
the C locale ordering.

If HEAD is a valid ref, HEAD MUST appear as the first advertised
ref.  If HEAD is not a valid ref, HEAD MUST NOT appear in the
advertisement list at all, but other refs may still appear.

The stream MUST include capability declarations behind a NUL on the
first ref. The peeled value of a ref (that is "ref^{}") MUST be
immediately after the ref itself, if presented. A conforming server
MUST peel the ref if it's an annotated tag.

----
  advertised-refs  =  (no-refs / list-of-refs)
		      *shallow
		      flush-pkt

  no-refs          =  PKT-LINE(zero-id SP "capabilities^{}"
		      NUL capability-list LF)

  list-of-refs     =  first-ref *other-ref
  first-ref        =  PKT-LINE(obj-id SP refname
		      NUL capability-list LF)

  other-ref        =  PKT-LINE(other-tip / other-peeled)
  other-tip        =  obj-id SP refname LF
  other-peeled     =  obj-id SP refname "^{}" LF

  shallow          =  PKT-LINE("shallow" SP obj-id)

  capability-list  =  capability *(SP capability)
  capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
  LC_ALPHA         =  %x61-7A
----

Server and client MUST use lowercase for obj-id, both MUST treat obj-id
as case-insensitive.

See protocol-capabilities.txt for a list of allowed server capabilities
and descriptions.

Packfile Negotiation
--------------------
After reference and capabilities discovery, the client can decide to
terminate the connection by sending a flush-pkt, telling the server it can
now gracefully terminate, and disconnect, when it does not need any pack
data. This can happen with the ls-remote command, and also can happen when
the client already is up-to-date.

Otherwise, it enters the negotiation phase, where the client and
server determine what the minimal packfile necessary for transport is,
by telling the server what objects it wants, its shallow objects
(if any), and the maximum commit depth it wants (if any).  The client
will also send a list of the capabilities it wants to be in effect,
out of what the server said it could do with the first 'want' line.

----
  upload-request    =  want-list
		       *shallow-line
		       *1depth-request
		       flush-pkt

  want-list         =  first-want
		       *additional-want

  shallow-line      =  PKT_LINE("shallow" SP obj-id)

  depth-request     =  PKT_LINE("deepen" SP depth)

  first-want        =  PKT-LINE("want" SP obj-id SP capability-list LF)
  additional-want   =  PKT-LINE("want" SP obj-id LF)

  depth             =  1*DIGIT
----

Clients MUST send all the obj-ids it wants from the reference
discovery phase as 'want' lines. Clients MUST send at least one
'want' command in the request body. Clients MUST NOT mention an
obj-id in a 'want' command which did not appear in the response
obtained through ref discovery.

The client MUST write all obj-ids which it only has shallow copies
of (meaning that it does not have the parents of a commit) as
'shallow' lines so that the server is aware of the limitations of
the client's history.

The client now sends the maximum commit history depth it wants for
this transaction, which is the number of commits it wants from the
tip of the history, if any, as a 'deepen' line.  A depth of 0 is the
same as not making a depth request. The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sent strbu The client does not want to receive
any commits beyond this depth, nor objects needed only to complete
those commits. Commits whose parents are not received as a result are
defined as shallow and marked as such in the server. This information
is sen