Commit Graph

33 Commits

Author SHA1 Message Date
Ævar Arnfjörð Bjarmason 98e2d9d6f7 upload-pack: document and rename --advertise-refs
The --advertise-refs documentation in git-upload-pack added in
9812f2136b (upload-pack.c: use parse-options API, 2016-05-31) hasn't
been entirely true ever since v2 support was implemented in
e52449b672 (connect: request remote refs using v2, 2018-03-15). Under
v2 we don't advertise the refs at all, but rather dump the
capabilities header.

This option has always been an obscure internal implementation detail,
it wasn't even documented for git-receive-pack. Since it has exactly
one user let's rename it to --http-backend-info-refs, which is more
accurate and points the reader in the right direction. Let's also
cross-link this from the protocol v1 and v2 documentation.

I'm retaining a hidden --advertise-refs alias in case there's any
external users of this, and making both options hidden to the bash
completion (as with most other internal-only options).

Signed-off-by: Ævar Arnfjörð Bjarmason <>
Signed-off-by: Junio C Hamano <>
2021-08-05 08:59:37 -07:00
Andrei Rybak abcb66c614 *: fix typos which duplicate a word
Fix typos in documentation, code comments, and RelNotes which repeat
various words.  In trivial cases, just delete the duplicated word and
rewrap text, if needed.  Reword the affected sentence in
Documentation/RelNotes/1.8.4.txt for it to make sense.

Signed-off-by: Andrei Rybak <>
Signed-off-by: Junio C Hamano <>
2021-06-14 10:16:06 +09:00
Junio C Hamano 644f4a2046 Merge branch 'jt/push-negotiation'
"git push" learns to discover common ancestor with the receiving
end over protocol v2.

* jt/push-negotiation:
  send-pack: support push negotiation
  fetch: teach independent negotiation (no packfile)
  fetch-pack: refactor command and capability write
  fetch-pack: refactor add_haves()
  fetch-pack: refactor process_acks()
2021-05-16 21:05:22 +09:00
Jonathan Tan 9c1e657a8f fetch: teach independent negotiation (no packfile)
Currently, the packfile negotiation step within a Git fetch cannot be
done independent of sending the packfile, even though there is at least
one application wherein this is useful. Therefore, make it possible for
this negotiation step to be done independently. A subsequent commit will
use this for one such application - push negotiation.

This feature is for protocol v2 only. (An implementation for protocol v0
would require a separate implementation in the fetch, transport, and
transport helper code.)

In the protocol, the main hindrance towards independent negotiation is
that the server can unilaterally decide to send the packfile. This is
solved by a "wait-for-done" argument: the server will then wait for the
client to say "done". In practice, the client will never say it; instead
it will cease requests once it is satisfied.

In the client, the main change lies in the transport and transport
helper code. fetch_refs_via_pack() performs everything needed - protocol
version and capability checks, and the negotiation itself.

There are 2 code paths that do not go through fetch_refs_via_pack() that
needed to be individually excluded: the bundle transport (excluded
through requiring smart_options, which the bundle transport doesn't
support) and transport helpers that do not support takeover. If or when
we support independent negotiation for protocol v0, we will need to
modify these 2 code paths to support it. But for now, report failure if
independent negotiation is requested in these cases.

Signed-off-by: Jonathan Tan <>
Signed-off-by: Junio C Hamano <>
2021-05-05 10:41:29 +09:00
Bruno Albuquerque a2ba162cda object-info: support for retrieving object info
Sometimes it is useful to get information of an object without having to
download it completely.

Add the "object-info" capability that lets the client ask for
object-related information with their full hexadecimal object names.

Only sizes are returned for now.

Signed-off-by: Bruno Albuquerque <>
Signed-off-by: Junio C Hamano <>
2021-04-20 17:41:13 -07:00
Junio C Hamano f47c3328ef Merge branch 'js/doc-proto-v2-response-end'

* js/doc-proto-v2-response-end:
  doc: fix naming of response-end-pkt
2021-02-25 16:43:30 -08:00
Joey Salazar 9d336655ba doc: fix naming of response-end-pkt
Git Protocol version 2[1] defines 0002 as a Message Packet that indicates
the end of a response for stateless connections.

Change the naming of the 0002 Packet to 'Response End' to match the
parsing introduced in Wireshark's MR !1922 for consistency. A subsequent
MR in Wireshark will address additional mismatches.


Signed-off-by: Joey Salazar <>
Reviewed-by: Denton Liu <>
Signed-off-by: Junio C Hamano <>
2021-02-17 16:30:43 -08:00
Jonathan Tan 59e1205d16 ls-refs: report unborn targets of symrefs
When cloning, we choose the default branch based on the remote HEAD.
But if there is no remote HEAD reported (which could happen if the
target of the remote HEAD is unborn), we'll fall back to using our local
init.defaultBranch. Traditionally this hasn't been a big deal, because
most repos used "master" as the default. But these days it is likely to
cause confusion if the server and client implementations choose
different values (e.g., if the remote started with "main", we may choose
"master" locally, create commits there, and then the user is surprised
when they push to "master" and not "main").

To solve this, the remote needs to communicate the target of the HEAD
symref, even if it is unborn, and "git clone" needs to use this

Currently, symrefs that have unborn targets (such as in this case) are
not communicated by the protocol. Teach Git to advertise and support the
"unborn" feature in "ls-refs" (by default, this is advertised, but
server administrators may turn this off through the lsrefs.unborn
config). This feature indicates that "ls-refs" supports the "unborn"
argument; when it is specified, "ls-refs" will send the HEAD symref with
the name of its unborn target.

This change is only for protocol v2. A similar change for protocol v0
would require independent protocol design (there being no analogous
position to signal support for "unborn") and client-side plumbing of the
data required, so the scope of this patch set is limited to protocol v2.

The client side will be updated to use this in a subsequent commit.

Signed-off-by: Jonathan Tan <>
Signed-off-by: Junio C Hamano <>
2021-02-05 13:49:53 -08:00
Josh Steadmon f5cdbe485f docs: new capability to advertise session IDs
In future patches, we will add the ability for Git servers and clients
to advertise unique session IDs via protocol capabilities. This
allows for easier debugging when both client and server logs are

Signed-off-by: Josh Steadmon <>
Signed-off-by: Junio C Hamano <>
2020-11-11 18:26:52 -08:00
Junio C Hamano 12210859da Merge branch 'bc/sha-256-part-2'
SHA-256 migration work continues.

* bc/sha-256-part-2: (44 commits)
  remote-testgit: adapt for object-format
  bundle: detect hash algorithm when reading refs
  t5300: pass --object-format to git index-pack
  t5704: send object-format capability with SHA-256
  t5703: use object-format serve option
  t5702: offer an object-format capability in the test
  t/helper: initialize the repository for test-sha1-array
  remote-curl: avoid truncating refs with ls-remote
  t1050: pass algorithm to index-pack when outside repo
  builtin/index-pack: add option to specify hash algorithm
  remote-curl: detect algorithm for dumb HTTP by size
  builtin/ls-remote: initialize repository based on fetch
  t5500: make hash independent
  serve: advertise object-format capability for protocol v2
  connect: parse v2 refs with correct hash algorithm
  connect: pass full packet reader when parsing v2 refs
  Documentation/technical: document object-format for protocol v2
  t1302: expect repo format version 1 for SHA-256
  builtin/show-index: provide options to determine hash algo
  t5302: modernize test formatting
2020-07-06 22:09:13 -07:00
Junio C Hamano 34e849b05a Merge branch 'jt/cdn-offload'
The "fetch/clone" protocol has been updated to allow the server to
instruct the clients to grab pre-packaged packfile(s) in addition
to the packed object data coming over the wire.

* jt/cdn-offload:
  upload-pack: fix a sparse '0 as NULL pointer' warning
  upload-pack: send part of packfile response as uri
  fetch-pack: support more than one pack lockfile
  upload-pack: refactor reading of pack-objects out
  Documentation: add Packfile URIs design doc
  Documentation: order protocol v2 sections
  http-fetch: support fetching packfiles by URL
  http-fetch: refactor into function
  http: refactor finish_http_pack_request()
  http: use --stdin when indexing dumb HTTP pack
2020-06-25 12:27:47 -07:00
Jonathan Tan cd8402e0fd Documentation: add Packfile URIs design doc
Signed-off-by: Jonathan Tan <>
Signed-off-by: Junio C Hamano <>
2020-06-10 18:06:34 -07:00
Jonathan Tan fd194dd56a Documentation: order protocol v2 sections
The current C Git implementation expects Git servers to follow a
specific order of sections when transmitting protocol v2 responses, but
this is not explicit in the documentation. Make the order explicit.

Signed-off-by: Jonathan Tan <>
Signed-off-by: Junio C Hamano <>
2020-06-10 18:06:34 -07:00
brian m. carlson 7f46e7ead1 Documentation/technical: document object-format for protocol v2
Document the object-format extension for protocol v2.

Signed-off-by: brian m. carlson <>
Signed-off-by: Junio C Hamano <>
2020-05-27 10:07:07 -07:00
Denton Liu b0df0c16ea stateless-connect: send response end packet
Currently, remote-curl acts as a proxy and blindly forwards packets
between an HTTP server and fetch-pack. In the case of a stateless RPC
connection where the connection is terminated before the transaction is
complete, remote-curl will blindly forward the packets before waiting on
more input from fetch-pack. Meanwhile, fetch-pack will read the
transaction and continue reading, expecting more input to continue the
transaction. This results in a deadlock between the two processes.

This can be seen in the following command which does not terminate:

	$ git -c protocol.version=2 clone --shallow-since=20151012
	Cloning into 'git'...

whereas the v1 version does terminate as expected:

	$ git -c protocol.version=1 clone --shallow-since=20151012
	Cloning into 'git'...
	fatal: the remote end hung up unexpectedly

Instead of blindly forwarding packets, make remote-curl insert a
response end packet after proxying the responses from the remote server
when using stateless_connect(). On the RPC client side, ensure that each
response ends as described.

A separate control packet is chosen because we need to be able to
differentiate between what the remote server sends and remote-curl's
control packets. By ensuring in the remote-curl code that a server
cannot send response end packets, we prevent a malicious server from
being able to perform a denial of service attack in which they spoof a
response end packet and cause the described deadlock to happen.

Reported-by: Force Charlie <>
Helped-by: Jeff King <>
Signed-off-by: Denton Liu <>
Signed-off-by: Junio C Hamano <>
2020-05-24 16:26:00 -07:00
Elijah Newren 031fd4b93b Documentation: fix a bunch of typos, both old and new
Reported-by: Jens Schleusener <>
Signed-off-by: Elijah Newren <>
Signed-off-by: Junio C Hamano <>
2019-11-07 13:42:00 +09:00
Mark Rushakoff 24966cd982 doc: fix repeated words
Inspired by 21416f0a07 ("restore: fix typo in docs", 2019-08-03), I ran
"git grep -E '(\b[a-zA-Z]+) \1\b' -- Documentation/" to find other cases
where words were duplicated, e.g. "the the", and in most cases removed
one of the repeated words.

There were many false positives by this grep command, including
deliberate repeated words like "really really" or valid uses of "that
that" which I left alone, of course.

I also did not correct any of the legitimate, accidentally repeated
words in old RelNotes.

Signed-off-by: Mark Rushakoff <>
Acked-by: Jeff King <>
Signed-off-by: Junio C Hamano <>
2019-08-11 17:40:07 -07:00
SZEDER Gábor b373e4d29b Documentation/technical/protocol-v2.txt: fix formatting
Asciidoctor versions v1.5.7 or later print the following warning while
building the documentation:

      ASCIIDOC technical/protocol-v2.html
  asciidoctor: WARNING: protocol-v2.txt: line 38: unterminated listing block

This highlights an issue (even with older Asciidoctor versions) where
the 'Initial Client Request' header is not rendered as a header but in
monospace.  I'm not sure what exactly causes this issue and why it's
an issue only with this particular header, but all headers in
'protocol-v2.txt' are written like this:

   Initial Client Request

i.e. the header itself is indented by a space, and the "underline" is
two characters longer than the header.

Dropping that indentation and making the length of the underline match
the length of the header apparently fixes this issue.

While at it, adjust all other headers 'protocol-v2.txt' as well, to
match the style we use everywhere else.

The page rendered with AsciiDoc doesn't have this formatting issue.

Signed-off-by: SZEDER Gábor <>
Signed-off-by: Junio C Hamano <>
2019-04-01 15:17:47 +09:00
Junio C Hamano 5f8b86db94 Merge branch 'jt/fetch-v2-sideband'
"git fetch" and "git upload-pack" learned to send all exchange over
the sideband channel while talking the v2 protocol.

* jt/fetch-v2-sideband:
  tests: define GIT_TEST_SIDEBAND_ALL
  {fetch,upload}-pack: sideband v2 fetch response
  sideband: reverse its dependency on pkt-line
  pkt-line: introduce struct packet_writer
  pack-protocol.txt: accept error packets in any context
  Use packet_reader instead of packet_read_line
2019-02-05 14:26:11 -08:00
Jonathan Tan 0bbc0bc574 {fetch,upload}-pack: sideband v2 fetch response
Currently, a response to a fetch request has sideband support only while
the packfile is being sent, meaning that the server cannot send notices
until the start of the packfile.

Extend sideband support in protocol v2 fetch responses to the whole
response. upload-pack will advertise it if the
uploadpack.allowsidebandall configuration variable is set, and
fetch-pack will automatically request it if advertised.

If the sideband is to be used throughout the whole response, upload-pack
will use it to send errors instead of prefixing a PKT-LINE payload with
"ERR ".

This will be tested in a subsequent patch.

Signed-off-by: Jonathan Tan <>
Signed-off-by: Junio C Hamano <>
2019-01-17 11:25:07 -08:00
Josh Steadmon 87c2d9d310 filter-options: expand scaled numbers
When communicating with a remote server or a subprocess, use
expanded numbers rather than numbers with scaling suffix in the
object filter spec (e.g.  "limit:blob=1k" becomes

Update the protocol docs to note that clients should always perform this
expansion, to allow for more compatibility between server

Signed-off-by: Josh Steadmon <>
Signed-off-by: Junio C Hamano <>
2019-01-15 15:42:31 -08:00
Junio C Hamano 88df0fa659 Merge branch 'jt/connectivity-check-after-unshallow'
"git fetch" failed to correctly validate the set of objects it
received when making a shallow history deeper, which has been

* jt/connectivity-check-after-unshallow:
  fetch-pack: write shallow, then check connectivity
  fetch-pack: implement ref-in-want
  fetch-pack: put shallow info in output parameter
  fetch: refactor to make function args narrower
  fetch: refactor fetch_refs into two functions
  fetch: refactor the population of peer ref OIDs
  upload-pack: test negotiation with changing repository
  upload-pack: implement ref-in-want
  test-pkt-line: add unpack-sideband subcommand
2018-07-24 14:50:44 -07:00
Junio C Hamano 5e6140e76f Merge branch 'bw/protocol-v2'
Doc fix.

* bw/protocol-v2:
  protocol-v2 doc: put HTTP headers after request
2018-07-18 12:20:30 -07:00
Brandon Williams 516e2b76bd upload-pack: implement ref-in-want
Currently, while performing packfile negotiation, clients are only
allowed to specify their desired objects using object ids.  This causes
a vulnerability to failure when an object turns non-existent during
negotiation, which may happen if, for example, the desired repository is
provided by multiple Git servers in a load-balancing arrangement and
there exists replication delay.

In order to eliminate this vulnerability, implement the ref-in-want
feature for the 'fetch' command in protocol version 2.  This feature
enables the 'fetch' command to support requests in the form of ref names
through a new "want-ref <ref>" parameter.  At the conclusion of
negotiation, the server will send a list of all of the wanted references
(as provided by "want-ref" lines) in addition to the generated packfile.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-06-28 09:33:29 -07:00
Josh Steadmon bb4d000e87 protocol-v2 doc: put HTTP headers after request
HTTP servers return 400 if you send headers before the GET request.

Signed-off-by: Josh Steadmon <>
Reviewed-by: Jonathan Nieder <>
Signed-off-by: Junio C Hamano <>
2018-06-22 13:03:42 -07:00
Junio C Hamano 54db5c0e1e Merge branch 'jt/partial-clone-proto-v2'
Transfer protocol v2 learned to support the partial clone.

* jt/partial-clone-proto-v2:
  {fetch,upload}-pack: support filter in protocol v2
  upload-pack: read config when serving protocol v2
  upload-pack: fix error message typo
2018-05-30 14:04:10 +09:00
Jonathan Tan ba95710a3b {fetch,upload}-pack: support filter in protocol v2
The fetch-pack/upload-pack protocol v2 was developed independently of
the filter parameter (used in partial fetches), thus it did not include
support for it. Add support for the filter parameter.

Like in the legacy protocol, the server advertises and supports "filter"
only if uploadpack.allowfilter is configured.

Like in the legacy protocol, the client continues with a warning if
"--filter" is specified, but the server does not advertise it.

Signed-off-by: Jonathan Tan <>
Signed-off-by: Junio C Hamano <>
2018-05-06 13:17:19 +09:00
Brandon Williams ecc3e5342d serve: introduce the server-option capability
Introduce the "server-option" capability to protocol version 2.  This
enables future clients the ability to send server specific options in
command requests when using protocol version 2.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-04-24 11:24:40 +09:00
Brandon Williams f7e2050105 fetch-pack: support shallow requests
Enable shallow clones and deepen requests using protocol version 2 if
the server 'fetch' command supports the 'shallow' feature.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-03-15 12:01:09 -07:00
Brandon Williams 685fbd3291 fetch-pack: perform a fetch using v2
When communicating with a v2 server, perform a fetch by requesting the
'fetch' command.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-03-15 12:01:08 -07:00
Brandon Williams 3145ea957d upload-pack: introduce fetch server command
Introduce the 'fetch' server command.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-03-15 12:01:08 -07:00
Brandon Williams 72d0ea0056 ls-refs: introduce ls-refs server command
Introduce the ls-refs server command.  In protocol v2, the ls-refs
command is used to request the ref advertisement from the server.  Since
it is a command which can be requested (as opposed to mandatory in v1),
a client can sent a number of parameters in its request to limit the ref
advertisement based on provided ref-prefixes.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-03-15 12:01:08 -07:00
Brandon Williams ed10cb952d serve: introduce git-serve
Introduce git-serve, the base server for protocol version 2.

Protocol version 2 is intended to be a replacement for Git's current
wire protocol.  The intention is that it will be a simpler, less
wasteful protocol which can evolve over time.

Protocol version 2 improves upon version 1 by eliminating the initial
ref advertisement.  In its place a server will export a list of
capabilities and commands which it supports in a capability
advertisement.  A client can then request that a particular command be
executed by providing a number of capabilities and command specific
parameters.  At the completion of a command, a client can request that
another command be executed or can terminate the connection by sending a
flush packet.

Signed-off-by: Brandon Williams <>
Signed-off-by: Junio C Hamano <>
2018-03-15 12:01:08 -07:00