Git Source Code Mirror - This is a publish-only repository and all pull requests are ignored. Please follow Documentation/SubmittingPatches procedure for any of your improvements. https://git-scm.com/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
git/unpack-trees.c

3027 lines
82 KiB

#include "cache.h"
#include "strvec.h"
#include "repository.h"
#include "config.h"
#include "dir.h"
#include "tree.h"
#include "tree-walk.h"
#include "cache-tree.h"
#include "unpack-trees.h"
#include "progress.h"
#include "refs.h"
#include "attr.h"
#include "split-index.h"
#include "sparse-index.h"
#include "submodule.h"
#include "submodule-config.h"
#include "fsmonitor.h"
#include "object-store.h"
#include "promisor-remote.h"
#include "entry.h"
unpack-trees: add basic support for parallel checkout This new interface allows us to enqueue some of the entries being checked out to later uncompress them, apply in-process filters, and write out the files in parallel. For now, the parallel checkout machinery is enabled by default and there is no user configuration, but run_parallel_checkout() just writes the queued entries in sequence (without spawning additional workers). The next patch will actually implement the parallelism and, later, we will make it configurable. Note that, to avoid potential data races, not all entries are eligible for parallel checkout. Also, paths that collide on disk (e.g. case-sensitive paths in case-insensitive file systems), are detected by the parallel checkout code and skipped, so that they can be safely sequentially handled later. The collision detection works like the following: - If the collision was at basename (e.g. 'a/b' and 'a/B'), the framework detects it by looking for EEXIST and EISDIR errors after an open(O_CREAT | O_EXCL) failure. - If the collision was at dirname (e.g. 'a/b' and 'A'), it is detected at the has_dirs_only_path() check, which is done for the leading path of each item in the parallel checkout queue. Both verifications rely on the fact that, before enqueueing an entry for parallel checkout, checkout_entry() makes sure that there is no file at the entry's path and that its leading components are all real directories. So, any later change in these conditions indicates that there was a collision (either between two parallel-eligible entries or between an eligible and an ineligible one). After all parallel-eligible entries have been processed, the collided (and thus, skipped) entries are sequentially fed to checkout_entry() again. This is similar to the way the current code deals with collisions, overwriting the previously checked out entries with the subsequent ones. The only difference is that, since we no longer create the files in the same order that they appear on index, we are not able to determine which of the colliding entries will survive on disk (for the classic code, it is always the last entry). Co-authored-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
1 year ago
#include "parallel-checkout.h"
sparse-checkout: integrate with sparse index When modifying the sparse-checkout definition, the sparse-checkout builtin calls update_sparsity() to modify the SKIP_WORKTREE bits of all cache entries in the index. Before, we needed the index to be fully expanded in order to ensure we had the full list of files necessary that match the new patterns. Insert a call to reset_sparse_directories() that expands sparse directories that are within the new pattern list, but only far enough that every necessary file path now exists as a cache entry. The remaining logic within update_sparsity() will modify the SKIP_WORKTREE bits appropriately. This allows us to disable command_requires_full_index within the sparse-checkout builtin. Add tests that demonstrate that we are not expanding to a full index unnecessarily. We can see the improved performance in the p2000 test script: Test HEAD~1 HEAD ------------------------------------------------------------------------ 2000.24: git ... (sparse-v3) 2.14(1.55+0.58) 1.57(1.03+0.53) -26.6% 2000.25: git ... (sparse-v4) 2.20(1.62+0.57) 1.58(0.98+0.59) -28.2% These reductions of 26-28% are small compared to most examples, but the time is dominated by writing a new copy of the base repository to the worktree and then deleting it again. The fact that the previous index expansion was such a large portion of the time is telling how important it is to complete this sparse index integration. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
4 months ago
#include "sparse-index.h"
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
/*
* Error messages expected by scripts out of plumbing commands such as
* read-tree. Non-scripted Porcelain is not required to use these messages
* and in fact are encouraged to reword them to better suit their particular
* situation better. See how "git checkout" and "git merge" replaces
* them using setup_unpack_trees_porcelain(), for example.
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
*/
static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {
/* ERROR_WOULD_OVERWRITE */
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
"Entry '%s' would be overwritten by merge. Cannot merge.",
/* ERROR_NOT_UPTODATE_FILE */
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
"Entry '%s' not uptodate. Cannot merge.",
/* ERROR_NOT_UPTODATE_DIR */
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
"Updating '%s' would lose untracked files in it",
/* ERROR_CWD_IN_THE_WAY */
"Refusing to remove '%s' since it is the current working directory.",
/* ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN */
"Untracked working tree file '%s' would be overwritten by merge.",
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
/* ERROR_WOULD_LOSE_UNTRACKED_REMOVED */
"Untracked working tree file '%s' would be removed by merge.",
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
/* ERROR_BIND_OVERLAP */
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
"Entry '%s' overlaps with '%s'. Cannot bind.",
/* ERROR_WOULD_LOSE_SUBMODULE */
"Submodule '%s' cannot checkout new HEAD.",
/* NB_UNPACK_TREES_ERROR_TYPES; just a meta value */
"",
/* WARNING_SPARSE_NOT_UPTODATE_FILE */
"Path '%s' not uptodate; will not remove from working tree.",
/* WARNING_SPARSE_UNMERGED_FILE */
"Path '%s' unmerged; will not remove from working tree.",
/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */
"Path '%s' already present; will not overwrite with sparse update.",
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
};
#define ERRORMSG(o,type) \
( ((o) && (o)->msgs[(type)]) \
? ((o)->msgs[(type)]) \
: (unpack_plumbing_errors[(type)]) )
unpack-trees: allow Porcelain to give different error messages The plumbing output is sacred as it is an API. We _could_ change it if it is broken in such a way that it cannot convey necessary information fully, but we just do not _reword_ for the sake of rewording. If somebody does not like it, s/he is complaining too late. S/he should have been here in early May 2005 and make the language used by the API closer to what humans read. S/he wasn't here. Too bad, and it is too late. And people who complain should look at a bigger picture. Look at what was suggested by one of them and think for five seconds: $ git checkout mytopic -fatal: Entry 'frotz' not uptodate. Cannot merge. +fatal: Entry 'frotz' has local changes. Cannot merge. If you do not see something wrong with this output, your brain has already been rotten with use of git for too long a time. Nobody asked us to "merge" but why are we talking about "Cannot merge"? This patch introduces a mechanism to allow Porcelains to specify messages that are different from the ones that is given by the underlying plumbing implementation of read-tree, so that we can reword the message Porcelains give without disrupting the output from the plumbing. $ git-checkout pu error: You have local changes to 'Makefile'; cannot switch branches. There are other places that ask unpack_trees() to n-way merge, detect issues and let it issue error message on its own, but I did this as a demonstration and replaced only one message. Yes I know about C99 structure initializers. I'd love to use them but we try to be nice to compilers without it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
15 years ago
unpack-trees: support super-prefix option In the future we want to support working tree operations within submodules, e.g. "git checkout --recurse-submodules", which will update the submodule to the commit as recorded in its superproject. In the submodule the unpack-tree operation is carried out as usual, but the reporting to the user needs to prefix any path with the superproject. The mechanism for this is the super-prefix. (see 74866d757, git: make super-prefix option) Add support for the super-prefix option for commands that unpack trees by wrapping any path output in unpacking trees in the newly introduced super_prefixed function. This new function prefixes any path with the super-prefix if there is one. Assuming the submodule case doesn't happen in the majority of the cases, we'd want to have a fast behavior for no super prefix, i.e. no reallocation/copying, but just returning path. Another aspect of introducing the `super_prefixed` function is to consider who owns the memory and if this is the right place where the path gets modified. As the super prefix ought to change the output behavior only and not the actual unpack tree part, it is fine to be that late in the line. As we get passed in 'const char *path', we cannot change the path itself, which means in case of a super prefix we have to copy over the path. We need two static buffers in that function as the error messages contain at most two paths. For testing purposes enable it in read-tree, which has no output of paths other than an unpack-trees.c. These are all converted in this patch. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
6 years ago
static const char *super_prefixed(const char *path)
{
/*
* It is necessary and sufficient to have two static buffers
* here, as the return value of this function is fed to
* error() using the unpack_*_errors[] templates we see above.
*/
static struct strbuf buf[2] = {STRBUF_INIT, STRBUF_INIT};
static int super_prefix_len = -1;
static unsigned idx = ARRAY_SIZE(buf) - 1;
if (super_prefix_len < 0) {
const char *super_prefix = get_super_prefix();
if (!super_prefix) {
super_prefix_len = 0;
} else {
int i;
for (i = 0; i < ARRAY_SIZE(buf); i++)
strbuf_addstr(&buf[i], super_prefix);
super_prefix_len = buf[0].len;
}
}
if (!super_prefix_len)
return path;
if (++idx >= ARRAY_SIZE(buf))
idx = 0;
strbuf_setlen(&buf[idx], super_prefix_len);
strbuf_addstr(&buf[idx], path);
return buf[idx].buf;
}
void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,
const char *cmd)
{
int i;
const char **msgs = opts->msgs;
const char *msg;
strvec_init(&opts->msgs_to_free);
if (!strcmp(cmd, "checkout"))
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("Your local changes to the following files would be overwritten by checkout:\n%%s"
"Please commit your changes or stash them before you switch branches.")
: _("Your local changes to the following files would be overwritten by checkout:\n%%s");
else if (!strcmp(cmd, "merge"))
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("Your local changes to the following files would be overwritten by merge:\n%%s"
"Please commit your changes or stash them before you merge.")
: _("Your local changes to the following files would be overwritten by merge:\n%%s");
else
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("Your local changes to the following files would be overwritten by %s:\n%%s"
"Please commit your changes or stash them before you %s.")
: _("Your local changes to the following files would be overwritten by %s:\n%%s");
msgs[ERROR_WOULD_OVERWRITE] = msgs[ERROR_NOT_UPTODATE_FILE] =
strvec_pushf(&opts->msgs_to_free, msg, cmd, cmd);
msgs[ERROR_NOT_UPTODATE_DIR] =
_("Updating the following directories would lose untracked files in them:\n%s");
msgs[ERROR_CWD_IN_THE_WAY] =
_("Refusing to remove the current working directory:\n%s");
if (!strcmp(cmd, "checkout"))
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("The following untracked working tree files would be removed by checkout:\n%%s"
"Please move or remove them before you switch branches.")
: _("The following untracked working tree files would be removed by checkout:\n%%s");
else if (!strcmp(cmd, "merge"))
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("The following untracked working tree files would be removed by merge:\n%%s"
"Please move or remove them before you merge.")
: _("The following untracked working tree files would be removed by merge:\n%%s");
else
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("The following untracked working tree files would be removed by %s:\n%%s"
"Please move or remove them before you %s.")
: _("The following untracked working tree files would be removed by %s:\n%%s");
msgs[ERROR_WOULD_LOSE_UNTRACKED_REMOVED] =
strvec_pushf(&opts->msgs_to_free, msg, cmd, cmd);
if (!strcmp(cmd, "checkout"))
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("The following untracked working tree files would be overwritten by checkout:\n%%s"
"Please move or remove them before you switch branches.")
: _("The following untracked working tree files would be overwritten by checkout:\n%%s");
else if (!strcmp(cmd, "merge"))
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("The following untracked working tree files would be overwritten by merge:\n%%s"
"Please move or remove them before you merge.")
: _("The following untracked working tree files would be overwritten by merge:\n%%s");
else
msg = advice_enabled(ADVICE_COMMIT_BEFORE_MERGE)
? _("The following untracked working tree files would be overwritten by %s:\n%%s"
"Please move or remove them before you %s.")
: _("The following untracked working tree files would be overwritten by %s:\n%%s");
msgs[ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN] =
strvec_pushf(&opts->msgs_to_free, msg, cmd, cmd);
/*
* Special case: ERROR_BIND_OVERLAP refers to a pair of paths, we
* cannot easily display it as a list.
*/
msgs[ERROR_BIND_OVERLAP] = _("Entry '%s' overlaps with '%s'. Cannot bind.");
msgs[ERROR_WOULD_LOSE_SUBMODULE] =
_("Cannot update submodule:\n%s");
msgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =
_("The following paths are not up to date and were left despite sparse patterns:\n%s");
msgs[WARNING_SPARSE_UNMERGED_FILE] =
_("The following paths are unmerged and were left despite sparse patterns:\n%s");
msgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =
_("The following paths were already present and thus not updated despite sparse patterns:\n%s");
opts->show_all_errors = 1;
/* rejected paths may not have a static buffer */
for (i = 0; i < ARRAY_SIZE(opts->unpack_rejects); i++)
opts->unpack_rejects[i].strdup_strings = 1;
}
void clear_unpack_trees_porcelain(struct unpack_trees_options *opts)
{
strvec_clear(&opts->msgs_to_free);
memset(opts->msgs, 0, sizeof(opts->msgs));
}
static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,
unsigned int set, unsigned int clear)
{
clear |= CE_HASHED;
if (set & CE_REMOVE)
set |= CE_WT_REMOVE;
ce->ce_flags = (ce->ce_flags & ~clear) | set;
return add_index_entry(&o->result, ce,
ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);
}
static void add_entry(struct unpack_trees_options *o,
const struct cache_entry *ce,
unsigned int set, unsigned int clear)
{
block alloc: allocate cache entries from mem_pool When reading large indexes from disk, a portion of the time is dominated in malloc() calls. This can be mitigated by allocating a large block of memory and manage it ourselves via memory pools. This change moves the cache entry allocation to be on top of memory pools. Design: The index_state struct will gain a notion of an associated memory_pool from which cache_entries will be allocated from. When reading in the index from disk, we have information on the number of entries and their size, which can guide us in deciding how large our initial memory allocation should be. When an index is discarded, the associated memory_pool will be discarded as well - so the lifetime of a cache_entry is tied to the lifetime of the index_state that it was allocated for. In the case of a Split Index, the following rules are followed. 1st, some terminology is defined: Terminology: - 'the_index': represents the logical view of the index - 'split_index': represents the "base" cache entries. Read from the split index file. 'the_index' can reference a single split_index, as well as cache_entries from the split_index. `the_index` will be discarded before the `split_index` is. This means that when we are allocating cache_entries in the presence of a split index, we need to allocate the entries from the `split_index`'s memory pool. This allows us to follow the pattern that `the_index` can reference cache_entries from the `split_index`, and that the cache_entries will not be freed while they are still being referenced. Managing transient cache_entry structs: Cache entries are usually allocated for an index, but this is not always the case. Cache entries are sometimes allocated because this is the type that the existing checkout_entry function works with. Because of this, the existing code needs to handle cache entries associated with an index / memory pool, and those that only exist transiently. Several strategies were contemplated around how to handle this: Chosen approach: An extra field was added to the cache_entry type to track whether the cache_entry was allocated from a memory pool or not. This is currently an int field, as there are no more available bits in the existing ce_flags bit field. If / when more bits are needed, this new field can be turned into a proper bit field. Alternatives: 1) Do not include any information about how the cache_entry was allocated. Calling code would be responsible for tracking whether the cache_entry needed to be freed or not. Pro: No extra memory overhead to track this state Con: Extra complexity in callers to handle this correctly. The extra complexity and burden to not regress this behavior in the future was more than we wanted. 2) cache_entry would gain knowledge about which mem_pool allocated it Pro: Could (potentially) do extra logic to know when a mem_pool no longer had references to any cache_entry Con: cache_entry would grow heavier by a pointer, instead of int We didn't see a tangible benefit to this approach 3) Do not add any extra information to a cache_entry, but when freeing a cache entry, check if the memory exists in a region managed by existing mem_pools. Pro: No extra memory overhead to track state Con: Extra computation is performed when freeing cache entries We decided tracking and iterating over known memory pool regions was less desirable than adding an extra field to track this stae. Signed-off-by: Jameson Miller <jamill@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
4 years ago
do_add_entry(o, dup_cache_entry(ce, &o->result), set, clear);
}
/*
* add error messages on path <path>
* corresponding to the type <e> with the message <msg>
* indicating if it should be display in porcelain or not
*/
static int add_rejected_path(struct unpack_trees_options *o,
enum unpack_trees_error_types e,
const char *path)
{
if (o->quiet)
return -1;
if (!o->show_all_errors)
unpack-trees: support super-prefix option In the future we want to support working tree operations within submodules, e.g. "git checkout --recurse-submodules", which will update the submodule to the commit as recorded in its superproject. In the submodule the unpack-tree operation is carried out as usual, but the reporting to the user needs to prefix any path with the superproject. The mechanism for this is the super-prefix. (see 74866d757, git: make super-prefix option) Add support for the super-prefix option for commands that unpack trees by wrapping any path output in unpacking trees in the newly introduced super_prefixed function. This new function prefixes any path with the super-prefix if there is one. Assuming the submodule case doesn't happen in the majority of the cases, we'd want to have a fast behavior for no super prefix, i.e. no reallocation/copying, but just returning path. Another aspect of introducing the `super_prefixed` function is to consider who owns the memory and if this is the right place where the path gets modified. As the super prefix ought to change the output behavior only and not the actual unpack tree part, it is fine to be that late in the line. As we get passed in 'const char *path', we cannot change the path itself, which means in case of a super prefix we have to copy over the path. We need two static buffers in that function as the error messages contain at most two paths. For testing purposes enable it in read-tree, which has no output of paths other than an unpack-trees.c. These are all converted in this patch. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
6 years ago
return error(ERRORMSG(o, e), super_prefixed(path));
/*
* Otherwise, insert in a list for future display by
* display_(error|warning)_msgs()
*/
string_list_append(&o->unpack_rejects[e], path);
return -1;
}
/*
* display all the error messages stored in a nice way
*/
static void display_error_msgs(struct unpack_trees_options *o)
{
int e;
unsigned error_displayed = 0;
for (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {
struct string_list *rejects = &o->unpack_rejects[e];
if (rejects->nr > 0) {
int i;
struct strbuf path = STRBUF_INIT;
error_displayed = 1;
for (i = 0; i < rejects->nr; i++)
strbuf_addf(&path, "\t%s\n", rejects->items[i].string);
unpack-trees: support super-prefix option In the future we want to support working tree operations within submodules, e.g. "git checkout --recurse-submodules", which will update the submodule to the commit as recorded in its superproject. In the submodule the unpack-tree operation is carried out as usual, but the reporting to the user needs to prefix any path with the superproject. The mechanism for this is the super-prefix. (see 74866d757, git: make super-prefix option) Add support for the super-prefix option for commands that unpack trees by wrapping any path output in unpacking trees in the newly introduced super_prefixed function. This new function prefixes any path with the super-prefix if there is one. Assuming the submodule case doesn't happen in the majority of the cases, we'd want to have a fast behavior for no super prefix, i.e. no reallocation/copying, but just returning path. Another aspect of introducing the `super_prefixed` function is to consider who owns the memory and if this is the right place where the path gets modified. As the super prefix ought to change the output behavior only and not the actual unpack tree part, it is fine to be that late in the line. As we get passed in 'const char *path', we cannot change the path itself, which means in case of a super prefix we have to copy over the path. We need two static buffers in that function as the error messages contain at most two paths. For testing purposes enable it in read-tree, which has no output of paths other than an unpack-trees.c. These are all converted in this patch. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
6 years ago
error(ERRORMSG(o, e), super_prefixed(path.buf));
strbuf_release(&path);
}
string_list_clear(rejects, 0);
}
if (error_displayed)
fprintf(stderr, _("Aborting\n"));
}
/*
* display all the warning messages stored in a nice way
*/
static void display_warning_msgs(struct unpack_trees_options *o)
{
int e;
unsigned warning_displayed = 0;
for (e = NB_UNPACK_TREES_ERROR_TYPES + 1;
e < NB_UNPACK_TREES_WARNING_TYPES; e++) {
struct string_list *rejects = &o->unpack_rejects[e];
if (rejects->nr > 0) {
int i;
struct strbuf path = STRBUF_INIT;
warning_displayed = 1;
for (i = 0; i < rejects->nr; i++)
strbuf_addf(&path, "\t%s\n", rejects->items[i].string);
warning(ERRORMSG(o, e), super_prefixed(path.buf));
strbuf_release(&path);
}
string_list_clear(rejects, 0);
}
if (warning_displayed)
fprintf(stderr, _("After fixing the above paths, you may want to run `git sparse-checkout reapply`.\n"));
}
static int check_submodule_move_head(const struct cache_entry *ce,
const char *old_id,
const char *new_id,
struct unpack_trees_options *o)
{
unsigned flags = SUBMODULE_MOVE_HEAD_DRY_RUN;
const struct submodule *sub = submodule_from_ce(ce);
if (!sub)
return 0;
if (o->reset)
flags |= SUBMODULE_MOVE_HEAD_FORCE;
if (submodule_move_head(ce->name, old_id, new_id, flags))
return add_rejected_path(o, ERROR_WOULD_LOSE_SUBMODULE, ce->name);
return 0;
}
/*
* Perform the loading of the repository's gitmodules file. This function is
* used by 'check_update()' to perform loading of the gitmodules file in two
* different situations:
* (1) before removing entries from the working tree if the gitmodules file has
* been marked for removal. This situation is specified by 'state' == NULL.
* (2) before checking out entries to the working tree if the gitmodules file
* has been marked for update. This situation is specified by 'state' != NULL.
*/
static void load_gitmodules_file(struct index_state *index,
struct checkout *state)
{
int pos = index_name_pos(index, GITMODULES_FILE, strlen(GITMODULES_FILE));
if (pos >= 0) {
struct cache_entry *ce = index->cache[pos];
if (!state && ce->ce_flags & CE_WT_REMOVE) {
repo_read_gitmodules(the_repository, 0);
} else if (state && (ce->ce_flags & CE_UPDATE)) {
submodule_free(the_repository);
checkout_entry(ce, state, NULL, NULL);
repo_read_gitmodules(the_repository, 0);
}
}
}
static struct progress *get_progress(struct unpack_trees_options *o,
struct index_state *index)
{
unsigned cnt = 0, total = 0;
if (!o->update || !o->verbose_update)
return NULL;
for (; cnt < index->cache_nr; cnt++) {
const struct cache_entry *ce = index->cache[cnt];
if (ce->ce_flags & (CE_UPDATE | CE_WT_REMOVE))
total++;
}
return start_delayed_progress(_("Updating files"), total);
}
static void setup_collided_checkout_detection(struct checkout *state,
struct index_state *index)
{
int i;
state->clone = 1;
for (i = 0; i < index->cache_nr; i++)
index->cache[i]->ce_flags &= ~CE_MATCHED;
}
static void report_collided_checkout(struct index_state *index)
{
struct string_list list = STRING_LIST_INIT_NODUP;
int i;
for (i = 0; i < index->cache_nr; i++) {
struct cache_entry *ce = index->cache[i];
if (!(ce->ce_flags & CE_MATCHED))
continue;
string_list_append(&list, ce->name);
ce->ce_flags &= ~CE_MATCHED;
}
list.cmp = fspathcmp;
string_list_sort(&list);
if (list.nr) {
warning(_("the following paths have collided (e.g. case-sensitive paths\n"
"on a case-insensitive filesystem) and only one from the same\n"
"colliding group is in the working tree:\n"));
for (i = 0; i < list.nr; i++)
fprintf(stderr, " '%s'\n", list.items[i].string);
}
string_list_clear(&list, 0);
}
static int must_checkout(const struct cache_entry *ce)
{
return ce->ce_flags & CE_UPDATE;
}
static int check_updates(struct unpack_trees_options *o,
struct index_state *index)
{
unsigned cnt = 0;
int errs = 0;
struct progress *progress;
struct checkout state = CHECKOUT_INIT;
parallel-checkout: add configuration options Make parallel checkout configurable by introducing two new settings: checkout.workers and checkout.thresholdForParallelism. The first defines the number of workers (where one means sequential checkout), and the second defines the minimum number of entries to attempt parallel checkout. To decide the default value for checkout.workers, the parallel version was benchmarked during three operations in the linux repo, with cold cache: cloning v5.8, checking out v5.8 from v2.6.15 (checkout I) and checking out v5.8 from v5.7 (checkout II). The four tables below show the mean run times and standard deviations for 5 runs in: a local file system on SSD, a local file system on HDD, a Linux NFS server, and Amazon EFS (all on Linux). Each parallel checkout test was executed with the number of workers that brings the best overall results in that environment. Local SSD: Sequential 10 workers Speedup Clone 8.805 s ± 0.043 s 3.564 s ± 0.041 s 2.47 ± 0.03 Checkout I 9.678 s ± 0.057 s 4.486 s ± 0.050 s 2.16 ± 0.03 Checkout II 5.034 s ± 0.072 s 3.021 s ± 0.038 s 1.67 ± 0.03 Local HDD: Sequential 10 workers Speedup Clone 32.288 s ± 0.580 s 30.724 s ± 0.522 s 1.05 ± 0.03 Checkout I 54.172 s ± 7.119 s 54.429 s ± 6.738 s 1.00 ± 0.18 Checkout II 40.465 s ± 2.402 s 38.682 s ± 1.365 s 1.05 ± 0.07 Linux NFS server (v4.1, on EBS, single availability zone): Sequential 32 workers Speedup Clone 240.368 s ± 6.347 s 57.349 s ± 0.870 s 4.19 ± 0.13 Checkout I 242.862 s ± 2.215 s 58.700 s ± 0.904 s 4.14 ± 0.07 Checkout II 65.751 s ± 1.577 s 23.820 s ± 0.407 s 2.76 ± 0.08 EFS (v4.1, replicated over multiple availability zones): Sequential 32 workers Speedup Clone 922.321 s ± 2.274 s 210.453 s ± 3.412 s 4.38 ± 0.07 Checkout I 1011.300 s ± 7.346 s 297.828 s ± 0.964 s 3.40 ± 0.03 Checkout II 294.104 s ± 1.836 s 126.017 s ± 1.190 s 2.33 ± 0.03 The above benchmarks show that parallel checkout is most effective on repositories located on an SSD or over a distributed file system. For local file systems on spinning disks, and/or older machines, the parallelism does not always bring a good performance. For this reason, the default value for checkout.workers is one, a.k.a. sequential checkout. To decide the default value for checkout.thresholdForParallelism, another benchmark was executed in the "Local SSD" setup, where parallel checkout showed to be beneficial. This time, we compared the runtime of a `git checkout -f`, with and without parallelism, after randomly removing an increasing number of files from the Linux working tree. The "sequential fallback" column below corresponds to the executions where checkout.workers was 10 but checkout.thresholdForParallelism was equal to the number of to-be-updated files plus one (so that we end up writing sequentially). Each test case was sampled 15 times, and each sample had a randomly different set of files removed. Here are the results: sequential fallback 10 workers speedup 10 files 772.3 ms ± 12.6 ms 769.0 ms ± 13.6 ms 1.00 ± 0.02 20 files 780.5 ms ± 15.8 ms 775.2 ms ± 9.2 ms 1.01 ± 0.02 50 files 806.2 ms ± 13.8 ms 767.4 ms ± 8.5 ms 1.05 ± 0.02 100 files 833.7 ms ± 21.4 ms 750.5 ms ± 16.8 ms 1.11 ± 0.04 200 files 897.6 ms ± 30.9 ms 730.5 ms ± 14.7 ms 1.23 ± 0.05 500 files 1035.4 ms ± 48.0 ms 677.1 ms ± 22.3 ms 1.53 ± 0.09 1000 files 1244.6 ms ± 35.6 ms 654.0 ms ± 38.3 ms 1.90 ± 0.12 2000 files 1488.8 ms ± 53.4 ms 658.8 ms ± 23.8 ms 2.26 ± 0.12 From the above numbers, 100 files seems to be a reasonable default value for the threshold setting. Note: Up to 1000 files, we observe a drop in the execution time of the parallel code with an increase in the number of files. This is a rather odd behavior, but it was observed in multiple repetitions. Above 1000 files, the execution time increases according to the number of files, as one would expect. About the test environments: Local SSD tests were executed on an i7-7700HQ (4 cores with hyper-threading) running Manjaro Linux. Local HDD tests were executed on an Intel(R) Xeon(R) E3-1230 (also 4 cores with hyper-threading), HDD Seagate Barracuda 7200.14 SATA 3.1, running Debian. NFS and EFS tests were executed on an Amazon EC2 c5n.xlarge instance, with 4 vCPUs. The Linux NFS server was running on a m6g.large instance with 2 vCPUSs and a 1 TB EBS GP2 volume. Before each timing, the linux repository was removed (or checked out back to its previous state), and `sync && sysctl vm.drop_caches=3` was executed. Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
1 year ago
int i, pc_workers, pc_threshold;
trace_performance_enter();
state.force = 1;
state.quiet = 1;
state.refresh_cache = 1;
state.istate = index;
clone_checkout_metadata(&state.meta, &o->meta, NULL);
unpack-trees: exit check_updates() early if updates are not wanted check_updates() has a lot of code that repeatedly checks whether o->update or o->dry_run are set. (Note that o->dry_run is a near-synonym for !o->update, but not quite as per commit 2c9078d05bf2 ("unpack-trees: add the dry_run flag to unpack_trees_options", 2011-05-25).) In fact, this function almost turns into a no-op whenever the condition !o->update || o->dry_run is met. Simplify the code by checking this condition at the beginning of the function, and when it is true, do the few things that are relevant and return early. There are a few things that make the conversion not quite obvious: * The fact that check_updates() does not actually turn into a no-op when updates are not wanted may be slightly surprising. However, commit 33ecf7eb61 (Discard "deleted" cache entries after using them to update the working tree, 2008-02-07) put the discarding of unused cache entries in check_updates() so we still need to keep the call to remove_marked_cache_entries(). It's possible this call belongs in another function, but it is certainly needed as tests will fail if it is removed. * The original called remove_scheduled_dirs() unconditionally. Technically, commit 7847892716 (unlink_entry(): introduce schedule_dir_for_removal(), 2009-02-09) should have made that call conditional, but it didn't matter in practice because remove_scheduled_dirs() becomes a no-op when all the calls to unlink_entry() are skipped. As such, we do not need to call it. * When (o->dry_run && o->update), the original would have two calls to git_attr_set_direction() surrounding a bunch of skipped updates. These two calls to git_attr_set_direction() cancel each other out and thus can be omitted when o->dry_run is true just as they already are when !o->update. * The code would previously call setup_collided_checkout_detection() and report_collided_checkout() even when o->dry_run. However, this was just an expensive no-op because setup_collided_checkout_detection() merely cleared the CE_MATCHED flag for each cache entry, and report_collided_checkout() reported which ones had it set. Since a dry-run would skip all the checkout_entry() calls, CE_MATCHED would never get set and thus no collisions would be reported. Since we can't detect the collisions anyway without doing updates, skipping the collisions detection setup and reporting is an optimization. * The code previously would call get_progress() and display_progress() even when (!o->update || o->dry_run). This served to show how long it took to skip all the updates, which is somewhat useless. Since we are skipping the updates, we can skip showing how long it takes to skip them. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
3 years ago
if (!o->update || o->dry_run) {
remove_marked_cache_entries(index, 0);
trace_performance_leave("check_updates");
return 0;