1 Use of index and Racy git problem
2 =================================
7 The index is one of the most important data structure in git.
8 It represents a virtual working tree state by recording list of
9 paths and their object names and serves as a staging area to
10 write out the next tree object to be committed. The state is
11 "virtual" in the sense that it does not necessarily have to, and
12 often does not, match the files in the working tree.
14 There are cases git needs to examine the differences between the
15 virtual working tree state in the index and the files in the
16 working tree. The most obvious case is when the user asks `git
17 diff` (or its low level implementation, `git diff-files`) or
18 `git-ls-files --modified`. In addition, git internally checks
19 if the files in the working tree is different from what are
20 recorded in the index to avoid stomping on local changes in them
21 during patch application, switching branches, and merging.
23 In order to speed up this comparison between the files in the
24 working tree and the index entries, the index entries record the
25 information obtained from the filesystem via `lstat(2)` system
26 call when they were last updated. When checking if they differ,
27 git first runs `lstat(2)` on the files and compare the result
28 with this information (this is what was originally done by the
29 `ce_match_stat()` function, which the current code does in
30 `ce_match_stat_basic()` function). If some of these "cached
31 stat information" fields do not match, git can tell that the
32 files are modified without even looking at their contents.
34 Note: not all members in `struct stat` obtained via `lstat(2)`
35 are used for this comparison. For example, `st_atime` obviously
36 is not useful. Currently, git compares the file type (regular
37 files vs symbolic links) and executable bits (only for regular
38 files) from `st_mode` member, `st_mtime` and `st_ctime`
39 timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members.
40 With a `USE_STDEV` compile-time option, `st_dev` is also
41 compared, but this is not enabled by default because this member
42 is not stable on network filesystems. With `USE_NSEC`
43 compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec`
44 members are also compared, but this is not enabled by default
45 because the value of this member becomes meaningless once the
46 inode is evicted from the inode cache on filesystems that do not
53 There is one slight problem with the optimization based on the
54 cached stat information. Consider this sequence:
56 $ git update-index 'foo'
57 : modify 'foo' in-place without changing its size
59 The first `update-index` computes the object name of the
60 contents of file `foo` and updates the index entry for `foo`
61 along with the `struct stat` information. If the modification
62 that follows it happens very fast so that the file's `st_mtime`
63 timestamp does not change, after this sequence, the cached stat
64 information the index entry records still exactly match what you
65 can obtain from the filesystem, but the file `foo` is modified.
66 This way, git can incorrectly think files in the working tree
67 are unmodified even though they actually are. This is called
68 the "racy git" problem (discovered by Pasky), and the entries
69 that appear clean when they may not be because of this problem
70 are called "racily clean".
72 To avoid this problem, git does two things:
74 . When the cached stat information says the file has not been
75 modified, and the `st_mtime` is the same as (or newer than)
76 the timestamp of the index file itself (which is the time `git
77 update-index foo` finished running in the above example), it
78 also compares the contents with the object registered in the
79 index entry to make sure they match.
81 . When the index file is updated that contains racily clean
82 entries, cached `st_size` information is truncated to zero
83 before writing a new version of the index file.
85 Because the index file itself is written after collecting all
86 the stat information from updated paths, `st_mtime` timestamp of
87 it is usually the same as or newer than any of the paths the
88 index contains. And no matter how quick the modification that
89 follows `git update-index foo` finishes, the resulting
90 `st_mtime` timestamp on `foo` cannot get the timestamp earlier
91 than the index file. Therefore, index entries that can be
92 racily clean are limited to the ones that have the same
93 timestamp as the index file itself.
95 The callers that want to check if an index entry matches the
96 corresponding file in the working tree continue to call
97 `ce_match_stat()`, but with this change, `ce_match_stat()` uses
98 `ce_modified_check_fs()` to see if racily clean ones are
99 actually clean after comparing the cached stat information using
100 `ce_match_stat_basic()`.
102 The problem the latter solves is this sequence:
104 $ git update-index 'foo'
105 : modify 'foo' in-place without changing its size
106 : wait for enough time
107 $ git update-index 'bar'
109 Without the latter, the timestamp of the index file gets a newer
110 value, and falsely clean entry `foo` would not be caught by the
111 timestamp comparison check done with the former logic anymore.
112 The latter makes sure that the cached stat information for `foo`
113 would never match with the file in the working tree, so later
114 checks by `ce_match_stat_basic()` would report the index entry
115 does not match the file and git does not have to fall back on more
116 expensive `ce_modified_check_fs()`.
122 The runtime penalty of falling back to `ce_modified_check_fs()`
123 from `ce_match_stat()` can be very expensive when there are many
124 racily clean entries. An obvious way to artificially create
125 this situation is to give the same timestamp to all the files in
126 the working tree in a large project, run `git update-index` on
127 them, and give the same timestamp to the index file:
130 $ git ls-files | xargs touch -r .datestamp
131 $ git ls-files | git update-index --stdin
132 $ touch -r .datestamp .git/index
134 This will make all index entries racily clean. The linux-2.6
135 project, for example, there are over 20,000 files in the working
136 tree. On my Athron 64X2 3800+, after the above:
138 $ /usr/bin/time git diff-files
139 1.68user 0.54system 0:02.22elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
140 0inputs+0outputs (0major+67111minor)pagefaults 0swaps
141 $ git update-index MAINTAINERS
142 $ /usr/bin/time git diff-files
143 0.02user 0.12system 0:00.14elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
144 0inputs+0outputs (0major+935minor)pagefaults 0swaps
146 Running `git update-index` in the middle checked the racily
147 clean entries, and left the cached `st_mtime` for all the paths
148 intact because they were actually clean (so this step took about
149 the same amount of time as the first `git diff-files`). After
150 that, they are not racily clean anymore but are truly clean, so
151 the second invocation of `git diff-files` fully took advantage
152 of the cached stat information.
155 Avoiding runtime penalty
156 ------------------------
158 In order to avoid the above runtime penalty, the recent "master"
159 branch (post 1.4.2) has a code that makes sure the index file
160 gets timestamp newer than the youngest files in the index when
161 there are many young files with the same timestamp as the
162 resulting index file would otherwise would have by waiting
163 before finishing writing the index file out.
165 I suspect that in practice the situation where many paths in the
166 index are all racily clean is quite rare. The only code paths
167 that can record recent timestamp for large number of paths I
170 . Initial `git add .` of a large project.
172 . `git checkout` of a large project from an empty index into an
173 unpopulated working tree.
175 Note: switching branches with `git checkout` keeps the cached
176 stat information of existing working tree files that are the
177 same between the current branch and the new branch, which are
178 all older than the resulting index file, and they will not
179 become racily clean. Only the files that are actually checked
180 out can become racily clean.
182 In a large project where raciness avoidance cost really matters,
183 however, the initial computation of all object names in the
184 index takes more than one second, and the index file is written
185 out after all that happens. Therefore the timestamp of the
186 index file will be more than one seconds later than the the
187 youngest file in the working tree. This means that in these
188 cases there actually will not be any racily clean entry in
191 So in summary I think we should not worry about avoiding the
192 runtime penalty and get rid of the "wait before finishing
projects / git / blob