6 git-fast-import - Backend for fast Git data importers.
11 frontend | 'git-fast-import' [options]
15 This program is usually not what the end user wants to run directly.
16 Most end users want to use one of the existing frontend programs,
17 which parses a specific type of foreign source and feeds the contents
18 stored there to git-fast-import (gfi).
20 gfi reads a mixed command/data stream from standard input and
21 writes one or more packfiles directly into the current repository.
22 When EOF is received on standard input, fast import writes out
23 updated branch and tag refs, fully updating the current repository
24 with the newly imported data.
26 The gfi backend itself can import into an empty repository (one that
27 has already been initialized by gitlink:git-init[1]) or incrementally
28 update an existing populated repository. Whether or not incremental
29 imports are supported from a particular foreign source depends on
30 the frontend program in use.
36 Maximum size of each output packfile, expressed in MiB.
37 The default is 4096 (4 GiB) as that is the maximum allowed
38 packfile size (due to file format limitations). Some
39 importers may wish to lower this, such as to ensure the
40 resulting packfiles fit on CDs.
43 Maximum delta depth, for blob and tree deltification.
46 --active-branches=<n>::
47 Maximum number of branches to maintain active at once.
48 See ``Memory Utilization'' below for details. Default is 5.
50 --export-marks=<file>::
51 Dumps the internal marks table to <file> when complete.
52 Marks are written one per line as `:markid SHA-1`.
53 Frontends can use this file to validate imports after they
57 Records every tag and commit made to a log file. (This file
58 can be quite verbose on large imports.) This particular
59 option has been primarily intended to facilitate debugging
60 gfi and has limited usefulness in other contexts. It may
61 be removed in future versions.
66 The design of gfi allows it to import large projects in a minimum
67 amount of memory usage and processing time. Assuming the frontend
68 is able to keep up with gfi and feed it a constant stream of data,
69 import times for projects holding 10+ years of history and containing
70 100,000+ individual commits are generally completed in just 1-2
71 hours on quite modest (~$2,000 USD) hardware.
73 Most bottlenecks appear to be in foreign source data access (the
74 source just cannot extract revisions fast enough) or disk IO (gfi
75 writes as fast as the disk will take the data). Imports will run
76 faster if the source data is stored on a different drive than the
77 destination Git repository (due to less IO contention).
82 A typical frontend for gfi tends to weigh in at approximately 200
83 lines of Perl/Python/Ruby code. Most developers have been able to
84 create working importers in just a couple of hours, even though it
85 is their first exposure to gfi, and sometimes even to Git. This is
86 an ideal situation, given that most conversion tools are throw-away
87 (use once, and never look back).
92 Like `git-push` or `git-fetch`, imports handled by gfi are safe to
93 run alongside parallel `git repack -a -d` or `git gc` invocations,
94 or any other Git operation (including `git prune`, as loose objects
95 are never used by gfi).
97 However, gfi does not lock the branch or tag refs it is actively
98 importing. After EOF, during its ref update phase, gfi blindly
99 overwrites each imported branch or tag ref. Consequently it is not
100 safe to modify refs that are currently being used by a running gfi
101 instance, as work could be lost when gfi overwrites the refs.
106 gfi tracks a set of branches in memory. Any branch can be created
107 or modified at any point during the import process by sending a
108 `commit` command on the input stream. This design allows a frontend
109 program to process an unlimited number of branches simultaneously,
110 generating commits in the order they are available from the source
111 data. It also simplifies the frontend programs considerably.
113 gfi does not use or alter the current working directory, or any
114 file within it. (It does however update the current Git repository,
115 as referenced by `GIT_DIR`.) Therefore an import frontend may use
116 the working directory for its own purposes, such as extracting file
117 revisions from the foreign source. This ignorance of the working
118 directory also allows gfi to run very quickly, as it does not
119 need to perform any costly file update operations when switching
124 With the exception of raw file data (which Git does not interpret)
125 the gfi input format is text (ASCII) based. This text based
126 format simplifies development and debugging of frontend programs,
127 especially when a higher level language such as Perl, Python or
130 gfi is very strict about its input. Where we say SP below we mean
131 *exactly* one space. Likewise LF means one (and only one) linefeed.
132 Supplying additional whitespace characters will cause unexpected
133 results, such as branch names or file names with leading or trailing
134 spaces in their name, or early termination of gfi when it encounters
139 gfi accepts several commands to update the current repository
140 and control the current import process. More detailed discussion
141 (with examples) of each command follows later.
144 Creates a new branch or updates an existing branch by
145 creating a new commit and updating the branch to point at
146 the newly created commit.
149 Creates an annotated tag object from an existing commit or
150 branch. Lightweight tags are not supported by this command,
151 as they are not recommended for recording meaningful points
155 Reset an existing branch (or a new branch) to a specific
156 revision. This command must be used to change a branch to
157 a specific revision without making a commit on it.
160 Convert raw file data into a blob, for future use in a
161 `commit` command. This command is optional and is not
162 needed to perform an import.
165 Forces gfi to close the current packfile, generate its
166 unique SHA-1 checksum and index, and start a new packfile.
167 This command is optional and is not needed to perform
172 Create or update a branch with a new commit, recording one logical
173 change to the project.
178 ('author' SP <name> SP LT <email> GT SP <time> SP <tz> LF)?
179 'committer' SP <name> SP LT <email> GT SP <time> SP <tz> LF
181 ('from' SP <committish> LF)?
182 ('merge' SP <committish> LF)?
183 (filemodify | filedelete)*
187 where `<ref>` is the name of the branch to make the commit on.
188 Typically branch names are prefixed with `refs/heads/` in
189 Git, so importing the CVS branch symbol `RELENG-1_0` would use
190 `refs/heads/RELENG-1_0` for the value of `<ref>`. The value of
191 `<ref>` must be a valid refname in Git. As `LF` is not valid in
192 a Git refname, no quoting or escaping syntax is supported here.
194 A `mark` command may optionally appear, requesting gfi to save a
195 reference to the newly created commit for future use by the frontend
196 (see below for format). It is very common for frontends to mark
197 every commit they create, thereby allowing future branch creation
198 from any imported commit.
200 The `data` command following `committer` must supply the commit
201 message (see below for `data` command syntax). To import an empty
202 commit message use a 0 length data. Commit messages are free-form
203 and are not interpreted by Git. Currently they must be encoded in
204 UTF-8, as gfi does not permit other encodings to be specified.
206 Zero or more `filemodify` and `filedelete` commands may be
207 included to update the contents of the branch prior to the commit.
208 These commands can be supplied in any order, gfi is not sensitive
209 to pathname or operation ordering.
213 An `author` command may optionally appear, if the author information
214 might differ from the committer information. If `author` is omitted
215 then gfi will automatically use the committer's information for
216 the author portion of the commit. See below for a description of
217 the fields in `author`, as they are identical to `committer`.
221 The `committer` command indicates who made this commit, and when
224 Here `<name>` is the person's display name (for example
225 ``Com M Itter'') and `<email>` is the person's email address
226 (``cm@example.com''). `LT` and `GT` are the literal less-than (\x3c)
227 and greater-than (\x3e) symbols. These are required to delimit
228 the email address from the other fields in the line. Note that
229 `<name>` is free-form and may contain any sequence of bytes, except
230 `LT` and `LF`. It is typically UTF-8 encoded.
232 The time of the change is specified by `<time>` as the number of
233 seconds since the UNIX epoc (midnight, Jan 1, 1970, UTC) and is
234 written in base-10 notation using US-ASCII digits. The committer's
235 timezone is specified by `<tz>` as a positive or negative offset
236 from UTC, in minutes. For example EST would be expressed in `<tz>`
241 Only valid for the first commit made on this branch by this
242 gfi process. The `from` command is used to specify the commit
243 to initialize this branch from. This revision will be the first
244 ancestor of the new commit.
246 Omitting the `from` command in the first commit of a new branch will
247 cause gfi to create that commit with no ancestor. This tends to be
248 desired only for the initial commit of a project. Omitting the
249 `from` command on existing branches is required, as the current
250 commit on that branch is automatically assumed to be the first
251 ancestor of the new commit.
253 As `LF` is not valid in a Git refname or SHA-1 expression, no
254 quoting or escaping syntax is supported within `<committish>`.
256 Here `<committish>` is any of the following:
258 * The name of an existing branch already in gfi's internal branch
259 table. If gfi doesn't know the name, its treated as a SHA-1
262 * A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
264 The reason gfi uses `:` to denote a mark reference is this character
265 is not legal in a Git branch name. The leading `:` makes it easy
266 to distingush between the mark 42 (`:42`) and the branch 42 (`42`
267 or `refs/heads/42`), or an abbreviated SHA-1 which happened to
268 consist only of base-10 digits.
270 Marks must be declared (via `mark`) before they can be used.
272 * A complete 40 byte or abbreviated commit SHA-1 in hex.
274 * Any valid Git SHA-1 expression that resolves to a commit. See
275 ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details.
277 The special case of restarting an incremental import from the
278 current branch value should be written as:
280 from refs/heads/branch^0
282 The `^0` suffix is necessary as gfi does not permit a branch to
283 start from itself, and the branch is created in memory before the
284 `from` command is even read from the input. Adding `^0` will force
285 gfi to resolve the commit through Git's revision parsing library,
286 rather than its internal branch table, thereby loading in the
287 existing value of the branch.
291 Includes one additional ancestor commit, and makes the current
292 commit a merge commit. An unlimited number of `merge` commands per
293 commit are permitted by gfi, thereby establishing an n-way merge.
294 However Git's other tools never create commits with more than 15
295 additional ancestors (forming a 16-way merge). For this reason
296 it is suggested that frontends do not use more than 15 `merge`
299 Here `<committish>` is any of the commit specification expressions
300 also accepted by `from` (see above).
304 Included in a `commit` command to add a new file or change the
305 content of an existing file. This command has two different means
306 of specifying the content of the file.
308 External data format::
309 The data content for the file was already supplied by a prior
310 `blob` command. The frontend just needs to connect it.
313 'M' SP <mode> SP <dataref> SP <path> LF
316 Here `<dataref>` can be either a mark reference (`:<idnum>`)
317 set by a prior `blob` command, or a full 40-byte SHA-1 of an
318 existing Git blob object.
321 The data content for the file has not been supplied yet.
322 The frontend wants to supply it as part of this modify
326 'M' SP <mode> SP 'inline' SP <path> LF
330 See below for a detailed description of the `data` command.
332 In both formats `<mode>` is the type of file entry, specified
333 in octal. Git only supports the following modes:
335 * `100644` or `644`: A normal (not-executable) file. The majority
336 of files in most projects use this mode. If in doubt, this is
338 * `100755` or `755`: A normal, but executable, file.
339 * `140000`: A symlink, the content of the file will be the link target.
341 In both formats `<path>` is the complete path of the file to be added
342 (if not already existing) or modified (if already existing).
344 A `<path>` string must use UNIX-style directory seperators (forward
345 slash `/`), may contain any byte other than `LF`, and must not
346 start with double quote (`"`).
348 If an `LF` or double quote must be encoded into `<path>` shell-style
349 quoting should be used, e.g. `"path/with\n and \" in it"`.
351 The value of `<path>` must be in canoncial form. That is it must not:
353 * contain an empty directory component (e.g. `foo//bar` is invalid),
354 * end with a directory seperator (e.g. `foo/` is invalid),
355 * start with a directory seperator (e.g. `/foo` is invalid),
356 * contain the special component `.` or `..` (e.g. `foo/./bar` and
357 `foo/../bar` are invalid).
359 It is recommended that `<path>` always be encoded using UTF-8.
364 Included in a `commit` command to remove a file from the branch.
365 If the file removal makes its directory empty, the directory will
366 be automatically removed too. This cascades up the tree until the
367 first non-empty directory or the root is reached.
373 here `<path>` is the complete path of the file to be removed.
374 See `filemodify` above for a detailed description of `<path>`.
378 Arranges for gfi to save a reference to the current object, allowing
379 the frontend to recall this object at a future point in time, without
380 knowing its SHA-1. Here the current object is the object creation
381 command the `mark` command appears within. This can be `commit`,
382 `tag`, and `blob`, but `commit` is the most common usage.
385 'mark' SP ':' <idnum> LF
388 where `<idnum>` is the number assigned by the frontend to this mark.
389 The value of `<idnum>` is expressed in base 10 notation using
390 US-ASCII digits. The value 0 is reserved and cannot be used as
391 a mark. Only values greater than or equal to 1 may be used as marks.
393 New marks are created automatically. Existing marks can be moved
394 to another object simply by reusing the same `<idnum>` in another
399 Creates an annotated tag referring to a specific commit. To create
400 lightweight (non-annotated) tags see the `reset` command below.
404 'from' SP <committish> LF
405 'tagger' SP <name> SP LT <email> GT SP <time> SP <tz> LF
410 where `<name>` is the name of the tag to create.
412 Tag names are automatically prefixed with `refs/tags/` when stored
413 in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
414 use just `RELENG-1_0-FINAL` for `<name>`, and gfi will write the
415 corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
417 The value of `<name>` must be a valid refname in Git and therefore
418 may contain forward slashes. As `LF` is not valid in a Git refname,
419 no quoting or escaping syntax is supported here.
421 The `from` command is the same as in the `commit` command; see
424 The `tagger` command uses the same format as `committer` within
425 `commit`; again see above for details.
427 The `data` command following `tagger` must supply the annotated tag
428 message (see below for `data` command syntax). To import an empty
429 tag message use a 0 length data. Tag messages are free-form and are
430 not interpreted by Git. Currently they must be encoded in UTF-8,
431 as gfi does not permit other encodings to be specified.
433 Signing annotated tags during import from within gfi is not
434 supported. Trying to include your own PGP/GPG signature is not
435 recommended, as the frontend does not (easily) have access to the
436 complete set of bytes which normally goes into such a signature.
437 If signing is required, create lightweight tags from within gfi with
438 `reset`, then create the annotated versions of those tags offline
439 with the standard gitlink:git-tag[1] process.
443 Creates (or recreates) the named branch, optionally starting from
444 a specific revision. The reset command allows a frontend to issue
445 a new `from` command for an existing branch, or to create a new
446 branch from an existing commit without creating a new commit.
450 ('from' SP <committish> LF)?
454 For a detailed description of `<ref>` and `<committish>` see above
455 under `commit` and `from`.
457 The `reset` command can also be used to create lightweight
458 (non-annotated) tags. For example:
465 would create the lightweight tag `refs/tags/938` referring to
466 whatever commit mark `:938` references.
470 Requests writing one file revision to the packfile. The revision
471 is not connected to any commit; this connection must be formed in
472 a subsequent `commit` command by referencing the blob through an
481 The mark command is optional here as some frontends have chosen
482 to generate the Git SHA-1 for the blob on their own, and feed that
483 directly to `commit`. This is typically more work than its worth
484 however, as marks are inexpensive to store and easy to use.
488 Supplies raw data (for use as blob/file content, commit messages, or
489 annotated tag messages) to gfi. Data can be supplied using an exact
490 byte count or delimited with a terminating line. Real frontends
491 intended for production-quality conversions should always use the
492 exact byte count format, as it is more robust and performs better.
493 The delimited format is intended primarily for testing gfi.
495 Exact byte count format:
502 where `<count>` is the exact number of bytes appearing within
503 `<raw>`. The value of `<count>` is expressed in base 10 notation
504 using US-ASCII digits. The `LF` on either side of `<raw>` is not
505 included in `<count>` and will not be included in the imported data.
510 'data' SP '<<' <delim> LF
515 where `<delim>` is the chosen delimiter string. The string `<delim>`
516 must not appear on a line by itself within `<raw>`, as otherwise
517 gfi will think the data ends earlier than it really does. The `LF`
518 immediately trailing `<raw>` is part of `<raw>`. This is one of
519 the limitations of the delimited format, it is impossible to supply
520 a data chunk which does not have an LF as its last byte.
524 Forces gfi to close the current packfile and start a new one.
525 As this requires a significant amount of CPU time and disk IO
526 (to compute the overall pack SHA-1 checksum and generate the
527 corresponding index file) it can easily take several minutes for
528 a single `checkpoint` command to complete.
535 Packfile Optimization
536 ---------------------
537 When packing a blob gfi always attempts to deltify against the last
538 blob written. Unless specifically arranged for by the frontend,
539 this will probably not be a prior version of the same file, so the
540 generated delta will not be the smallest possible. The resulting
541 packfile will be compressed, but will not be optimal.
543 Frontends which have efficient access to all revisions of a
544 single file (for example reading an RCS/CVS ,v file) can choose
545 to supply all revisions of that file as a sequence of consecutive
546 `blob` commands. This allows gfi to deltify the different file
547 revisions against each other, saving space in the final packfile.
548 Marks can be used to later identify individual file revisions during
549 a sequence of `commit` commands.
551 The packfile(s) created by gfi do not encourage good disk access
552 patterns. This is caused by gfi writing the data in the order
553 it is received on standard input, while Git typically organizes
554 data within packfiles to make the most recent (current tip) data
555 appear before historical data. Git also clusters commits together,
556 speeding up revision traversal through better cache locality.
558 For this reason it is strongly recommended that users repack the
559 repository with `git repack -a -d` after gfi completes, allowing
560 Git to reorganize the packfiles for faster data access. If blob
561 deltas are suboptimal (see above) then also adding the `-f` option
562 to force recomputation of all deltas can significantly reduce the
563 final packfile size (30-50% smaller can be quite typical).
567 There are a number of factors which affect how much memory gfi
568 requires to perform an import. Like critical sections of core
569 Git, gfi uses its own memory allocators to ammortize any overheads
570 associated with malloc. In practice gfi tends to ammoritize any
571 malloc overheads to 0, due to its use of large block allocations.
575 gfi maintains an in-memory structure for every object written in
576 this execution. On a 32 bit system the structure is 32 bytes,
577 on a 64 bit system the structure is 40 bytes (due to the larger
578 pointer sizes). Objects in the table are not deallocated until
579 gfi terminates. Importing 2 million objects on a 32 bit system
580 will require approximately 64 MiB of memory.
582 The object table is actually a hashtable keyed on the object name
583 (the unique SHA-1). This storage configuration allows gfi to reuse
584 an existing or already written object and avoid writing duplicates
585 to the output packfile. Duplicate blobs are surprisingly common
586 in an import, typically due to branch merges in the source.
590 Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
591 bytes, depending on pointer size) per mark. Although the array
592 is sparse, frontends are still strongly encouraged to use marks
593 between 1 and n, where n is the total number of marks required for
598 Branches are classified as active and inactive. The memory usage
599 of the two classes is significantly different.
601 Inactive branches are stored in a structure which uses 96 or 120
602 bytes (32 bit or 64 bit systems, respectively), plus the length of
603 the branch name (typically under 200 bytes), per branch. gfi will
604 easily handle as many as 10,000 inactive branches in under 2 MiB
607 Active branches have the same overhead as inactive branches, but
608 also contain copies of every tree that has been recently modified on
609 that branch. If subtree `include` has not been modified since the
610 branch became active, its contents will not be loaded into memory,
611 but if subtree `src` has been modified by a commit since the branch
612 became active, then its contents will be loaded in memory.
614 As active branches store metadata about the files contained on that
615 branch, their in-memory storage size can grow to a considerable size
618 gfi automatically moves active branches to inactive status based on
619 a simple least-recently-used algorithm. The LRU chain is updated on
620 each `commit` command. The maximum number of active branches can be
621 increased or decreased on the command line with `--active-branches=`.
625 Trees (aka directories) use just 12 bytes of memory on top of the
626 memory required for their entries (see ``per active file'' below).
627 The cost of a tree is virtually 0, as its overhead ammortizes out
628 over the individual file entries.
630 per active file entry
631 ~~~~~~~~~~~~~~~~~~~~~
632 Files (and pointers to subtrees) within active trees require 52 or 64
633 bytes (32/64 bit platforms) per entry. To conserve space, file and
634 tree names are pooled in a common string table, allowing the filename
635 ``Makefile'' to use just 16 bytes (after including the string header
636 overhead) no matter how many times it occurs within the project.
638 The active branch LRU, when coupled with the filename string pool
639 and lazy loading of subtrees, allows gfi to efficiently import
640 projects with 2,000+ branches and 45,114+ files in a very limited
641 memory footprint (less than 2.7 MiB per active branch).
646 Written by Shawn O. Pearce <spearce@spearce.org>.
650 Documentation by Shawn O. Pearce <spearce@spearce.org>.
654 Part of the gitlink:git[7] suite
projects / git / blob