Tyler Cipriani: pages tagged vcs

Delete Local Git Commits

Tyler Cipriani — Wed, 07 Sep 2016 17:48:08 +0000

Usually when I need to delete a commit from a local git repo I use:

git rebase -i

This starts an interactive rebase session for all commits from HEAD..@{u}. Once I’m in the git rebase editor I put a drop on the line that has the commit I want to delete and save the file. Magic. Commit gone.

drop c572357 [LOCAL] Change to be removed

# Rebase dbfcb3a..c572357 onto dbfcb3a (1 command(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

This is fine if you’re there to drive an editor for an interactive rebase.

The problem with interactivity ¶

The problem with interactive rebase is you can’t easily script it.

At work I help to maintain our “beta cluster” which is a bunch of virtual machines that is supposed to mirror the setup of production. The extent to which production is actually mirrored is debatable; however, it is true that we use the same puppet repository as production. Further, this puppet repository resides on a virtual machine within the cluster that acts as the beta cluster’s puppetmaster.

Often, to test a patch for production puppet, folks will apply patches on top of beta’s local copy of the production puppet repo.

root@puppetmaster# cd /var/lib/git/operations/puppet
root@puppetmaster# git apply --check --3way /home/thcipriani/0001-some.patch
root@puppetmaster# git am --3way /home/thcipriani/0001-some.path
root@puppetmaster# git status
On branch production
Your branch is ahead of 'origin/production' by 6 commits.
  (use "git push" to publish your local commits)

nothing to commit, working directory clean

The problem here is that a lot of these commits are old and busted.

root@puppetmaster# git log -6 --format='%h %ai'
ad4f419 2016-08-25 20:46:23 +0530
d3e9ab8 2016-09-05 15:49:48 +0100
59b7377 2016-08-30 16:21:39 -0700
5d2609e 2016-08-17 14:17:02 +0100
194ee1c 2016-07-19 13:19:54 -0600
057c9f7 2016-01-07 17:11:12 -0800

It’d be nice to have a script that periodically cleans up these patches based on some criteria. This script would have to potentially be able to remove, say, 59b7377 while leaving all the others in-tact, and it would have to do so without any access to git rebase -i or any user input.

Non-interactive commit deletion ¶

The way to do this is with git rebase --onto.

Selective quoting of man git-rebase

SYNOPSIS
    git rebase [--onto <newbase>] [<upstream>]

DESCRIPTION
    All changes made by commits in the current branch but that are not
    in <upstream> are saved to a temporary area. This is the same set
    of commits that would be shown by git log <upstream>..HEAD

    The current branch is reset to <newbase> --onto option was supplied.

    The commits that were previously saved into the temporary area are
    then reapplied to the current branch, one by one, in order.

The commits we want are all the commits surrounding 59b7377 without 59b7377.

root@puppetmaster# git log --format='%h %ai' 59b7377..HEAD
ad4f419 2016-08-25 20:46:23 +0530
d3e9ab8 2016-09-05 15:49:48 +0100
root@puppetmaster# git log --format='%h %ai' @{u}..59b7377^
5d2609e 2016-08-17 14:17:02 +0100
194ee1c 2016-07-19 13:19:54 -0600
057c9f7 2016-01-07 17:11:12 -0800

So we really want to take the commits 59b7377..HEAD and rebase them onto the commit before 59b7377 (which is 59b7377^)

root@puppetmaster# git rebase --onto 59b7377^ 59b7377
root@puppetmaster# git log -6 --format='%h %ai'
ad4f419 2016-08-25 20:46:23 +0530
d3e9ab8 2016-09-05 15:49:48 +0100
5d2609e 2016-08-17 14:17:02 +0100
194ee1c 2016-07-19 13:19:54 -0600
057c9f7 2016-01-07 17:11:12 -0800
d5e1340 2016-09-07 23:45:25 +0200

Tada! Commit gone.

(ab)Using Git Notes

Tyler Cipriani — Fri, 26 Aug 2016 22:35:15 +0000

Flickr ¶

I like to post my photos on the internet. I used to post all of my photos on Flickr, but that site has been getting worse and worse and worse. More recently, I’ve been using a static photo gallery generator I ~~wrote~~ hacked together that I (perhaps unfortunately) named hiraeth.

Hiraeth is, to put it mildly, missing some features. There are a few reasons that I opted to create hiraeth rather than use something that was already built:

I need to host my own photos now, evidently. The internet has seemingly decided to sell its user-base to the highest bidder, so hosted services are out.
I think it makes sense to host my photos statically, since they’re static (mostly).
I really like the way my ~/Pictures directory is organized—I can find stuff—and I don’t want to mess all that up to generate a crappy website out of my photos.
I use git-annex to manage my ~/Pictures, which creates…unique challenges :)

Hiraeth is invoked like: publish [edited-photo-dir] [output-dir]. Hiraeth looks for a file named _metadata.yaml inside the directory of edited photos and uses that to map photo files to photo descriptions and add titles and whatnot to the page. It makes a few different sized thumbnails of each photo, grabs the exif info, and generates some html.

Hiraeth was designed to look and behave like a static version of Flickr circa 2007. There are still features to add, but there is a base that works in place at least.

Home Pictures ¶

I mange my ~/Pictures directory using git-annex (which I’ve wanted to write something about for a long time). This is mostly amazing and great. Git-annex has a lot of cool features. For instance, in git-annex once you’ve copied files to a remote, it will allow you to “drop” a file locally to save space. You can still get the file back from the remote any time you rootin’ tootin’ feel like, so nbd. Occasionally, when I’m running out of space on one machine or another, I’ll drop a bunch of photos.

The ability to drop a bunch of photos means that hiraeth needs to be able to get photo metadata from a picture without having the file actually be on disk.

Gerrit ¶

We use gerrit at work and I genuinely like it.

<rant> The web-UI is one of the worst interfaces I’ve ever used. The web interface is an unfortunate mix of late-90s, designed-by-engineers, impossibly-option-filled interface mashed together in an unholy union with a fancy-schmancy new-fangled javascripty single-page application. It’s basically a mix of two interface paradigms I hate, yet rarely see in concert: back-button breakage + no design aesthetic whatsoever. </rant>

HOWEVER, The workflow gerrit enforces, the git features it uses, and the beautiful repository history that results makes gerrit a really nice code review system.

Gerrit is the first system I’ve seen use git-notes.

Gerrit has a cool feature where it keeps all of the patch review in git-notes:

tyler@taskmaster:mediawiki-core$ git fetch origin refs/notes/*:refs/notes/*
remote: Counting objects: 176401, done
remote: Finding sources: 100% (147886/147886)
remote: Getting sizes: 100% (1723/1723)
remote: Compressing objects: 100% (116810/116810)
remote: Total 147886 (delta 120436), reused 147854 (delta 120434)
Receiving objects: 100% (147886/147886), 14.91 MiB | 3.01 MiB/s, done.
Resolving deltas: 100% (120449/120449), done.
From ssh://gerrit.wikimedia.org:29418/mediawiki/core
 * [new ref]         refs/notes/commits -> refs/notes/commits
 * [new ref]         refs/notes/review -> refs/notes/review
tyler@taskmaster:mediawiki-core$ ls -l .git/refs/notes
total 8
-rw-r--r-- 1 tyler tyler 41 Aug 28 16:44 commits
-rw-r--r-- 1 tyler tyler 41 Aug 28 16:44 review
tyler@taskmaster:mediawiki-core$ git log --show-notes=review --author='Tyler Cipriani'
 commit ab131d4be475bf87b0f0a86fa356a2b1a188a673
 Author: Tyler Cipriani <tcipriani@wikimedia.org>
 Date:   Tue Mar 22 09:08:52 2016 -0700
 
 Revert "Add link to anon's user page; remove "Not logged in""
 
 This reverts change I049d0671a7050.
 
 This change was reverted in the wmf/1.27.0-wmf.17. Since there is no
 clear consensus, revert in master before branching wmf/1.27.0-wmf.18.
 Bug: T121793
 Change-Id: I2dc0f2562c908d4e419d34e80a64065843778f3d
 
 Notes (review):
     Verified+2: jenkins-bot
     Code-Review+2: Legoktm <legoktm.wikipedia@gmail.com>
     Submitted-by: jenkins-bot
     Submitted-at: Tue, 22 Mar 2016 18:08:27 +0000
     Reviewed-on: https://gerrit.wikimedia.org/r/278923
     Project: mediawiki/core
     Branch: refs/heads/master

This is super cool. You can have, effectively, an offline backup of lots of information you’d usually have to brave the gerrit web-ui to find. Plus, you don’t have to have this information in your local repo taking up space, it’s only there if you fetch it down.

There is another project from Google that uses git-notes for review called git-appraise.

This is the stated use of git-notes in the docs: store extra information about a commit, without changing the SHA1 of the commit by modifying its contents.

It is, however, noteworthy that you can store a note that points to any object in your repository and not just commit objects.

EXIF data without pictures ¶

After some minor testing it seems that I can store all the EXIF info I need about my images in git-notes without actually having those images on disk; i.e., I can have git-annex drop the actual files and just have broken symlinks that point to where the files live in annex.

I wrote a small bash script to play with some of these ideas.

tyler@taskmaster:Pictures$ git photo show fish.jpg
+ git notes --ref=pictures show d4a9c57715ce63a228577900d1abc027
error: No note found for object d4a9c57715ce63a228577900d1abc0273396e8ef.
tyler@taskmaster:Pictures$ git photo add fish.jpg
+ git notes --ref=pictures add -m 'FileName: fish.jpg
FileTypeExtension: jpg
Make: NIKON CORPORATION
Model: NIKON D610
LensID: AF-S Zoom-Nikkor 24-70mm f/2.8G ED
FocalLength: 62.0 mm
FNumber: 2.8
ISO: 3200' d4a9c57715ce63a228577900d1abc027
+ set +x
tyler@taskmaster:Pictures$ git photo show fish.jpg
+ git notes --ref=pictures show d4a9c57715ce63a228577900d1abc027
FileName: fish.jpg
FileTypeExtension: jpg
Make: NIKON CORPORATION
Model: NIKON D610
LensID: AF-S Zoom-Nikkor 24-70mm f/2.8G ED
FocalLength: 62.0 mm
FNumber: 2.8
ISO: 3200
+ set +x

Now it seems like it should be possible to git push origin refs/notes/pictures, fetch them on the other side, and modify hiraeth to read EXIF from notes when the symlink target doesn’t exist.

We’ll see how any of that goes in practice :

Visualizing Git’s Merkle DAG with D3.js

Tyler Cipriani — Mon, 21 Mar 2016 00:00:00 +0000

The Directed Acyclic Graph (DAG) is a concept I run into over-and-over again; which is, perhaps, somewhat ironic.

A DAG is a representation of vertices (nodes) that are connected by directional edges (arcs—i.e., lines) in such a way that there are no cycles (e.g., starting at Node A, you shouldn’t be able to return to Node A).

DAGs have lots of uses in computer science problems and in discrete mathematics. You’ll find DAGs in build-systems, network problems, and, importantly (for this blog post, if not generally) in Git.

One way to think of a DAG is as a set of dependencies—each node may have a dependency on one or more other nodes. That is, in order to get to Node B you must route through Node A, so Node B depends on Node A:

// Node → [dependent nodes]
var dag = {
    'Node A': ['Node B'],
    'Node B': []
};

The visualization of dependencies in a JSON object is (SURPRISE!) different from the input format needed to visualize a DAG using the D3.js Force layout. To change the above object into Force’s expected input, I created a little helper function:

var forceFormat = function(dag) {
    var orderedNodes = [],
        nodes = [],
        links = [],
        usesPack = false;

    // Basically a dumb Object.keys
    for (node in dag) {
        if ( !dag.hasOwnProperty( node ) ) continue;
        orderedNodes.push(node);
    }

    orderedNodes.forEach(function(node) {
        var sources = dag[node];

        if (!sources) return;

        sources.forEach(function(source) {
            var source = orderedNodes.indexOf(source);

            // If the source isn't in the Git DAG, it's in a packfile
            if (source < 0) {
                if (usesPack) return;
                source = orderedNodes.length;
                usesPack = true;
            }

            links.push({
                'source': source,
                'target': orderedNodes.indexOf(node)
            });
        });
        nodes.push({'name': node});
    });

    // Add pack file to end of list
    if (usesPack) nodes.push({'name': 'PACK'});

    return { 'nodes': nodes, 'links': links };
};

var forceInput = forceFormat(dag);

forceFormat outputs a JSON object that can be used as input for the Force layout.

{
    "links": [
        {
            "source": "Node A",
            "target": "Node B"
        }
    ],
    "nodes": [
        { "name": "Node A" },
        { "name": "Node B" }
    ]
}

I can pass this resulting JSON object off to a function that I created after a long time staring at one of mbostock’s many amazing examples to create a D3 Force graph of verticies and edges:

// http://bl.ocks.org/mbostock/1138500
var makeGraph  = function(target, graphData) {
    var target = d3.select(target),
        bounds = target.node().getBoundingClientRect(),
        fill   = d3.scale.category20(),
        radius = 25;

    var svg = target.append('svg')
        .attr('width', bounds.width)
        .attr('height', bounds.height);

    // Arrow marker for end-of-line arrow
    svg.append('defs').append('marker')
        .attr('id', 'arrowhead')
        .attr('refX', 17.5)
        .attr('refY', 2)
        .attr('markerWidth', 8)
        .attr('markerHeight', 4)
        .attr('orient', 'auto')
        .attr('fill', '#ccc')
        .append('path')
        .attr('d', 'M 0,0 V 4 L6,2 Z');

    var link = svg.selectAll('line')
        .data(graphData.links)
        .enter()
            .append('line')
                .attr('class', 'link')
                .attr('marker-end', 'url(#arrowhead)');

    // Create a group for each node
    var node = svg.selectAll('g')
        .data(graphData.nodes)
        .enter()
            .append('g');

    // Color the node based on node's git-type (otherwise, hot pink!)
    node.append('circle')
        .attr('r', radius)
        .attr('class', 'node')
        .attr('fill', function(d) {
            var blue  = '#1BA1E2',
                red   = 'tomato',
                green = '#5BB75B',
                pink  = '#FE57A1';

            if (d.name.endsWith('.b')) { return red; }
            if (d.name.endsWith('.t')) { return blue; }
            if (d.name.endsWith('.c')) { return green; }
            return pink;
        });

    node.append('text')
        .attr('y', radius * 1.5)
        .attr('text-anchor', 'middle')
        .attr('fill', '#555')
        .text(function(d) {
            if (d.name.length > 10) {
                return d.name.substring(0, 8) + '...';
            }

            return d.name;
         });

    // If the node has a type: tag it
    node.append('text')
        .attr('text-anchor', 'middle')
        .attr('y', 4)
        .attr('fill', 'white')
        .attr('class', 'bold-text')
        .text(function(d) {
            if (d.name.endsWith('.b')) { return 'BLOB'; }
            if (d.name.endsWith('.t')) { return 'TREE'; }
            if (d.name.endsWith('.c')) { return 'COMMIT'; }
            return '';
         });

    var charge = 700 * graphData.nodes.length;

    var force = d3.layout.force()
        .size([bounds.width, bounds.height])
        .nodes(graphData.nodes)
        .links(graphData.links)
        .linkDistance(150)
        .charge(-(charge))
        .gravity(1)
        .on('tick', tick);

    // No fancy animation, tick amount varies based on number of nodes
    force.start();
    for (var i = 0; i < graphData.nodes.length * 100; ++i) force.tick();
    force.stop();

    function tick(e) {
        // Push sources up and targets down to form a weak tree.
        var k = -12 * e.alpha;

        link
            .each(function(d) { d.source.y -= k, d.target.y += k; })
                .attr('x2', function(d) { return d.source.x; })
                .attr('y2', function(d) { return d.source.y; })
                .attr('x1', function(d) { return d.target.x; })
                .attr('y1', function(d) { return d.target.y; });

        node
            .attr('transform', function(d) {
                return 'translate(' + d.x + ',' + d.y + ')';
            });
    }
};
makeGraph('.merkle-1', forceInput);

You’d be forgiven for thinking that is a line.

This directional line is a DAG—albeit a simple one. Node B depends on Node A and that is the whole graph. If you want to get to Node B then you have to start at Node A. Depending on your problem-space, Node B could be many things: A place in Königsberg, a target in a Makefile (or a Rakefile), or (brace yourself) a Git object.

Git Object Anatomy ¶

In order to understand how Git is a DAG, you need to understand Git “objects”:

$ mkdir merkle
$ cd merkle
$ echo 'This is the beginning' > README
$ git init
$ git add .
$ git -m 'Initial Commit'
$ find .git/objects/ -type f
.git/objects/1b/9f426a8407ffee551ad2993c5d7d3780296353
.git/objects/09/8e6de29daf4e55f83406b49f5768df9bc7d624
.git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da

What are Git objects? Because they look like nonsense:

$ cat .git/objects/1b/9f426a8407ffee551ad2993c5d7d3780296353
xKOR02,V¢T¤̼¼̼t.

After a little digging through the Pro Git book, Git objects are a little less non-sensicle. Git objects are simply zlib compressed, formatted messages:

$ python2 -c 'import sys,zlib; \
  print zlib.decompress(sys.stdin.read());' \
    < .git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da
commit 195tree 098e6de29daf4e55f83406b49f5768df9bc7d624
author Tyler Cipriani <tcipriani@wikimedia.org> 1458604120 -0700
committer Tyler Cipriani <tcipriani@wikimedia.org> 1458604120 -0700

Initial Commit

Parts of that message are obvious: author and committer obviously come from my .gitconfig. There is a Unix epoch timestamp with a timezone offset. commit is the type of object. 195 is the byte-length of the remainder of the message.

There are a few parts of that message that aren’t immediately obvious. What is tree 098e6de29daf4e55f83406b49f5768df9bc7d624? And why would we store this message in .git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da and not .git/objects/commit-message? Is a merkle what I think it is? The answer to all of these questions and many more is the same: Cryptographic Hash Functions.

Hash Functions ¶

A cryptographic hash function is a function that when given an input of any length it creates a fixed-length output. Furthermore (and more importantly), the fixed-length output should be unique to a given input; any change in input will likely cause a big change in the output. Git uses a cryptographic hash function called Secure Hash Algorithm 1 (SHA-1).

You can play with the SHA-1 function on the command line:

$ echo 'message' | sha1sum
1133e3acf0a4cbb9d8b3bfd3f227731b8cd2650b  -
$ echo 'message' | sha1sum
1133e3acf0a4cbb9d8b3bfd3f227731b8cd2650b  -
$ echo 'message1' | sha1sum
c133514a60a4641b83b365d3dc7b715dc954e010  -

Note the big change in the output of sha1sum from a tiny change in input. This is what cryptographic hash functions do.

Hash that DAG! ¶

Now that we have some idea of what is inside a commit object, let’s reverse-engineer the commit object from the HEAD of our merkle repo:

$  python2 -c 'import sys,zlib; \
print zlib.decompress(sys.stdin.read());' \
< .git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da | \
od -c
 0000000   c   o   m   m   i   t       1   9   5  \0   t   r   e   e    
 0000020   0   9   8   e   6   d   e   2   9   d   a   f   4   e   5   5
 0000040   f   8   3   4   0   6   b   4   9   f   5   7   6   8   d   f
 0000060   9   b   c   7   d   6   2   4  \n   a   u   t   h   o   r    
 0000100   T   y   l   e   r       C   i   p   r   i   a   n   i       <
 0000120   t   c   i   p   r   i   a   n   i   @   w   i   k   i   m   e
 0000140   d   i   a   .   o   r   g   >       1   4   5   8   6   0   4
 0000160   1   2   0       -   0   7   0   0  \n   c   o   m   m   i   t
 0000200   t   e   r       T   y   l   e   r       C   i   p   r   i   a
 0000220   n   i       <   t   c   i   p   r   i   a   n   i   @   w   i
 0000240   k   i   m   e   d   i   a   .   o   r   g   >       1   4   5
 0000260   8   6   0   4   1   2   0       -   0   7   0   0  \n  \n   I
 0000300   n   i   t   i   a   l       C   o   m   m   i   t  \n  \n
 0000317

$ printf 'tree 098e6de29daf4e55f83406b49f5768df9bc7d62k4\n' >> commit-msg
$ printf 'author Tyler Cipriani <tcipriani@wikimedia.org> 1458604120 -0700\n' >> commit-msg
$ printf 'committer Tyler Cipriani <tcipriani@wikimedia.org> 1458604120 -0700\n' >> commit-msg
$ printf '\nInitial Commit\n' >> commit-msg

$ sha1sum <(cat \
    <(printf "commit ") \
    <(wc -c < commit-msg | tr -d '\n') \
    <(printf '%b' '\0') commit-msg)
1a06ce381ac14f7a5baa1670691c2ff8a73aa6da  /dev/fd/63

Hmm… that seems familiar

$ export COMMIT_HASH=$(sha1sum <(cat <(printf "commit ") <(wc -c < commit-msg | tr -d '\n') <(printf '%b' '\0') commit-msg) | cut -d' ' -f1)
$ find ".git/objects/${COMMIT_HASH:0:2}" -type f -name "${COMMIT_HASH:(-38)}"
.git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da

The commit object is a zlib-compressed, formatted message that is stored in a file named after the SHA-1 hash of the file’s un-zlib compressed contents.

(/me wipes brow)

Let’s use git-cat-file to see if we can explore the tree 098e6de29daf4e55f83406b49f5768df9bc7d62k4-part of the commit message object:

$ cat .git/HEAD
ref: refs/heads/master

$ cat .git/refs/heads/master
1a06ce381ac14f7a5baa1670691c2ff8a73aa6da

$ git cat-file -p 1a06ce381ac14f7a5baa1670691c2ff8a73aa6da
tree 098e6de29daf4e55f83406b49f5768df9bc7d624
author Tyler Cipriani <tcipriani@wikimedia.org> 1458604120 -0700
committer Tyler Cipriani <tcipriani@wikimedia.org> 1458604120 -0700

$ git cat-file -p 098e6de29daf4e55f83406b49f5768df9bc7d624
100644 blob 1b9f426a8407ffee551ad2993c5d7d3780296353    README

$ git cat-file -p 1b9f426a8407ffee551ad2993c5d7d3780296353
This is the beginning

Hey that’s the text I put into README!

So .git/HEAD refers to .git/refs/heads/master, calling git-cat-file on the object found inside that file shows that it’s the commit object we recreated. The commit object points to 098e6de29daf4e55f83406b49f5768df9bc7d624, which is a tree object with the contents: 100644 blob 1b9f426a8407ffee551ad2993c5d7d3780296353 README The blob object 1b9f426a8407ffee551ad2993c5d7d3780296353 is the contents of README! So it seems each commit object points to a tree object that points to other objects.

What was I talking about? Merkle DAGs? D3.js? ¶

Let’s see if we can paste together what Git is doing at a low-level when we make a new commit:

Take the contents of README, hash the contents using SHA-1, and store as a blob object in .git/objects.
Create a directory listing of the git working directory, listing each file, with its directory permissions and its hash value. Hash this directory listing and store as a tree in .git/objects.
Take the commit message, along with info from .gitconfig and the hash of the top-level tree. Hash this information and store it as a commit object in .git/objects.

It seems that there may be a chain of dependencies:

var gitDag = {
    // blob (add .b for blob)
    '1b9f426a8407ffee551ad2993c5d7d3780296353.b': [],
    // tree (.t == tree) is a hash that includes the hash from blob
    '098e6de29daf4e55f83406b49f5768df9bc7d624.t': ['1b9f426a8407ffee551ad2993c5d7d3780296353.b'],
    // commit (.c == commit) is a hash that includes the hash from tree
    '1a06ce381ac14f7a5baa1670691c2ff8a73aa6da.c': ['098e6de29daf4e55f83406b49f5768df9bc7d624.t'],
};

makeGraph('.merkle-2', forceFormat(gitDag));

You’d be forgiven for thinking that is a line.

What’s really happening is that there is a commit object (1a06ce38) that depends on a tree object (098e6de2) that depends on a blob (1b9f426a).

Since it’s running each of these objects through a hash function and each of them contains a reference up the chain of dependencies, a minor change to either the blob or the tree will create a drastically different commit object.

Applying a cryptographic hash function on top of a graph was Ralph Merkle’s big idea. This scheme makes magic possible. Transferring verifiable and trusted information through an untrusted medium is toatz for realz possible with Ralph’s little scheme.

The idea is that if you have the root-node hash, that is, the cryptographic hash of the node that depends on all other nodes (the commit object in Git), and you obtained that root-node hash from a trusted source, you can trust all sub-nodes that stem from that root node if the hash of all those sub-root-nodes matches the root-node hash!

This is the mechanism by which things like Git, IPFS, Bitcoin, and BitTorrent are made possible: changing any one node in the graph changes all nodes that depend on that node all the way to the root-node (the commit in Git).

Tales from the Merkle Graph ¶

I wrote a simple NodeJS script that creates a graph that is suitable for input into the JavaScript that I’ve already written that will create a D3.js force graph with whatever it finds in .git/objects.

#!/usr/bin/env nodejs
/* makeDag - creates a JSON dependency graph from .git/objects */

var glob = require('glob'),
    fs = require('fs'),
    zlib = require('zlib');

var types = ['tree', 'commit', 'blob'],
    treeRegex = {
        // 100644 README\0[20 byte sha1]
        regex: /[0-9]+\s[^\0]+\0((.|\n){20})/gm,
        fn: function(sha) {
            var buf = new Buffer(sha[1], 'binary');
            return buf.toString('hex') + '.b';
        }
    },
    commitRegex = {
        // tree 098e6de29daf4e55f83406b49f5768df9bc7d624
        regex: /(tree|parent)\s([a-f0-9]{40})/gm,
        fn: function(sha) {
            if (sha[1] === 'tree') {
                return sha[2] + '.t';
            }
            return sha[2] + '.c';
        }
    },
    total = 0,
    final = {};

// determine file type, parse out SHA1s
var handleObjects = function(objData, name) {
    types.forEach(function(type) {
        var re, regex, match, key;

        if (!objData.startsWith(type)) { return; }

        key = name + '.' + type[0];
        final[key] = [];
        if (type === 'tree') { objType = treeRegex; }
        if (type === 'commit') { objType = commitRegex; }
        if (type === 'blob') { return; }

        // Remove the object-type and size from file
        objData = objData.split('\0');
        objData.shift();
        objData = objData.join('\0');

        // Recursive regex match remainder
        while ((match = objType.regex.exec(objData)) !== null) {
            final[key].push(objType.fn(match));
        }
    });

    // Don't output until you've got it all
    if (Object.keys(final).length !== total) {
        return;
    }

    // Output what ya got.
    console.log(final);
};

// Readable object names not file names
var getName = function(file) {
    var fileParts = file.split('/'),
        len = fileParts.length;
    return fileParts[len - 2] + fileParts[len - 1];
}

// Inflate the deflated git object file
var handleFile = function(file, out) {
    var name = getName(file);

    fs.readFile(file, function(e, data) {
        zlib.inflate(data, function(e, data) {
            if (e) { console.log(file, e); return; }
            handleObjects(data.toString('binary'), name);
        });
    });
};

// Sort through the gitobjects directory
var handleFiles = function(files) {
    files.forEach(function(file) {
        fs.stat(file, function(e, f) {
            if (e) { return; }
            if (f.isFile()) {
                // Don't worry about pack files for now
                if (file.indexOf('pack') > -1) { return; }
                total++;
                handleFile(file);
            }
        });

    });
};

(function() {
    glob('.git/objects/**/*', function(e, f) {
        if (e) { throw e; }
        handleFiles(f);
    });
})();

Merkle graph transformations are often difficult to describe, but easy to see. Using this last piece of code to create and view graphs for several repositories has been illuminating. The graph visualization both illuminates and challenges my understanding of Git in ways I didn’t anticipate.

The Tale of Commit Message Bike-shedding ¶

When you change your commit message, what happens to the graph? What depends on a commit? Where is the context for a commit?

$ git commit --amend -m 'This is the commit message now'
[master 585448a] This is the commit message now
 Date: Mon Mar 21 16:48:40 2016 -0700
  1 file changed, 1 insertion(+)
   create mode 100644 README
$ find .git/objects -type f
.git/objects/1b/9f426a8407ffee551ad2993c5d7d3780296353
.git/objects/09/8e6de29daf4e55f83406b49f5768df9bc7d624
.git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da
.git/objects/da/94af3a21ac7e0c875bbbe6162aa1d26d699c73

Now the DAG is a bit different:

var gitDag = { '098e6de29daf4e55f83406b49f5768df9bc7d624.t': [ '1b9f426a8407ffee551ad2993c5d7d3780296353.b' ],
  '1a06ce381ac14f7a5baa1670691c2ff8a73aa6da.c': [ '098e6de29daf4e55f83406b49f5768df9bc7d624.t' ],
  '1b9f426a8407ffee551ad2993c5d7d3780296353.b': [],
  'da94af3a21ac7e0c875bbbe6162aa1d26d699c73.c': [ '098e6de29daf4e55f83406b49f5768df9bc7d624.t' ] }

makeGraph('.merkle-3', forceFormat(gitDag));

Here we see that there are now two commit objects (1a06ce38 and da94af3a) that both depend on a single tree object (098e6de2) that depends on a single blob (1b9f426a).

One of these commit objects will never be seen with git log.

The Orphan Blob That Dared to Dream ¶

TIL: Git creates blob objects as soon as a file is added to the staging area.

$ echo 'staged' > staged
$ find .git/objects -type f
.git/objects/1b/9f426a8407ffee551ad2993c5d7d3780296353
.git/objects/09/8e6de29daf4e55f83406b49f5768df9bc7d624
.git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da
.git/objects/da/94af3a21ac7e0c875bbbe6162aa1d26d699c73

Notice that nothing depends on this object just yet. It’s a lonely orphan blob.

$ git add staged
$ find .git/objects -type f
.git/objects/1b/9f426a8407ffee551ad2993c5d7d3780296353
.git/objects/09/8e6de29daf4e55f83406b49f5768df9bc7d624
.git/objects/1a/06ce381ac14f7a5baa1670691c2ff8a73aa6da
.git/objects/da/94af3a21ac7e0c875bbbe6162aa1d26d699c73
.git/objects/19/d9cc8584ac2c7dcf57d2680375e80f099dc481
$ makeDag
{ '098e6de29daf4e55f83406b49f5768df9bc7d624.t': [ '1b9f426a8407ffee551ad2993c5d7d3780296353.b' ],
  '19d9cc8584ac2c7dcf57d2680375e80f099dc481.b': [],
  '1a06ce381ac14f7a5baa1670691c2ff8a73aa6da.c': [ '098e6de29daf4e55f83406b49f5768df9bc7d624.t' ],
  'da94af3a21ac7e0c875bbbe6162aa1d26d699c73.c': [ '098e6de29daf4e55f83406b49f5768df9bc7d624.t' ],
  '1b9f426a8407ffee551ad2993c5d7d3780296353.b': [] }

Even unstaging and deleting the file doesn’t remove the object. Orphan objects in git are only garbage collected as part of git gc --prune.

When this object is committed to the repo, it creates a whole new layer of the graph:

$ git commit -m 'Add staged file'
[master 4f407b3] Add staged file
 1 file changed, 1 insertion(+)
 create mode 100644 staged
$ makeDag
{ '098e6de29daf4e55f83406b49f5768df9bc7d624.t': [ '1b9f426a8407ffee551ad2993c5d7d3780296353.b' ],
  '19d9cc8584ac2c7dcf57d2680375e80f099dc481.b': [],
  '1a06ce381ac14f7a5baa1670691c2ff8a73aa6da.c': [ '098e6de29daf4e55f83406b49f5768df9bc7d624.t' ],
  '1b9f426a8407ffee551ad2993c5d7d3780296353.b': [],
  '4f407b396e6ecbb65de6cf192259c18ecd4d1e9b.c': 
   [ '7ce38101e91de29ee0fee3aa9940cc81159e0f8d.t',
     'da94af3a21ac7e0c875bbbe6162aa1d26d699c73.c' ],
  '7ce38101e91de29ee0fee3aa9940cc81159e0f8d.t': 
   [ '1b9f426a8407ffee551ad2993c5d7d3780296353.b',
     '19d9cc8584ac2c7dcf57d2680375e80f099dc481.b' ],
  'da94af3a21ac7e0c875bbbe6162aa1d26d699c73.c': [ '098e6de29daf4e55f83406b49f5768df9bc7d624.t' ] }

So we’ve created a new commit (4f407b39) that is the parent of a different commit (da94af3a) and a new tree (7ce38101) that contains our old README blob (1b9f426a) and our new, previously orphaned, blob (19d9cc85).

The Tale of Powerful Software ¶

I’ve always enjoyed the idea that software (and computer science more generally) is nothing but an abstraction to manage complexity. Good software— powerful software—like Git—is a software that manages an incredible amount of complexity and hides it completely from the user.

In recognition of this idea, I’ll leave you with the graph of my local copy of clippy—a small command line tool I created that is like man(1) except it shows Clippy at the end of the man output (yes, it’s dumb).

This should give you an idea of the complexity that is abstracted by the Git merkle graph: this repo contains 5 commits!

Integrating git with your existing website

Tyler Cipriani — Sun, 21 Oct 2012 00:00:00 +0000

Better git it in your soul*.

Git has a lot of great tutorials for getting started. There are also a number of great articles on how to use git and github for your workflow.

What I haven’t seen is an article on how to integrate git with your current site without storing any code on github. I’m writing this blog to create a quick reference for how to get up and running using git on your existing site.

Needed ¶

I’m making the assumption that you have the following:

Knowledge of Linux
A local development environment
git-core installed both locally and on your webserver
rsync installed both locally and on your webserver
Keyless SSH access to your webserver

I’m using Ubuntu 12.04 locally, but I’d assume that most of this won’t be too different on a different distribution or on a Mac—but I’m probably totally wrong about that ☺

Step One: ¶

RSync your site to your local development environment

In order to begin to develop locally (and break the old cowboy-coding habits that you’ve undoubtedly developed over the years) you need a local copy of your site.

Open your terminal emulator and cd to the directory in which you will be storing these files (i.e. cd /srv/www/tylercipriani.com/public_html)
Rsync the htdocs or public_html from your webserver into this local directory:

$ rsync -av -e "ssh -p <port>" <rsync-username>@<webserver>:/path/to/public_html/ .

The command breaks down like this:

a means “Archive”—keeps permissions, mtimes, etc the same
v means “Verbose”—increases verbosity of the command
e means “RSH”—allows you to use remote shell (same as RSH=command)
:/path/to/htdocs/—the path to you htdocs folder. The trailing / is significant—it means copy the content of the htdocs directory rather than the directory by name
. is the current directory

Step Two: ¶

Initialize git in local development environment.

This step will create a new git repository on your local machine and add all the code that you’ve rsynced in the previous step to that repo.

cd to the directory to which you previously rsynced your site and initialize a git repository by running git init
```
$ cd /srv/www/tylercipriani.com/public_html/
$ git init
```
Add the contents of the current directory to the git repository by running git add .
Commit all your newly added files to the repo by running your first commit git commit -m “First Commit”

Step Three: ¶

Setup a bare repo on your web server

You need a bare repo out on your webserver that will act as a mirror to your local development environment.

ssh into your webserver and make a new directory, I usually make it above the webroot (i.e. htdocs)
```
$ mkdir tylercipriani.com.git && cd tylercipriani.com.git
```
Once inside the new directory initialize a bare repository by using the --bare flag:
```
$ git init --bare
```
Now we can define a new post-receive hook that will be triggered whenever an update is pushed to this new bare repository. The post-receive hook can be any type of script you want, the script below is written in bash. cd into the .git/hooks directory and create a file called “post-receive”. Copy the code below into the file:
```
#!/bin/bash
GIT_WORK_TREE=/path/to/your/htdocs/direcotry git checkout -f
```
make sure that this code is executable by running chmod +x .git/hooks/post-receive

You’re Done! ¶

Push to your new repo, you beautiful command-line ninja, you!

Back on your local machine, in the webroot of your local development environment, add your bare webserver repo as your remote and push your git repo up to your server. The post-receive hook will take care of the rest!

$ git remote add web ssh://user@tylercipriani.com/home/user/tylercipriani.com.git
$ git push -u origin master

By using the -u flag you’re setting the upstream which means you can just run git pull without further arguments to merge origin and master.