I really want to blog more. The problem is that most of the posts to this blog are fairly long-form. This will probably change as I do blog more.

I've also moved to Pandoc for rendering posts via ikiwiki-pandoc.

In that process, I've also come to realize that pandoc handles org-mode files almost completely unambiguously—which is not actually something you get in most markup languages/markup language interpreters. BONUS: I enjoy writing in org-mode.

As is my wont, I've also created some (not-so) fancy scripts to help me:

#!/usr/bin/env bash

set -eu

POST=$*
BLOG_DIR=$(grep '^srcdir' ~/ikiwiki.setup | cut -d':' -f2 | xargs)
DATE=$(date -Ins)
BLOG_PATH=$(date +'blog/%Y/%m/%d')
EXT='.org'

SAFE_POST=$(echo "$POST" | \
    tr '[:upper:]' '[:lower:]' | \
    sed -e 's/\W/-/g')

OUT="${BLOG_DIR}/${BLOG_PATH}/${SAFE_POST}${EXT}"

mkdir -p "$(dirname "$OUT")"

{
    printf '#+TITLE: %b\n' "$POST"
    printf '\n' "$POST"
    printf '[[!meta <span class="error">Error: cannot parse date/time: &#37;b</span>]]\n\n' "$DATE"
} >> "$OUT"

# http://www.emacswiki.org/emacs/EmacsClient
exec emacsclient --no-wait --alternate-editor="" --create-frame "$OUT"

sensible-editor is a shell script in Debian at /usr/bin/sensible-editor that selects a sensible editor.

See also: sensible-{browser,editor,pager}

Usually when I need to delete a commit from a local git repo I use:

git rebase -i

This starts an interactive rebase session for all commits from HEAD..@{u}. Once I’m in the git rebase editor I put a drop on the line that has the commit I want to delete and save the file. Magic. Commit gone.

drop c572357 [LOCAL] Change to be removed

# Rebase dbfcb3a..c572357 onto dbfcb3a (1 command(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

This is fine if you’re there to drive an editor for an interactive rebase.

root@puppetmaster# cd /var/lib/git/operations/puppet
root@puppetmaster# git apply --check --3way /home/thcipriani/0001-some.patch
root@puppetmaster# git am --3way /home/thcipriani/0001-some.path
root@puppetmaster# git status
On branch production
Your branch is ahead of 'origin/production' by 6 commits.
  (use "git push" to publish your local commits)

nothing to commit, working directory clean

root@puppetmaster# git log -6 --format='%h %ai'
ad4f419 2016-08-25 20:46:23 +0530
d3e9ab8 2016-09-05 15:49:48 +0100
59b7377 2016-08-30 16:21:39 -0700
5d2609e 2016-08-17 14:17:02 +0100
194ee1c 2016-07-19 13:19:54 -0600
057c9f7 2016-01-07 17:11:12 -0800

SYNOPSIS
    git rebase [--onto <newbase>] [<upstream>]

DESCRIPTION
    All changes made by commits in the current branch but that are not
    in <upstream> are saved to a temporary area. This is the same set
    of commits that would be shown by git log <upstream>..HEAD

    The current branch is reset to <newbase> --onto option was supplied.

    The commits that were previously saved into the temporary area are
    then reapplied to the current branch, one by one, in order.

root@puppetmaster# git log --format='%h %ai' 59b7377..HEAD
ad4f419 2016-08-25 20:46:23 +0530
d3e9ab8 2016-09-05 15:49:48 +0100
root@puppetmaster# git log --format='%h %ai' @{u}..59b7377^
5d2609e 2016-08-17 14:17:02 +0100
194ee1c 2016-07-19 13:19:54 -0600
057c9f7 2016-01-07 17:11:12 -0800

root@puppetmaster# git rebase --onto 59b7377^ 59b7377
root@puppetmaster# git log -6 --format='%h %ai'
ad4f419 2016-08-25 20:46:23 +0530
d3e9ab8 2016-09-05 15:49:48 +0100
5d2609e 2016-08-17 14:17:02 +0100
194ee1c 2016-07-19 13:19:54 -0600
057c9f7 2016-01-07 17:11:12 -0800
d5e1340 2016-09-07 23:45:25 +0200

From gitrevisions(7)

SPECIFYING REVISIONS

    :/<text>, e.g. :/fix nasty bug
        A colon, followed by a slash, followed by a text, names a commit whose
        commit message matches the specified regular expression. This name
        returns the youngest matching commit which is reachable from any ref.
        The regular expression can match any part of the commit message. To
        match messages starting with a string, one can use e.g.  :/^foo. The
        special sequence :/! is reserved for modifiers to what is matched.
        :/!-foo performs a negative match, while :/!!foo matches a literal !
        character, followed by foo. Any other sequence beginning with :/! is
        reserved for now.

This is the same revision that is returned by

git log --grep '<regex>' -1

Where it seems awesome is in combination with git revert or git commit --fixup

I could see running this in a panic:

git revert --no-edit ':/that thing that broke'
git commit
git push origin HEAD:refs/for/master

Found via a comment on hackernews

Today I was working with a PDF that was 27,000 pages long (god help me).

I never tried to open that PDF. This is just the page count I got from a python script I wrote to parse that PDF to a CSV file.

When the time came to spot-check the results of that python script I needed to compare some pages deep within the PDF with the output on the CSV file.

I use the Zathura document viewer to view PDFs, but I was reasonably certain that it would choke on such a large document. Instead I extracted one page at a time using ImageMagick.

convert 'big.pdf[2000]' big-pg2000.pdf

Then I opened the generated PDF using Zathura.

zathura big-pg2000.pdf

I was able to compare that side-by-side with the generated CSV.

Easy peasy.

A jargon-less definition of code review is a difficult thing to pin- down. Code review is what happens when someone submits some code to a software project and a different person, who has knowledge about that project, checks to see if there are any problems in the submitted code before making the submitted code a part of the project.

Code review is based around the simple idea that with enough eyes all bugs are shallow.

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

– Linus’s Law, ESR, The Cathedral and the Bazaar

This is all part of a collaborative model of software development. A model in which anyone can submit code or feedback to an open project. This model is the engine that drives the development of some of the world’s largest and most successful software projects. It’s an accepted Good Thing™. It is tested. It works. As a result, almost every software project uses code review for almost every code change—big or small.

Code review is meant to ensure that only high-quality code can be merged into a project. Code review may be used for many other purposes, some good and some bad.

I know that Linode has had its share of problems recently, but it's a service I won't be leaving any time soon.

In July of 2014, as part of the Electronic Frontier Foundation's Tor Challenge I decided to build a Tor node. Not only did I decide to build a Tor node, I decided to build an exit node. Running an exit node means that instead of passing traffic to another node on the Tor network, my node will be where Tor traffic exits the network, so it'll be my IP address that any end-points see as the source of traffic.

When I decided to run an exit node, I was using Comcast as my ISP—running a Tor exit node is definitely against Comcast's Terms of Service (TOS). By this point, I had been using Linode for many years to host a bunch of personal web projects, and I had no complaints. On a whim I perused the Linode TOS:

Linode does not prohibit the use of distributed, peer to peer network services such as Tor, nor does Linode routinely monitor the network communications of customer Linodes as a normal business practice.

So with that, I created intothetylerzone (because naming things is hard). It's a fast (1MB/s), stable, exit node supporting exits on a number of ports:

ExitPolicy accept *:22  # ssh
ExitPolicy accept *:465 # smtps (SMTP over SSL)
ExitPolicy accept *:993 # imaps (IMAP over SSL)
ExitPolicy accept *:994 # ircs (IRC over SSL)
ExitPolicy accept *:995 # pop3s (POP3 over SSL)
ExitPolicy accept *:5222 # xmpp
ExitPolicy accept *:6660-6697 # allow irc ports, very widely
ExitPolicy accept *:443 # http is dead
ExitPolicy accept *:80 # but not *that* dead
ExitPolicy reject *:* # no other exits allowed

My Tor exit node is sometimes abused. There are fewer abuse-reports than I would have expected—I probably get 1 report every 3 or 4 months. I just handled an abuse report today from Linode (heavily edited):

We have received a report of malicious activity originating from your Linode. It appears that your Linode is being used to scan hosts. We ask that you investigate this matter as soon as you are able.

There was a follow-up message on the ticket with the specific IPs that were being scanned. I blocked any exit to those targeted IPs in my /etc/tor/torrc and replied to the ticket (most of this language comes from the operator-tools exit notice html):

This router is part of the Tor Anonymity Network, which is dedicated to providing people with anonymity who need it most: average computer users. This router IP should be generating no other traffic.

While Tor is not designed for malicious computer users, it is inevitable that some may use the network for malicious ends. In the mind of this operator, the social need for easily accessible censorship-resistant anonymous communication trumps the risk. Tor sees use by many important segments of the population, including whistle blowers, journalists, Chinese dissidents skirting the Great Firewall and oppressive censorship, abuse victims, stalker targets, the US military, and law enforcement, just to name a few.

I've updated my /etc/tor/torrc to include all ports on the destination IP:

ExitPolicy reject [targeted-ip]:*

I've also restarted the tor service. The destination IP should no longer receive any traffic from this machine.

Not 5 minutes later this reply came:

Hi,

Thank you for your response. At this time, it looks like we can consider this case resolved. We will continue to monitor for additional complaints and let you know should we receive more.

I’ve set this ticket to close automatically in 48 hours, so there’s no need to respond since this case is now resolved.

Let us know if there's anything else we can do for you, and we'll be happy to help.

<3 Linode.

Without Linode I would be unable to host a stable and fast node. Linode supports online freedom, so I support Linode.

While I was looking through my ~/.gitconfig today, I learned some new tricks.

I've had the color section in my ~/.gitconfig without really understanding it:

[color]
    ui = true

First, it seems that git config color.ui true is superfluous as it has been set to true by default since Git 1.8.4. More importantly, I learned that you can customize the colors of git-branch, git-diff, git-status, git log --decorate, git-grep, and git {add,clean} --interactive

The easiest way I could find to view what properties are available to colorize is to search for <slot> on the git-config man page.

The other neat thing I learned is that you can include other config files by adding an include section in your ~/.gitconfig:

[include]
    path = ~/.config/git/colors.config

By combining these two tricks, you can create one colorscheme for a terminal that uses the Tomorrow Theme and one for the terminal that uses Solarized and swap between them easily by swapping the symlink pointed at by ~/.config/git/color.config

# Tomorrow Night Eighties in ~/.config/git/tomorrow-night-eighties.config
# Included in ~/.gitconfig via:
# [include]
#     path = ~tyler/.config/git/tomorrow-night-eighties.config
[color "status"]
    header = "#999999"
    added = "#99cc99"
    changed = "#f2777a"
    untracked = "#ffcc66"
    branch = "#2d2d2d" "#6699cc"
    # Because the phrase "Detached HEAD" isn't unnerving enough
    nobranch = bold ul blink "#f99157"
[color "diff"]
    meta = "#515151"
    func = "#cc99cc"
    frag = "#66cccc"
    context = "#999999"
    old = "#f2777a" "#393939"
    new = "#bef2be" "#515151"

I wish the symlink swap didn't have to happen – I wish you could just use environment variables.

As far as I can tell the only variables that are expanded in ~/.gitconfig are ~ to point at the current user's home directory and ~[user] points to [user]'s home directory.

include also fails silently by design.

I feel like there are some other neat tricks to uncover here, but I haven't quite grokked this to fullness just yet.

Today I went to coffee with a buddy of mine from $DAY_JOB - 2. During coffee we were discussing all the folk theories of “fast” websites. I made some off-the-cuff remark that I didn’t really understand – that you should try to squeeze the most important bits of your website into the first X bytes (X being a value I didn’t remember) for an opaque reason that has to do with TCP congestion algorithms that I’ve never understood.

We spent a good chunk of time looking at Wireshark captures of an HTTP request. It got muddled and we got off-track because we were looking at an HTTPS request and we started talking about TLS1.2, CLIENT-HELLO, and the TLS handshake in general.

I eventually found the source for what I was yammering about.

Due to how TCP estimates the capacity of a connection (i.e. TCP Slow Start), a new TCP connection cannot immediately use the full available bandwidth between the client and the server. Because of this, the server can send up to 10 TCP packets on a new connection (~14KB) in first roundtrip, and then it must wait for client to acknowledge this data before it can grow its congestion window and proceed to deliver more data.

After I got home and ate some ice cream, I did some more reading trying to understand this stuff. I think a lot of what was confusing about the Wireshark stuff we were looking at is it also had TLS1.2 mixed in there. Requesting a plain-old HTTP, port 80 site made looking at packets a bit more sane.

It seems like the size of a TCP packet is negotiated during the SYN, SYN-ACK phase of the TCP three-way handshake. In the Options section of the TCP segment (for a segment that has the SYN-flag set in the TCP segment header) a sender or receiver may indicate a Max Segment Size (MSS). An MSS is derived from a machine’s Maximum Transmission Unit (MTU). In the case of my machine the MTU is set to 1500. I found this via the ip command:

tyler@magneto:~$ ip -4 link show dev wlp3s0
2: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000

The MSS is derived from the MTU and is (at most?) the MTU minus the 40 bytes used for the TCP header and the IP header, so my machine sent the MSS size of 1460 to the server in the Options section of the SYN TCP segment that it initially sent my webserver.

So the data portion of a TCP segment sent in this particular connection should never exceed 1460 bytes.

While we were looking at Wireshark packet captures we kept noticing a TCP segment size of 1486 in the “length” column with a TCP segment data length of 1432. I noticed that my webserver sent an MSS size of 1432, which I suppose became the limiting factor in how much data could be sent per segment (rather than the 1460 MSS sent by my client – the browser):

Each TCP segment had a header-size of 20 bytes, so that leaves 34 unexplained bytes which (I guess) were part of the IP header.

So why try to squeeze a website into ~14KB? Seems like you should be trying to squeeze it into 1432 bytes.

I found my answer in IETF RFC 6928 – a proposal to increase the TCP Initial Window to 10 segments. The Initial Congestion Window (cwnd) is the maximum number of bytes that a server can send without receiving any acknowledgement that the bytes were received via a client ACK packet. This maximum number of bytes to send is calculated by multiplying some number by MSS. As of RFC 6928 some number is equal to 10 (initially).

I checked the value of cwnd on some established connections on my server:

tyler@magneto:~$ sudo ss -ti
    ...cwnd:10...

And it is indeed set to 10.

This means that my server can send cwnd * MSS bytes of information before the client has to acknowledge that anything has been receieved. So, in the case of the Wireshark connection I recorded, my webserver can send 1432 * 10 bytes, or 14320 bytes:

tyler@magneto:~$ units
Currency exchange rates from www.timegenie.com on 2016-06-22
202926 units, 109 prefixes, 88 nonlinear units

You have: 14320 bytes
You want: kilobytes
        * 14.32
        / 0.069832402

OR around 14KB of data can be sent to a browser before another round-trip from the client to the webserver has to happen.

So, overall, a nice, relaxing coffee and catch-up session with an old coworker.

Git annex goes mindbogglingly deep.

I use git annex to manage my photos – I have tons of RAW photos in ~/Pictures that are backed-up to a few git annex special remotes.

Whenever I'm done editing a set of raw files I go through this song and dance:

cd raw
git annex add
git annex copy --to=s3
git annex copy --to=nas
git commit -m 'Added a bunch of files to s3'
git push origin master
git push origin git-annex
git annex drop

And magically my raw files are available on s3 and my home NAS.

If I jump over to a different machine I can run:

cd ~/Pictures
git fetch
git rebase
git annex get [new-raw-file]

Now that raw file is available on my new machine, I can open it in Darktable, I can do whatever I want to it: it's just a file.

This is a pretty powerful extension of git.

While I was reading the git annex internals page today I stumbled across an even more powerful feature: metadata. You can store and retrieve arbitrary metadata about any git annex file. For instance, if I wanted to store EXIF info for a particular file, I could do:

git annex metadata 20140118-BlazeyAndTyler.jpg --set exif="$(exiftool -S \
  -filename \
  -filetypeextensions \
  -make \
  -model \
  -lensid \
  -focallength \
  -fnumber \
  -iso 20140118-BlazeyAndTyler.jpg)"

And I can drop that file and still retrieve the EXIF data

$ git annex drop 20140118-BlazeyAndTyler.jpg
drop 20140118-BlazeyAndTyler.jpg (checking tylercipriani-raw...) (checking tylercipriani-raw...) (checking tylercipriani-raw...) ok
(recording state in git...)
$ git annex metadata --get exif !$
git annex metadata --get exif 20140118-BlazeyAndTyler.jpg
FileName: 20140118-BlazeyAndTyler.jpg
Make: SAMSUNG
Model: SPH-L720
FocalLength: 4.2 mm
FNumber: 2.2
ISO: 125

This is pretty neat, but it can also be achieved with git notes so it's nothing too spectacular.

But git annex metadata doesn't quite stop there.

Pictures/
└── 2015
    └── 2015-08-14_Project-name
        ├── bin
        │   └── convert-and-resize.sh
        ├── edit
        │   ├── 2015-08-14_Project-name_00001.jpg
        │   └── 2015-08-14_Project-name_00002.jpg
        └── raw
            └── 2015-08-14_Project-name_00001.NEF

cd 2015/2015-10-05_RMNP-hike/edit
git annex get
geeqie .

$ cd ./2016/2016-05-01_wikimedia-reading-offsite/edit/
$ git annex metadata --tag rmnp Elk.jpg
metadata Elk.jpg 
  lastchanged=2016-09-29@04-14-44
  tag=rmnp
  tag-lastchanged=2016-09-29@04-14-44
ok
(recording state in git...)
$ git annex metadata --tag rmnp Reading\ folks\ bing\ higher\ up\ than\ it\ looks.jpg
metadata Reading folks bing higher up than it looks.jpg 
  lastchanged=2016-09-29@04-14-57
  tag=rmnp
  tag-lastchanged=2016-09-29@04-14-57
ok
(recording state in git...)

$ cd 2015/2016-01-25_cody-family-adventuretime/edit
$ git annex metadata --tag rmnp alberta-falls.jpg
metadata alberta-falls.jpg 
  lastchanged=2016-09-29@04-17-48
  tag=rmnp
  tag-lastchanged=2016-09-29@04-17-48
ok
(recording state in git...)

$ tree -d -L 1
.
├── 2011
├── 2012
├── 2013
├── 2014
├── 2015
├── 2016
├── instagram
├── lib
├── lossy
├── nasa
└── Webcam

$ git annex view tag=rmnp
view  (searching...) 
Switched to branch 'views/(tag=rmnp)'
ok
$ ls
alberta-falls_%2015%2016-01-25_cody-family-adventuretime%edit%.jpg
Elk_%2016%2016-05-01_wikimedia-reading-offsite%edit%.jpg
Reading folks bing higher up than it looks_%2016%2016-05-01_wikimedia-reading-offsite%edit%.jpg