Tyler Cipriani: TIL

The Square Knot

2019-12-31T18:38:37Z

TIL how to tie my shoes 😐.

I figured this out today – the last day of 2019 – while I was tying together two ends of cotton sash cord to make a loop. I suddenly noticed my square knot didn’t look like the cover of the Boy Scouts knot book.

I looked up the square knot on animatedknots.com – which is an amazing site with step-by-step instructions for all kinds of knots. Sure enough, I had been doing wrong – left-over-right, left-over-right – a granny knot vs a square knot.

Loose Granny Knot

Loose Square/Reef Knot

This was satisfying, if a little embarassing – I somehow made it through Cub Scouts without learning to tie a square knot. As I kept reading animatedknots.com I learned something else:

The Square (Reef) Knot […] is usually learned when we tie the laces on our first pair of shoes. Admittedly it is usually a bow that we tie – but the underlying knot is a Square (Reef) Knot.

I quickly untied and retied my shoes – left-over-right, left-over-right – damnit – a granny knot. It’s especially frustrating given that I’d seen Ian’s Shoelace Site’s page on Granny Knots forever ago and dismissed it because my shoelace bows weren’t askew.

So here I am – a living cautionary tale. It’s the last day of 2019. I’m 33. And I just learned to tie my shoes 🌠.

Git Annex Metadata Filtered Views

2017-10-12T03:25:12Z

Git annex goes mindbogglingly deep.

I use git annex to manage my photos – I have tons of RAW photos in ~/Pictures that are backed-up to a few git annex special remotes.

Whenever I'm done editing a set of raw files I go through this song and dance:

cd raw
git annex add
git annex copy --to=s3
git annex copy --to=nas
git commit -m 'Added a bunch of files to s3'
git push origin master
git push origin git-annex
git annex drop

And magically my raw files are available on s3 and my home NAS.

If I jump over to a different machine I can run:

cd ~/Pictures
git fetch
git rebase
git annex get [new-raw-file]

Now that raw file is available on my new machine, I can open it in Darktable, I can do whatever I want to it: it's just a file.

This is a pretty powerful extension of git.

While I was reading the git annex internals page today I stumbled across an even more powerful feature: metadata. You can store and retrieve arbitrary metadata about any git annex file. For instance, if I wanted to store EXIF info for a particular file, I could do:

git annex metadata 20140118-BlazeyAndTyler.jpg --set exif="$(exiftool -S \
  -filename \
  -filetypeextensions \
  -make \
  -model \
  -lensid \
  -focallength \
  -fnumber \
  -iso 20140118-BlazeyAndTyler.jpg)"

And I can drop that file and still retrieve the EXIF data

$ git annex drop 20140118-BlazeyAndTyler.jpg
drop 20140118-BlazeyAndTyler.jpg (checking tylercipriani-raw...) (checking tylercipriani-raw...) (checking tylercipriani-raw...) ok
(recording state in git...)
$ git annex metadata --get exif !$
git annex metadata --get exif 20140118-BlazeyAndTyler.jpg
FileName: 20140118-BlazeyAndTyler.jpg
Make: SAMSUNG
Model: SPH-L720
FocalLength: 4.2 mm
FNumber: 2.2
ISO: 125

This is pretty neat, but it can also be achieved with git notes so it's nothing too spectacular.

But git annex metadata doesn't quite stop there.

Random background ¶

My picture directory is laid out like this:

Pictures/
└── 2015
    └── 2015-08-14_Project-name
        ├── bin
        │   └── convert-and-resize.sh
        ├── edit
        │   ├── 2015-08-14_Project-name_00001.jpg
        │   └── 2015-08-14_Project-name_00002.jpg
        └── raw
            └── 2015-08-14_Project-name_00001.NEF

I have directories for each year, under those directories I create directories that are prefixed with the ISO 8601 import date for the photo, some memorable project name (like mom-birthday or rmnp-hike). Inside that directory I have 2 directories: raw and edit. Inside each one of those directories, I have photos that are named with ISO 8601 import date, project name, and 5-digit import number and a file extension – raw files go in raw and edited/finished files edit.

I got this system from Riley Brandt (I can't recommend the Open Source Photography Course enough – it's amazing!) and it's served me well. I can find stuff! But git annex really expands the possibilities of this system.

Fictional, real-world, totally real actually happening scenario ¶

I go to Rocky Mountain National Park (RMNP) multiple times per year. I've taken a lot of photos there. If I take a trip there in October I will generally import those photos in October and create 2015/2015-10-05_RMNP-hike/{raw,edit}, and then if I go there again next March I'd create 2016/2016-03-21_RMNP-daytrip-with-blazey/{raw,edit}. So if I want to preview my RMNP edited photos from October I'd go:

cd 2015/2015-10-05_RMNP-hike/edit
git annex get
geeqie .

But what happens if I want to see all the photos I've ever taken in RMNP? I could probably cook up some script to do this. Off the top of my head I could do something like find . -iname '*rmnp*' -type l, but that would undoubtedly miss some files from a project in RMNP that I didn't name with the string rmnp. Git annex gives me a different option: metadata tags.

Metadata Tags ¶

Git annex supports a special type of short metadata – --tag. With --tag, you can tag individual files in your repo.

The WMF reading team offsite in 2016 was partially in RMNP, but I didn't name any photos RMNP because that wasn't the most memorable bit of information about those photos (reading-team-offsite seemed like a better project name) nor did RMNP represent all the photos from the offsite. I should tag a few of those photos rmnp with git annex:

$ cd ./2016/2016-05-01_wikimedia-reading-offsite/edit/
$ git annex metadata --tag rmnp Elk.jpg
metadata Elk.jpg 
  lastchanged=2016-09-29@04-14-44
  tag=rmnp
  tag-lastchanged=2016-09-29@04-14-44
ok
(recording state in git...)
$ git annex metadata --tag rmnp Reading\ folks\ bing\ higher\ up\ than\ it\ looks.jpg
metadata Reading folks bing higher up than it looks.jpg 
  lastchanged=2016-09-29@04-14-57
  tag=rmnp
  tag-lastchanged=2016-09-29@04-14-57
ok
(recording state in git...)

Also, when my old roommate came to town we went to RMNP, but I tagged those photos cody-family-adventure-time. So let's tag a few of those rmnp, too:

$ cd 2015/2016-01-25_cody-family-adventuretime/edit
$ git annex metadata --tag rmnp alberta-falls.jpg
metadata alberta-falls.jpg 
  lastchanged=2016-09-29@04-17-48
  tag=rmnp
  tag-lastchanged=2016-09-29@04-17-48
ok
(recording state in git...)

Metadata views ¶

Now the thing that was really surprising to me, you can filter the whole pictures directory based on a particular tag with git annex by using a metadata driven view.

$ tree -d -L 1
.
├── 2011
├── 2012
├── 2013
├── 2014
├── 2015
├── 2016
├── instagram
├── lib
├── lossy
├── nasa
└── Webcam

$ git annex view tag=rmnp
view  (searching...) 
Switched to branch 'views/(tag=rmnp)'
ok
$ ls
alberta-falls_%2015%2016-01-25_cody-family-adventuretime%edit%.jpg
Elk_%2016%2016-05-01_wikimedia-reading-offsite%edit%.jpg
Reading folks bing higher up than it looks_%2016%2016-05-01_wikimedia-reading-offsite%edit%.jpg

I can even filter this view using other tags with git annex vfilter tag=whatever. And I can continue to edit, refine, and work with the photo files from there.

This feature absolutely blew my mind – I dropped what I was doing to write this – I'm trying to think of a good way to work it into my photo workflow

The 14KB in the TCP Initial Window

2017-02-14T15:11:05Z

Today I went to coffee with a buddy of mine from $DAY_JOB - 2. During coffee we were discussing all the folk theories of “fast” websites. I made some off-the-cuff remark that I didn’t really understand – that you should try to squeeze the most important bits of your website into the first X bytes (X being a value I didn’t remember) for an opaque reason that has to do with TCP congestion algorithms that I’ve never understood.

We spent a good chunk of time looking at Wireshark captures of an HTTP request. It got muddled and we got off-track because we were looking at an HTTPS request and we started talking about TLS1.2, CLIENT-HELLO, and the TLS handshake in general.

I eventually found the source for what I was yammering about.

Due to how TCP estimates the capacity of a connection (i.e. TCP Slow Start), a new TCP connection cannot immediately use the full available bandwidth between the client and the server. Because of this, the server can send up to 10 TCP packets on a new connection (~14KB) in first roundtrip, and then it must wait for client to acknowledge this data before it can grow its congestion window and proceed to deliver more data.

After I got home and ate some ice cream, I did some more reading trying to understand this stuff. I think a lot of what was confusing about the Wireshark stuff we were looking at is it also had TLS1.2 mixed in there. Requesting a plain-old HTTP, port 80 site made looking at packets a bit more sane.

It seems like the size of a TCP packet is negotiated during the SYN, SYN-ACK phase of the TCP three-way handshake. In the Options section of the TCP segment (for a segment that has the SYN-flag set in the TCP segment header) a sender or receiver may indicate a Max Segment Size (MSS). An MSS is derived from a machine’s Maximum Transmission Unit (MTU). In the case of my machine the MTU is set to 1500. I found this via the ip command:

tyler@magneto:~$ ip -4 link show dev wlp3s0
2: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000

The MSS is derived from the MTU and is (at most?) the MTU minus the 40 bytes used for the TCP header and the IP header, so my machine sent the MSS size of 1460 to the server in the Options section of the SYN TCP segment that it initially sent my webserver.

So the data portion of a TCP segment sent in this particular connection should never exceed 1460 bytes.

While we were looking at Wireshark packet captures we kept noticing a TCP segment size of 1486 in the “length” column with a TCP segment data length of 1432. I noticed that my webserver sent an MSS size of 1432, which I suppose became the limiting factor in how much data could be sent per segment (rather than the 1460 MSS sent by my client – the browser):

Each TCP segment had a header-size of 20 bytes, so that leaves 34 unexplained bytes which (I guess) were part of the IP header.

So why try to squeeze a website into ~14KB? Seems like you should be trying to squeeze it into 1432 bytes.

I found my answer in IETF RFC 6928 – a proposal to increase the TCP Initial Window to 10 segments. The Initial Congestion Window (cwnd) is the maximum number of bytes that a server can send without receiving any acknowledgement that the bytes were received via a client ACK packet. This maximum number of bytes to send is calculated by multiplying some number by MSS. As of RFC 6928 some number is equal to 10 (initially).

I checked the value of cwnd on some established connections on my server:

tyler@magneto:~$ sudo ss -ti
    ...cwnd:10...

And it is indeed set to 10.

This means that my server can send cwnd * MSS bytes of information before the client has to acknowledge that anything has been receieved. So, in the case of the Wireshark connection I recorded, my webserver can send 1432 * 10 bytes, or 14320 bytes:

tyler@magneto:~$ units
Currency exchange rates from www.timegenie.com on 2016-06-22
202926 units, 109 prefixes, 88 nonlinear units

You have: 14320 bytes
You want: kilobytes
        * 14.32
        / 0.069832402

OR around 14KB of data can be sent to a browser before another round-trip from the client to the webserver has to happen.

So, overall, a nice, relaxing coffee and catch-up session with an old coworker.

Git-Config Colors and Include

2017-02-14T15:11:05Z

While I was looking through my ~/.gitconfig today, I learned some new tricks.

I've had the color section in my ~/.gitconfig without really understanding it:

[color]
    ui = true

First, it seems that git config color.ui true is superfluous as it has been set to true by default since Git 1.8.4. More importantly, I learned that you can customize the colors of git-branch, git-diff, git-status, git log --decorate, git-grep, and git {add,clean} --interactive

The easiest way I could find to view what properties are available to colorize is to search for <slot> on the git-config man page.

The other neat thing I learned is that you can include other config files by adding an include section in your ~/.gitconfig:

[include]
    path = ~/.config/git/colors.config

By combining these two tricks, you can create one colorscheme for a terminal that uses the Tomorrow Theme and one for the terminal that uses Solarized and swap between them easily by swapping the symlink pointed at by ~/.config/git/color.config

# Tomorrow Night Eighties in ~/.config/git/tomorrow-night-eighties.config
# Included in ~/.gitconfig via:
# [include]
#     path = ~tyler/.config/git/tomorrow-night-eighties.config
[color "status"]
    header = "#999999"
    added = "#99cc99"
    changed = "#f2777a"
    untracked = "#ffcc66"
    branch = "#2d2d2d" "#6699cc"
    # Because the phrase "Detached HEAD" isn't unnerving enough
    nobranch = bold ul blink "#f99157"
[color "diff"]
    meta = "#515151"
    func = "#cc99cc"
    frag = "#66cccc"
    context = "#999999"
    old = "#f2777a" "#393939"
    new = "#bef2be" "#515151"

I wish the symlink swap didn't have to happen – I wish you could just use environment variables.

As far as I can tell the only variables that are expanded in ~/.gitconfig are ~ to point at the current user's home directory and ~[user] points to [user]'s home directory.

include also fails silently by design.

I feel like there are some other neat tricks to uncover here, but I haven't quite grokked this to fullness just yet.

ImageMagick to Extract a PDF Page

2017-02-14T15:11:05Z

Today I was working with a PDF that was 27,000 pages long (god help me).

I never tried to open that PDF. This is just the page count I got from a python script I wrote to parse that PDF to a CSV file.

When the time came to spot-check the results of that python script I needed to compare some pages deep within the PDF with the output on the CSV file.

I use the Zathura document viewer to view PDFs, but I was reasonably certain that it would choke on such a large document. Instead I extracted one page at a time using ImageMagick.

convert 'big.pdf[2000]' big-pg2000.pdf

Then I opened the generated PDF using Zathura.

zathura big-pg2000.pdf

I was able to compare that side-by-side with the generated CSV.

Easy peasy.

Git Revision Colon Slash

2017-02-14T15:11:05Z

From gitrevisions(7)

SPECIFYING REVISIONS

    :/<text>, e.g. :/fix nasty bug
        A colon, followed by a slash, followed by a text, names a commit whose
        commit message matches the specified regular expression. This name
        returns the youngest matching commit which is reachable from any ref.
        The regular expression can match any part of the commit message. To
        match messages starting with a string, one can use e.g.  :/^foo. The
        special sequence :/! is reserved for modifiers to what is matched.
        :/!-foo performs a negative match, while :/!!foo matches a literal !
        character, followed by foo. Any other sequence beginning with :/! is
        reserved for now.

This is the same revision that is returned by

git log --grep '<regex>' -1

Where it seems awesome is in combination with git revert or git commit --fixup

I could see running this in a panic:

git revert --no-edit ':/that thing that broke'
git commit
git push origin HEAD:refs/for/master

Found via a comment on hackernews

sensible-editor

2017-02-14T15:11:05Z

sensible-editor is a shell script in Debian at /usr/bin/sensible-editor that selects a sensible editor.

See also: sensible-{browser,editor,pager}