EDIT: 2016-08-19
I got an email a very long time ago that I meant to post about on ye old weblog from a fellow named Tom Duck who built a CMS based around some of the ideas presented here.
It’s called Bassclef and it looks pretty amazing and folks should check it out.
I used to use del.icio.us to keep track of links, then it went away. After del.icio.us shutdown, I used a ton of uniquely awful services to keep track of links. Eventually, I came around to the idea that all I needed was a series of markdown files and github: BOOM! Public link repositiory—just like del.icio.us back in the day.
After a while I started thinking, I could make these files a lot more presentable if I did some jekyll-ifying and served them out on github.
Previously Jekyllfied ¶
My linuxtips repo is just a dumb jekyll repo. Esentially all linuxtips is is just a big README file. So, for that repo, I created a gh-pages branch with a _config.yml and a _layout directory and popped in a Makefile:
INDEX = ${CURDIR}/index.md
$(INDEX):
@ git show origin/master:README.md > $@
@ perl -i -pe 'print "---\nlayout: default\ntitle: Linux Tips\n---\n\n" if $$. == 1;' $@and then I got tylercipriani.com/linuxtips; neat.
I ran into some problems with that approach along the way. Mostly problems with git and the separate branches and the order in which I’d commit and pull and whatever, it was/is a headache.
Pandoc ¶
I started thinking about Pandoc. Pandoc is this haskell library that makes miracles of text happen.
Got an org-mode file and need TeX? Done.
Got a markdown slideshow that needs to become a beamer slide show? OK, sure.
Fuck Beamer, how about markdown slides → Reveal.js slides? You bet your sweet sensual bologna.
Imma install Pandoc… ¶
Debian? ¶
then add it to your path in your .${SHELL}rc file:
OSX? ¶
Imma Use Pandoc… ¶
Alright, so I’ve got tons of markdown files, fairly structured, with bunches of links and I need html5. I’ll create a Makefile proof-of-concept for this:
Running make takes my README.md and makes this:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<title></title>
<style type="text/css">code{white-space: pre;}</style>
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body>
<h1 id="tyler-ciprianis-bookmarks">Tyler Cipriani's Bookmarks</h1>
<p>In an effort to <em>not</em> have 100+ tabs open…</p>
<ul>
<li><a href="http://www.flickr.com/photos/tylercipriani/">My Photography</a></li>
<li><a href="Design.md">Design</a></li>
<li><a href="Development.md">Development</a></li>
<li><a href="Business.md">Business</a></li>
<li><a href="Fun.md">Fun</a></li>
</ul>
</body>
</html>Title/Layout/CSS ¶
So now that I’m outputting html, I still need to be able to:
- Configure a layout in which to render html
- Include a css file in said layout
- Add post metadata to my layout (e.g., title, headline, etc.)
Pandoc is able to do all of these things—easy-peasy-lemon-squeezy.
First, to establish a layout, let’s copy the default html5 layout file for Pandoc:
I’ll make some small tweaks to that file, keep the variables I need, ditch the variables I don’t need. Here is the html5 layout file I came up with:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>$if(title-prefix)$$title-prefix$ - $endif$$pagetitle$</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="http://fonts.googleapis.com/css?family=Open+Sans:800">
$for(css)$
<link rel="stylesheet" href="$css$">
$endfor$
</head>
<body>
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<main class="container">
$body$
</main>
</body>
</html>Next, I need to figure out how to include a css stylesheet. A quick search for css in pandoc(1) turns up the --css flag which enables you to link to a css stylesheet.
Updated Makefile:
index.html: README.md
pandoc -s --template "_layout" --css "css/main.css" -f markdown -t html5 -o "$@" "$<"Finally, I need to be able to include a unique <title> tag string for each page. Again, a search through pandoc(1) for variable yields results; using the -V flag enables you to set template variables using -V [KEY]=[val].
A bit more digging in the online docs, however, nets better solution: YAML Metadata blocks—just like jekyll!
So, for each markdown file in my repo, I’ll add a block to the top that looks like this:
$pagetitle$ is the variable I defined in my _layout.html5 that I’m now passing as a template to Pandoc.
Makefile fanciness ¶
Alright, so now that I have the basics of Pandoc down, I need to whip my Makefile into shape.
First thing is I want to convert ALL of my markdown files to html, not just the README.md. So howzabout I add a wildcard target for all the html files? Also, the whole point of this is to make a github pages site, so let’s add that to the Makefile too
REPO := $(shell git config --get remote.origin.url)
GHPAGES = gh-pages
all: $(GHPAGES) $(addprefix $(GHPAGES)/, $(addsuffix .html, $(basename $(wildcard *.md))))
$(GHPAGES):
git clone "$(REPO)" "$(GHPAGES)"
(cd $(GHPAGES) && git checkout $(GHPAGES)) || (cd $(GHPAGES) && git checkout --orphan $(GHPAGES) && git rm -rf .)
$(GHPAGES)/%.html: %.md
pandoc -s --template "_layout" -c "css/main.css" -f markdown -t html5 -o "$@" "$<"Running make at this point should checkout your current git repository to a subdirectory called gh-pages (which should be added to .gitignore on master).
This Makefile first tries to checkout an existing gh-pages branch, otherwise it creates a new orphan branch for gh-pages. After that, we glob the current directory for any file name *.md and run it through pandoc, placing the result in gh-pages/[whatever].html
Niceities ¶
I’m a big fan of pre-processors, so the css/main.css file (which doesn’t actually exist as of yet) should be converted from less. The easiest way to do that: add a package.json with less as a dependency.
Now running npm install should create a new node_modules directory (which should be added to .gitignore on master). Now we need to add a lessc step to our Makefile.
LESSC = node_modules/less/bin/lessc
LESSFILE = less/main.less
CSSDIR = $(GHPAGES)/css
CSSFILE = $(CSSDIR)/main.css
$(CSSFILE): $(CSSDIR) $(LESSFILE)
$(LESSC) "$(LESSFILE)" "$(CSSFILE)"
$(CSSDIR):
mkdir -p "$(CSSDIR)"Also, it’s always nice to have a clean target in any Makefile
I’d also like to be able to preview before commiting this file by typing make serve
Finally, speaking of commiting this file, let’s make commit a target, too.
commit:
cd $(GHPAGES) && \
git add . && \
git commit --edit --message="Publish @$$(date)"
cd $(GHPAGES) && \
git push origin $(GHPAGES)Now when I update my links repo’s markdown files I issue a simple series of commands: make checks-out my gh-pages branch and builds the html and css files, make serve lets me look at the output html at localhost:8000, and, finally, make commit pushes those changes live.
So here’s the result and the final Makefile
REPO := $(shell git config --get remote.origin.url)
GHPAGES = gh-pages
LESSC = node_modules/less/bin/lessc
LESSFILE = less/main.less
CSSDIR = $(GHPAGES)/css
CSSFILE = $(CSSDIR)/main.css
all: init clean $(GHPAGES) $(CSSFILE) $(addprefix $(GHPAGES)/, $(addsuffix .html, $(basename $(wildcard *.md))))
$(GHPAGES)/%.html: %.md
pandoc -s --template "_layout" -c "css/main.css" -f markdown -t html5 -o "$@" "$<"
$(CSSFILE): $(CSSDIR) $(LESSFILE)
$(LESSC) "$(LESSFILE)" "$(CSSFILE)"
$(CSSDIR):
mkdir -p "$(CSSDIR)"
$(GHPAGES):
@echo $(REPO)
git clone "$(REPO)" "$(GHPAGES)"
@echo "Donezo"
(cd $(GHPAGES) && git checkout $(GHPAGES)) || (cd $(GHPAGES) && git checkout --orphan $(GHPAGES) && git rm -rf .)
init:
@command -v pandoc > /dev/null 2>&1 || (echo 'pandoc not found http://johnmacfarlane.net/pandoc/installing.html' && exit 1)
@[ -x $(LESSC) ] || npm install
serve:
cd $(GHPAGES) && python -m SimpleHTTPServer
clean:
rm -rf $(GHPAGES)
commit:
cd $(GHPAGES) && \
git add . && \
git commit --edit --message="Publish @$$(date)"
cd $(GHPAGES) && \
git push origin $(GHPAGES)
.PHONY: init clean commit serveEDIT: 2017-01-08
Igor Rzegocki has a project that uses the same technique outlined here to make some incredible ready-made Street FIghter MotDs – it is really cool, and if you are interested in abusing ansi escape sequences to make MotDs you should check it out :)
Everyone universally agrees that most Message of the Days (MOTDs) are stupid and suck. By the end of reading this post, your mind grapes should be swollen with the knowledge of how to make an MOTD that isn’t stupid and, some would say, doesn’t suck.
Prerequisites ¶
- Imagemagick
- OpenJDK
- coreutils
- perl
- git
This should have you covered:
Creating the Util-say file ¶
I use Util-Say to create motd messages. I started out using img2xterm, but I’ve found I get better results with Util-Say.
$ git clone https://github.com/maandree/util-say
$ cd util-say
$ ./img2ponysay -- yourimg.png > yourimg.txtYou can also try ./img2ponysay -2 -- youimg.png > yourimg.txt but I’ve never had good results with that
MOTD-ifying ¶
On CentOS and Debian, I usually just throw the ponysay file directly into /etc/motd and maybe add on some other useful info:
$ sudo cat yourimg.txt > /etc/motd
$ sudo figlet "$(hostname)" >> /etc/motd
$ sudo printf "Public IP: $(dig +short myip.opendns.com @resolver1.opendns.com)\n" >> /etc/motdOn Ubuntu Boxes (since they use update-motd(1)), I do pretty much the same thing except I just make a bash script in /etc/update-motd.d/25-baller-motd
#!/usr/bin/env bash
cat yourimg.txt
figlet "$(hostname)"
printf "Public IP: $(dig +short myip.opendns.com @resolver1.opendns.com)\n"
command -v fortune &> /dev/null && fortuneThere are likely better articles on creating a useful MOTD, (here’s one that looks kinda cool) but there are exactly none-better articles on creating MOTDs that are so flossy!
I’ve used Vagrant a fair amount in my day, and it’s great. I enjoy being able to spin-up toy linux environments to test out ideas. I tend to use the Chef provisioner with Vagrant to build-out a local environment that matches my server fairly closely.
My Ever-Evolving Rant About DevOps ¶
I’ve been thinking about Chef a lot lately. Often is the time, in moments of profound frustration, that I’ve had the thought that Chef is nothing more than a useless, leaky abstraction that separates me from something I know fairly well—Linux.
This thought is usually fleeting: Chef provides many needed abstractions that are, ultimately, much easier to grok than the underlying Linux system. Further, Chef allows you to keep a(n ostensibly) well-tested system under version control.
I’ve come to the realization that my problem with Chef is not really a problem with Chef, but a problem with Linux itself.
Linux system administration is difficult because Linux commands are non-deterministic and rely heavily on system state (e.g., installed software, permissions, network settings and availability). Maintaining a bare-metal, long-running server non-interactively using Chef sucks because any hand-tinkering via ssh is going to fuck with the “state” of the system—creating different results for subsequent chef-client runs. This system state adjustment may or may not be reflected in the chef repository (which double sucks).
Why Docker Curtails My Rage ¶
I started to think about Docker. I feel Docker addresses the problem of program state better than other currently available solutions (although, Nix is looking pretty promising as well). While Docker is still a Linux system—and, ipso facto, state-dependant—it’s also ephemeral and therefore, by not persisting changes to state, Docker has created a previously unavailable (on bare metal hardware), lightweight workaround to the problem of system state.
As is my wont, I decided today to play a bit with Docker on Vagrant and, lo-and-below, I found that the newest version of Vagrant (1.6.2, as of May 26th) can actually use docker as a provider, that is, as an alternative to VirtualBox. Using Docker as a provider means that you can run a fully-independent development enviroment, on your host machine without the overhead of VirtualBox. Neat.
“Imma setup a local development environment for Ubuntu 14.04, nginx and php-fpm using Vagrant, Supervisord and Docker,” says I.
Project Layout ¶
To keep my project directory nice and tidy, I’ve separated-out most of the files needed by the Docker provider into a Docker folder. This results in the directory structure below.
├── Docker
│ ├── Dockerfile
│ ├── nginx
│ │ └── default
│ ├── php-fpm
│ │ └── php-fpm.conf
│ └── supervisor
│ └── supervisord.conf
├── Vagrantfile
└── www
└── index.phpThe Dockerfile is used to build the main docker machine and the subfolders in the Docker directory contain configuration used in the Dockerfile.
The www folder is my fake php project folder.
VagrantFile ¶
Since docker handles so much of what was previously handled by Vagrant provisioner, the Vagrantfile for a Docker-backed Vagrant instance is pretty sparse.
In mine, I’ve got:
Dockerfile ¶
Most of the work of provisioning a container is handled by Docker and the Dockerfile. In fact, if you were only ever going to run this container on a Linux machine, I don’t think that Vagrant adds any needed functionality to the docker.io cli. In terms of portability, however, Vagrant is, at this time, a necessary evil to run docker on OSX and Windows.
FROM ubuntu:latest
MAINTAINER Tyler Cipriani, tyler@tylercipriani.com
# Download and install php, nginx, and supervisor, hey, just linux for a change!
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:nginx/stable
RUN apt-get update
RUN apt-get -y dist-upgrade
RUN apt-get install -y php5-fpm nginx supervisor
# Setup config files
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
ADD ./nginx/default /etc/nginx/sites-enabled/default
ADD ./supervisor/supervisord.conf /etc/supervisor/supervisord.conf
ADD ./php-fpm/php-fpm.conf /etc/php5/fpm/php-fpm.conf
# Shared volume
RUN mkdir -p /var/www
VOLUME ["/var/www"]
# Default command for container, start supervisor
CMD ["supervisord", "--nodaemon"]
USER root
# Expose port 80 of the container
EXPOSE 80This Dockerfile takes care of building a docker container from the latest Ubuntu image (14.04 as of May 26th, 2014). Running this code installs:
- Nginx 1.6.0
- PHP 5.5.9
- Supervisor
This config also starts supervisor with the --nodaemon flag by default. Docker can run a container running a non-daemonized program as a daemon (much like supervisor can run non-daemonized programs as daemons). Supervisor is used as a way of running both nginx and php-fpm as non-daemonized programs. It is also noteworthy that the dockerfile creates and/or modifies configuration files for php-fpm and nginx to make sure they both run in non-daemon mode.
nginx/default
server {
listen 80 default_server;
root /var/www;
index index.php index.html;
# pass the PHP scripts to FastCGI server
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_index index.php;
include fastcgi_params;
}
}php-fpm/php-fpm.conf
[global]
pid = /var/run/php5-fpm.pid
error_log = /var/log/php5-fpm.log
daemonize = no
include=/etc/php5/fpm/pool.d/*.confsupervisor/supervisord.conf
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
[program:php5-fpm]
command=/usr/sbin/php5-fpm -c /etc/php5/fpm
stdout_events_enabled=true
stderr_events_enabled=true
[program:nginx]
command=/usr/sbin/nginx
stdout_events_enabled=true
stderr_events_enabled=trueJam Time ¶
With all of our configuration in place there isn’t much left to do aside from running the vagrant instance and allowing docker to create our container.
$ sudo docker pull ubuntu # to grab the latest Ubuntu image, Vagrant should probably do this but doesn't
$ sudo vagrant up --provider=docker --debug # use debug if you don't want to sit waiting with no info for a long time on the first runWith that, you now have a container running nginx and php-fpm that is sharing a folder with you at /var/www. Navigating to http://localhost:8080/index.php should show you the contents of your ./www/index.php file.
This process is really simple AND super lightweight. I’ve been running my docker/vagrant instance for about 45 minutes alongside chrome and tmux/xterm without any noticeable jankyness on a notoriously janky laptop.