Tar, XKCD 1168 by Randall Monroe Licensed: CC-by-NC 2.5

I’ve never been able to trust ~/.bash_history. Regardless of my configuration, I was forever searching for that one gnarly jq filter that had somehow disappeared. Then, I found a better way.

Over the past eight years, I’ve hoarded ¾ million lines of bash history:

$ wc -l < ~/.muh_history

This accounts for every shell command I’ve run since 2016—all saved to a 102MB file: ~/.muh_history.

$ ls -lh ~/.muh_history
-rw------- 1 thcipriani thcipriani 102M Feb 27 21:23 /home/thcipriani/.muh_history

It’s a perfect, searchable archive of my work.

hist a command to search ~/.muh_history

I’ve come to rely on ~/.muh_history to take notes for me while I focus on solving problems. I trust this system, because it’s so simple.

🛟 Simple history via prompt command

To save my history, I exploit the bash PROMPT_COMMAND variable.

Bash will execute anything you assign to PROMPT_COMMAND before showing your prompt:

$ export PROMPT_COMMAND='fortune && echo 🥳'
Q:  Do you know what the death rate around here is?
A:  One per person.

You can use this variable to write the output of history 1 to a file:

$ export PROMPT_COMMAND='history 1 >> ~/.muh-history.test'
$ fortune
Caution: breathing may be hazardous to your health.
$ cat ~/.muh-history.test
6851  2024-02-25 18:44:30-0700 export PROMPT_COMMAND='history 1 >> ~/.test-history'
6852  2024-02-25 18:44:35-0700 fortune

I started doing this in 20151, but the more metadata I added—bash process ID, current directory, user—the more powerful my history became.

Today, my history file looks like this:

$ tail -1 ~/.muh-history
847008 /home/thcipriani thcipriani 2024-02-27T15:52:16-0700 echo 'Weeee!' | figlet | lolcat

Now, I can crawl these bits of metadata to look at my history in interesting ways. For example:

  • Trace the history of a single shell session.
  • Show commands within a time range.
  • Search for commands run in the current directory.

🧐 Problems and tradeoffs of ~/.muh_history

My eternal shell history has been working well for eight years. But every solution has tradeoffs and problems:

  • One machine – This works on one machine. If you need history saved across more than one machine, Atuin seems like what you want (though I’ve never tried it).
  • Local security – This stores any command you run, including those with woopsied passwords. The HISTCONTROL=ignorespace setting helps—it makes history ignore commands that start with a space.
  • Bash only – I use the default Linux shell for most operating systems: bash. Nicer shells can do the same things as PROMPT_COMMAND and history (e.g., zsh’s preexec, fish’s fish_preexec).
  • Too simplisticMcFly and Atuin seem like robust, active alternatives. And both use PROMPT_COMMAND under the hood. Maybe I’d start there if I were starting today, but ~/.muh_history pre-dates both projects.
  • Remote security – It would be Bad™ to install this on a remote machine unless you’re the admin of that machine. Never cross a sysadmin.

⚠️ Flailing at bash’s built-in history

@michaelhoffman.bsky.social, Wed, 02 Sep 2015

Goals of ~/.muh_history:

  1. Eternal – I want the option of keeping history forever.
  2. History builtin commands – “↑” and ctrl-r should work in the default way.
  3. History builtin depth – I want a few weeks of history available to ctrl-r.
  4. Instant startup – Minimize lag when starting new bash sessions.
  5. No logrotate – Avoid logrotate, systemd timers, and cronjobs to manage history—nothing to troubleshoot or maintain.

And the hacks I’ve seen to juice bash’s built-in ~/.bash_history fail at least one of these:

  • HIST(FILE)SIZE=<huge number>2 – On startup, bash loads a HISTSIZE amount of HISTFILE into memory—this could cause lag during bash startup unless you logrotate regularly. And there’s a risk of losing commands if/when your sessions crash.
  • unset HIST(FILE)SIZE/HIS(FILE)SIZE=-1 – This should make HIST(FILE)SIZE infinite, so the same caveats apply as using a huge number. Plus, this has a history of failing in some instances.
  • HISTFILE=~/.history/$(date -I) – For the first bash session you start in the morning ctrl-r will give you nothing.
  • PROMPT_COMMAND='history -a && history -r' – Before showing your prompt, write your in-memory history to HISTFILE and re-read it into memory. This mixes up the history of different sessions, so hitting “↑” may show history from a different session.

Other pitfalls:

  1. Write on exit – when you exit a session, bash dumps HISTSIZE lines of your command history into ~/.bash_history. If your session crashes: all gone. This makes HIST(FILE)SIZE unappealing to me.
  2. Append vs. overwrite – if your shell initialization files skip setting histappend, bash will overwrite ~/.bash_history rather than append your session history on exit.
  3. Small defaults – bash stores your 500 most recent commands in your session history and 500 commands from old sessions in ~/.bash_history.

I’ve set some sensible defaults and ignored clever ideas. Here are my settings:

# append to history vs. overwrite
shopt -s histappend
# ignore commands starting with space; ignore duplicates
# Up the history in memory: 500 → 10,000
# Up the history on disk: 500 → 20,000
# RFC 3339 format; e.g., 2024-02-27T15:52:16-0700

  1. When I read the wonderful, shorter version of this article: Andy Teijelo Pérez’s Bash eternal history↩︎

  2. I scraped of GitHub for HISTSIZE=. It’s all over the place. Max: 10,000,000,000,000,000; 25 percentile: 1,000; 75th percentile: 100,000; median: 10,000. I set mine at the median. Works fine.↩︎