Emacs in the server room

I manage a lot of servers. Some are serving static content like this blog, with others running services like Nextcloud, ZNC, Shadowsocks, or Mumble. I also have one or two game servers to play with my family and friends. These are spread across two providers for cost and geographic reasons.

In addition, I have several machines running in my house, one running FreeNAS with some jails, another running Proxmox with several VMs and containers. I also have a couple smaller single board PCs like Raspberry Pis scattered around.

Needless to say, I've got a lot to keep track of. I tried using a couple methods of keeping inventory of what I had running where, the user names, IP addresses, and links, but found that none suited my needs particularly well. I also had to worry about making sure my ~/.ssh/config file was always up to date with VMs and containers I create.

I'd already been playing with the idea of using an org mode file to keep track of servers with VMs and containers, as it seemed like that would fit well with the hierarchical structure of org files.

What I came up was a system where each server location/provider gets a heading, with the machines in that location as headings under it. If the machine runs VMs or containers, I just put those as headings under the host machine.

* Scaleway
** example.com
** example.org

* Vultr
** lambda.cx

* Home
** proxmox
*** pi-hole
*** openbsd-1
** freenas
*** web-jail

Each machine gets a bullet point list of what's running on it. For services with web interfaces, I add a link with the name of the service to the list, so I can click it to open it in my browser. I also write details about services underneath their bullet points if I there's more I want to remember.

* Vultr
** example.com
   - Minecraft
     Save directory: =/home/minecraft/survival=
     Port 4587
   - [[https://example.com][nginx]]
   - [[https://example.com:8080/][znc]]
** example.org
  - Shadowsocks
  - Mumble
  - [[https://example.net][nginx]]

From there I added either an IP or Hostname properties to each heading, along with other information about the system like OS, SSH_USER, etc. This allows me to use org-mode's sparse trees to search for, say, all VMs running OpenBSD. Using org mode also allows me to manage servers like anything else in an org mode document; adding tags, TODO entries, code blocks, hyperlinks, tables, attachments, putting details in drawers, etc.

* home
** proxmox
   :PROPERTIES:
   :IP:       192.168.0.6
   :SSH_USER: dante
   :OS:       Proxmox-VE
   :END:
   - [[https://192.168.0.6:8006][WebUI]]

*** VMs
**** openindiana
     :PROPERTIES:
     :OS:       OpenIndiana
     :IP:       192.168.0.11
     :SSH_USER: dante
     :END:

**** pfsense
     :PROPERTIES:
     :IP:       192.168.0.12
     :OS:       FreeBSD
     :SSH_USER: admin
     :END:
     - [[https://192.168.0.12][WebUI]]
     - DHCP
     - DNS
     - OSPF

*** Containers
**** pihole
     :PROPERTIES:
     :IP:       192.168.0.21
     :OS:       Debian
     :SSH_USER: pi
     :END:
     - [[http://192.168.0.23/admin/][WebUI]]

**** ubuntu
     :PROPERTIES:
     :IP:       192.168.0.22
     :OS:       Ubuntu
     :SSH_USER: dante
     :END:
     - Prometheus
     - Shadowsocks
The code above rendered in Emacs
The code above in Emacs with all headings unfolded

Finally to keep my SSH config up to date, I wrote ox-ssh. A backend for the org mode export engine that lets me export my buffer as an SSH configuration file. It takes the properties from the all headings with either an IP or HOSTNAME property and turns them into entries in a configuration file. It supports every client configuration option OpenSSH has, so I can maintain my entire SSH client list from within my org mode file.

An example ox-ssh export
Property drawers are folded by default in org mode, but expanded here for context.

For completeness, I also added a variable that lets me set a header to the configuration when exporting. This lets me add options which apply to all hosts, like keyring support for MacOS.

With this new setup, I just have a single version controlled Servers.org file which I keep all the relevant information in. Whenever I change any details related to a server, I simply press Ctrl + c Ctrl + e to bring up the org export dispatch, then s to select Export to SSH config, and x to overwrite my existing ssh config file with the newly generated one.

BONUS

On reddit, user Funkmaster_Lincoln mentioned in response to this post that it would be interesting if one could press a key on the server heading to connect, without having to add the server to one's SSH configuration. I had a go at trying to write out what that might look like, and this is what I came up with.

(defun org-ssh-connect (&optional arg)
  "Connect to the host at point and open `dired'.
If ARG is non-nil, open `eshell' instead of `dired'."
  (interactive "P")
  (let* ((properties (org-entry-properties))
         (name (alist-get "ITEM" properties nil nil #'string=))
         (user (alist-get "SSH_USER" properties nil nil #'string=))
         (port (alist-get "SSH_PORT" properties nil nil #'string=))
         (host (or (alist-get "IP" properties nil nil #'string=)
                   (alist-get "HOSTNAME" properties nil nil #'string=))))
    (if host
        (let ((default-directory (format "/ssh:%s%s%s:"
                                         (if user (format "%s@" user) "")
                                         name
                                         (if port (format "#%s" port) ""))))
          (message "Connecting to %s..." name)
          (if arg
              (eshell t)
            (dired ".")))
      (user-error "Not an SSH host"))))