First, I apologise for not noticing that the comments had been broken for a while. This was entirely my fault and not fault of ISSO, which I’m still super happy with as a self-hosted comments system. So in this post I’m going to describe what went wrong, and also how I made the system a little more resilient at the same time.

First, what did go wrong?

My web server is using FreeBSD as its OS, with a bunch of software installed via FreeBSD’s ports system. For those not that familiar with FreeBSD’s ports, the system essentially acts like a rolling distribution. As a result, you sometimes have to upgrade tools, especially languages like Perl, Ruby, and in this case, Python. A little while ago, the default Python version on FreeBSD was upgraded from Python 3.7 to Python 3.8, and I eventually followed along with that upgrade. ISSO is run out of a virtualenv as a regular user and the virtualenv was still using Python 3.7, but I decided I didn’t want to keep multiple Python versions on this machine. So, I upgraded the version in the virtualenv to 3.8 as well. So far, so good, especially as ISSO seemed to restart without issue.

Muggins here didn’t look into ISSO’s log file for a while so I didn’t notice that there were several records in there complaining that the code that is executed when a new comment is posted was triggering exceptions because it couldn’t find cgi.escape(). Which was correct for Python 3.8 because this function had been deprecated in the past and has been removed in Python 3.8. In newer versions of Python, you’re supposed to use html.escape() instead. Unfortunately I was running a slightly out of date version of ISSO - 0.10.6, I believe - that didn’t yet include this fix.

pip3 to the rescue, right? Well, not so fast - for some reason, the first attempt at upgrading ISSO to the latest version failed with a build error in cffi. Which was a bit odd to me because the system has a global Python 3.8 cffi package installed, again via the ports system. I’m a bit of a Python dabbler so instead of trying to track down the underlying issue, I simply installed cffi in the virtualenv as well, which allowed me to upgrade ISSO to the latest version. This, finally, unborked the posting of new comments. As mentioned, this was an entirely preventable failure because I didn’t check the logs for a while after an upgrade. Of course we all have excuses, but I really should’ve checked sooner.

Anyway, after installing cffi in the virtualenv, the comments were working fine again.

Making ISSO’s comment notifications more resilient

Which brings us to the second part of the work on configuring ISSO, making the notifications more resilient. I have comment moderation turned on, so ISSO sends me an email with a delete and an approve link every time someone leaves a comment. For that to work, I had to configure SMTP for ISSO, and I went for the simplest approach - just send it to my existing email server. Thus, the basic configuration for SMTP I had employed looked something like this:

[smtp]
host = <my email server>
port = 587
security = starttls
to = <my approval email>
from = "ISSO" <isso user's email>
timeout = 10
username = <email server authentication user>
password = <email server authentication password>

The above works fine until I have to take down the mail server for maintenance, which is exactly what had to happen over this weekend. ISSO - correctly, in my opinion - uses a simple SMTP client and expects the server it submits the email to to, well, be there. Which it is, until I have to shut it down for maintenance. And yes, I do run my own email server and have done so for over 20 years.

Anyway, queuing up email is really an MTA’s job, not a client’s job. So the next question is, where would I find such an MTA, especially one that isn’t spelled “sendmail”, and ideally simple to configure. After all, it really only has to do two things - deliver local email, and send the comment notification emails to my main email server using an authenticated connection. My main email server has been running Postfix for a long time, but for this simple task, Postfix seemed overkill. Enter OpenSMTPD, ie the SMTP server that is part of OpenBSD but also available as a standalone server. And most importantly, as a FreeBSD port. I’ve recently started using it for simpler tasks - for example, my secondary email server is running OpenSMTPD instead of Postfix - and I really like its simple configuration for simple setups. So OpenSMTPD it was, which gave me a convenient excuse to finally ditch the stock sendmail on the web server - as the web server really only sends local emails, I considered sendmail as a safe enough choice for an MTA but I don’t voluntarily configure it. In order to get OpenSMTPD to do what I needed it to do, I needed a configuration that looks roughly like this:

# This is the smtpd server system-wide configuration file.
# See smtpd.conf(5) for more information.

# Use the standard local aliases from the standard location
table aliases file:/etc/mail/aliases
# 'secrets' contains the usernames and passwords needed to authenticate
# receiving mail server
table secrets file:/etc/mail/secrets

# To accept external mail, replace with: listen on all
# We don't want this as this is a local MTA only
#
listen on localhost

action "local" mbox alias <aliases>
action "relay" relay
action "internal-mail" relay host smtp+tls://relay@<recipient mail server>:587 auth <secrets> tls no-verify

match for local action "local"
match mail-from "@<sender domain>" for any action "internal-mail"
match from local for any action "relay"

The above configuration is pretty simple, but really contains everything we need. If you’re used to Postfix, you can probably see why I’ve come to prefer OpenSMTPD for simple setups.

The ‘secrets’ file is pretty simple and contains a mapping of the relay host’s tag - in the case above, ‘relay’ - to a username/password combo used to authenticate. It looks like this:

relay user:password

Note that the username and password have to be unencrypted for this use case.

With the changes above, I could then rework the smtp section of my ISSO configuration to this much simpler version:

[smtp]
host = localhost
port = 25
security = none
to = <recipient email>
from = "ISSO" <ISSO specific email address>
timeout = 10

The security = none is important because otherwise, the built in SMTP client will attempt to use STARTTLS, which this server isn’t configured for as it only accepts submissions on localhost.

With this configuration in place and OpenSMTPD running, the MTA now takes care of queueing up emails. If the recipient email server is not available, OpenSMTPD will deliver them later as needed rather than having the emails go into the bit bucket. Yes, this kind of setup might be a little over the top - especially if you don’t run your own email server and use one of the big providers - but it provides some much needed resilience for when you have to occasionally shut down your email server for maintenance and upgrades.