We started using Ghost a few weeks ago, and honestly have been impressed with it so far. The backend is lightweight, easy to navigate, and very simple. The focus on writing and quality content is amazing. While there are a few features, Ghost is very good at keeping the features to the bare necessities. Remember, this is all about the content.
So when Ghost does add a feature, it's important to take notice. They obviously think it's worth incorporating into (and potentially bloating) their software. One of those really cool features they placed in their content platform is Scheduled Posts.
Ghost has the ability to set a date and time for posts to be published. In short, you can write one, two, or ten articles, schedule them to publish at a later date, then have a week worth of content automatically posting to your site at predefined times.
It's a cool feature, and Ghost made a big deal about scheduling posts, too. So when it doesn't work, the breaks jam on quickly! Unfortunately, aside from a couple of quasi-related forum posts, there is little in the way of understanding what is happening.
We start by creating a post in Ghost, like any other. After we create our lovely post, we then schedule it.
- Click on Publish
- Set the date and time to publish
- Click Schedule
The post is now scheduled, and at the predetermined date and time you selected, the post will magically exist on your blog! While that's cool and all, there's another issue which seems to plague a small percentage of people (including us).
Obviously a post can't go live in the past, so the posts will never actually publish. When digging further, it was apparent this was not working for one of several reasons. Going down all routes, eventually we addressed the issue.
If your Ghost blog is not posting your scheduled posts, there are several things I've found in the search to end this battle with scheduled posts. The most helpful by far was this:
Seriously, we can go on and on about everything you can try, but start here. It's so simple to do, and takes seconds to try. We aren't sure if this resolved the problems for us, BUT it was also the last thing we tried which seemed to address the problem. So we aren't sure if a restart would have fixed this issue to begin with, without diving down the rabbit hole, but we will never know. YOU can by simply restarting ghost to see if that resolves the issue.
Restarting Ghost didn't solve the scheduled post issue
Okay, so it wasn't as simple as a restart? While that would have been nice, it's possible there are other configuration issues at play. With our setup, we are behind a reverse proxy (nginx) which also sits behind another upstream reverse proxy (haproxy). Since the configuration was a bit more advanced, it may be worth while to check the logs Ghost is producing when trying to publish the scheduled post.
Take a look at the logs
Looking at the log files is a great place to start, after attempting the restart. Log files can indicate issues preventing operation of the site. The initial thought was the proxy was sending bad information to nginx / ghost, causing the bad calls to the API. So off to the logs.
ghost log -f
This command will work very similar to
tail -f /some/file where the output is displayed in real time as it hits the log. Running the
ghost log -f command, watch and wait for the time of a scheduled post; or, set a scheduled test post to fire off in a couple minutes (needs to be at least 2 minutes in the future). This will allow you to see exactly what's going on during the attempted scheduled posting.
After a quick look at the log, it was obvious there really was something not working correctly. The log showed something rather important.
[2019-03-14 18:35:06] ERROR NAME: InternalServerError CODE: EPROTO MESSAGE: The server has encountered an error. level: critical InternalServerError: The server has encountered an error. at new GhostError (/var/www/ghost/versions/2.16.4/core/server/lib/common/errors.js:10:26) at request.catch (/var/www/ghost/versions/2.16.4/core/server/adapters/scheduling/SchedulingDefault.js:250:30) at tryCatcher (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/promise.js:512:31) at Promise._settlePromise (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/promise.js:569:18) at Promise._settlePromise0 (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/promise.js:614:10) at Promise._settlePromises (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/promise.js:690:18) at _drainQueueStep (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/async.js:138:12) at _drainQueue (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/async.js:131:9) at Async._drainQueues (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/async.js:147:5) at Immediate.Async.drainQueues (/var/www/ghost/versions/2.16.4/node_modules/bluebird/js/release/async.js:17:14) at runCallback (timers.js:810:20) at tryOnImmediate (timers.js:768:5) at processImmediate [as _immediateCallback] (timers.js:745:5) RequestError: write EPROTO 139736587589440:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802: at ClientRequest.req.once.err (/var/www/ghost/versions/2.16.4/node_modules/got/index.js:182:22) at Object.onceWrapper (events.js:315:30) at emitOne (events.js:116:13) at ClientRequest.emit (events.js:211:7) at TLSSocket.socketErrorListener (_http_client.js:401:9) at emitOne (events.js:116:13) at TLSSocket.emit (events.js:211:7) at onwriteError (_stream_writable.js:417:12) at onwrite (_stream_writable.js:439:5) at _destroy (internal/streams/destroy.js:39:7) at TLSSocket.Socket._destroy (net.js:568:3) at TLSSocket.destroy (internal/streams/destroy.js:32:8) at WriteWrap.afterWrite [as oncomplete] (net.js:870:10) at ClientRequest.req.once.err (/var/www/ghost/versions/2.16.4/node_modules/got/index.js:182:22) at Object.onceWrapper (events.js:315:30) at emitOne (events.js:116:13) at ClientRequest.emit (events.js:211:7) at TLSSocket.socketErrorListener (_http_client.js:401:9) at emitOne (events.js:116:13) at TLSSocket.emit (events.js:211:7) at onwriteError (_stream_writable.js:417:12) at onwrite (_stream_writable.js:439:5) at _destroy (internal/streams/destroy.js:39:7) at TLSSocket.Socket._destroy (net.js:568:3) at TLSSocket.destroy (internal/streams/destroy.js:32:8) at WriteWrap.afterWrite [as oncomplete] (net.js:870:10)
If you glazed over that, no worries. The important line was:
RequestError: write EPROTO 139736587589440:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:
It appears that at some point, Ghost was failing because SSL was failing. By the looks of the error, it was a guess, but maybe something was off with our SSL settings? Pretty sure sslv3 is disabled, but at the same time, it doesn't hurt to take a look.
Testing your SSL
We need to look at the current SSL setup to make sure there's nothing grossly misconfigured. The error message looked like there was an error when attempting to make an SSL connection. Off to ssllabs.com to check what was going on.
After a quick review of the SSL on the site, it appeared that maybe we were using unsupported TLS versions. Going into our configuration, we added
no-tlsv11 to essentially force TLS 1.2. The alternative would be
force-tlsv12 which would also force TLS 1.2.
After switching our haproxy configuration to only use TLS 1.2, we tried the attempted scheduled post again.
Failure Round 2
Unfortunately, setting the reverse proxy to only use TLS 1.2 didn't work, either. Digging deeper, it looks like there may have been another issue at play. While not in the scope of this article, our haproxy setup was explicitly forcing TLS 1.0, TLS 1.1, and TLS 1.2, and ignoring our TLS 1.2 only setup.
This can happen when you have layered configurations, or multiple backend / frontend setups. Because of this, it was prudent to remove all our SSL settings, and start over with haproxy. If you are using haproxy, it may be prudent to do the same.
If you are an adopter of HTTP/2, good for you! Everyone should be using it. That said, HTTP/2 can cause issues in some older versions of node.js, in particular is fails. node version 8, the current LTS, doesn't support HTTP/2. If you upgrade to node version 10, the issues with unsupported HTTP/2 go away. There were a number of changes to HTTP/2 in node 10. It may be worthwhile to update your version of node. A great guide on d.o. on updating node is available, as well.
While I would be curious why it would be an issue, I didn't want to rule out anything. Looking at OpenSSL, we are running an older supported version. The latest LTS release is version 1.1.1, and it may be worthwhile to upgrade your OpenSSL to the latest supported version.
Upgrade OpenSSL to version 1.1.1a
In Ubuntu (the supported OS for Ghost), upgrading OpenSSL can be accomplished by the following:
- Get the tarball file:
- Unpack the tarball:
tar -zxf openssl-1.1.1a.tar.gz && cd openssl-1.1.1a
- Configure the source:
- Issue the make command:
- Install the code:
- Backup the current OpenSSL binary file:
sudo mv /usr/bin/openssl ~/tmp
- Create the symbolic link for the new OpenSSL files you installed:
sudo ln -s /usr/local/bin/openssl /usr/bin/openssl
- Update your symlinks:
- Check your OpenSSL Version:
- You should see:
OpenSSL 1.1.1a 20 Nov 2018, or whatever version you installed.
When you have done everything you think you can, and are still scratching your head, perhaps restarting Ghost is a viable option? We mentioned it before, we troubleshooted the entire stack, and came up empty; just to have everything work on a restart.
Moral of the story? Restart Ghost FIRST and save yourself the trouble of guessing later!