It's a blog

wikibase-docker, Mediawiki & Wikibase update

Today on the Wikibase Community User Group Telegram chat I noticed some people discussing issues with upgrading Mediawiki and Wikibase using the docker images provided for Wikibase.

As the wikibase-registry is currently only running Mediawiki 1.30 I should probably update it to 1.31, which is the next long term stable release.

This blog post was written as I performed the update and is yet to be proofread, so expect some typos. I hope it can help those that were chatting on Telegram today.

Starting state

Documentation

There is a small amount of documentation in the wikibase docker image README file that talks about upgrading, but this simply tells you to run update.php.

Update.php has its own documentation on mediawiki.org.
None of this helps you piece everything together for the docker world.

Installation

The installation creation process is documented in this blog post, and some customization regarding LocalSettings and extensions was covered here.
The current state of the docker-compose file can be seen below with private details redacted.

This docker-compose files is found in /root/wikibase-registry on the server hosting the installation. (Yes I know that’s a dumb place, but that’s not the point of this post)

version: '3'

services:
  wikibase:
    image: wikibase/wikibase:1.30-bundle
    restart: always
    links:
      - mysql
    ports:
     - "8181:80"
    volumes:
      - mediawiki-images-data:/var/www/html/images
      - ./LocalSettings.php:/var/www/html/LocalSettings.php:ro
      - ./Nuke:/var/www/html/extensions/Nuke
      - ./ConfirmEdit:/var/www/html/extensions/ConfirmEdit
    depends_on:
    - mysql
    environment:
      MW_ADMIN_NAME: "private"
      MW_ADMIN_PASS: "private"
      MW_SITE_NAME: "Wikibase Registry"
      DB_SERVER: "mysql.svc:3306"
      DB_PASS: "private"
      DB_USER: "private"
      DB_NAME: "private"
      MW_WG_SECRET_KEY: "private"
    networks:
      default:
        aliases:
         - wikibase.svc
         - wikibase-registry.wmflabs.org
  mysql:
    image: mariadb:latest
    restart: always
    volumes:
      - mediawiki-mysql-data:/var/lib/mysql
    environment:
      MYSQL_DATABASE: 'private'
      MYSQL_USER: 'private'
      MYSQL_PASSWORD: 'private'
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
    networks:
      default:
        aliases:
         - mysql.svc
  wdqs-frontend:
    image: wikibase/wdqs-frontend:latest
    restart: always
    ports:
     - "8282:80"
    depends_on:
    - wdqs-proxy
    environment:
      BRAND_TITLE: 'Wikibase Registry Query Service'
      WIKIBASE_HOST: wikibase.svc
      WDQS_HOST: wdqs-proxy.svc
    networks:
      default:
        aliases:
         - wdqs-frontend.svc
  wdqs:
    image: wikibase/wdqs:0.3.0
    restart: always
    volumes:
      - query-service-data:/wdqs/data
    command: /runBlazegraph.sh
    environment:
      WIKIBASE_HOST: wikibase-registry.wmflabs.org
    networks:
      default:
        aliases:
         - wdqs.svc
  wdqs-proxy:
    image: wikibase/wdqs-proxy
    restart: always
    environment:
      - PROXY_PASS_HOST=wdqs.svc:9999
    ports:
     - "8989:80"
    depends_on:
    - wdqs
    networks:
      default:
        aliases:
         - wdqs-proxy.svc
  wdqs-updater:
    image: wikibase/wdqs:0.3.0
    restart: always
    command: /runUpdate.sh
    depends_on:
    - wdqs
    - wikibase
    environment:
      WIKIBASE_HOST: wikibase-registry.wmflabs.org
    networks:
      default:
        aliases:
         - wdqs-updater.svc

volumes:
  mediawiki-mysql-data:
  mediawiki-images-data:
  query-service-data:

Backups

docker-compose.yml

So that you can always return to your previous configuration take a snapshot of your docker-compose file.

If you have any other mounted files it also might be worth taking a quick snapshot of those.

Volumes

The wikibase docker-compose example README has a short section about backing up docker volumes using the loomchild/volume-backup docker image.
So let’s give that a go.

I’ll run the backup command for all 3 volumes used in the docker compose file which cover the 3 locations that I care about that persist data.

docker run -v wikibase-registry_mediawiki-mysql-data:/volume -v /root/volumeBackups:/backup --rm loomchild/volume-backup backup mediawiki-mysql-data_20190129
docker run -v wikibase-registry_mediawiki-images-data:/volume -v /root/volumeBackups:/backup --rm loomchild/volume-backup backup mediawiki-images-data_20190129
docker run -v wikibase-registry_query-service-data:/volume -v /root/volumeBackups:/backup --rm loomchild/volume-backup backup query-service-data_20190129

Looking in the /root/volumeBackups directory I can see that the backup files have been created.

ls -lahr /root/volumeBackups/ | grep 2019
-rw-r--r-- 1 root root 215K Jan 29 16:40 query-service-data_20190129.tar.bz2
-rw-r--r-- 1 root root  57M Jan 29 16:40 mediawiki-mysql-data_20190129.tar.bz2
-rw-r--r-- 1 root root  467 Jan 29 16:40 mediawiki-images-data_20190129.tar.bz2

I’m not going to bother checking that the backups are actually complete here, but you might want to do that!

Prepare the next version

Grab new versions of extensions


The wikibase-registry has a couple of extension shoehorned into it mounted through mounts in the docker-compose file (see above).

We need new versions of these extensions for Mediawiki 1.31 while leaving the old versions in place for the still running 1.30 version.

I’ll do this by creating a new folder, copying the existing extension code into it, and then changing and fetching the branch.

# Make copies of the current 1.30 versions of extensions
root@wbregistry-01:~/wikibase-registry# mkdir mw131
root@wbregistry-01:~/wikibase-registry# cp -r ./Nuke ./mw131/Nuke
root@wbregistry-01:~/wikibase-registry# cp -r ./ConfirmEdit ./mw131/ConfirmEdit

# Update them to the 1.31 branch of code
root@wbregistry-01:~/wikibase-registry# cd ./mw131/Nuke/
root@wbregistry-01:~/wikibase-registry/mw131/Nuke# git fetch origin REL1_31
From https://github.com/wikimedia/mediawiki-extensions-Nuke
 * branch            REL1_31    -> FETCH_HEAD
root@wbregistry-01:~/wikibase-registry/mw131/Nuke# git checkout REL1_31
Branch REL1_31 set up to track remote branch REL1_31 from origin.
Switched to a new branch 'REL1_31'
root@wbregistry-01:~/wikibase-registry/mw131/Nuke# cd ./../ConfirmEdit/
root@wbregistry-01:~/wikibase-registry/mw131/ConfirmEdit# git fetch origin REL1_31
From https://github.com/wikimedia/mediawiki-extensions-ConfirmEdit
 * branch            REL1_31    -> FETCH_HEAD
root@wbregistry-01:~/wikibase-registry/mw131/ConfirmEdit# git checkout REL1_31
Branch REL1_31 set up to track remote branch REL1_31 from origin.
Switched to a new branch 'REL1_31'

Define an updated Wikibase container / service

We can run a container with the new Mediawiki and Wikibase code in alongside the old container without causing any problems, it just needs a name.

So below I define this new service, called wikibase-131 using the same general details as my previous wikibase service, but pointing to the new versions of my extensions, and add it to my docker-compose file.

Note that no port is exposed, as I don’t want public traffic here yet, and also no network aliases are yet defined. We will switch those from the old service to the new service at a later stage.

  wikibase-131:
    image: wikibase/wikibase:1.31-bundle
    restart: always
    links:
      - mysql
    volumes:
      - mediawiki-images-data:/var/www/html/images
      - ./LocalSettings.php:/var/www/html/LocalSettings.php:ro
      - ./mw131/Nuke:/var/www/html/extensions/Nuke
      - ./mw131/ConfirmEdit:/var/www/html/extensions/ConfirmEdit
    depends_on:
    - mysql
    environment:
      MW_ADMIN_NAME: "private"
      MW_ADMIN_PASS: "private"
      MW_SITE_NAME: "Wikibase Registry"
      DB_SERVER: "mysql.svc:3306"
      DB_PASS: "private"
      DB_USER: "private"
      DB_NAME: "private"
      MW_WG_SECRET_KEY: "private"

I tried running this service as is but ran into an issue with the change from 1.30 to 1.31. (Your output will be much more verbose if you need to pull the image)

root@wbregistry-01:~/wikibase-registry# docker-compose up wikibase-131
wikibase-registry_mysql_1 is up-to-date
Creating wikibase-registry_wikibase-131_1 ... done
Attaching to wikibase-registry_wikibase-131_1
wikibase-131_1   | wait-for-it.sh: waiting 120 seconds for mysql.svc:3306
wikibase-131_1   | wait-for-it.sh: mysql.svc:3306 is available after 0 seconds
wikibase-131_1   | wait-for-it.sh: waiting 120 seconds for mysql.svc:3306
wikibase-131_1   | wait-for-it.sh: mysql.svc:3306 is available after 1 seconds
wikibase-131_1   | /extra-entrypoint-run-first.sh: line 3: MW_ELASTIC_HOST: unbound variable
wikibase-registry_wikibase-131_1 exited with code 1

The wikibase:1.31-bundle docker image includes the Elastica and CirrusSearch extensions which were not a part of the 1.30 bundle, and due to the entrypoint infrastructure added along with it I will need to change some things to continue without using Elastic for now.

Fix MW_ELASTIC_HOST requirement with a custom entrypoint.sh

The above error message shows that the error occurred while running extra-entrypoint-run-first.sh which is provided as part of the bundle.
It is automatically loaded by the base image entry point.
The bundle now also runs some extra steps as part of the install for wikibase that we don’t want if we are not using Elastic.

If you give the entrypoint file a read through you can see that it does a few things:

  • Makes sure the required environment variables are passed in
  • Waits for the DB server to be online
  • Runs extra scripts added by the bundle image
  • Does the Mediawiki / Wikibase install on the first run (if LocalSettings does not exist)
  • Run apache

This is a bit excessive for what the wikibase-registry requires right now, so lets strip this down, saving next to our docker-compose file, so /root/wikibase-registry/entrypoint.sh for the wikibase-registry

#!/bin/bash

REQUIRED_VARIABLES=(MW_ADMIN_NAME MW_ADMIN_PASS MW_WG_SECRET_KEY DB_SERVER DB_USER DB_PASS DB_NAME)
for i in ${REQUIRED_VARIABLES[@]}; do
    eval THISSHOULDBESET=\$$i
    if [ -z "$THISSHOULDBESET" ]; then
    echo "$i is required but isn't set. You should pass it to docker. See: https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file";
    exit 1;
    fi
done

set -eu

/wait-for-it.sh $DB_SERVER -t 120
sleep 1
/wait-for-it.sh $DB_SERVER -t 120

docker-php-entrypoint apache2-foreground

And mount it in the wikibase-131 service that we have created by adding a new volume.

    volumes:
      - ./entrypoint.sh:/entrypoint.sh

Run the new service alongside the old one

Running the service now works as expected.

root@wbregistry-01:~/wikibase-registry# docker-compose up -d wikibase-131
wikibase-registry_mysql_1 is up-to-date
Recreating wikibase-registry_wikibase-131_1 ... done

And the service appears in the list of running containers.

root@wbregistry-01:~/wikibase-registry# docker-compose ps
              Name                             Command               State          Ports
-------------------------------------------------------------------------------------------------
wikibase-registry_mysql_1           docker-entrypoint.sh mysqld      Up      3306/tcp
wikibase-registry_wdqs-frontend_1   /entrypoint.sh nginx -g da ...   Up      0.0.0.0:8282->80/tcp
wikibase-registry_wdqs-proxy_1      /bin/sh -c "/entrypoint.sh"      Up      0.0.0.0:8989->80/tcp
wikibase-registry_wdqs-updater_1    /entrypoint.sh /runUpdate.sh     Up      9999/tcp
wikibase-registry_wdqs_1            /entrypoint.sh /runBlazegr ...   Up      9999/tcp
wikibase-registry_wikibase-131_1    /bin/bash /entrypoint.sh         Up      80/tcp
wikibase-registry_wikibase_1        /bin/bash /entrypoint.sh         Up      0.0.0.0:8181->80/tcp

Update.php

From here you should now be able to get into your new container with the new code.

root@wbregistry-01:~/wikibase-registry# docker-compose exec wikibase-131 bash
root@40de55dc62fc:/var/www/html#

And then run update.php

In theory updates to the database, and anything else, will always be backward compatible for at least 1 major version, which is why we can run this update while the site is still being served from Mediawiki 1.30

root@40de55dc62fc:/var/www/html# php ./maintenance/update.php --quick
MediaWiki 1.31.1 Updater

Your composer.lock file is up to date with current dependencies!
Going to run database updates for wikibase_registry
Depending on the size of your database this may take a while!
{snip boring output}
Purging caches...done.

Done in 0.9 s.

Switching versions

The new service is already running alongside the old one, and the database has already been updated, now all we have to do is switch the services over.

If you want a less big bangy approach you could probably setup a second port exposing the updated version and direct a different domain or sub domain to that location, but I don’t go into that at all here.

Move the “ports” definition and “networks” definition from the “wikibase” service to the “wikibase-131” service. Then recreate the container for each service using the update configuration. (If you have any other references to the “wikibase” service in the docker-compose.yml file such as in depends-on then you will also need to change this.

The reason that we run the “up” command specifically on the 2 “wikibase” services is to avoid touching any of the other services at all.

root@wbregistry-01:~/wikibase-registry# docker-compose up -d wikibase
wikibase-registry_mysql_1 is up-to-date
Recreating wikibase-registry_wikibase_1 ... done
root@wbregistry-01:~/wikibase-registry# docker-compose up -d wikibase-131
wikibase-registry_mysql_1 is up-to-date
Recreating wikibase-registry_wikibase-131_1 ... done

If everything has worked you should see Special:Version reporting the newer version, which we now see on the wikibase-registry.

Cleanup

Now that everything is updated we can stop and remove the previous “wikibase” service container.

root@wbregistry-01:~/wikibase-registry# docker-compose stop wikibase
Stopping wikibase-registry_wikibase_1 ... done
root@wbregistry-01:~/wikibase-registry# docker-compose rm wikibase
Going to remove wikibase-registry_wikibase_1
Are you sure? [yN] y
Removing wikibase-registry_wikibase_1 ... done

You can then do some cleanup:

  • Remove the “wikibase” service definition from the docker-compose.yml file, leaving “wikibase-131” in place.
  • Remove any files or extensions (older versions) that are only loaded by the old service that you have now removed.

Further notes

There are lots of other things I noticed while writing this blog post:

  • It would be great to move the env vars out of the docker-compose and into env var files.
  • The default entrypoint in the docker images is quite annoying after the initial install and if you don’t use all of the features in the bundle.
  • We need a documentation hub? ;)

2 Comments

  1. D063520

    Hi,

    thank you for this tutorial! At some point you write “I’m not going to bother checking that the backups are actually complete here, but you might want to do that!”. Could you explain how to do this? Is is just checking that the bz2 file is fine, like with bunzip2 -t dump?

    Thank you
    D063520

    • addshore

      Checking that the bz2 file is okay is definitely a good first check. But the ultimate check would be restoring the backup volume to a different container and ensuring that everything looks good from there.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© 2020 Addshore

Theme by Anders NorénUp ↑

%d bloggers like this: