It's a blog

wikibase-docker, Mediawiki & Wikibase update

Today on the Wikibase Community User Group Telegram chat I noticed some people discussing issues with upgrading Mediawiki and Wikibase using the docker images provided for Wikibase.

As the wikibase-registry is currently only running Mediawiki 1.30 I should probably update it to 1.31, which is the next long term stable release.

This blog post was written as I performed the update and is yet to be proofread, so expect some typos. I hope it can help those that were chatting on Telegram today.

Starting state

Documentation

There is a small amount of documentation in the wikibase docker image README file that talks about upgrading, but this simply tells you to run update.php.

Update.php has its own documentation on mediawiki.org.
None of this helps you piece everything together for the docker world.

Installation

The installation creation process is documented in this blog post, and some customization regarding LocalSettings and extensions was covered here.
The current state of the docker-compose file can be seen below with private details redacted.

This docker-compose files is found in /root/wikibase-registry on the server hosting the installation. (Yes I know that’s a dumb place, but that’s not the point of this post)

version: '3' services: wikibase: image: wikibase/wikibase:1.30-bundle restart: always links: - mysql ports: - "8181:80" volumes: - mediawiki-images-data:/var/www/html/images - ./LocalSettings.php:/var/www/html/LocalSettings.php:ro - ./Nuke:/var/www/html/extensions/Nuke - ./ConfirmEdit:/var/www/html/extensions/ConfirmEdit depends_on: - mysql environment: MW_ADMIN_NAME: "private" MW_ADMIN_PASS: "private" MW_SITE_NAME: "Wikibase Registry" DB_SERVER: "mysql.svc:3306" DB_PASS: "private" DB_USER: "private" DB_NAME: "private" MW_WG_SECRET_KEY: "private" networks: default: aliases: - wikibase.svc - wikibase-registry.wmflabs.org mysql: image: mariadb:latest restart: always volumes: - mediawiki-mysql-data:/var/lib/mysql environment: MYSQL_DATABASE: 'private' MYSQL_USER: 'private' MYSQL_PASSWORD: 'private' MYSQL_RANDOM_ROOT_PASSWORD: 'yes' networks: default: aliases: - mysql.svc wdqs-frontend: image: wikibase/wdqs-frontend:latest restart: always ports: - "8282:80" depends_on: - wdqs-proxy environment: BRAND_TITLE: 'Wikibase Registry Query Service' WIKIBASE_HOST: wikibase.svc WDQS_HOST: wdqs-proxy.svc networks: default: aliases: - wdqs-frontend.svc wdqs: image: wikibase/wdqs:0.3.0 restart: always volumes: - query-service-data:/wdqs/data command: /runBlazegraph.sh environment: WIKIBASE_HOST: wikibase-registry.wmflabs.org networks: default: aliases: - wdqs.svc wdqs-proxy: image: wikibase/wdqs-proxy restart: always environment: - PROXY_PASS_HOST=wdqs.svc:9999 ports: - "8989:80" depends_on: - wdqs networks: default: aliases: - wdqs-proxy.svc wdqs-updater: image: wikibase/wdqs:0.3.0 restart: always command: /runUpdate.sh depends_on: - wdqs - wikibase environment: WIKIBASE_HOST: wikibase-registry.wmflabs.org networks: default: aliases: - wdqs-updater.svc volumes: mediawiki-mysql-data: mediawiki-images-data: query-service-data:
Code language: PHP (php)

Backups

docker-compose.yml

So that you can always return to your previous configuration take a snapshot of your docker-compose file.

If you have any other mounted files it also might be worth taking a quick snapshot of those.

Volumes

The wikibase docker-compose example README has a short section about backing up docker volumes using the loomchild/volume-backup docker image.
So let’s give that a go.

I’ll run the backup command for all 3 volumes used in the docker compose file which cover the 3 locations that I care about that persist data.

docker run -v wikibase-registry_mediawiki-mysql-data:/volume -v /root/volumeBackups:/backup --rm loomchild/volume-backup backup mediawiki-mysql-data_20190129 docker run -v wikibase-registry_mediawiki-images-data:/volume -v /root/volumeBackups:/backup --rm loomchild/volume-backup backup mediawiki-images-data_20190129 docker run -v wikibase-registry_query-service-data:/volume -v /root/volumeBackups:/backup --rm loomchild/volume-backup backup query-service-data_20190129
Code language: JavaScript (javascript)

Looking in the /root/volumeBackups directory I can see that the backup files have been created.

ls -lahr /root/volumeBackups/ | grep 2019 -rw-r--r-- 1 root root 215K Jan 29 16:40 query-service-data_20190129.tar.bz2 -rw-r--r-- 1 root root 57M Jan 29 16:40 mediawiki-mysql-data_20190129.tar.bz2 -rw-r--r-- 1 root root 467 Jan 29 16:40 mediawiki-images-data_20190129.tar.bz2

I’m not going to bother checking that the backups are actually complete here, but you might want to do that!

Prepare the next version

Grab new versions of extensions


The wikibase-registry has a couple of extension shoehorned into it mounted through mounts in the docker-compose file (see above).

We need new versions of these extensions for Mediawiki 1.31 while leaving the old versions in place for the still running 1.30 version.

I’ll do this by creating a new folder, copying the existing extension code into it, and then changing and fetching the branch.

# Make copies of the current 1.30 versions of extensions root@wbregistry-01:~/wikibase-registry# mkdir mw131 root@wbregistry-01:~/wikibase-registry# cp -r ./Nuke ./mw131/Nuke root@wbregistry-01:~/wikibase-registry# cp -r ./ConfirmEdit ./mw131/ConfirmEdit # Update them to the 1.31 branch of code root@wbregistry-01:~/wikibase-registry# cd ./mw131/Nuke/ root@wbregistry-01:~/wikibase-registry/mw131/Nuke# git fetch origin REL1_31 From https://github.com/wikimedia/mediawiki-extensions-Nuke * branch REL1_31 -> FETCH_HEAD root@wbregistry-01:~/wikibase-registry/mw131/Nuke# git checkout REL1_31 Branch REL1_31 set up to track remote branch REL1_31 from origin. Switched to a new branch 'REL1_31' root@wbregistry-01:~/wikibase-registry/mw131/Nuke# cd ./../ConfirmEdit/ root@wbregistry-01:~/wikibase-registry/mw131/ConfirmEdit# git fetch origin REL1_31 From https://github.com/wikimedia/mediawiki-extensions-ConfirmEdit * branch REL1_31 -> FETCH_HEAD root@wbregistry-01:~/wikibase-registry/mw131/ConfirmEdit# git checkout REL1_31 Branch REL1_31 set up to track remote branch REL1_31 from origin. Switched to a new branch 'REL1_31'
Code language: PHP (php)

Define an updated Wikibase container / service

We can run a container with the new Mediawiki and Wikibase code in alongside the old container without causing any problems, it just needs a name.

So below I define this new service, called wikibase-131 using the same general details as my previous wikibase service, but pointing to the new versions of my extensions, and add it to my docker-compose file.

Note that no port is exposed, as I don’t want public traffic here yet, and also no network aliases are yet defined. We will switch those from the old service to the new service at a later stage.

wikibase-131: image: wikibase/wikibase:1.31-bundle restart: always links: - mysql volumes: - mediawiki-images-data:/var/www/html/images - ./LocalSettings.php:/var/www/html/LocalSettings.php:ro - ./mw131/Nuke:/var/www/html/extensions/Nuke - ./mw131/ConfirmEdit:/var/www/html/extensions/ConfirmEdit depends_on: - mysql environment: MW_ADMIN_NAME: "private" MW_ADMIN_PASS: "private" MW_SITE_NAME: "Wikibase Registry" DB_SERVER: "mysql.svc:3306" DB_PASS: "private" DB_USER: "private" DB_NAME: "private" MW_WG_SECRET_KEY: "private"
Code language: JavaScript (javascript)

I tried running this service as is but ran into an issue with the change from 1.30 to 1.31. (Your output will be much more verbose if you need to pull the image)

root@wbregistry-01:~/wikibase-registry# docker-compose up wikibase-131 wikibase-registry_mysql_1 is up-to-date Creating wikibase-registry_wikibase-131_1 ... done Attaching to wikibase-registry_wikibase-131_1 wikibase-131_1 | wait-for-it.sh: waiting 120 seconds for mysql.svc:3306 wikibase-131_1 | wait-for-it.sh: mysql.svc:3306 is available after 0 seconds wikibase-131_1 | wait-for-it.sh: waiting 120 seconds for mysql.svc:3306 wikibase-131_1 | wait-for-it.sh: mysql.svc:3306 is available after 1 seconds wikibase-131_1 | /extra-entrypoint-run-first.sh: line 3: MW_ELASTIC_HOST: unbound variable wikibase-registry_wikibase-131_1 exited with code 1
Code language: PHP (php)

The wikibase:1.31-bundle docker image includes the Elastica and CirrusSearch extensions which were not a part of the 1.30 bundle, and due to the entrypoint infrastructure added along with it I will need to change some things to continue without using Elastic for now.

Fix MW_ELASTIC_HOST requirement with a custom entrypoint.sh

The above error message shows that the error occurred while running extra-entrypoint-run-first.sh which is provided as part of the bundle.
It is automatically loaded by the base image entry point.
The bundle now also runs some extra steps as part of the install for wikibase that we don’t want if we are not using Elastic.

If you give the entrypoint file a read through you can see that it does a few things:

  • Makes sure the required environment variables are passed in
  • Waits for the DB server to be online
  • Runs extra scripts added by the bundle image
  • Does the Mediawiki / Wikibase install on the first run (if LocalSettings does not exist)
  • Run apache

This is a bit excessive for what the wikibase-registry requires right now, so lets strip this down, saving next to our docker-compose file, so /root/wikibase-registry/entrypoint.sh for the wikibase-registry

#!/bin/bash REQUIRED_VARIABLES=(MW_ADMIN_NAME MW_ADMIN_PASS MW_WG_SECRET_KEY DB_SERVER DB_USER DB_PASS DB_NAME) for i in ${REQUIRED_VARIABLES[@]}; do eval THISSHOULDBESET=\$$i if [ -z "$THISSHOULDBESET" ]; then echo "$i is required but isn't set. You should pass it to docker. See: https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file"; exit 1; fi done set -eu /wait-for-it.sh $DB_SERVER -t 120 sleep 1 /wait-for-it.sh $DB_SERVER -t 120 docker-php-entrypoint apache2-foreground
Code language: PHP (php)

And mount it in the wikibase-131 service that we have created by adding a new volume.

volumes: - ./entrypoint.sh:/entrypoint.sh

Run the new service alongside the old one

Running the service now works as expected.

root@wbregistry-01:~/wikibase-registry# docker-compose up -d wikibase-131 wikibase-registry_mysql_1 is up-to-date Recreating wikibase-registry_wikibase-131_1 ... done
Code language: PHP (php)

And the service appears in the list of running containers.

root@wbregistry-01:~/wikibase-registry# docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------- wikibase-registry_mysql_1 docker-entrypoint.sh mysqld Up 3306/tcp wikibase-registry_wdqs-frontend_1 /entrypoint.sh nginx -g da ... Up 0.0.0.0:8282->80/tcp wikibase-registry_wdqs-proxy_1 /bin/sh -c "/entrypoint.sh" Up 0.0.0.0:8989->80/tcp wikibase-registry_wdqs-updater_1 /entrypoint.sh /runUpdate.sh Up 9999/tcp wikibase-registry_wdqs_1 /entrypoint.sh /runBlazegr ... Up 9999/tcp wikibase-registry_wikibase-131_1 /bin/bash /entrypoint.sh Up 80/tcp wikibase-registry_wikibase_1 /bin/bash /entrypoint.sh Up 0.0.0.0:8181->80/tcp
Code language: PHP (php)

Update.php

From here you should now be able to get into your new container with the new code.

root@wbregistry-01:~/wikibase-registry# docker-compose exec wikibase-131 bash root@40de55dc62fc:/var/www/html#
Code language: PHP (php)

And then run update.php

In theory updates to the database, and anything else, will always be backward compatible for at least 1 major version, which is why we can run this update while the site is still being served from Mediawiki 1.30

root@40de55dc62fc:/var/www/html# php ./maintenance/update.php --quick MediaWiki 1.31.1 Updater Your composer.lock file is up to date with current dependencies! Going to run database updates for wikibase_registry Depending on the size of your database this may take a while! {snip boring output} Purging caches...done. Done in 0.9 s.
Code language: PHP (php)

Switching versions

The new service is already running alongside the old one, and the database has already been updated, now all we have to do is switch the services over.

If you want a less big bangy approach you could probably setup a second port exposing the updated version and direct a different domain or sub domain to that location, but I don’t go into that at all here.

Move the “ports” definition and “networks” definition from the “wikibase” service to the “wikibase-131” service. Then recreate the container for each service using the update configuration. (If you have any other references to the “wikibase” service in the docker-compose.yml file such as in depends-on then you will also need to change this.

The reason that we run the “up” command specifically on the 2 “wikibase” services is to avoid touching any of the other services at all.

root@wbregistry-01:~/wikibase-registry# docker-compose up -d wikibase wikibase-registry_mysql_1 is up-to-date Recreating wikibase-registry_wikibase_1 ... done
Code language: PHP (php)
root@wbregistry-01:~/wikibase-registry# docker-compose up -d wikibase-131 wikibase-registry_mysql_1 is up-to-date Recreating wikibase-registry_wikibase-131_1 ... done
Code language: PHP (php)

If everything has worked you should see Special:Version reporting the newer version, which we now see on the wikibase-registry.

Cleanup

Now that everything is updated we can stop and remove the previous “wikibase” service container.

root@wbregistry-01:~/wikibase-registry# docker-compose stop wikibase Stopping wikibase-registry_wikibase_1 ... done root@wbregistry-01:~/wikibase-registry# docker-compose rm wikibase Going to remove wikibase-registry_wikibase_1 Are you sure? [yN] y Removing wikibase-registry_wikibase_1 ... done
Code language: PHP (php)

You can then do some cleanup:

  • Remove the “wikibase” service definition from the docker-compose.yml file, leaving “wikibase-131” in place.
  • Remove any files or extensions (older versions) that are only loaded by the old service that you have now removed.

Further notes

There are lots of other things I noticed while writing this blog post:

  • It would be great to move the env vars out of the docker-compose and into env var files.
  • The default entrypoint in the docker images is quite annoying after the initial install and if you don’t use all of the features in the bundle.
  • We need a documentation hub? ;)

2 Comments

  1. D063520

    Hi,

    thank you for this tutorial! At some point you write “I’m not going to bother checking that the backups are actually complete here, but you might want to do that!”. Could you explain how to do this? Is is just checking that the bz2 file is fine, like with bunzip2 -t dump?

    Thank you
    D063520

    • addshore

      Checking that the bz2 file is okay is definitely a good first check. But the ultimate check would be restoring the backup volume to a different container and ensuring that everything looks good from there.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© 2021 Addshore

Theme by Anders NorénUp ↑

%d bloggers like this: