It's a blog

mediawiki-docker-dev v1 rewrite

Back in 2017 at the Wikimedia Hackathon, I played around with Docker and docker-compose in relation to MediaWiki and testing with multiple setups at once while developing, meaning multiple PHP versions, web servers and databases. My original slides can still be found here.

Since then mediawiki-docker-dev evolved into less of a testing system and more of a development environment, allowing the use of a master replica DB setup, easily swappable PHP versions, debugging and more. The project on GitHub currently has 40 stars, 38 forks and has seen 17 people contributing back.

Over the past couple of years, developer productivity and development environments have been a big discussion area. The Wikimedia technical conference in 2019 had the main topic of Developer Productivity. There have also been a few efforts in a few directions trying to figure out what is best for the majority of people. These include local-charts (Kubernetes based environment) and MediaWiki-Docker (simple docker-compose based environment).

Prototyping improvements

I personally still love the level of convenience and control that mediawiki-docker-dev has brought my development workflow to this day, and thus I decided to see what I could change in 1 week to make it better.

The version of mediawiki-docker-dev that has developed over the past years has a few major problems, and these are the main points that the v1 branch tries to solve:

  • Fewer startup services – To this day mediawiki-docker dev starts up 9 services, even if you don’t want them all.
  • More control over what is running – Currently everything is on or off, there is no fine-grained control.
  • Tie in with utility images – There is not yet an easy or nice way to run npm, composer or other such MediaWiki related utility services.
  • Only tied together with a collection of bash scripts – Doing anything fancy is painful

Language and Framework

To quickly prototype, I chose a language that I spend most of my current days in, PHP. This is not the ideal language to write a CLI application in, but it will work and allow me to move quickly. (At the time of writing this blog post this work has already all been ported to Go)

With PHP as a language, I chose symfony/console to do the bulk of the command line work, leaving me to just fill in the logic gaps.

The result

After 40 new commits over 6 days on the v1 branch everything was working, was faster and with more functionality.

You can try out the v1 branch now, which generally works in the same way as the original mediawiki-docker-dev. Make sure you read the README as some things have changed and the initial setup has already been updated for ease of testing.

Overview

The environment has switched from a single docker-compose file to 12 files each containing a set of services that fit into a logical group. One of these groups is for example “base”, which contains a DNS service and a reverse proxy, and another is “mw”, which contains the MediaWiki web service.

All of these docker-compose files are now orchestrated using a control application that shells out to docker-compose. The main function of this layer of abstraction is specifying all of the files to the docker-compose command so that the user doesn’t need to worry about them. However a user can still easily send command directly to docker-compose through the control applciation.

This control application also provides a nice CLI interface for interacting with the various moving parts. You can turn services on or off and interact with utility images such as composer, npm, fresh or quibble. Running tests is easy, as is getting into the redis or MySQL CLI to look at your data.

Setup is very similar to the original mediawiki-docker-dev, but with an additional requirement of PHP being installed and needing to composer install the control application. Ultimately, it is still docker-compose, and a .env file is still used for custom settings, with the same docker images being used with the same glue tieing it all together.

Measurable improvements

Well, I didn’t exactly measure these but:

  • Changed from 9 services at startup time to 4.
  • No replica DB server at startup time now, so 1 less slow DB to wait for.
  • CLI is now a useful framework provided interface rather than a collection of bash scripts
  • New functionality provided: adminer, fresh, quibble, composer,

Lessons along the way

One of the things that has been discussed is the idea of having separate docker-compose files in separate git repositories, all tie together to make a development environment. I briefly tried that when prototyping but immediately realized that it would be a bad idea as it appears docker-compose requires all files in a setup to specify the same version. Thus, they probably all need to be controlled in the same place.

The time-old issue of developing Docker focused solutions in Windows came up once again with this prototype where I repeatedly forget to ass +x bits in git for executable scripts, and also generally forgot to think about what I was doing with various file permissions. This lead to a lot of fixes being needed when I first tested everything on a Debian box.

Although Symfony Console allowed me to create a fairly rapid prototype with little effort, I don’t think it would be very nice for a final solution, mainly due to the command style, joining command parts with a “:” character, and its verbose command listings. I imagine these could be solved but rapid searching didn’t find any out of the box solutions here.

I did have the idea of running the control application within Docker as well, to avoid the extra dependency on PHP or composer for the environment. I managed to get this working to a point but decided it wasn’t worth investing the time in as I shouldn’t be writing this in PHP anyway. This idea should be possible with a custom image and mounting the docker socket, but there are much cleaner ways to get the same result.

Whats next?

The Wikimedia Foundation release engineering team has been working towards a MediaWiki CLI tools for development purposes. This tool will likely be the tool to control whatever “advanced” developer environment ends up being used beyond the current docker-composer setup in MediaWiki core.

A port of the prototype described in this blog post is already in the process of being ported to the Golang CLI application so that we can continue the discussion of the idea and direction. You can find the main patch on Gerrit, and discussion in the associated Phabricator ticket.

The port is already going fairly well, but I’ll leave that for another blog post…

2 Comments

  1. Yuri Astrakhan

    Having multiple git repos is a major pain. It is usually easier to have a monorepo, but with different docker images (contexts). Works nicely both with github actions and docker hub (i prefer the former — much faster to build and it can push to docker hub).
    it would be awesome to have as little local host requirements as possible to do the real dev work. Ideally – just add a few shell scripts (one per host OS) that do a docker run …. and possibly pass the -v /var/run/docker.sock:/var/run/docker.sock so that it can manage other docker containers. I’m not sure if this approach will work on Windows though.

    • addshore

      The golang solution, when done, will be all in a single binary, and only require docker and docker-compose.
      And mounting the docker.sock does work on Windows, but figuring out what to mount and making sure all of the paths are correct when running the control app within a docker container ends up being quite a lot of overhead.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© 2020 Addshore

Theme by Anders NorénUp ↑

%d bloggers like this: