It's a blog

Tag: docker (Page 1 of 2)

mediawiki-docker-dev v1 rewrite

Back in 2017 at the Wikimedia Hackathon, I played around with Docker and docker-compose in relation to MediaWiki and testing with multiple setups at once while developing, meaning multiple PHP versions, web servers and databases. My original slides can still be found here.

Since then mediawiki-docker-dev evolved into less of a testing system and more of a development environment, allowing the use of a master replica DB setup, easily swappable PHP versions, debugging and more. The project on GitHub currently has 40 stars, 38 forks and has seen 17 people contributing back.

Over the past couple of years, developer productivity and development environments have been a big discussion area. The Wikimedia technical conference in 2019 had the main topic of Developer Productivity. There have also been a few efforts in a few directions trying to figure out what is best for the majority of people. These include local-charts (Kubernetes based environment) and MediaWiki-Docker (simple docker-compose based environment).

Continue reading

Reducing Java JVM memory usage in Containers and on Kubernetes

For a while I have been running a Wikibase query service update script for WBStack, which is a Java application on a Kubernetes cluster. Part of that journey has included the updater using all available memory, hitting into the kubernetes memory limit and being OOM killed. The title of the post is a little verbose, but I wanted to include all of the keywords that might help people find the answers to the memory issues that I was running into.

Before getting into my weeds, if you want the answers head straight to https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/

UPDATE: This blog post was written with Java 8. Java 10+ now automatically recognizes memory limits and enforces them. https://www.docker.com/blog/improved-docker-container-integration-with-java-10/

Continue reading

Automatic cleanup of old gcloud container images

I have been using Google Cloud Build for a budget project for roughly a year now. Cloud Build stores built images in a storage bucket which you are of course billed for. Within the first weeks of using it I realized that I needed some automated way to cleanup unused and old images that were built there.

At the time I had a quick search around on the web for something already implemented that I could copy, but I came up blank, and decided putting my problem off would be the best solution. I filed issue number 6 for my project and left it for future me.

Now it’s time to finally close that issue, and I hope others might also find the small bash script useful.

Continue reading

WBStack Infrastructure

WBStack currently runs on a Google Cloud Kubernetes cluster made up of 2 virtual machines, one e2-medium and one e2-standard-2. This adds up to a current total of 4 vCPUs and 12GB of memory. No Google specific services make up any part of the core platform at this stage meaning WBStack can run wherever there is a Kubernetes cluster with little to no modification.

A simplified overview of the internals can be seen in the diagram below where blue represents the Google provided services, with green representing everything running within the kubernetes cluster.

Continue reading

Changing the concept URI of an existing Wikibase with data

Many users of Wikibase find themselves in a position where they need to change the concept URI of an existing Wikibase for one or more reasons, such as a domain name update or desire to have https concept URIs instead of HTTP.

Below I walk through a minimal example of how this can be done using a small amount of data and the Wikibase Docker images. If you are not using the Docker images the steps should still work, but you do not need to worry about copying files into and out of containers or running commands inside containers.

Continue reading

Your own Wikidata Query Service, with no limits (part 1)

The Wikidata Query Service allows anyone to use SPARQL to query the continuously evolving data contained within the Wikidata project, currently standing at nearly 65 millions data items (concepts) and over 7000 properties, which translates to roughly 8.4 billion triples.

Screenshot of the Wikidata Query Service home page including the example query which returns all Cats on Wikidata.

You can find a great write up introducing SPARQL, Wikidata, the query service and what it can do here. But this post will assume that you already know all of that.

Guide

Here we will focus on creating a copy of the query service using data from one of the regular TTL data dumps and the query service docker image provided by the wikibase-docker git repo supported by WMDE.

Continue reading

Creating a Dockerfile for the Wikibase Registry

Currently the Wikibase Registry(setup post) is deployed using the shoehorning approach described in one of my earlier posts. After continued discussion on the Wikibase User Group Telegram chat about different setups and upgrade woes I have decided to convert the Wikibase Registry to use the prefered approach of a custom Dockerfile building a layer on top of one of the wikibase images.

I recently updated updated the Wikibase registry from Mediawiki version 1.30 to 1.31 and described the process in a recent post, so if you want to see what the current setup and docker-compose file looks like, head there.

As a summary the Wikibase Registry uses:

  • The wikibase/wikibase:1.31-bundle image from docker hub
  • Mediawiki extensions:
    • ConfirmEdit
    • Nuke
Continue reading

wikibase-docker, Mediawiki & Wikibase update

Today on the Wikibase Community User Group Telegram chat I noticed some people discussing issues with upgrading Mediawiki and Wikibase using the docker images provided for Wikibase.

As the wikibase-registry is currently only running Mediawiki 1.30 I should probably update it to 1.31, which is the next long term stable release.

This blog post was written as I performed the update and is yet to be proofread, so expect some typos. I hope it can help those that were chatting on Telegram today.

Starting state

Documentation

There is a small amount of documentation in the wikibase docker image README file that talks about upgrading, but this simply tells you to run update.php.

Update.php has its own documentation on mediawiki.org.
None of this helps you piece everything together for the docker world.

Installation

The installation creation process is documented in this blog post, and some customization regarding LocalSettings and extensions was covered here.
The current state of the docker-compose file can be seen below with private details redacted.

This docker-compose files is found in /root/wikibase-registry on the server hosting the installation. (Yes I know that’s a dumb place, but that’s not the point of this post)

Continue reading

A firewall is blocking sharing between Windows and the containers – Docker

I recently encountered this error while trying to run one of my docker setups.

ERROR: for mediawiki-docker-dev_db-slave_1  Cannot start service db-slave: b'OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/tmp/mwdd/entrypoint.sh\": stat /tmp/mwdd/entrypoint.sh: no such file or directory": unknown'

I have encountered errors like this before and it has always ended up being related to docker and sharing my drives to the linux VM that actually runs my containers.

Checking the shared drives menu of the docker UI everything seemed to be fine.

However when removing the drive share and re sharing the drive I got an error message saying that there was a “Firewall detected” and that “A firewall is blocking file Sharing between Windows and the containers. See documentation for more info”.

The docs seem to suggest that my firewall rule had vanished somehow. I spent a few minutes digging around and tinkering with firewall rules before determining the rule itself was not the issue. The next suggestion was to “reinstall the File and Print sharing service on the Hyper-V virtual network card

Opening PowerShell as administrator and running the commands suggested on stackoverflow (switching the DockerNAT network) seemed to do the trick.

Set-NetConnectionProfile -interfacealias "vEthernet (DockerNAT)" -NetworkCategory Private

I still have no idea why this happened. Did I accidently set the DockerNAT network to be public? Could the Docker UI really not check the firewall rule itself and propose the fix?

Customizing Wikibase config in the docker-compose example

2019 Edit: If you want to use a Dockerfile and custom image take a look a here.

Just over a month ago I setup the Wikibase registry project on Wikimedia Cloud VPS using the docker-compose example provided by Wikibase docker images. The Wikibase registry is the first Wikibase install that I control that uses the Wikibase docker images, so I’ll be using it as an example showing how the docker images can be manipulated to configure MediaWiki, Wikibase, and load custom extensions and skins.

The example docker-compose file at the time of writing this post can be found at https://github.com/wmde/wikibase-docker/blob/5919016eac16c5f0aefc448240fdf6a09bb56bec/docker-compose.yml

Since the last blog post new wikibase image tags have been created (the ‘bundle’ tags) that include some extensions you might want to enable, as well a quickstatements image for the quickstatements service used on Wikidata written by Magnus Manske.

Continue reading

« Older posts

© 2020 Addshore

Theme by Anders NorénUp ↑