Wikidata Architecture Overview (diagrams)
Over the years diagrams have appeared in a variety of forms covering various areas of the architecture of Wikidata. Now, as the current tech lead for Wikidata it is my turn. Wikidata has slowly become a more and more complex system, including multiple extensions, services and storage backends. Those of us that work with it…
Grafana, Graphite and maxDataPoints confusion for totals
The title is a little wordy, but I hope you get the gist. I just spent 10 minutes staring at some data on a Grafana dashboard, comparing it with some other data, and finding the numbers didn’t add up. Here is the story in case it catches you out. The dashboard The dashboard in question…
Using Hue & Hive to quickly determine Wikidata API maxlag usage
Hue, or Hadoop User Experience is described by its documentation pages as “a Web application that enables you to easily interact with an Hadoop cluster”. The Wikimedia Foundation has a Hue frontend for their Hadoop cluster, which contains various datasets including web requests, API usage and the MediaWiki edit history for all hosted sites. The install…
The Wikimedia Server Admin Logs
The Wikimedia Server Admin Log or SAL for short is a timestamped log of actions performed on the Wikimedia cluster by users such as roots and deployers. The log is stored on the WikiTech Wikimedia project and can be found at the following URL: https://wikitech.wikimedia.org/wiki/Server_Admin_Log An example entry in the log could be: 09:04 addshore: addshore@terbium:~$…
WMDE: CI, Deploys & Config changes intro
This is an internal WMDE presentation made to introduce people to the land of Wikimedia CI, MediaWiki deployment and config changes. This briefly covers: Jenkins & Zuul CI Config Beta Cluster SWAT The MediaWiki train Monitoring mediawiki-config The Docker & Kubernetes future?
From 0 to Kubernetes cluster with Ingress on custom VMs
While working on a new Mediawiki project, and trying to setup a Kubernetes cluster on Wikimedia Cloud VPS to run it on, I hit a couple of snags. These were mainly to do with ingress into the cluster through a single static IP address and some sort of load balancer, which is usually provided by…
Wikibase docker images
This is a belated post about the Wikibase docker images that I recently created for the Wikidata 5th birthday. You can find the various images on docker hub and matching Dockerfiles on github. These images combined allow you to quickly create docker containers for Wikibase backed by MySQL and with a SPARQL query service running…
Wikimedia Hackathon 2017: mediawiki-docker-dev showcase presentation
This presentation was used during the 2017 Wikimedia Hackathon showcase presentations. The code shown can be found @ https://github.com/addshore/mediawiki-docker-dev
Wikimedia Commons Android App Pre-Hackathon
The Wikimedia Commons Android App allows users to upload photos to Commons directly from their phone. The website for the app details some of the features and the code can be found on GitHub. A hackathon was organized in Prague to work on the app in the run up to the yearly Wikimedia Hackathon which…