With all of the goings on recently, as well as the new GDPR (General Data Protection Regulation) in the EU, Facebook have attempted to make the data that they have about an individual easier to see, understand and edit or remove. One such section of this data covers your “ad preferences“.
Wikidata.org runs on MediaWiki with the Wikibase extension. But there is more to it than just that. The Wikibase extension itself is split into 3 different sections, being Lib, Repo and Client. There are also 6 other extensions all providing extra functionality to the site and it’s sisters. The extensions are also loaded on a different combination of Clients (such a Wikipedia) and the Repo itself (wikidata.org).
A few people I know are now running WordPress installs, or are currently setting them up. The one question that always seems to come up while discussing them is “what plugins are best”? I try to sum up what I use below (in alphabetical order), as well as how these might be changing in the near future due to the new Gutenberg WordPress editor.
I won’t bother to list some of the more common plugins such as Akismet Anti-Spam or Jetpack, as these are already pretty visible in the WordPress world.
The title is a little wordy, but I hope you get the gist. I just spent 10 minutes staring at some data on a Grafana dashboard, comparing it with some other data, and finding the numbers didn’t add up. Here is the story in case it catches you out.
The dashboard in question is the Wikidata Edits dashboard hosted on the Wikimedia Grafana instance that is public for all to see. The top of the dashboard features a panel that shows the total number of edits on Wikidata in the past 7 days. The rest of the dashboard breaks these edits down further, including another general edits panel on the left of the second row.
Once logged in Hue can be used to write Hive queries with syntax highlighting, auto suggestions and formatting, as well as allowing users to save queries with names and descriptions, run queries from the browser and watch hadoop job execution state.
The Wikidata & maxlag bit
MediaWiki has a maxlag API parameter that can be passed alongside API requests in order to cause errors / stop writes from happening when the DB servers are lagging behind the master. Within MediaWiki this lag can also be raised when the JobQueue is very full. Recently Wikibase introduced the ability to raise this lag when the Dispatching of changes to client projects is also lagged behind. In order to see how effective this will be, we can take a look at previous API calls.
The asyncio module was added to Python in version 3.4 to “provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives”. If you want an introduction to asyncio take a look at this blog post.
This post will look at how to share state between multiple async methods that may be running at the same time without using anything silly like globals. The examples should be easy to follow and easy to extrapolate from. Also I’ll be embedding trinket.io code snippets which you should be able to run in the browser.
Disclaimer: I don’t really write much python so there might be better solutions. This post is only meant to serve as a consolidation of knowledge to be passed to another.
The Wikimedia Server Admin Log or SAL for short is a timestamped log of actions performed on the Wikimedia cluster by users such as roots and deployers. The log is stored on the WikiTech Wikimedia project and can be found at the following URL: https://wikitech.wikimedia.org/wiki/Server_Admin_Log
2019 Edit: If you want to use a Dockerfile and custom image take a look a here.
Just over a month ago I setup the Wikibase registry project on Wikimedia Cloud VPS using the docker-compose example provided by Wikibase docker images. The Wikibase registry is the first Wikibase install that I control that uses the Wikibase docker images, so I’ll be using it as an example showing how the docker images can be manipulated to configure MediaWiki, Wikibase, and load custom extensions and skins.
Since the last blog post new wikibase image tags have been created (the ‘bundle’ tags) that include some extensions you might want to enable, as well a quickstatements image for the quickstatements service used on Wikidata written by Magnus Manske.
The aim of the registry is to act as a central point for details of public Wikibase installs hosted around the web. Data held about the installs currently includes the URL for the home page, Query frontend URL and SPARQL API endpoint URL (if a query service exists).
During the workshop an initial data set was added, and this can be easily seen using the timeline view of the query service and a query that is explained within this post.
2017 has been a great year with continued work at WMDE on both technical wishes projects and also Wikibase / Wikidata related areas. Along the way I shared a fair amount of this through this blog, although not as much as I would have liked. Hopefully I’ll be slightly more active in 2018. Here are some fun stats:
7,992 page views
4 total posts (that’s terrible, but its April 2018 now and I already have 6 under my belt)