Wikimedia – Page 3

Wikidata query service updater evolution

April 12, 2022 by addshore

The Wikidata Query Service (WDQS) sits in front of Wikidata and provides access to query its data via a SPARQL API. The query service itself is built on top of Blazegraph, but in many regards is very similar to any other triple store that provides a SPARQL API.

In the early days of the query service (circa 2015), the service was only run by Wikidata, hence the name. However, as interest and usage of Wikibase continued to grow more people started running a query service of their own, for data in their own Wikibase. But you’ll notice most people still refer to it as WDQS today.

Whereas most core Wikibase functionality is developed by Wikimedia Deutschland, the query service is developed by the search platform team at the Wikimedia Foundation, with a focus on wikidata.org, but also a goal of keeping it useable outside of Wikimedia infrastructure.

The query service itself currently works as a whole application rather than just a database. Under the surface, this can roughly be split into 2 key parts

Backend Blazegraph database that stores and indexes data
Updater process that takes data from a Wikibase and puts it in the database

This actually means that you can run your own query service, without running a Wikibase at all. For example, you can load the whole of Wikidata into a query service that you operate, and have it stay up to date with current events. Though in practice this is quite some work, and expense on storage and indexing and I expect not many folks do this.

Over time the updater element of the query service updater has iterated through some changes. The updater now packaged with Wikibase as used by most folks outside of the Wikimedia infrastructure is now 2 steps behind the updater used for Wikidata itself.

The updater generations look something like this:

HTTP API Recent Changes polling updater (used by most Wikibases)
Kafka based Recent Changes polling updater
Streaming updater (used on Wikidata)

Let’s take a look at a high-level overview of these updaters, what has changed and why. I’ll also be applying some pretty arbitrary / gut feeling scores to 4 categories for each updater.

WikiCrowd at 50k answers

April 4, 2022April 2, 2022 by addshore

In January 2022 I published a new Wikimedia tool called WikiCrowd.

This tool allows people to answer simple questions to contribute edits to Wikimedia projects such as Wikimedia Commons and Wikidata.

It’s designed to be able to deal with a wide variety of questions, but due to time constraints, the extent of the current questions covers Aliases for Wikidata, and Depict statements for Wikimedia Commons.

Altering a Gerrit change (git-review workflow)

February 25, 2022 by addshore

I don’t use git-review for Gerrit interactions. This is primarily because back in 2012/2013 I couldn’t get git-review installed, and someone presented me with an alternative that worked. Years later I realized that this was actually the documented way of pushing changes to Gerrit.

As a little introduction to what this workflow looks, and a comparison with git-review I have created 2 overview posts altering a gerrit change on the Wikimedia gerrit install. I’m not trying to convince you, either way, is better, merely show the similarities/difference and what is happening behind the scenes.

Be sure to take a look at the other post “Altering a Gerrit change (git workflow)“

One prerequisite of this workflow is that you have git-review installed and a .gitreview file in your repository!

I’ll be taking a change from the middle of last year, rebasing it, making a change, and pushing it back for review. Fundamentally the 2 approaches do the same thing, just one (git-review) requires an external tool.

Altering a Gerrit change (git workflow)

April 2, 2022February 25, 2022 by addshore

Be sure to take a look at the other post “Altering a Gerrit change (git-review workflow)“

Profiling a Wikibase item creation on test.wikidata.org

April 2, 2022February 3, 2022 by addshore

Today I was in a Wikibase Stakeholder group call, and one of the discussions was around Wikibase importing speed, data loading, and the APIs. My previous blog post covering what happens when you make a new Wikibase item was raised, and we also got onto the topic of profiling.

So here comes another post looking at some of the internals of Wikibase, through the lens of profiling on test.wikidata.org.

The tools used to write this blog post for Wikimedia infrastructure are both open source, and also public. You can do similar profiling on both your own Wikibase, or for your requests that you suspect are slow on Wikimedia sites such as Wikidata.

Wikimedia Profiling

Profiling of Wikimedia sites is managed and maintained by the Wikimedia performance team. They have a blog, and one of the most recent posts was actually covering profiling PHP at scale in production, so if you want to know the details of how this is achieved give it a read.

Throughout this post I will be looking at data collected from a production Wikimedia request, by setting the X-Wikimedia-Debug header in my request. This header has a few options, and you can find the docs on wikitech.wikimedia.org. There are also browser extensions available to easily set this header on your requests.

I will be using the Wikimedia hosted XHGui to visualize the profile data. Wikimedia specific documentation for this interface also exists on wikitech.wikimedia.org. This interface contains a random set of profiled requests, as well as any requests that were specifically requested to be profiled.

Profiling PHP & MediaWiki

If you want to profile your own MediaWiki or Wikibase install, or PHP in general, then you should take a look at the mediawiki.org documentation page for this. You’ll likely want to use either Tideways or XDebug, but probably want to avoid having to setup any extra UI to visualize the data.

This profiling only covered the main PHP application (MediaWiki & Wikibase extension). Other services such as the query service would require separate profiling.

A first look at WikiCrowd

April 2, 2022January 6, 2022 by addshore

I have quite enjoyed the odd contribution to an app by Google called Crowdsource. You can find it either on the web, or also as an app.

Crowdsource allows people to throw data at Google in controlled ways to add to the massive pile of data that Google uses to improve its services and at the end of the day beat its competition.

It does this by providing a collection of micro contribution tasks in a marginally gamified way, similar to how Google Maps contributions get you Local Guide points etc. In Crowdsource you get a contribution count, a level, and a metric for agreements.

While I enjoy making the odd contribution when bored out of my mind and enjoy looking at the new challenges (currently at 2625 contributions), I always think that data like this should just be going out into the world under a free licence to benefit everyone.

So finally, introducing WikiCrowd, an interface, and soon to be app, that I developed over the new year period.

WikiCrowd Overview

WikiCrowd is hosted on toolforge and can be found at https://wikicrowd.toolforge.org/ (Source code on Github)

In order to contribute, you need some knowledge of the world, a Wikimedia account and that’s it!

Screenshot showing the wikicrowd application, listing various groups of questions users can contribute to

Laravel 8 with Wikimedia OAuth login

January 6, 2022January 6, 2022 by addshore

I recently wrote a little app called wikicrowd (blog post to follow) using Laravel and MediaWiki / Wikimedia authentication. It certainly wasn’t entirely out of the box, and the existing docs still need some tweaking.

This post reflects the steps I went through to set this app up, and it should only take a few minutes.

You can find a tag of the code at the end of this walkthrough on Github for PHP 8. (There is also a tag for PHP 7.4)

Shout out to the developers that worked on the Wikidata Mismatch Finder which is also a Laravel app with MediaWiki OAuth and was used as inspiration when writing this post, along with the documentation for the package used by Taavi.

Setup Laravel

First off I need a Laravel installation. Currently 8.x is the stream of the latest versions, and the installation docs say to run the below command.

curl -s https://laravel.build/demo-laravel-mediawiki-auth | bashCode language: JavaScript (javascript)

I’m not a fan of running random code on the internet on my machine, but this is what the docs say. It creates a directory at the location you specify at the end of the URL, in my case demo-laravel-mediawiki-auth , creates a laravel/laravel project, and does a composer install.

Tech Lead Digest – Q3/4 2021

August 28, 2023December 6, 2021 by addshore

This entry is part 5 of 5 in the series Tech Lead Digest (wmde)

It’s time for the 5th instalment of my tech lead digest posts. I switched to monthly for 2 months, but decided to back down to quarterlyish. You can find the other digests by checking out the series.

🧑‍🤝‍🧑Wikidata & Wikibase

The biggest event of note in the past months was WikidataCon 2021 which took place toward the end of October 2021. Spread over 3 days the event celebrated Wikidatas 9th birthday. We are still awaiting the report from the event to know how many folks participated, and recordings of talks will likely not be available until early 2022. At which point I’ll try to write another blog post.

Just before WikidataCon the updated strategy for Linked Open Data was published by Wikimedia Deutschland which includes sub-strategies for Wikidata and the Wikibase Ecosystem. This strategy is much easier to digest than the strategy papers published in 2019 and I highly recommend the read. Part of the Wikidata strategy talks about “sharing workload” which reminds me of some thoughts I recently had comparing Wikipedia and Wikidata editing. Wikibase has a focus on Ecosystem enablement, which I am looking forward to working on.

The Wikibase stakeholder group continues to grow and organize. A Twitter account (@wbstakeholders) now exists tweeting relevant updates. Now with over 14 organizational members and 15 individual members, the budget is now public and the group is working on getting some desired features implemented. If you are an organization or individual working in the Wikibase space, be sure to check them out! The group recently published a prioritized list of institutional requirements, and I’m happy to say that some parts of the “Automatic maintenance processes and updating cascades should work out of the box” area that scored 4 have already been tackled by the Wikidata / Wikibase teams.

mwcli CI in Wikimedia GitLab (docker in docker)

April 2, 2022November 1, 2021 by addshore

mwcli is a golang CLI tool that I have been working on over the past year to replace the mediawiki-docker-dev development environment that I accidently created a few years back (among other things). I didn’t start the CLI, but I did this mediawiki-docker-dev like functionality.

As some point through the development journey it became clear that one of the ways to set the new and old environments apart would be through some rigorous CI and testing.

This started with CI running on a Qemu node as part of the shared Wikimedia Jenkins CI infrastructure that is hooked up to Gerrit, where the code was being developed. This ended up being quite slow, and involved lots of manual steps.

A next iteration saw the majority of development take place in my own fork on Github, making use of Github Actions. Changes would then be copied over to Gerrit for final review once CI tests had run.

And finally the repository moved to the new Wikimedia GitLab instance (work in progress), where I could make use of GitLab Runners powered by a machine in Wikimedia Cloud VPS.

Screenshot of GitLab pipelines in action for the mwcli project

Covid-19 Wikipedia pageviews, a first look

May 17, 2020March 22, 2020 by addshore

World events often have a dramatic impact on online services. A past example would be the death of Michael Jackson which brought down Twitter and Wikipedia and made Google believe that they were under attack according to the BBC.

Events like the COVID-19 (Coronavirus) pandemic have less instantaneous affect but trends can still be seen to change. Cloudflare recently posted about some of the internet wide traffic changes due to the pandemic and various government announcements, quarantines and lockdowns.

Currently the main English Wikipedia article for the COVID-19 pandemic is receiving roughly 1.2 million page views per day (14 per second). This article has already gone through 4 different names over the past months, and the pageview rate continues to climb.