Addshore

It's a blog

Category: Posts (page 1 of 9)

All Posts

Wikidata Map October 2018

October 5, 2018 / / 7 Comments

It has been another 6 months since my last post in the Wikidata Map series. In that time Wikidata has gained 4 million items, 1 property with the globe-coordinate data type (coordinates of geographic centre) and 1 million items with coordinates [1]. Each Wikidata item with a coordinate is represented on the map with a single dim pixel. Below you can see the areas of change between this new map and the once generated in March. To see the equivalent change in the previous 4 months take a look at the previous post.

Comparison of March 26th and October 1st maps in 2018
Continue reading

Quickly clearing out your Facebook advert ‘interests’

October 4, 2018 / / 0 Comments

Over the past years Facebook have had a few privacy related issues. First came the ‘scandal’ with Cambridge Analytica and more recently a bug (or series of bugs) that apparently affected 50 million accounts allowing peoples access tokens to be stolen. Oh, and there was also the story about Facebook using your 2 factor authentication phone number for targeting advertising.

With all of the goings on recently, as well as the new GDPR (General Data Protection Regulation) in the EU, Facebook have attempted to make the data that they have about an individual easier to see, understand and edit or remove. One such section of this data covers your “ad preferences“.

Screenshot of Facebook ad preferences
Continue reading

Wikibase extensions on Wikidata.org

September 20, 2018 / / 0 Comments

Wikidata.org runs on MediaWiki with the Wikibase extension. But there is more to it than just that. The Wikibase extension itself is split into 3 different sections, being Lib, Repo and Client. There are also 6 other extensions all providing extra functionality to the site and it’s sisters. The extensions are also loaded on a different combination of Clients (such a Wikipedia) and the Repo itself (wikidata.org).

A diagram of current dependencies between the various Wikibase extensions running on wikidata.org
Continue reading

WordPress plugins

September 18, 2018 / / 0 Comments

A few people I know are now running WordPress installs, or are currently setting them up. The one question that always seems to come up while discussing them is “what plugins are best”? I try to sum up what I use below (in alphabetical order), as well as how these might be changing in the near future due to the new Gutenberg WordPress editor.

I won’t bother to list some of the more common plugins such as Akismet Anti-Spam or Jetpack, as these are already pretty visible in the WordPress world.

Continue reading

Grafana, Graphite and maxDataPoints confusion for totals

September 10, 2018 / / 0 Comments

The title is a little wordy, but I hope you get the gist. I just spent 10 minutes staring at some data on a Grafana dashboard, comparing it with some other data, and finding the numbers didn’t add up. Here is the story in case it catches you out.

The dashboard

The dashboard in question is the Wikidata Edits dashboard hosted on the Wikimedia Grafana instance that is public for all to see. The top of the dashboard features a panel that shows the total number of edits on Wikidata in the past 7 days. The rest of the dashboard breaks these edits down further, including another general edits panel on the left of the second row. 

Continue reading

Using Hue & Hive to quickly determine Wikidata API maxlag usage

July 3, 2018 / / 0 Comments

Hue, or Hadoop User Experience is described by its documentation pages as “a Web application that enables you to easily interact with an Hadoop cluster”.

The Wikimedia Foundation has a Hue frontend for their Hadoop cluster, which contains various datasets including web requests, API usage and the MediaWiki edit history for all hosted sites. The install can be accessed at https://hue.wikimedia.org/ using Wikimedia LDAP for authentication.

Once logged in Hue can be used to write Hive queries with syntax highlighting, auto suggestions and formatting, as well as allowing users to save queries with names and descriptions, run queries from the browser and watch hadoop job execution state.

The Wikidata & maxlag bit

MediaWiki has a maxlag API parameter that can be passed alongside API requests in order to cause errors / stop writes from happening when the DB servers are lagging behind the master. Within MediaWiki this lag can also be raised when the JobQueue is very full. Recently Wikibase introduced the ability to raise this lag when the Dispatching of changes to client projects is also lagged behind. In order to see how effective this will be, we can take a look at previous API calls.

Continue reading

Python3, Using some shared state in 2 async methods

June 28, 2018 / / 0 Comments

Python logoThe asyncio module was added to Python in version 3.4 to “provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives”. If you want an introduction to asyncio take a look at this blog post.

This post will look at how to share state between multiple async methods that may be running at the same time without using anything silly like globals. The examples should be easy to follow and easy to extrapolate from. Also I’ll be embedding trinket.io  code snippets which you should be able to run in the browser.

Disclaimer: I don’t really write much python so there might be better solutions. This post is only meant to serve as a consolidation of knowledge to be passed to another.

Continue reading

The Wikimedia Server Admin Logs

June 27, 2018 / / 0 Comments

The Wikimedia Server Admin Log or SAL for short is a timestamped log of actions performed on the Wikimedia cluster by users such as roots and deployers. The log is stored on the WikiTech Wikimedia project and can be found at the following URL: https://wikitech.wikimedia.org/wiki/Server_Admin_Log

An example entry in the log could be:

As well as the main cluster SAL there are also logs for release engineering (jenkins, zuul, and other CI things) and individual logs for each project that uses Wikimedia Cloud VPS.

A tool has been created for easy SAL navigation which can be found at https://tools.wmflabs.org/sal

Each SAL can be selected at the top of the tool, with ‘Other’ providing you with a list of all Cloud VPS SALs.

The search and date filters can then be used to find entries throughout history.

Continue reading

Customizing Wikibase config in the docker-compose example

June 9, 2018 / / 1 Comment

Just over a month ago I setup the Wikibase registry project on Wikimedia Cloud VPS using the docker-compose example provided by Wikibase docker images. The Wikibase registry is the first Wikibase install that I control that uses the Wikibase docker images, so I’ll be using it as an example showing how the docker images can be manipulated to configure MediaWiki, Wikibase, and load custom extensions and skins.

The example docker-compose file at the time of writing this post can be found at https://github.com/wmde/wikibase-docker/blob/5919016eac16c5f0aefc448240fdf6a09bb56bec/docker-compose.yml

Since the last blog post new wikibase image tags have been created (the ‘bundle’ tags) that include some extensions you might want to enable, as well a quickstatements image for the quickstatements service used on Wikidata written by Magnus Manske.

Continue reading

Wikibase of Wikibases

April 30, 2018 / / 1 Comment

The Wikibase registry was one of the outcomes of the first in a series of Federated wikibase workshops organised in partnership with the European research council.

The aim of the registry is to act as a central point for details of public Wikibase installs hosted around the web. Data held about the installs currently includes the URL for the home page, Query frontend URL and SPARQL API endpoint URL (if a query service exists).

During the workshop an initial data set was added, and this can be easily seen using the timeline view of the query service and a query that is explained within this post.

Continue reading

Older posts

Tags

addwiki API Cambridge Analytica Chrome Conference Data DeleteFacebook docker event eventlogging extension facebook for-wmde git GitHub google grafana graphite Hackathon hadoop hive Java JavaScript Jenkins Joomla map MassAction Mediafire mediawiki metrics Orain PHP Presentation Prezi query SWA traveling UK visualization wdqs wikibase Wikidata Wikimedia Wikipedia wmde

Recent Comments

Search

RSS

Privacy

© 2018 Addshore

Theme by Anders Noren — Up ↑