There has been some recent chat once again on the Wikibase telegram groups around importing, and the best approach to import a large amount of data into a Wikibase instance. 2 years ago I started a little GitHub project aimed at profiling the speed of loading using the action API, and various settings, DB versions etc, as well as trying out a bulk load API. And I have just taken the opportunity to take another look at it and try to visualize some of the comparisons given changes through the last 2 years.
In case you don’t want to read and follow everything below, the key takeaways are:
EPS (edits per second) of around 150 are achievable on a single laptop
When testing imports, you really need to test at least 50k items to get some good figures
The 2 ID generation related settings are VERY IMPORTANT if you want to maximise import times
Make async requests, but not too many, likely tuned to the number of CPUs you have serving web requests. You wan near 100% utilization
On the morning of 22nd March 2025 I received and read an email from Google giving me an “update” on my “Google Maps Timeline”, little did I know this was actually telling me they had just lost some of my data…
The email read…
We briefly experienced a technical issue that caused the deletion of Timeline data for some people. We’re reaching out as your account may have been impacted. If you have encrypted backups enabled, you may be able to restore your data. Make sure that you have the latest version of Google Maps, then go to your Timeline. Tap the cloud icon near the top of your screen, and choose a backup to import your data. If you did not have backups turned on, unfortunately you will not be able to recover lost data. We understand that this can be frustrating if you use Timeline to remember places that you’ve visited, and we are taking steps to improve our systems for the future.
I have heard of Google loosing data before (drive files and or photos disappearing and such), or making it inaccessible for people, and so far I’m glad to not have been affected, and have never really dived into these cases before to see if it has happened.
However, it was easy to see in a matter of minutes that ~10 years of location data was indeed gone from my phone… With data only showing from the 6th or 7th of March.
A year or so ago, I imported a bunch of existing AWS resources into a Spacelift stack using Terraform. Parts of this stack included provisioning Github actions secrets from AWS into Github itself. Due to the way the Github provider and Github API work, I was starting to hit into rate limits due to my ever-increasing number of secrets.
Rather than do anything fancy with additional authentications with the Github API, or higher limits or refactorings within the stack, I opted to split the stack out into the more manageable and focused stacks, which I had already started with my latest deployment which had a stack all to itself.
Unfortunately, there is no “super easy” way to do this. I was dreaming of clicking a button and being able to drag and drop configuration and or state between the various stacks, that would be dreamy. But instead I had to code up some simple scripts to help me migrate the state locally.
Do you often find yourself wanting to make a basic (or complex) web app that is client side only and will log users into Wikimedia sites with ease? Me to!
I have been trying this every year or so, and it’s gradually been getting easier. This year it only took me a couple of to get a really nice template web app setup using Vue, Vuetify and a OAuth 2.0 Wikimedia consumer (thanks to the OAuth extension).
Starting off with a default Vuetify app install using vite (commit c3edb0f), you’ll end up with a basic web page that just says welcome to Vuetify. You can copy the code in my commit, or just follow the Vuetify instructions.
In October last year, I wrote a post starting to visualize the connections between Wikibases in the ecosystem that had been found and collected on wikibase.world thanks to my bot that I occasionally run. That post made use of the query service visualizations, and in this post I’ll take the visualizations a step further, making use of IPython notebooks and plotly.
Previously I reported the total number of Wikibases tracked in wikibase.world being around 784, with around 755 being active (however I didn’t write down exactly how I determined this). So I’m going to take another stab at that with some code backing up the determinations, rather than just my late night data ramblings.
All of the data shown in this post is generated from the IPython notebook available on Github, on 16 Feb 2025, based on the data on wikibase.world which is maintained as a best effort system.
General numbers
Metric
Value
Wikibases with properties
777
Wikibases with properties, and more than 10 pages
600
Wikibases with properties, and more than 10 pages, and 1 or more active users
264
Wikibases with properties, and more than 10 pages, and 2 or more active users
129
Wikibases that link to other wikibases
194
Wikibases that only link to non Wikimedia Foundation wikibases
5
Wikibases that link to other wikibases, excluding Wikimedia Foundation
35
A few things of note:
“with properties” is used, as a clear indicator that Wikibase is not only installed, but also used in at least a very basic way. (ie, it has a created Wikibase property). I would use the number of items ideally as a measure here, however as far as I can tell, this is hard to figure out?)
“with more than 10 pages” is my baseline measure of the site having some content, however this applies across all namespaces, so can also be wikitext pages…
“active users” are taken from MediaWiki statistics, and apply across all namespaces. These numbers also rely on MediaWiki being correctly maintained and these numbers actually being updated. (Users who have performed an action in the last 30 days)
“link to other wikibases” are links extracted from sites by Addbot either via external links or specific properties that state they are links to other wikibases. (The code is not pretty, but gives us an initial view)
And summarized in words:
264 Wikibases with some content that have been edited in the past 30 days
194 Wikibases link in some way to other Wikibases
Excluding links to Wikidata and Commons, this number comes down to 35 (So Wikidata is very much the centre)
And of course, take all of this with a pinch of salt, these numbers are an initial stab at trying to have an overview of the ecosystem.
An updated web
My October post included some basic visualizations from the query service of wikibase.world.
However, it’s time to get a little more fancy and interactive. (As well as showing all wikibases, not just the linked ones)
mcwli includes the third of so generation of “developer environments” that I have made for MediaWiki over the years. You can see the backstory in this earlier post.
Since the early days of 2022, there has been optional metric collection included within the mwcli tool.
This metric collection simply collects what command you run, and when you run it (without any parameters or inputs) so that the data can be aggregated, and the various commands usage can be graphed.
Each command run includes something like:
Command
docker mediawiki exec
DateTime
2025-01-07T12:45:18.213Z
Version
0.25.1
I used to have live (ish) graphs in the Wikimedia Superset installation, however, the queries there appear to fail now. So I took some time to export the dataset as a CSV, and shove it around a bit in a Python notebook.
As one of the initial steps in my smart home journey back in November 2023, I purchased a set of 12 Aqara Temperature and Humidity Sensors for £101 (approximately £8 each) from Aliexpress. Today, this same set is available for around £89, or £7.40 each.
Since then, my Zigbee mesh has continued to expand with a bunch of powered devices, and I have had to start my first round of battery changes on the Aqara sensors. There have also been some power cuts leading to my hub turning off, many home assistant restarts and upgrades, and I feel that I have some more to share.
To this date I have still only permanently fixed one of these sensors in any way, most of them are just resting in nice out-of-the-way places, or on top of door frames.
A very different year when compared to 2023 (where I spent most of the year on a boat).
I’m back working full-time, and primarily in a closed source space (currently) at Lightbug. I’m primarily cloud focused, but have been moving closer and closer to the firmware and hardware month by month. I’m looking forward to releasing some new things this year, including something with an ESP on board!
It’s been the first year that I have stepped back from Wikimedia since starting as a volunteer, and then also working there full-time, and that has been a refreshing break.
Within the Wikimedia world, my primarily volunteer focuses have been and will continue to be:
At some point in 2024 I bought an Aqara FP2 human presence sensor to try and automate the lights in my main open-plan living space. I have been running the installation for nearly a year, now have plans for the future and want to take this opportunity to do a quick write-up on how I have things configured, and how they have been working.
The room
Firstly, a little introduction to the space that I’m playing with…
It’s a kitchen, dinning, living space, with a main single entrance door, as well as large opening bifold doors to the back (both highlighted with red lines). The main flow of people in and out is indicated by the orange arrow on the right, which comes through the main single entrance door.
