addshore

Using skaffold.dev with helmfile deployments on minikube

October 7, 2021 by addshore

Recently I setup a small skaffold.dev development environment for a platform deployed using helmfile, deployed to a GKE cluster in production that is moving towards also having a local setup using minikube.

There is an open Github issue requesting helmfile support, so my first fear was that this wouldn’t be possible. Though after a little hacking around this seemed fairly easy by using the skaffold pre deploy hooks.

Here is a quick introduction to this stack, and a solution for using skaffold with helmfile deployments on minikube before “native” support is added.

Tech Lead Digest – August 2021

August 28, 2023September 18, 2021 by addshore

This entry is part 4 of 5 in the series Tech Lead Digest (wmde)

Welcome to the 4th instalment of my Wikidata & Wikibase Tech lead Digest for August 2021. For previous instalments see Q1, Q2 & July.

🧑‍🤝‍🧑Wikidata & Wikibase

The Wikidata Query Builder has been deployed. The Wikidata Query Builder provides a visual interface for building a simple Wikidata query. It is ideal for users with little or no experience in SPARQL.

The Wikibase fall release, which will be compatible with MediaWiki 1.36 will be made in the next month or so. At some point in the next 3-6 months we will likely also make a Wikibase 1.37 release. Keep an eye out on the mailing list for these.

Work is about to wrap up on the next iteration of Wikibase Federated Properties which will enable the use of properties from multiple sources at once, such as Wikidata and also the local Wikibase.

Work continues on the Wikidata Mismatch Finder. You can read more about this future tool on wikidata.org.

The campsite worked on many other things. Most notably Ladsgroup spotted that SpamBlacklist was rendering content on Wikidata twice (phabricator). This fix resulted in a rather significant improvement in save times for Wikidata, and users of Wikibase and SpamBlacklist in combination.

Pulling data into Google Sheets from Phabricator

August 21, 2021 by addshore

Phabricator is a suite of web-based software development collaboration tools, used by Wikimedia and many other organizations. One of these tools is a fairly customizable task tracker.

In a recent post I described the current Technical Debt backlog prioritization for Wikidata and Wikibase. Part of that process currently makes use of Google Sheets for a consolidated overview of a whole collection of tasks that are tracked in Phabricator. (I’m looking forward to trying out the new Github Issues table views soon, but I won’t be able to make use of this for Phabricator tasks)

Copying data between Phabricator and Google Sheets constantly would be a complete pain, especially as new tasks get added to the sheet every day and details of tasks can also change on Phabricator itself, such as titles, and statuses.

Which is where Google Apps Script for Google Sheets and the Phabricator API come in to automate this part of the process, at least in one direction.

If you don’t want to read any more of this post, you can find the source on the addshore/phabricator-google-apps-script Github repository.

All you need to do is paste this into a script for your Google Sheet. You can find this by heading to Tools -> Script Editor.

sudo / elevate on Windows 11

December 3, 2021August 1, 2021 by addshore

Windows has never had a native sudo or elevation feature.

Of course if you write the correct Power Shell commands you can make this happen, but what we really want is convenience.

There are various packages around to help out and various ways to install them.

And in Windows 11 with a new package manager, installing one of them as never been so easy.

Tech Lead Digest – July 2021

August 28, 2023July 31, 2021 by addshore

This entry is part 3 of 5 in the series Tech Lead Digest (wmde)

Welcome to the third installment of my tech lead digest digest. In order to allow myself some extra space to write, and also to provide these public updates and thoughts on a more regular basis, this is now becoming a monthly digest.

I’m going to try to incorporate some of the ongoings from other Wikidata / Wikibase projects, as well as my regular digest and reading.

🧑‍🤝‍🧑Wikidata & Wikibase

Work continues on the next iteration of Wikibase Federated Properties (phabricator board). This work will allow use of properties from multiple sources at once, such as Wikidata and also the local Wikibase.

Work also continues on the Wikidata Mismatch Finder (phabricator board) which is a tool to enable finding mismatches between Wikidata’s data and data in other databases.

The Campsite continues to work on a variety of smaller tasks, in the last month including a new release of our Design System, dealing with the Query Builder security review and preparing for deployment, performing some maintenance on WBStack including preparing for 1.36 and adding Elasticsearch. We also continue to support a university team in deploying a new Property Suggester algorithm (announcement coming soon), work towards tagging all edits made from the UI (T236893), as well as many other smaller tasks.

What happens in Wikibase when you make a new Item?

February 3, 2022July 19, 2021 by addshore

A recent Wikibase email list post on the topic of Wikibase and bulk imports caused me to write up a mostly human readable version of what happens, in what order, and when, for Wikibase action API edits, for the specific case of item creation.

There are a fair few areas that could be improved and optimized for a bulk import use case in the existing APIs and code. Some of which are actively being worked on today (T285987). Some of which are on the roadmap, such as the new REST APIs for Wikibase. And others which are out there, waiting to be considered.

This post is is written looking at Wikibase and MediaWiki 1.36 with links to Github for code references. Same areas may be glossed over or even slightly inaccurate, so take everything here with a pinch of salt.

Reach out to me on Twitter if you have questions or fancy another deep dive.

A first look at Wikidata through Github Copilot

December 3, 2021July 11, 2021 by addshore

I was added to the Github Copilot preview in the past days, and the first thing I tested out was code suggestions around Wikidata.

Copilot allows you to start writing code, and have a model trained on public Github code suggest block of code that you might want to use.

For example, if you start defining a method called onePlusOne then it might suggest the body return 1+1;.

function onePlusOne() {
    return 1 + 1;
}Code language: JavaScript (javascript)

More targeted toward Wikidata, if you start defining a function called whatIsWikidata then it might suggest return "Wikidata";

function whatIsWikidata(){
    return "Wikidata";
}Code language: JavaScript (javascript)

In some cases copilot suggests multiple code blocks. Some of these seem useful, others less so.

Tackling Technical Debt, big and small, in Wikidata and Wikibase

August 21, 2021June 30, 2021 by addshore

If you’re working with legacy code, chances are you’ve inherited some technical debt. Infact, if you’re working with code, chances you’re already surrounded by technical debt of varying sizes, at least by some measures.

Some believe that technical debt is something to be avoided, and that technical debt that exists is a dirty secret that should be hidden. The reality is that technical debt is a fact of life when code iteratively changes to deliver product solutions.

Striving for programming perfection is great in principle, but ultimately code is meant to deliver features, and there is always a good, better and best approach, with many other variations in-between.

Over the last year at Wikimedia Deutschland we have worked on refining how we record, triage, prioritize and tackle technical debt within the Wikidata and Wikibase product family.

There are many thoughts out there about how to track, tackle, and prioritize technical debt. This post is meant to represent the current status of the Wikidata / Wikibase team. Hopefully you find this useful.

Tech Lead Digest – Q2 2021

August 28, 2023June 26, 2021 by addshore

This entry is part 2 of 5 in the series Tech Lead Digest (wmde)

This is the second installment of my tech lead digest digest with my tech lead hat on for the Wikidata Wikibase team.

This is a digest of my internal digest emails. These contain lots of links to reading, podcasts and general goings on that could be useful to a wider audience.

🧑‍🤝‍🧑Wikidata & Wikibase

Federated Properties, Mismatch finder & WBaaS maintenance teams have spun up from the Campsite
The “experimental” patch for Github Dependabot -> Gerrit was merged
ADR 11 rollout it at 26% (slow and steady)
The wikibase-docker repository has been flagged as not in use / legacy and will soon be archived
wikibase-containers phabricator tickets are being closed or moved to the release strategy phabricator tag
There was no strong team consensus on ADR 18, and thus we will likely not maintain backward compatibility with MW 1.36 on master of Wikibase for now
Team Site changes:
- Team list of repositories now includes the languages, tooling used, creation and last touched dates.
- The purtle library (for RDF output) was also added to the list of repositories that we own
- Team tech vision has been updated (using markmap JS)
- An auto collected list of team ADRS has been added.
- A documentation page / portal has been added

Tech Lead Digest – Q1 2021

August 28, 2023May 24, 2021 by addshore

This entry is part 1 of 5 in the series Tech Lead Digest (wmde)

At some point last year I started sending a weekly internal digest to the Wikidata Wikibase team with my tech lead hat on.

The emails are internal only but contain lots of links to reading, podcasts and general goings on that could be useful to everyone.

So here is my first Wikidata Wikibase tech lead digest digest!

🧑‍🤝‍🧑Wikidata & Wikibase

A collection of Wikibase specific logstash dashboards were created
We moves our secondary CI from Travis to Github Actions (ADR)
We archived WikibaseCodesniffer, choosing to use the upstream mediawiki-codesniffer
Deployment of the Wikidata Query Service UI was moved to microsites
wdwb-tech phabricator boards were recreated, funneling towards a prioritized tech backlog.
We created a team site, which contains a list of all code repositories that we own and interact with as a core part of Wikidata / Wikibase. (git repo)
Browser tests came up as a hot topic. A deep dive and some central analysis occurred seemingly correlation failures with “memory compaction” on VMs. This is out of our control, so we increased timeouts in some key areas.