Review & Removal of Wikidata query service Blazegraph JNL on Cloudflare R2

Back in August, I uploaded a new Wikidata query service Blazegraph JNL file to both Cloudflare and the Internet Archive. 4 months on, it is time for me to remove the R2 version of this file, which is costing me around 18 USD per month to store, and fall back to the Internet Archive version … Read more

Smart home: Starting with OPNSense Router, Eero Wi-Fi and a pile of cables

I recently moved into a home, that whether I like it or note, is rather “smart”.

There is a Ring video doorbell, Ring camera out the back and Wi-Fi radiators throughout, not to mention the Wi-Fi fridge, dishwasher, washing machine, hot water tank and Amazon Echo Dot that I was recently gifted.

In total, I think there are around 18 Wi-Fi devices in the house before I add any of my own.

Router choice

When first moving in, I only had an old Netgear WNR1000v3 router that would barely give a single client 40MBps connection, let alone be able to support 20+ devices moving forward. It also really couldn’t handle the fibre internet speed, given it is from 2011.

I also tried an old Plusnet hub that I had kicking around, but apparently they lock the PPPoe details to only allow Plusnet accounts?

Trusting the advice of my dear friend Ollie after discussing his “2022.11.14 Low Power Firewall” research spreadsheet, I settled on buying a Beelink U59 Pro N5105 mini PC (that would also arrive the next day).

You can find a good performance review including power benchmarks for this device on CNX Software.

Read more

COVID-19 Wikipedia pageview spikes, 2019-2022

Back in 2019 at the start of the COVID-19 outbreak, Wikipedia saw large spikes in page views on COVID-19 related topics while people here hunting for information.

I briefly looked at some of the spikes in March 2020 using the easy-to-use pageview tool for Wikimedia sites. But the problem with viewing the spikes through this tool is that you can only look at 10 pages at a time on a single site, when in reality you’d want to look at many pages relating to a topic, across multiple sites at once.

I wrote a notebook to do just this, submitted it for privacy review, and I am finally getting around to putting some of those moving parts and visualizations in public view.

Methodology

It certainly isn’t perfect, but the representation of spikes is much more accurate than looking at a single Wikipedia or set of hand selected pages.

  1. Find statements on Wikidata that relate to COVID-19 items
  2. Find Wikipedia site links for these items
  3. Find previous names of these pages if they have been moved
  4. Lookup pageviews for all titles in the pageview_hourly dataset
  5. Compile into a gigantic table and make some graphs using plotly

I’ll come onto the details later, but first for the…

Graphics

All graphics generally show an initial peak in the run-up to the WHO declaring an international public health emergency (12 Feb 2020), and another peak starting prior to the WHO declaring a pandemic.

Be sure to have a look at the interactive views of each diagram to really see the details.

COVID-19 related Wikimedia pageviews (interactive view)

Read more

See all Windows 11 network data usage

Windows 11 (and possibly previous versions of windows) have a data usage view built into the advanced network settings view.

This feature allows users to monitor and manage their data usage on both Wi-Fi and wired connections, and I assume also data connections if your device can be connected via a SIM.

The Data usage page only allows you to see the current usage of networks that you are connected to, and doesn’t allow you to get a view of the whole picture.

For example, my current “Ethernet 4” usage is 7.2GB in the last 30 days, and the current Wi-Fi network that I am on has 97.1GB usage in the last 30 days.

However, I spend lots of time on other networks, and would love to know my overall data usage in the past 30 days.

Where is the data?

I figured all of the data was stored somewhere on disk, the real questions was where.

After a fair bit of googling I came accros “SRUM” or “System Resource Usage Monitor”, and “SRUDB.dat” referenced quite a lot:

Read more

If you have a sandwich and cut it in half, do you have one or two sandwiches

A few years ago, I asked a similarly silly question about lasagne to a group of people I know and summarized their responses. This time I’m bored on a coach, so asked 30–40 people the question “If you have a sandwich and cut it in half, do you have one or two sandwiches”.

In this post, I’ll occasionally refer to individuals by 2 letter initials to keep their personal sandwich views mostly private, while still allowing for a good laugh and for them to follow the analysis.

A few people that have ended up in both lasagne and sandwich groups noticed what I was doing and started reminiscing about the lasagne topic, shout out to HD, GC & LM…

From the provided opinions, there are several distinct groupings based on the perspectives shared.

But to introduce the problem, let’s take a look at this sandwich platter I found online, and how many sandwiches people think it delivers.

Read more

Autoreload go code on code change

For developers, the ability to autoreload code upon any changes is nothing short of a game-changer. It not only streamlines the development process but also fosters an environment of continuous improvement and experimentation.

There are so many packages for languages such as JavaScript for such a behaviour, but I struggled to easily find a simple-to-use go version with an example.

And that’s where this post comes in, as a simple to-the-point example extracted from what I learnt while reading https://dev.to/jacobsngoodwin/full-stack-memory-app-01-setup-go-server-with-reload-in-docker-62n

We will be making use of https://github.com/cespare/reflex, which is a go package and small tool to watch a directory and rerun a command when certain files change. 

Read more

Automatic cobra command registration with fx

Cobra is a popular Go package for creating CLIs. It provides a lot of functionality for creating commands, subcommands, and flags. However, it can be tedious to manually register all of your commands.

fx is a Go package that provides a dependency injection framework. It can be used to automatically register your application components, including but not limited to Cobra commands.

In this blog post, I will show you how you can use fx to automatically register your Cobra commands.

This code was written 5 minutes ago, but works, and I imagine it could help folks bootstrap more complex CLIs rapidly.

The problem

When you create a Cobra command, you need to manually register it with the root command. This can be tedious, especially if you have a lot of commands.

For example, the following code registers a command called hello:

func NewHelloCmd() *cobra.Command {
  cmd := &cobra.Command{
    Use:   "hello",
    Short: "Say hello",
    Run: func(cmd *cobra.Command, args []string) {
      cmd.Println("Hello, world!")
    },
  }
  return cmd
}

func main() {
  rootCmd := &cobra.Command{
    Use:   "myapp",
    Short: "My application",
  }
  rootCmd.AddCommand(NewHelloCmd())
  rootCmd.Execute()
}Code language: Go (go)

Read more

A copy-paste go SQL mock for GORM

When it comes to writing robust and reliable tests for your Go applications, having a well-structured and efficient testing setup is crucial. One common challenge in testing Go applications is dealing with database interactions. To ensure that your code functions correctly, it’s essential to create a controlled environment for database operations during testing. In this blog post, we’ll explore how to create a mock GORM database for testing purposes, allowing you to isolate and verify your database interactions in a controlled manner.

Here is some copy-and-paste code (explained below) which should get you started.

import (
	"github.com/DATA-DOG/go-sqlmock"
	"gorm.io/gorm"
	"gorm.io/gorm/logger"
)

func NewMockGORM() (GORM *gorm.DB, close func()) {
	db, mock, err := sqlmock.New()
	if err != nil {
		panic(err)
	}

	// GORM always runs this query, so mock it for all tests
	mock.ExpectQuery("SELECT VERSION()").WillReturnRows(sqlmock.NewRows([]string{"version"}).AddRow("5.7.0"))

	GORM, err = gorm.Open(mysql.New(mysql.Config{Conn: db}), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
	if err != nil {
		panic(err)
	}

	return GORM, func() { db.Close() }
}Code language: PHP (php)

The code block above defines a NewMockGORM function that sets up a mock GORM database instance for testing. Let’s break down what this code does and how it can be a valuable addition to your testing toolkit.

Setting up a Mock GORM Database

Read more

Verifying Wikimedia user page links on Mastodon

While reviewing the ongoings of the 2023 Wikimedia hackathon, I learned about the RealMe MediaWiki extension, which is already deployed to Wikimedia sites and allows verification of URLs that appear on user pages within other software or platforms, such as Mastodon.

Link verification for dummies

Imagine you want to show that your online profiles, like on Mastodon, truly belong to you. One way to do this is by using a special code called “rel=me”. It’s like saying, “Hey, this link over here is connected to me.” However, there’s a catch: both the link and the page it points to need to say they’re connected.

On platforms like Mastodon, you can add links to your other profiles. The platform then checks if those profiles also point back to your original page using the same “rel=me” code. If they do, your link gets a stamp of approval, showing it’s really yours.

The RealMe extension allows you to configure a set of links on your user page that include this “rel=me” special code that other systems, such as Mastodon, can check.

Configuring it

This one took me a few minutes to get working after reading the instructions, but on meta.wikimedia.org I added a link to my Mastodon profile, enabled the setting on meta, headed over to my Mastodon profile to add the link, and tada, it is verified!

Read more

Dependency injection in go using fx, and replacing services for test

I’m writing a new go application and ended up giving fx (by uber) a try for dependency injection. The getting started docs were brilliant for my use case (creating an API), but the examples for how to inject mock services for tests were lacking, so I decided to write some code examples of how I am currently using fx in tests.

General app setup

I set up my whole application in a app package, which pulls in all services that are needed.

Most of these are provided as constructor functions, with some more dynamic registration of routes as is done in the getting started docs.

package app

func New(additionalOpts ...fx.Option) *fx.App {
	return fx.New(
		fx.Provide(
			config.NewDefault,
			ses.NewSES,
			ses.NewEmailSender,
			mysql.NewGormGB,
			middleware.NewLimiterFactory,
			middleware.NewAuth,
			AsRoute(handlers.UserLogin),
			AsRoute(handlers.ListStuff),
			fx.Annotate(
				router.NewGinRouter,
				fx.ParamTags(`group:"routes"`),
			),
			server.NewHTTPServer,
		),
		fx.Invoke(func(*http.Server) {}),
		fx.Options(additionalOpts...),
	)
}Code language: Go (go)

Read more