The Wikimedia Server Admin Log or SAL for short is a timestamped log of actions performed on the Wikimedia cluster by users such as roots and deployers. The log is stored on the WikiTech Wikimedia project and can be found at the following URL: https://wikitech.wikimedia.org/wiki/Server_Admin_Log

An example entry in the log could be:

As well as the main cluster SAL there are also logs for release engineering (jenkins, zuul, and other CI things) and individual logs for each project that uses Wikimedia Cloud VPS.

A tool has been created for easy SAL navigation which can be found at https://tools.wmflabs.org/sal

Each SAL can be selected at the top of the tool, with ‘Other’ providing you with a list of all Cloud VPS SALs.

The search and date filters can then be used to find entries throughout history.

Usages of the SAL

The timestamps of the SAL can be used to determine the causes of cluster or projects issues. The entries can be used alongside other timestamped data in incident reports, for example https://wikitech.wikimedia.org/wiki/Incident_documentation/20180229-Train-1.31.0-wmf.27#Timeline

The SAL messages can sometimes be used to find changes that have otherwise gone entirely undocumented (not in puppet or any git repo).

An example of finding such an undocumented change can be found below with the tmp1 index that was recently found on the wb_terms wikidata table.

Finding the wikidata tmp1 index

Searching for ‘tmp1’ only finds log entries from 2018:

Changing the search a little to look for the table name instead of the index name reveals some different log entries:

The key entry here mentioning gerrit change 99660 which in turn links to bugzilla bug 45529 which is now recorded as phabricator ticket T47529.

Looking through the phabricator comments we can see a first mention of tmp1, https://phabricator.wikimedia.org/T47529#518889.

Writing to SAL

The SAL can of course be written to directly on the various wiki pages however this is not advised at it can of course lead to edit conflicts.

Take a look at https://wikitech.wikimedia.org/wiki/Tool:Stashbot