Wikimedia Grafana graphs of Wikidata profiling information

September 4, 2015 0 By addshore

I recently discovered the Wikimedia Grafana instance. After poking it for a little while here are some slightly interesting graphs that I managed to extract.

The graphs

Now most of these are quite pointless, at this early stage I am still trying to work out what half of the data stored in Graphite means. But they are still interesting to look at…

The image above relates to one of my recent blogposts. It shows the removal of the lib serialization code from wikibase and the switch to the new DataModel serialization code. The top graph shows that the Lib code was totally removed and is no longer called at all. The lower graph shows that the DataModel serialization code was infact called in various parts of the code base prior to the switch of the API, although the API is clearly be biggest user of the serialization code.

For quite a while now I have wanted to know which of the wikidata api modules get the most use and which are not used at all / barely used. This graph shows that the wbgetclaims and wbgetentities modules are the most used, which makes sense, accessing data happens more often than writing data. Because of the 1:1000 sample rate from the profiling however the details stored for the less used modules is rather useless.

Now this is quite a pointless graph, simply the construction counts of what I assume are 2 of the most used objects in the datamodel.

The WikibaseLuaBindings class contains the methods that Lua binds to in order to access data. This graph shows use that the most used method is getEntityId and that getDescription is barely used.

This is again quite a pointless graph but it does show us the general construction amounts for out DataValue objects and thus we can see which is generally used more. This makes the composition of objects very apparent with GlobeCoordinateValue constructions basically matching LatLongValue constructions, as they use each other.

The Legacy Deserializer code is in a separate component and is used to deserialize old / legacy versions of entities that are still stored in the database. If we could look at this over a period of a year for example rather than a month we would probably expect this to slowly trend down. The graph above shows a month which doesn’t really show us anything other than that it is still used!

The DataModel Services library is new and hence had no use prior to the deployment clearly visible above. You can read a blog post about the component here. Below you can see green and purple lines vanishing around the time of this deployment as again code moves from using deprecated classes in Lib to this new component.

The dashboard

Graphana is quite a nice tool, although it is a shame that the whole UI just seemed to be a bit broken, which is very similar to my first impressions of most of these graphing tools…

As a result of not being able to create dashboards using the tool I had to resort to downloading some sample JSON snippets from other dashboards and trying to glue them all together.

The JSON used to generate the graphs above can be seen below, to use it you need to go to http://grafana.wikimedia.org/#/dashboard/file/empty.json, click on ‘Search’ in the top right and then click on ‘Import’ and load the JSON! File downloadable here.

{
  "id": null,
  "title": "Xhprof General",
  "originalTitle": "Xhprof General",
  "tags": [
    "API"
  ],
  "style": "dark",
  "timezone": "utc",
  "editable": true,
  "hideControls": false,
  "sharedCrosshair": true,
  "rows": [
    {
      "title": "xhprof $class $method",
      "height": "350px",
      "editable": true,
      "collapse": false,
      "panels": [
        {
          "id": 13,
          "span": 12,
          "type": "graph",
          "x-axis": true,
          "y-axis": true,
          "scale": 1,
          "y_formats": [
            "short",
            "short"
          ],
          "grid": {
            "max": null,
            "min": null,
            "leftMax": null,
            "rightMax": null,
            "leftMin": 0,
            "rightMin": null,
            "threshold1": null,
            "threshold2": null,
            "threshold1Color": "rgba(216, 200, 27, 0.27)",
            "threshold2Color": "rgba(234, 112, 112, 0.22)",
            "thresholdLine": false
          },
          "resolution": 100,
          "lines": true,
          "fill": 0,
          "linewidth": 2,
          "points": false,
          "pointradius": 5,
          "bars": false,
          "stack": false,
          "spyable": true,
          "options": false,
          "legend": {
            "show": true,
            "values": true,
            "min": false,
            "max": true,
            "current": false,
            "total": true,
            "avg": true,
            "hideEmpty": true,
            "alignAsTable": true
          },
          "interactive": true,
          "legend_counts": true,
          "timezone": "browser",
          "percentage": true,
          "zerofill": true,
          "nullPointMode": "null",
          "steppedLine": false,
          "tooltip": {
            "value_type": "cumulative",
            "query_as_alias": true,
            "shared": false
          },
          "targets": [
            {
              "target": "aliasByNode(sortByMaxima(MediaWiki.xhprof.$class.$method.$what.$value), 2, 3, 4, 5)",
              "hide": false
            }
          ],
          "aliasColors": {},
          "aliasYAxis": {},
          "title": "",
          "datasource": "graphite",
          "renderer": "flot",
          "annotate": {
            "enable": false
          },
          "seriesOverrides": [],
          "leftYAxisLabel": "",
          "height": "",
          "timeFrom": null,
          "timeShift": null,
          "links": []
        }
      ],
      "showTitle": true
    }
    ],
  "nav": [
    {
      "type": "timepicker",
      "collapse": false,
      "enable": true,
      "status": "Stable",
      "time_options": [
        "1h",
        "6h",
        "12h",
        "24h",
        "2d",
        "7d",
        "30d",
        "6M"
      ],
      "refresh_intervals": [
        "1m",
        "5m",
        "15m",
        "30m",
        "1h="
      ],
      "now": true,
      "notice": false,
      "nowDelay": ""
    }
  ],
  "time": {
    "from": "now-30d",
    "to": "now"
  },

  "templating": {
    "list": [
      {
        "type": "query",
        "datasource": "graphite",
        "refresh_on_load": false,
        "name": "class",
        "options": [],
        "includeAll": true,
        "allFormat": "wildcard",
        "refresh": true,
        "query": "MediaWiki.xhprof.{Wikibase,Wikidata}*",
        "current": {
          "text": "",
          "value": ""
        }
      },
      {
        "type": "query",
        "datasource": "graphite",
        "refresh_on_load": false,
        "name": "method",
        "options": [],
        "includeAll": true,
        "allFormat": "wildcard",
        "refresh": true,
        "query": "MediaWiki.xhprof.$class.*",
        "current": {
          "text": "",
          "value": ""
        }
      },
      {
        "type": "query",
        "datasource": "graphite",
        "refresh_on_load": false,
        "name": "what",
        "options": [],
        "includeAll": true,
        "allFormat": "wildcard",
        "refresh": true,
        "query": "MediaWiki.xhprof.$class.$method.*",
        "current": {
          "text": "null",
          "value": "null"
        }
      },
      {
        "type": "query",
        "datasource": "graphite",
        "refresh_on_load": false,
        "name": "value",
        "options": [],
        "includeAll": true,
        "allFormat": "wildcard",
        "refresh": true,
        "query": "MediaWiki.xhprof.$class.$method.$what.*",
        "current": {
          "text": "null",
          "value": "null"
        }
      }
    ],
    "enable": true
  },
  
  "annotations": {
    "list": [
      {
        "name": "Show deployments",
        "datasource": "graphite",
        "showLine": true,
        "iconColor": "rgba(203, 244, 238, 0.82)",
        "lineColor": "rgba(31, 134, 168, 0.59)",
        "iconSize": 9,
        "enable": false,
        "target": "exclude(aliasByNode(deploy.*.count,-2),\"all\")"
      }
    ],
    "enable": true
  },
  "refresh": "1m",
  "version": 6,
  "hideAllLegends": false
}

Enjoy!

And who knows, I might be able to pull some more things out soon……