Open Sourcing WBStack
Open Sourcing the code and config for WBStack has always been part of the plan, although functionality came first throughout the first year or so. Finally there is a github organization for wbstack containing 16 public repositories that make up the entire deployment for wbstack.com.
This effort took a few weeks trying to split sensible components out of the original mono repo that was started back in 2017 that now has over 1600 commits, making sure that no secrets were swept up along the way, and also trying to preserve git history where possible.
Although everything is now on Github that doesn’t mean that it is clearly understandable just yet, or in the most sensible layout, that will come with time.
The Repositories
- docs – A new VuePress docs site, currently on Github pages.
- deploy – All deployment code and config (minus secrets) for Google Cloud, and Kubernetes for wbstack.com
- ui – The main VueJS based frontend for wbstack.com
- api – The main platform API in PHP using Laravel. Currently acting as a bit of a kitchen sink for prototyping
- mediawiki – A full version of MediaWiki, skins, extensions and config logic for all wbstack.com wikis
- queryservice – Tiny modifications to the regular query service
- queryservice-updater – Java wrapper around the regular query service updater to make it better for running on multiple sites
- queryservice-gateway – Custom node application to route different SPARQL queries to different backend hosts and namespaces
- queryservice-ui – Slightly modified queryservice frontend, mainly styles and config
- mediawiki-extensions-WikibaseInWikitext – A custom MediaWiki extension for a <sparql> tag in wikitext
- quickstatements – Fork of quickstatments with style and config modifications
- widar – Fork of widar with style and config modifications
- cradle – Fork of cradle with style and config modifications
- magnustools – Fork of magnustools library with config modifications
- docker-build-util – Docker utility image used in some build steps
- bitnami-charts – Fork with some small modifications that I should upstream
Builds, Deployment & Reuse
All of the repositories that get deployed now have Github Actions built Docker images, as well as tagged images and a CHANGELOG. The images that are available on Github are the images that are used for wbstack.com. You can currently find a full list of the Docker images here.
I believe all of these images should be usable off the shelf if anyone wanted to try and deploy all of the components, the exception being the ui, which has the API url built into the image. Though any reuse of these components would likely want a custom UI anyway. Anything that seems to get in the way of reuse should be easily fixable.
Upstreaming
Lots of the customizations in these repositories work around the fact that most components have some sort of hardcoded or simplistic way of determining configuration. This is normally a file that can be provided for a single site.
The modifications mostly turn this into a dynamic decision. For both frontend and backend application this means switching or configuring based on the request domain.
Almost all of the modifications that have been made for the wbstack.com use case can likely be upstreamed in one way or another, and I look forward to working toward that in 2021.
Thank you Adam. Oh and by the way, THANK you. One last thing, THANK YOU!
Did you test by chance on Windows 10 WSL2 Docker? If not, I can do that perhaps over the upcoming holidays and let you know and send PR’s for improvements.
I’m running WIndows 10 WSL2 Docker ;)
[…] Open Sourcing WBStack, by […]
[…] year is up, and what a year it has been. I finally open sourced wbstack, I complained about fake news, looked at Minecraft mods and took a look at how COVID-19 was […]
[…] commitment was one of the things that lead to the open-sourcing of the WBStack components at the end of 2020. The plan also lead to a continued investigation of the Wikibase as a Service […]
[…] will be based on the WBStack open-source platform, but will be maintained by a full team of engineers at Wikimedia Germany, included in our annual […]
[…] WBStack, the first Wikibase as a service, was open sourced. […]
[…] always been controlled in git, but the first commit came to the deploy repository when wbstack was fully open-sourced a year or so ago. Before that time it was controlled in a private git repository which remains private […]