Downfall of Orain
So, from where I am sat right now, it looks like Orain is dead. It could just be me but Orain has been struggling with issues for a while now and the events that took place last week were basically the final nail in the coffin.
As final nails in coffins go, I don’t see this nail being removed any time soon.
Some brief history and facts
Orain was founded by Dusti and Kudu in July 2013, which means it is currently just over 2 years old. Orain has no paid ‘staff’ but instead is kept on-line by a small team of volunteers. The number of volunteers trying to keep the sites up at any given time has varied although in the past month of so that number dropped to 2. During the 2 years the Orain services have been on a variety of different hosts, including AWS, Ramnode and most recently DigitalOcean.
Last week ( 16 Sept 2015 )
Firstly I will say that I still do not know exactly what happened, or how, but it must have been one of the following things:
- Someone did something stupid with a password. This could have been accidentally posting it somewhere, sharing it with someone or not keeping it in a secure location.
- Someone with access to the firstname.lastname@example.org email forwarder had their email address compromised.
- Someone on the inside decided that it was time for Orain to die…
There are other options but frankly the likely hood of those compared with the list above is slim.
Rough Timeline (UTC)
- 01:44 the CloudFlare password was reset (we have a an IP address relating to the reset of this password).
- At some point the mail DNS records for orain.org were changed, pointing to an external server (not in orain control).
- At some point the password for DigitalOcean was reset, made easy by the fact that this person had control of the email accounts.
- At some point 1 in 2 requests were redirected to a questionable device. You can find an image of the change that was made here.
- 09:20 I woke up to see Orain in a mess and Informed Dusti and others by email while trying to see what on earth happened.
- 16:00 Confirmed that the ATT databases was no longer on the server. A screenshot can be seen here.
- Also confirmed that someone had root access to the servers using the DigitalOcean panel, screenshot can be seen here. (It should be noted this shows the root user as idle for 9 hours at 16:00 UTC, meaning at least for prod5 the user was active last at roughly 07:00.
- At some point in the afternoon / evening all machines were powered down.
What I can say with 100% certainty.
I have backups from 15th June 2015 @ 18:00 UTC for all wikis that existed at that time and I am more than happy to give these to people.
- EDIT: Backups from August 2015 are available on archive.org
- The ATT database was deleted, but I was not able to SSH to the primary database server so those databases may not have been deleted.
- As the user had root on all servers via the DigitalOcean control panel it should be assumed that ALL data was / could have been compromised. This includes usernames, email addresses, names and hashed & salted passwords. This also includes access logs meaning IP addresses, user agents and request data which could all be tied to users.
- I do not have any backups of the uploads, although these had not been deleted before the machines were powered down.
- Right now I have no idea if the machines were simply powered down or deleted (they are only VPSs after all)
- At this time I believe Dusti is trying to gain access back to the DigitalOcean and Cloudflare accounts, until this happens it’s hard to really say or do anything more.
Possible conclusions to all of this
- Orain gets access back to DO, the servers are still there, it is powered up and the dbs & uploads are still there
- Orain gets access back to DO, the servers are still there, it is powered up and a mixture of dbs and uploads are still there.
- Orain gets access back to DO, the servers are still there, it is powered up and all the dbs are gone & the uploads are gone.
- Orain gets access back to DO, the servers are gone…
- Orain does not get access back to DO…
EDIT (well, option 6 here happened.)
I am happy to answer any questions I can, although basically everything I can say is written above.
As I previously said I would have expected the founders of Orain to inform the users of Orain of the events, but apparently they haven’t found the time to, or don’t want to, or a mixture. I hope that they will soon.
Personally I want to make try to help everyone that did have a wiki with Orain, I have the backups and am of course willing to give them to the wiki owners so that they can move to new hosting.
Other Orain posts
I have a few other posts about Orain, you can find them below.