[Users] Jenkins down due to suspected security compromise

[Ian Hinder]

On 20 Jun 2017, at 13:45, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

> On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
> 
>> Hi all,
>> 
>> The security team at NCSA have blocked access to the ET Jenkins server due to a suspected security compromise.  We are investigating.
>> 
>> If you have in the past configured a jenkins build node which can be accessed from the jenkins master via ssh (i.e. you have added the jenkins public ssh key to an authorized_keys file), then you should immediately remove this key.  
>> 
>> Note that none of the jenkins build nodes apart from the one also hosted at NCSA was working at the time, so it's unlikely that any further attack was possible to those machines.
>> 
>> We have backups from before the incident, so assuming we can fix the vulnerability, we should be able to get the system up and running in a few days.
> 
> Hi,
> 
> A quick update:
> 
> I have recreated the Jenkins master and build nodes from backups, and have the new machines running. I am still waiting to hear from the NCSA security team concerning exactly what the vulnerability was.  I can't make Jenkins available publicly until we are confident that the vulnerability is not still exposed.

Hi all,

We are fairly sure that the problem was due to a security flaw in Jenkins itself, which was exploited to install a bitcoin miner (see https://security.stackexchange.com/questions/160068/kworker34-malware-on-linux,).  Apart from the presence of this miner, we can't find any other indication of a problem. Nevertheless, we have restored from backup and upgraded Jenkins.

This flaw was announced and patched on 26-Apr-2017 (https://jenkins.io/security/advisory/2017-04-26/), but Jenkins was not being updated automatically on this machine (other ubuntu security updates were being applied automatically, but Jenkins was obtained from the Jenkins PPA, and not included in the unattended-upgrades list).  We have added Jenkins to unattended-upgrades.

There are a number of outdated plugins which should also be upgraded to be safe, but these are not all backward compatible, so this needs to be done with care (and backups).  I don't want to make Jenkins accessible publicly until this is done.

For ET members with ssh accounts on login.barrywardell.net, you can access Jenkins using

ssh -L 8080:192.168.0.28:443 -Nv login.barrywardell.net

and browsing to https://localhost:8080.  You will need to agree to the mismatched SSL certificate.

-- 
Ian Hinder
http://members.aei.mpg.de/ianhin