[Users] Jenkins down due to suspected security compromise

[Ian Hinder]

On 3 Jul 2017, at 17:32, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

> 
> On 20 Jun 2017, at 13:45, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
> 
>> On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
>> 
>>> Hi all,
>>> 
>>> The security team at NCSA have blocked access to the ET Jenkins server due to a suspected security compromise.  We are investigating.
>>> 
>>> If you have in the past configured a jenkins build node which can be accessed from the jenkins master via ssh (i.e. you have added the jenkins public ssh key to an authorized_keys file), then you should immediately remove this key.  
>>> 
>>> Note that none of the jenkins build nodes apart from the one also hosted at NCSA was working at the time, so it's unlikely that any further attack was possible to those machines.
>>> 
>>> We have backups from before the incident, so assuming we can fix the vulnerability, we should be able to get the system up and running in a few days.
>> 
>> Hi,
>> 
>> A quick update:
>> 
>> I have recreated the Jenkins master and build nodes from backups, and have the new machines running. I am still waiting to hear from the NCSA security team concerning exactly what the vulnerability was.  I can't make Jenkins available publicly until we are confident that the vulnerability is not still exposed.
> 
> Hi all,
> 
> We are fairly sure that the problem was due to a security flaw in Jenkins itself, which was exploited to install a bitcoin miner (seehttps://security.stackexchange.com/questions/160068/kworker34-malware-on-linux,).  Apart from the presence of this miner, we can't find any other indication of a problem. Nevertheless, we have restored from backup and upgraded Jenkins.
> 
> This flaw was announced and patched on 26-Apr-2017 (https://jenkins.io/security/advisory/2017-04-26/), but Jenkins was not being updated automatically on this machine (other ubuntu security updates were being applied automatically, but Jenkins was obtained from the Jenkins PPA, and not included in the unattended-upgrades list). We have added Jenkins to unattended-upgrades.
> 
> There are a number of outdated plugins which should also be upgraded to be safe, but these are not all backward compatible, so this needs to be done with care (and backups).  I don't want to make Jenkins accessible publicly until this is done.
> 
> For ET members with ssh accounts on login.barrywardell.net, you can access Jenkins using
> 
> ssh -L 8080:192.168.0.28:443 -Nv login.barrywardell.net
> 
> and browsing to https://localhost:8080.  You will need to agree to the mismatched SSL certificate.

Hi all,

The Einstein Toolkit build and test system "Jenkins" server is online and accessible again via

	https://build-test.barrywardell.net

We still have the 6 failures (https://build-test.barrywardell.net/job/EinsteinToolkit/lastCompletedBuild/testReport/) which were previously reported.  

I have:

- Recovered the build machine from a backup from before the intrusion
- Reset the ssh host key
- Regenerated the Jenkins ssh private key
- Upgraded the OS to Ubuntu 16.04 LTS
- Upgraded Jenkins to the latest version
- Ensured that automatic security updates are applied to Jenkins (previously they were not, due to an oversight)
- Upgraded all plugins to the latest versions
- Reset all system and Jenkins passwords

If you have a Jenkins account and would like to know the new password, please let me know.

-- 
Ian Hinder
http://members.aei.mpg.de/ianhin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20170929/bde84882/attachment.html