[Users] Jenkins down due to suspected security compromise

[Ian Hinder]

On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

> Hi all,
> 
> The security team at NCSA have blocked access to the ET Jenkins server due to a suspected security compromise.  We are investigating.
> 
> If you have in the past configured a jenkins build node which can be accessed from the jenkins master via ssh (i.e. you have added the jenkins public ssh key to an authorized_keys file), then you should immediately remove this key.  
> 
> Note that none of the jenkins build nodes apart from the one also hosted at NCSA was working at the time, so it's unlikely that any further attack was possible to those machines.
> 
> We have backups from before the incident, so assuming we can fix the vulnerability, we should be able to get the system up and running in a few days.

Hi,

A quick update:

I have recreated the Jenkins master and build nodes from backups, and have the new machines running. I am still waiting to hear from the NCSA security team concerning exactly what the vulnerability was.  I can't make Jenkins available publicly until we are confident that the vulnerability is not still exposed.

The same 5 tests that had been failing before are still failing, but I don't see any failures in McLachlan.

 GRHydro.GRHydro_test_shock_weno/1procs	
 GRHydro.GRHydro_test_shock_weno/2procs	
 SphericalHarmonicRecon.regression_test/2procs	
 SphericalHarmonicReconGen.SpEC-dat-test/2procs	
 SphericalHarmonicReconGen.SpEC-h5-test/2procs

-- 
Ian Hinder
http://members.aei.mpg.de/ianhin