On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
Hi all,
The security team at NCSA have blocked access to the ET Jenkins server due to a suspected security compromise. We are investigating.
If you have in the past configured a jenkins build node which can be accessed from the jenkins master via ssh (i.e. you have added the jenkins public ssh key to an authorized_keys file), then you should immediately remove this key.
Note that none of the jenkins build nodes apart from the one also hosted at NCSA was working at the time, so it's unlikely that any further attack was possible to those machines.
We have backups from before the incident, so assuming we can fix the vulnerability, we should be able to get the system up and running in a few days.
Hi,
A quick update:
I have recreated the Jenkins master and build nodes from backups, and have the new machines running. I am still waiting to hear from the NCSA security team concerning exactly what the vulnerability was. I can't make Jenkins available publicly until we are confident that the vulnerability is not still exposed.
The same 5 tests that had been failing before are still failing, but I don't see any failures in McLachlan.
GRHydro.GRHydro_test_shock_weno/1procs GRHydro.GRHydro_test_shock_weno/2procs SphericalHarmonicRecon.regression_test/2procs SphericalHarmonicReconGen.SpEC-dat-test/2procs SphericalHarmonicReconGen.SpEC-h5-test/2procs
Hello Ian,
The same 5 tests that had been failing before are still failing, but I don't see any failures in McLachlan.
GRHydro.GRHydro_test_shock_weno/1procs GRHydro.GRHydro_test_shock_weno/2procs SphericalHarmonicRecon.regression_test/2procs SphericalHarmonicReconGen.SpEC-dat-test/2procs SphericalHarmonicReconGen.SpEC-h5-test/2procs
I pushed fixed for the last two * SphericalHarmonicReconGen.SpEC-dat-test/2procs * SphericalHarmonicReconGen.SpEC-h5-test/2procs that failed with a segfault and hdf5 error, respectively, yesterday Mon Jun 19 13:43:37 2017 -0500 . Do you know if they are still failing for you? Finally this test * SphericalHarmonicRecon.regression_test/2procs is also passing on my laptop.
So: They are no longer failing on my laptop (Sky Lake cpu) using the Ubuntu option list that the jenkins slaves were using.
I attach the option list and cpuinfo output.
Yours, Roland
On 20 Jun 2017, at 17:18, Roland Haas rhaas@illinois.edu wrote:
Hello Ian,
The same 5 tests that had been failing before are still failing, but I don't see any failures in McLachlan.
GRHydro.GRHydro_test_shock_weno/1procs GRHydro.GRHydro_test_shock_weno/2procs SphericalHarmonicRecon.regression_test/2procs SphericalHarmonicReconGen.SpEC-dat-test/2procs SphericalHarmonicReconGen.SpEC-h5-test/2procs
I pushed fixed for the last two
- SphericalHarmonicReconGen.SpEC-dat-test/2procs
- SphericalHarmonicReconGen.SpEC-h5-test/2procs
that failed with a segfault and hdf5 error, respectively, yesterday Mon Jun 19 13:43:37 2017 -0500 . Do you know if they are still failing for you?
SphericalHarmonicReconGen.SpEC-dat-test/2procs is failing with
NewsB_scri.L02Mm01.asc: substantial differences significant differences on 1 (out of 2) lines maximum absolute difference in column 1 is 963 maximum absolute difference in column 2 is 0.000185770963653907 maximum absolute difference in column 3 is 0.000142466608463344 maximum relative difference in column 1 is 1 maximum relative difference in column 2 is 1 maximum relative difference in column 3 is 1
as well as many others. So the segfault has been fixed, but the results are not matching the test data.
The same for SphericalHarmonicReconGen.SpEC-h5-test/2procs.
Finally this test
- SphericalHarmonicRecon.regression_test/2procs
is also passing on my laptop.
I get
Jn_l[0]_2D.asc: substantial differences significant differences on 7440 (out of 11163) lines maximum absolute difference in column 1 is 60 maximum absolute difference in column 2 is 60 maximum absolute difference in column 3 is 0.482964312185403 maximum absolute difference in column 4 is 0.482843028679412 maximum relative difference in column 1 is 1 maximum relative difference in column 2 is 1 maximum relative difference in column 3 is 0.482964312185403 maximum relative difference in column 4 is 0.482843028679412 (insignificant differences on 2989 lines)
-- Ian Hinder http://members.aei.mpg.de/ianhin
Hi
These are results with the change: both 1 and 2-process testsuites:
mpirun noticed that process rank 0 with PID 0 on node spine exited on signal 11 (Segmentation fault).
Other testsuites succeed, so it shouldn't be the machine setup.
Frank
On Tue, Jun 20, 2017 at 03:10:29PM -0500, Frank Loeffler wrote:
These are results with the change: both 1 and 2-process testsuites:
mpirun noticed that process rank 0 with PID 0 on node spine exited on signal 11 (Segmentation fault).
gdb tells me:
#0 NaNChecker::CHECK_DATA<double> () at /home/knarf/TMP/configs/L2/build/NaNChecker/NaNCheck.cc:644 #1 0x00007ffff35b36b6 in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1 #2 0x00007ffff6eb9494 in start_thread (arg=0x7fffd4031700) at pthread_create.c:333 #3 0x00007ffff30cfaff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
So the segfault actually happens in NanChecker, trying to access _data[i=0:nelems] in CHECK_DATA.
Is SphericalHarmonicReconGen doing something strange with the Cactus infrastructure, or does this more point to some memory garbage?
Frank
Hello Frank,
Is SphericalHarmonicReconGen doing something strange with the Cactus infrastructure, or does this more point to some memory garbage?
Are you on the newest version of CactusUtils, in particular have:
1ea4f65 - NaNChecker: fix handling of variables without storage (26 hours ago) <Roland Haas
?
Yours, Roland
On Tue, Jun 20, 2017 at 03:52:59PM -0500, Roland Haas wrote:
Are you on the newest version of CactusUtils, in particular have:
1ea4f65 - NaNChecker: fix handling of variables without storage (26 hours ago) <Roland Haas
I was indeed not, thanks for the pointer (pun intended). Now both testsuites succeed, both using 1 and 2 processes.
Frank
On 20 Jun 2017, at 13:45, Ian Hinder ian.hinder@aei.mpg.de wrote:
On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
Hi all,
The security team at NCSA have blocked access to the ET Jenkins server due to a suspected security compromise. We are investigating.
If you have in the past configured a jenkins build node which can be accessed from the jenkins master via ssh (i.e. you have added the jenkins public ssh key to an authorized_keys file), then you should immediately remove this key.
Note that none of the jenkins build nodes apart from the one also hosted at NCSA was working at the time, so it's unlikely that any further attack was possible to those machines.
We have backups from before the incident, so assuming we can fix the vulnerability, we should be able to get the system up and running in a few days.
Hi,
A quick update:
I have recreated the Jenkins master and build nodes from backups, and have the new machines running. I am still waiting to hear from the NCSA security team concerning exactly what the vulnerability was. I can't make Jenkins available publicly until we are confident that the vulnerability is not still exposed.
Hi all,
We are fairly sure that the problem was due to a security flaw in Jenkins itself, which was exploited to install a bitcoin miner (see https://security.stackexchange.com/questions/160068/kworker34-malware-on-lin...,). Apart from the presence of this miner, we can't find any other indication of a problem. Nevertheless, we have restored from backup and upgraded Jenkins.
This flaw was announced and patched on 26-Apr-2017 (https://jenkins.io/security/advisory/2017-04-26/), but Jenkins was not being updated automatically on this machine (other ubuntu security updates were being applied automatically, but Jenkins was obtained from the Jenkins PPA, and not included in the unattended-upgrades list). We have added Jenkins to unattended-upgrades.
There are a number of outdated plugins which should also be upgraded to be safe, but these are not all backward compatible, so this needs to be done with care (and backups). I don't want to make Jenkins accessible publicly until this is done.
For ET members with ssh accounts on login.barrywardell.net, you can access Jenkins using
ssh -L 8080:192.168.0.28:443 -Nv login.barrywardell.net
and browsing to https://localhost:8080. You will need to agree to the mismatched SSL certificate.
On 3 Jul 2017, at 17:32, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 20 Jun 2017, at 13:45, Ian Hinder ian.hinder@aei.mpg.de wrote:
On Fri, Jun 2, 2017 at 11:25 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
Hi all,
The security team at NCSA have blocked access to the ET Jenkins server due to a suspected security compromise. We are investigating.
If you have in the past configured a jenkins build node which can be accessed from the jenkins master via ssh (i.e. you have added the jenkins public ssh key to an authorized_keys file), then you should immediately remove this key.
Note that none of the jenkins build nodes apart from the one also hosted at NCSA was working at the time, so it's unlikely that any further attack was possible to those machines.
We have backups from before the incident, so assuming we can fix the vulnerability, we should be able to get the system up and running in a few days.
Hi,
A quick update:
I have recreated the Jenkins master and build nodes from backups, and have the new machines running. I am still waiting to hear from the NCSA security team concerning exactly what the vulnerability was. I can't make Jenkins available publicly until we are confident that the vulnerability is not still exposed.
Hi all,
We are fairly sure that the problem was due to a security flaw in Jenkins itself, which was exploited to install a bitcoin miner (seehttps://security.stackexchange.com/questions/160068/kworker34-malware-on-lin...,). Apart from the presence of this miner, we can't find any other indication of a problem. Nevertheless, we have restored from backup and upgraded Jenkins.
This flaw was announced and patched on 26-Apr-2017 (https://jenkins.io/security/advisory/2017-04-26/), but Jenkins was not being updated automatically on this machine (other ubuntu security updates were being applied automatically, but Jenkins was obtained from the Jenkins PPA, and not included in the unattended-upgrades list). We have added Jenkins to unattended-upgrades.
There are a number of outdated plugins which should also be upgraded to be safe, but these are not all backward compatible, so this needs to be done with care (and backups). I don't want to make Jenkins accessible publicly until this is done.
For ET members with ssh accounts on login.barrywardell.net, you can access Jenkins using
ssh -L 8080:192.168.0.28:443 -Nv login.barrywardell.net
and browsing to https://localhost:8080. You will need to agree to the mismatched SSL certificate.
Hi all,
The Einstein Toolkit build and test system "Jenkins" server is online and accessible again via
https://build-test.barrywardell.net
We still have the 6 failures (https://build-test.barrywardell.net/job/EinsteinToolkit/lastCompletedBuild/t...) which were previously reported.
I have:
- Recovered the build machine from a backup from before the intrusion - Reset the ssh host key - Regenerated the Jenkins ssh private key - Upgraded the OS to Ubuntu 16.04 LTS - Upgraded Jenkins to the latest version - Ensured that automatic security updates are applied to Jenkins (previously they were not, due to an oversight) - Upgraded all plugins to the latest versions - Reset all system and Jenkins passwords
If you have a Jenkins account and would like to know the new password, please let me know.
users@lists.einsteintoolkit.org
-
Frank Loeffler -
Ian Hinder -
Roland Haas