Hi Roland,

It almost worked -- but not quite. The final link failed because, I believe, of a single missing entry point: "__gfortran_os_error_at" .

The error message is:

Undefined symbols for architecture x86_64:
  "__gfortran_os_error_at", referenced from:
      _ahfinder_ in AHFinder.F.o
      _ahfinder_dis_ in AHFinder_dis.F.o
      _ahfinder_int_ in AHFinder_int.F.o
      _linmin_ in AHFinder_pow.F.o
      _ahfinder_initguess_ in AHFinder_initguess.F.o
      _ahfinder_gau_ in AHFinder_gau.F.o
      ___carpet_boxtypes_MOD_splitregionsrecursive in carpet_boxtypes.F90.o
      ...
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[1]: *** [/Users/murray/Cactus/exe/cactus_sim] Error 1
make: *** [sim] Error 2

The compilations completed without errors, but with a fair number of warnings.

I'm googling ("mac gfortran_os_error_at") to see what I can learn. I've found some possibilities, but nothing that seems terribly promising. I brew reinstalled gfortran and I re-downloaded the CactusTutorial.ipynb notebook and I'm trying again.

The file OptionList I used before sets
FPP = cpp-10
F90 = gfortran

That file also specifies:
FPPFLAGS = -traditional
F90FLAGS = -g -fcray-pointer -ffixed-line-length-none
F90_OPTIMISE_FLAGS = -O2
F90_WARN_FLAGS = -Wall
LDFLAGS = -rdynamic
F90_OPENMP_FLAGS =
F90_PROFILE_FLAGS = -pg

I get
> which gfortran
/usr/local/bin/gfortran
❯ gfortran --version
GNU Fortran (Homebrew GCC 10.2.0_4) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> which gfortran-10
/usr/local/bin/gfortran-10
❯ gfortran-10 --version
GNU Fortran (Homebrew GCC 10.2.0_4) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Possibly I should add a link library somewhere, but I don't know what, where or how. (Is it possible I inadvertently deleted a link library specification in the OptionList file or elsewhere? I thought I saved the original file, but I overwrote it, which is part of the reason I'm trying again from the start. The only difference I found from the OptionsList file I used is that the correct file sets F90 = gfortran-10. But gfortran is symlinked to gfortran-10, so I don't expect that'll make a difference.)

Any thoughts on fixing the missing reference?

Again, thanks!

Murray

__________________________________________________________________

The build process takes slightly over 3 hours on my Mac. The final output of the sim build, clean, reconfig run is:

---------------------------------------------
Creating cactus_sim in /Users/murray/Cactus/exe from EinsteinAnalysis/ADMAnalysis EinsteinBase/ADMBase EinsteinBase/ADMCoupling Llama/ADMDerivatives EinsteinBase/ADMMacros EinsteinAnalysis/ADMMass Numerical/AEILocalInterp EinsteinAnalysis/AHFinder EinsteinAnalysis/AHFinderDirect ExternalLibraries/BLAS WVUThorns/Baikal WVUThorns/BaikalVacuum CactusBase/Boundary CTThorns/CT_Analytic CTThorns/CT_MultiLevel EinsteinAnalysis/CalcK Carpet/Carpet Carpet/CarpetEvolutionMask Carpet/CarpetIOASCII Carpet/CarpetIOBasic Carpet/CarpetIOHDF5 Carpet/CarpetIOScalar Carpet/CarpetIntegrateTest Carpet/CarpetInterp Carpet/CarpetInterp2 Carpet/CarpetLib Carpet/CarpetMask Carpet/CarpetProlongateTest Carpet/CarpetReduce Carpet/CarpetRegrid Carpet/CarpetRegrid2 Carpet/CarpetRegridTest Carpet/CarpetSlab Carpet/CarpetTracker CactusBase/CartGrid3D CactusNumerical/Cartoon2D EinsteinBase/Constants WVUThorns/Convert_to_HydroBase CactusBase/CoordBase EinsteinBase/CoordGauge Llama/Coordinates Llama/CoordinatesSymmetry Carpet/CycleClock CactusExamples/DemoInterp CactusNumerical/Dissipation EinsteinInitialData/DistortedBHIVP EinsteinAnalysis/EHFinder EinsteinBase/EOS_Base EinsteinEOS/EOS_Hybrid EinsteinEOS/EOS_IdealFluid EinsteinEOS/EOS_Omni EinsteinEOS/EOS_Polytrope EinsteinExact/EinsteinExact_Test CactusElliptic/EllBase CactusElliptic/EllSOR EinsteinInitialData/Exact EinsteinAnalysis/Extract ExternalLibraries/FFTW3 WVUThorns/FishboneMoncriefID CactusExamples/FleshInfo CactusUtils/Formaline CactusBase/Fortran EinsteinEvolve/GRHydro EinsteinEvolve/GRHydro_InitData ExternalLibraries/GSL EinsteinExact/GaugeWave KrancNumericalTools/GenericFD WVUThorns/GiRaFFE WVUThorns/GiRaFFE_to_HydroBase WVUThorns/GiRaFFEfood Llama/GlobalDerivative ExternalLibraries/HDF5 CactusConnect/HTTPD CactusConnect/HTTPDExtra CactusExamples/HelloWorld Carpet/HighOrderWaveTest EinsteinBase/HydroBase EinsteinAnalysis/Hydro_Analysis EinsteinInitialData/Hydro_InitExcision EinsteinInitialData/Hydro_RNSID EinsteinInitialData/IDAnalyticBH EinsteinInitialData/IDAxiBrillBH EinsteinInitialData/IDAxiOddBrillBH EinsteinInitialData/IDBrillData EinsteinInitialData/IDConstraintViolate EinsteinInitialData/IDFileADM EinsteinInitialData/IDLinearWaves CactusWave/IDScalarWave CactusWave/IDScalarWaveC CactusWave/IDScalarWaveCXX CactusWave/IDScalarWaveElliptic CactusExamples/IDWaveMoL WVUThorns/ID_converter_GiRaFFE WVUThorns/ID_converter_ILGRMHD CactusBase/IOASCII CactusBase/IOBasic CactusPUGHIO/IOHDF5 CactusPUGHIO/IOHDF5Util CactusIO/IOJpeg CactusBase/IOUtil WVUThorns/IllinoisGRMHD CactusBase/InitBase CactusNumerical/InterpToArray Llama/Interpolate2 EinsteinExact/KerrSchild ExternalLibraries/LAPACK ExternalLibraries/LORENE Lean/LeanBSSNMoL Llama/LlamaWaveToy CactusNumerical/LocalInterp CactusNumerical/LocalInterp2 CactusNumerical/LocalReduce Carpet/LoopControl McLachlan/ML_ADMConstraints McLachlan/ML_ADMQuantities McLachlan/ML_BSSN McLachlan/ML_BSSN_Helper McLachlan/ML_BSSN_Test McLachlan/ML_CCZ4 McLachlan/ML_CCZ4_Helper McLachlan/ML_CCZ4_Test McLachlan/ML_WaveToy McLachlan/ML_WaveToy_Test ExternalLibraries/MPI CactusUtils/MemSpeed EinsteinInitialData/Meudon_Bin_BH EinsteinInitialData/Meudon_Bin_NS EinsteinInitialData/Meudon_Mag_NS EinsteinExact/Minkowski CactusNumerical/MoL EinsteinExact/ModifiedSchwarzschildBL EinsteinAnalysis/Multipole Lean/NPScalars Proca/NPScalars_Proca CactusUtils/NaNCatcher CactusUtils/NaNChecker EinsteinEvolve/NewRad CactusUtils/Nice EinsteinInitialData/NoExcision CactusUtils/NoMPI CactusNumerical/Noise CactusNumerical/Norms PITTNullCode/NullConstr PITTNullCode/NullDecomp PITTNullCode/NullEvolve PITTNullCode/NullExact PITTNullCode/NullGrid PITTNullCode/NullInterp PITTNullCode/NullNews PITTNullCode/NullPsiInt PITTNullCode/NullSHRExtract PITTNullCode/NullVars ExternalLibraries/OpenSSL EinsteinAnalysis/Outflow ExternalLibraries/PAPI CactusPUGH/PUGH CactusPUGH/PUGHInterp CactusPUGH/PUGHReduce CactusPUGH/PUGHSlab CactusNumerical/Periodic Carpet/PeriodicCarpet CactusExamples/Poisson Proca/ProcaBase Proca/ProcaEvolve Proca/Proca_simpleID EinsteinAnalysis/PunctureTracker EinsteinAnalysis/QuasiLocalMeasures EinsteinInitialData/ReadInterpolate Carpet/ReductionTest Carpet/ReductionTest2 Carpet/ReductionTest3 CactusNumerical/ReflectionSymmetry Carpet/RegridSyncTest EinsteinInitialData/RotatingDBHIVP CactusNumerical/RotatingSymmetry180 CactusNumerical/RotatingSymmetry90 CactusExamples/SampleBoundary CactusExamples/SampleIO WVUThorns/Seed_Magnetic_Fields WVUThorns_Diagnostics/Seed_Magnetic_Fields_BNS EinsteinUtils/SetMask_SphericalSurface EinsteinExact/ShiftedGaugeWave WVUThorns/ShiftedKerrSchild CactusNumerical/Slab CactusNumerical/SlabTest CactusConnect/Socket CactusNumerical/SpaceMask PITTNullCode/SphericalHarmonicDecomp PITTNullCode/SphericalHarmonicRecon PITTNullCode/SphericalHarmonicReconGen CactusNumerical/SphericalSurface EinsteinBase/StaticConformal CactusNumerical/SummationByParts CactusBase/SymBase CactusUtils/SystemStatistics CactusUtils/SystemTopology CactusElliptic/TATelliptic EinsteinUtils/TGRtensor EinsteinInitialData/TOVSolver CactusNumerical/TensorTypes CactusUtils/TerminationTrigger CactusTest/TestArrays Carpet/TestCarpetGridInfo CactusTest/TestComplex CactusTest/TestCoordinates CactusTest/TestFortranCrayPointers CactusTest/TestFortranDependencies1 CactusTest/TestFortranDependencies2 CactusTest/TestFpointerNULL CactusTest/TestFreeF90 CactusTest/TestGlobalReduce CactusTest/TestInclude1 CactusTest/TestInclude2 CactusNumerical/TestLocalInterp2 CactusNumerical/TestLocalReduce CactusTest/TestLoop Carpet/TestLoopControl CactusTest/TestMath CactusTest/TestMoL CactusTest/TestPar CactusTest/TestReadWrite CactusTest/TestReduce CactusTest/TestSchedule CactusTest/TestStrings CactusTest/TestTable CactusTest/TestTimers CactusTest/TestTypes CactusBase/Time CactusExamples/TimerInfo CactusUtils/TimerReport Carpet/Timers EinsteinBase/TmunuBase CactusUtils/Trigger EinsteinInitialData/TwoPunctures Proca/TwoPunctures_KerrProca EinsteinExact/Vaidya2 CactusUtils/Vectors WVUThorns_Diagnostics/VolumeIntegrals_GRMHD WVUThorns_Diagnostics/VolumeIntegrals_vacuum CactusUtils/WatchDog CactusWave/WaveBinarySource Llama/WaveExtractL CactusExamples/WaveMoL CactusExamples/WaveToy1DF77 CactusExamples/WaveToy2DF77 CactusWave/WaveToyC CactusWave/WaveToyCXX CactusWave/WaveToyExtra CactusWave/WaveToyF77 CactusWave/WaveToyF90 CactusWave/WaveToyFreeF90 EinsteinAnalysis/WeylScal4 ExternalLibraries/hwloc ExternalLibraries/libjpeg WVUThorns_Diagnostics/particle_tracerET ExternalLibraries/pthreads WVUThorns_Diagnostics/smallbPoynET ExternalLibraries/zlib
Undefined symbols for architecture x86_64:
  "__gfortran_os_error_at", referenced from:
      _ahfinder_ in AHFinder.F.o
      _ahfinder_dis_ in AHFinder_dis.F.o
      _ahfinder_int_ in AHFinder_int.F.o
      _linmin_ in AHFinder_pow.F.o
      _ahfinder_initguess_ in AHFinder_initguess.F.o
      _ahfinder_gau_ in AHFinder_gau.F.o
      ___carpet_boxtypes_MOD_splitregionsrecursive in carpet_boxtypes.F90.o
      ...
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[1]: *** [/Users/murray/Cactus/exe/cactus_sim] Error 1
make: *** [sim] Error 2

============================================================
On Tue, Apr 13, 2021 at 11:50 AM Roland Haas <rhaas@illinois.edu> wrote:
Hello Murray,

ok. Good to hear that.

My initial email was a send a bit off the cuff since I had not quite
realized that you were already using the notebook.

OSX is a bit tricky for the ET to support for the reasons that you
outlined.

When you run "./simfactory/bin/sim setup-silent" simulation factory will
find the gcc-10 compiler and creates a set of of make options for your
laptop (there's explicit code in simfactory to recongnize OSX). So
running this command is essential.

You should not have to adjust anything (though given that the last test
with Catalina rather than BigSur was about a year ago that may not be
true anymore if HomeBrew changed too much).

This set of options is tied to your laptop's hostname as reported by
"hostname -f". Unfortunately this hostname is not "stable" in the
sense that it may change with time since it can be derived from your
laptop's IP address obtained from the wireless network. Eg on my test
laptop it currently reads:

wirelessprvnat-172-16-210-128.near.illinois.edu

So you may have to check that

"./simfactory/bin/sim whoami"

returns the expected machine name and run "./simfactory/bin/sim
setup-silent" again if it reports an unknown machine.

You can also create a file $HOME/.hostname and add a name that you
would like simfactory to use for your laptop eg:

echo >$HOME/.hostname "murrays-laptop"

(no spaces allowed).

Yours,
Roland

> Hi Roland,
>
> Thanks so much for your response! I've been struggling with the -fopenmp
> issue for a couple of days now. I'll try your suggestions and let you know
> if it works. It's encouraging to know that the ET works with Catalina: that
> gives me hope I'll get it to work eventually.
>
> I'm using the notebook CactusTutorial.ipynb, and I certainly don't want to
> use clang. But the Mac makes using gcc instead of clang tricky. On my
> machine, gcc gets clang; you need to enter gcc-10 to get gcc.
>
> I verified that gcc (gcc-10) accepts the -fopenmp flag properly. So both
> gcc and OpenMP seem to work properly on my machine. My problem is
> convincing ET to use gcc instead of clang! BTW, I'm using brew. It's
> possible that MacPorts would work better, but I think brew and MacPorts are
> incompatible.
>
> Again, thank you and I'll let you know my progress later today.
>
> Yours,
>
> Murray
>
> On Tue, Apr 13, 2021 at 9:55 AM Roland Haas <rhaas@illinois.edu> wrote:
>
> > Hello Murray,
> >
> > thinking about this a bit more.
> >
> > The notebook currently (as of last week) lets me compile and run the ET
> > using Homebrew (and worked on Catalina in the past).
> >
> > One thing to keep in mind is that the various CC/CXX and other
> > exported env variables are only used the *first* time you attempt to
> > build and not afterwards.
> >
> > You may have more luck having the options be respected by using:
> >
> > ./simfactory/bin/sim build --clean --reconfig
> >
> > which will recompile from scratch and re-run the configuration stage or
> > even by manually removing the "configs/sim" directory completely.
> >
> > Did you run "simfactory/bin/sim setup-silent" before trying to compile?
> >
> > Yours,
> > Roland
> > 
> > > Hello Murray,
> > >
> > > clang (in particular Apple's version delivered in OSX) is not fully
> > > supported by the ET (due to, at least historically, no OpenMP support,
> > > no paired Fortran compiler).
> > >
> > > Please see the initial section of:
> > >
> > > https://urldefense.com/v3/__https://github.com/nds-org/jupyter-et/blob/master/CactusTutorial.ipynb__;!!DZ3fjg!vBkInm5NFkd2yeuitJ1Bdg6aPGIDCDXMQZsV-k7fG50QV1uy1SqxbWDXQTfC3_p7$
> > >
> > > for how to use the ET on OSX using homebrew or macports.
> > >
> > > If you have a Fortran compiler installed you can try disabling openmp
> > > by setting OPENMP=no in your option list. I strongly advise consulting
> > > the link I provided above and following the (tested and
> > > verified) instructions there in particular if this is your first time
> > > using the toolkit.
> > >
> > > Yours,
> > > Roland
> > > 
> > > > Hi All,
> > > >
> > > > Thanks so much for making the Einstein Toolkit software available!
> > > >
> > > > I'm using the CactusTutorial.ipynb that I got from your site. I'm 
> > trying to 
> > > > build ET on my 2015 Macbook Pro running OS 10.15.7.
> > > >
> > > > I'm stuck at the cell (after) *Building the Einstein Toolkit* .
> > > > I modified that cell to get past issues with zlib . The cell now looks 
> > like: 
> > > >
> > > > %%bash
> > > > export ZLIB_DIR=/usr/local/opt/zlib/lib
> > > > export LDFLAGS="-L/usr/local/opt/zlib/lib"
> > > > export CPPFLAGS="-I/usr/local/opt/zlib/include"
> > > > export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"
> > > > ./simfactory/bin/sim build -j2 --thornlist ../einsteintoolkit.th
> > > >
> > > > Running that cell eventually stops with the errors
> > > >
> > > > COMPILING CactusBase/CoordBase/src/Domain.c
> > > > COMPILING configs/sim/bindings/build/ADMCoupling/cctk_ThornBindings.c
> > > > clang: error: unsupported option '-fopenmp'
> > > > make[3]: *** [Domain.c.o] Error 1
> > > > make[2]: *** [make.checked] Error 2
> > > > make[1]: *** 
> > [/Users/murray/Cactus/configs/sim/lib/libthorn_CoordBase.a] Error 2 
> > > > make[1]: *** Waiting for unfinished jobs....
> > > > clang: error: unsupported option '-fopenmp'
> > > > make[3]: *** [cctk_ThornBindings.c.o] Error 1
> > > > make[2]: *** [cctk_Bindings/make.checked] Error 2
> > > > make[1]: *** 
> > [/Users/murray/Cactus/configs/sim/lib/libthorn_ADMCoupling.a] 
> > > > Error 2
> > > > make: *** [sim] Error 2
> > > >
> > > >
> > > > Can you help? I'm confused by the error message coming from clang, 
> > since 
> > > > the ET seems to use gcc, but my attempt to force gcc, by including the
> > > > exports below, made no difference:
> > > >
> > > > export CC=gcc-10
> > > > export CXX=gcc-10
> > > >
> > > > I really appreciate any help you are able to provide!
> > > >
> > > > Thank you.
> > > >
> > > > Murray (Wolinsky)
> > > > ______________________________________________________
> > > >
> > > > Additional (likely irrelevant) notes:
> > > >
> > > > I used brew to install the prerequisites. It seemed to have gone
> > > > more-or-less ok.
> > > >
> > > > Brew did issue a somewhat concerning message regarding hdf5:
> > > > 
> > > > ==> Pouring hdf5-1.12.0_1.catalina.bottle.1.tar.gz 
> > > >
> > > > Warning: hdf5 dependency gcc was built with a different C++ standard
> > > >
> > > > library (libstdc++ from clang). This may cause problems at runtime.
> > > >
> > > >
> > > > But it didn't provide any guidance into fixing the potential problem. 
> > Itried 
> > > >
> > > >
> > > > brew reinstall hdf5
> > > >
> > > >
> > > > and brew no longer complains.
> > > >
> > > >
> > > > I also had some issues with the subversion installation. But I don't 
> > think 
> > > > they're responsible for my current problem.
> > > >
> > > >
> > > > Also, when I did the step *Configuring SimFactory for your machine*, I 
> > got 
> > > > the output:
> > > >
> > > > Here we will define some necessary Simulation Factory defaults.
> > > >
> > > > Determining local machine name: mw-3.local
> > > >
> > > > --------------------SUMMARY--------------------:
> > > >
> > > >
> > > > [default]
> > > > user            = murray
> > > > email           = murray
> > > > allocation      = NO_ALLOCATION
> > > >
> > > > ------------------END SUMMARY------------------:
> > > >
> > > > Contents successfully written to
> > > > /Users/murray/Cactus/repos/simfactory2/etc/defs.local.ini
> > > >
> > > >
> > > > which looks OK, but the NO_ALLOCATION message seems like it might be 
> > of concern? 
> > >
> > > 
> >
> >
> > --
> > My email is as private as my paper mail. I therefore support encrypting
> > and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!vBkInm5NFkd2yeuitJ1Bdg6aPGIDCDXMQZsV-k7fG50QV1uy1SqxbWDXQUQagYP8$  .
> > 


--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .