[WIP] compute and save the ISRF in every cell #231

dnarayanan · 2023-02-03T15:14:37Z

the upshot is that by setting in the python code:

m.compute_isrf(True)

hyperion will save the spectrum of energy deposited in every cell. this can be accessed in the hdf5 file via:

import numpy as np
import h5py

f = h5py.File('pd_skirt_comparison.134.rtout.sed')
dset = f['iteration_00007']
print(np.max(dset['specific_energy_sum_nu']))
print(dset['ISRF_frequency_bins'][:])

where the units are erg/s/g (and the "g" is the dust mass in the cell).

this, however, doesn't work for AMR grids because I'm having trouble understanding the file structure, and am unable to get the code to compile. As a result, I'm labeling this PR as a work in progress, and happy to iterate on this. Two main areas I could use eyes from @astrofrog are is:

accuracy of code!
if Reduce RAM usage for AMR grids #1 is okay, then to see by analogy how to update the AMR code in the fortran modules

astrofrog · 2023-02-03T15:31:44Z

Thanks for all your work on this! Will try and review soon 😊

astrofrog · 2023-08-18T11:48:30Z

@dnarayanan - could you rebase this on the latest main branch from this repository? I have finally managed to fix the CI so that it can give meaningful results again.

dnarayanan · 2023-08-18T11:52:58Z

Happy to ! (Will need to be Monday likely).

Just to confirm since I’m not super with git, I want to follow a procedure like this one (that I googled)?

https://verdantfox.com/blog/view/how-to-git-rebase-mainmaster-onto-your-feature-branch-even-with-merge-conflicts

astrofrog · 2023-08-18T12:12:26Z

Yes similar to that - but make sure you use this repository as the one you are rebasing on. So assuming you don't already have this repository set up as 'upstream':

git remote add upstream https://github.com/hyperion-rt/hyperion.git
git fetch upstream
git rebase upstream/main

You'll then need to do:

git push -f origin master

to force push the changes since the git history has changed.

I'll keep a backup of your work here in case anything goes wrong 😄

dnarayanan · 2023-08-21T14:53:57Z

thanks for the step by step :) i would have never figured out that git order of operations. i've just rebased (i believe)!

astrofrog · 2023-08-21T15:23:11Z

Great! There are some real test failures, you can run the tests locally by running:

tox -e test

dnarayanan · 2023-09-05T20:08:51Z

thanks tom! so i have to admit i'm not quite sure how to interpret the tox results. when i run the test locally, i get:

(py38) [desika.narayanan@login6 hyperion_dn]$ tox -e test
warning: no previously-included files found matching 'scripts/mctherm2hyperion'
warning: no previously-included files matching '*.pyx' found under directory 'hyperion'
warning: no previously-included files matching '*.pyc' found under directory 'hyperion'
warning: no previously-included files matching '*.o' found under directory 'hyperion'
warning: no previously-included files found matching 'scripts/mctherm2hyperion'
warning: no previously-included files matching '*.pyx' found under directory 'hyperion'
warning: no previously-included files matching '*.pyc' found under directory 'hyperion'
warning: no previously-included files matching '*.o' found under directory 'hyperion'
test inst-nodeps: /home/desika.narayanan/hyperion_dn/.tox/.tmp/package/1/Hyperion-0.9.11.dev67+gac54b35.tar.gz
test installed: astropy==5.2.2,asttokens==2.4.0,backcall==0.2.0,cmyt==1.3.1,comm==0.1.4,contourpy==1.1.0,cycler==0.11.0,decorator==5.1.1,ewah-bool-utils==1.0.2,exceptiongroup==1.1.3,executing==1.2.0,fonttools==4.42.1,h5py==3.9.0,Hyperion @ file:///home/desika.narayanan/hyperion_dn/.tox/.tmp/package/1/Hyperion-0.9.11.dev67%2Bgac54b35.tar.gz,importlib-resources==6.0.1,iniconfig==2.0.0,ipython==8.12.2,ipywidgets==8.1.0,jedi==0.19.0,jupyterlab-widgets==3.0.8,kiwisolver==1.4.5,matplotlib==3.7.2,matplotlib-inline==0.1.6,more-itertools==10.1.0,mpmath==1.3.0,numpy==1.21.6,packaging==23.1,parso==0.8.3,pexpect==4.8.0,pickleshare==0.7.5,Pillow==10.0.0,pluggy==1.3.0,prompt-toolkit==3.0.39,ptyprocess==0.7.0,pure-eval==0.2.2,pyerfa==2.0.0.3,Pygments==2.16.1,pyparsing==3.0.9,pytest==7.4.1,python-dateutil==2.8.2,PyYAML==6.0.1,six==1.16.0,stack-data==0.6.2,sympy==1.12,tomli==2.0.1,tomli_w==1.0.0,tqdm==4.66.1,traitlets==5.9.0,typing_extensions==4.7.1,unyt==2.9.5,wcwidth==0.2.6,widgetsnbextension==4.0.8,yt==4.2.2,zipp==3.16.2
test run-test-pre: PYTHONHASHSEED='1699501006'
test run-test: commands[0] | pip freeze
astropy==5.2.2
asttokens==2.4.0
backcall==0.2.0
cmyt==1.3.1
comm==0.1.4
contourpy==1.1.0
cycler==0.11.0
decorator==5.1.1
ewah-bool-utils==1.0.2
exceptiongroup==1.1.3
executing==1.2.0
fonttools==4.42.1
h5py==3.9.0
Hyperion @ file:///home/desika.narayanan/hyperion_dn/.tox/.tmp/package/1/Hyperion-0.9.11.dev67%2Bgac54b35.tar.gz
importlib-resources==6.0.1
iniconfig==2.0.0
ipython==8.12.2
ipywidgets==8.1.0
jedi==0.19.0
jupyterlab-widgets==3.0.8
kiwisolver==1.4.5
matplotlib==3.7.2
matplotlib-inline==0.1.6
more-itertools==10.1.0
mpmath==1.3.0
numpy==1.21.6
packaging==23.1
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.0
pluggy==1.3.0
prompt-toolkit==3.0.39
ptyprocess==0.7.0
pure-eval==0.2.2
pyerfa==2.0.0.3
Pygments==2.16.1
pyparsing==3.0.9
pytest==7.4.1
python-dateutil==2.8.2
PyYAML==6.0.1
six==1.16.0
stack-data==0.6.2
sympy==1.12
tomli==2.0.1
tomli_w==1.0.0
tqdm==4.66.1
traitlets==5.9.0
typing_extensions==4.7.1
unyt==2.9.5
wcwidth==0.2.6
widgetsnbextension==4.0.8
yt==4.2.2
zipp==3.16.2
test run-test: commands[1] | /home/desika.narayanan/hyperion_dn/configure --prefix=/home/desika.narayanan/hyperion_dn/.tox/test
checking for h5fc... h5fc
checking for h5pfc... no
checking for mpif90... mpif90
configure: creating ./config.status
config.status: creating Makefile
test run-test: commands[2] | mv Makefile /home/desika.narayanan/hyperion_dn/
test run-test: commands[3] | make -C /home/desika.narayanan/hyperion_dn serial
make: Entering directory `/home/desika.narayanan/hyperion_dn'
h5fc -module src/modules -heap-arrays -error-limit 1 -fp-model strict -diag-disable 8291    src/mpi/mpi_core_nompi.f90 fortranlib/src/posix_default.f90 fortranlib/src/base_types.f90 fortranlib/src/lib_messages.f90 fortranlib/src/lib_io.f90 fortranlib/src/lib_version.f90 fortranlib/src/lib_algebra.f90 fortranlib/src/lib_conf.f90 fortranlib/src/lib_array.f90 fortranlib/src/lib_statistics.f90 fortranlib/src/lib_random.f90 fortranlib/src/lib_hdf5_110.f90 fortranlib/src/lib_constants.f90 fortranlib/src/type_angle3d.f90 fortranlib/src/type_vector3d.f90 fortranlib/src/type_stokes.f90 fortranlib/src/type_pdf.f90 src/core/core_lib.f90 src/main/counters.f90 src/mpi/mpi_io_nompi.f90 src/grid/type_grid_cartesian_3d.f90 src/core/type_cell_id_3d.f90 src/core/type_photon.f90 src/main/settings.f90 src/main/performance.f90 src/images/image_type.f90 src/dust/dust_type_4elem.f90 src/dust/dust.f90  src/grid/grid_io.f90 src/grid/grid_geometry_cartesian_3d.f90 src/grid/grid_geometry_common_3d.f90 src/grid/grid_pda_cartesian_3d.f90 src/grid/grid_physics_3d.f90 src/grid/grid_monochromatic.f90 src/dust/dust_interact.f90 src/sources/source_type.f90 src/sources/source.f90 src/grid/grid_propagate_3d.f90 src/grid/grid_mrw_3d.f90 src/grid/grid_pda_3d.f90 src/grid/grid_generic.f90 src/images/images_binned.f90 src/images/images_peeled.f90 src/main/forced_interaction.f90 src/main/setup_rt.f90 src/mpi/mpi_routines_nompi.f90 src/main/iter_lucy.f90 src/main/iter_final.f90 src/main/iter_final_mono.f90 src/main/iter_raytracing.f90 src/main/main.f90 -o bin/hyperion_car

Error: A license for Comp-FL could not be obtained (-1,359,2).

Is your license file in the right location and readable?
The location of your license file should be specified via
the $INTEL_LICENSE_FILE environment variable.

License file(s) used were (in this order):
**  1.  /apps/compilers/intel/2018/1.163/compilers_and_libraries_2018.1.163/linux/bin/intel64/../../Licenses
**  2.  /opt/intel/licenses
**  3.  /Users/Shared/Library/Application Support/Intel/Licenses
**  4.  /apps/compilers/intel/2018/1.163/compilers_and_libraries_2018.1.163/linux/bin/intel64/*.lic

Please refer http://software.intel.com/sites/support/ for more information..

ifort: error #10052: could not checkout FLEXlm license
make: *** [car_serial] Error 1
make: Leaving directory `/home/desika.narayanan/hyperion_dn'
ERROR: InvocationError for command /usr/bin/make -C /home/desika.narayanan/hyperion_dn serial (exited with code 2)

but i'm guessing that that's not the issue for why the tests are failing. when i click on the details of the tox tests reported here in github, i'm not sure what they're reporting. very happy to hunt these down, but i'm afraid i'm not even sure what i'm looking for here.

dnarayanan · 2023-09-05T20:09:14Z

indeed, a license does seem to exist:

(py38) [desika.narayanan@login6 hyperion_dn]$ echo $INTEL_LICENSE_FILE
/apps/compilers/intel/licenses

dnarayanan · 2024-01-11T19:02:01Z

hi tom -

it seems in some tests of mine, that i find that the ISRF saved is much higher in runs where i employ more than 1 MPI core. do you have any sense as to where i might begin looking in the code to debug this? i can't tell what might be going on...if there's a normalization by the number of cores that needs to happen somewhere perhaps.

it's not something as simple as the ISRF with 32 cores (for example) is 32x the ISRF energy deposited for a single core...it's like many orders of mag higher.

astrofrog · 2024-04-22T08:22:56Z

@dnarayanan - sorry for not getting back to you sooner. I am currently in the process of trying to fix compatibility of Hyperion with recent Python/Numpy/Astropy versions and will take a look at this soon.

dnarayanan · 2024-04-22T11:44:17Z

sounds good thanks!

astrofrog · 2024-04-23T12:22:26Z

@dnarayanan - now that I've unbroken the CI, could you rebase on top of the latest main branch? I'll then try and take a look at this.

…on of frequency. note, this has a manual setting of frequencies and right now only computes, not saves

…think

…d m.read is broken still with the new specific_energy_sum_nu array dimensions

… it to compile

… code

…in now when running without ISRF calculation turned on

… naturally, random segfaults]

dnarayanan · 2024-05-09T21:10:44Z

sorry about the delay - i just rebased and pushed!

astrofrog

About to try this out, but spotted a typo below. Note that you can run the tests locally using e.g.:

tox -e test-nobinaries

this tests just the basic functionality of the Python package excluding tests that have to run the fortran binaries. You can then test more extensively with:

tox -e test

I'll try and look into the MPI issue shortly.

astrofrog · 2024-05-12T21:09:24Z

hyperion/conf/conf_files.py

@@ -719,6 +743,7 @@ def read_run_conf(self, group):  # not a class method because inherited
        self._read_max_reabsorptions(group)
        self._read_pda(group)
        self._read_mrw(group)
+        self._reada_isrf(group)


Typo here - should be read_isrf?

dnarayanan · 2024-05-15T13:32:44Z

thanks for finding the typo! i'm trying to figure out how to push the fix but i think i need to set a rule for reconciling divergent branches after the rebase before i can push to github. any advice is helpful!

 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'https://github.com/dnarayanan/hyperion.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
[desika.narayanan@login7 conf]$ git pull
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

not sure which of the 3 options above are best

astrofrog · 2024-05-15T15:09:49Z

Can you let me know the output of 'git remote'?

dnarayanan · 2024-05-15T15:14:22Z

[desika.narayanan@login7 hyperion_dn_rebase]$ git remote
origin
upstream

astrofrog · 2024-05-15T15:54:38Z

Thabks! And the output of 'git log' and 'git status'?

dnarayanan · 2024-05-15T15:56:31Z

(py38) [desika.narayanan@login7 hyperion_dn_rebase]$ git log
commit 9d028bf01cdd651ef63ca765e6c460909ce48181 (HEAD -> master)
Author: dnarayanan <[email protected]>
Date:   Wed May 15 09:30:47 2024 -0400

    fixing typo noticed by tom

commit ac54b3506c054fabf7c1e5ad689bf8477693fa16
Author: dnarayanan <[email protected]>
Date:   Thu Aug 11 11:49:45 2022 -0400

    critical bug fix changing indexes starting from 0 to 1 [which caused, naturally, random segfaults]

commit a31bd0cdafe1ea7b6f1ded20ac528960bca4338c
Author: dnarayanan <[email protected]>
Date:   Tue May 24 12:42:56 2022 -0400

    one more 5d file turned off

commit 9d0f4d5d2c2b27f546a2edef7e2858694524b67b
Author: dnarayanan <[email protected]>
Date:   Tue May 10 12:14:36 2022 -0400

    getting rid of read_grid_5d since its not actually used

commit 3ab74b0c43584fab966093946e6b593250580c34
Author: dnarayanan <[email protected]>
Date:   Mon May 9 12:22:59 2022 -0400

    bugs fixed that kept code from running when ISRF is turned off

commit f1696d36df79478a588e20e47738a19d2448aab3
Author: dnarayanan <[email protected]>
Date:   Mon May 9 08:51:16 2022 -0400

    some bugs corrected fixing the ISRF calculation.   but some issues remain now when running without ISRF calculation turned on
    ```
    
    ```
    (py38) [desika.narayanan@login7 hyperion_dn_rebase]$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 15 and 68 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	.tox/
	hyperion_dn_rebase/
	src/grid/#grid_generic.f90#
	src/mpi/#mpi_io.f90#
	tatus

nothing added to commit but untracked files present (use "git add" to track)

astrofrog · 2024-05-15T16:05:41Z

Thanks! Will take a close look this evening and suggest a fix

dnarayanan force-pushed the master branch from af0e272 to ac54b35 Compare August 21, 2023 14:53

dnarayanan added 14 commits May 9, 2024 17:09

strawperson architecture in for computing specific energy as a functi…

53862f3

…on of frequency. note, this has a manual setting of frequencies and right now only computes, not saves

specific_energy_sum_nu is in all places specific_energy_sum is now i …

5aab88c

…think

code now saves a ISRF in every cell. notably: AMR is still broken, an…

ce76800

…d m.read is broken still with the new specific_energy_sum_nu array dimensions

specific_energy_sum_nu saves and reads. AMR still needs to be dealt with

8c8145d

grid_io_amr_template updated but commetned out since i cant quite get…

d38e048

… it to compile

grid modules now takes the n_isrf_wavelengths from the dust files

033472e

outputting the ISRF frequency bins

74fd6e6

can now call whether ISRF is computed and saved via m.compute_isrf(True)

b54d608

getting rid of internal comments showing where ive added new lines of…

7c588b1

… code

some bugs corrected fixing the ISRF calculation. but some issues rema…

97b2dc6

…in now when running without ISRF calculation turned on

bugs fixed that kept code from running when ISRF is turned off

6d0f51d

getting rid of read_grid_5d since its not actually used

2ffe74d

one more 5d file turned off

8717014

critical bug fix changing indexes starting from 0 to 1 [which caused,…

a0184e5

… naturally, random segfaults]

dnarayanan force-pushed the master branch from ac54b35 to a0184e5 Compare May 9, 2024 21:09

astrofrog requested changes May 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] compute and save the ISRF in every cell #231

[WIP] compute and save the ISRF in every cell #231

dnarayanan commented Feb 3, 2023

astrofrog commented Feb 3, 2023

astrofrog commented Aug 18, 2023

dnarayanan commented Aug 18, 2023

astrofrog commented Aug 18, 2023

dnarayanan commented Aug 21, 2023

astrofrog commented Aug 21, 2023

dnarayanan commented Sep 5, 2023

dnarayanan commented Sep 5, 2023

dnarayanan commented Jan 11, 2024

astrofrog commented Apr 22, 2024

dnarayanan commented Apr 22, 2024

astrofrog commented Apr 23, 2024

dnarayanan commented May 9, 2024

astrofrog left a comment

astrofrog May 12, 2024

dnarayanan commented May 15, 2024

astrofrog commented May 15, 2024

dnarayanan commented May 15, 2024

astrofrog commented May 15, 2024

dnarayanan commented May 15, 2024

astrofrog commented May 15, 2024 •

edited

Loading

[WIP] compute and save the ISRF in every cell #231

Are you sure you want to change the base?

[WIP] compute and save the ISRF in every cell #231

Conversation

dnarayanan commented Feb 3, 2023

astrofrog commented Feb 3, 2023

astrofrog commented Aug 18, 2023

dnarayanan commented Aug 18, 2023

astrofrog commented Aug 18, 2023

dnarayanan commented Aug 21, 2023

astrofrog commented Aug 21, 2023

dnarayanan commented Sep 5, 2023

dnarayanan commented Sep 5, 2023

dnarayanan commented Jan 11, 2024

astrofrog commented Apr 22, 2024

dnarayanan commented Apr 22, 2024

astrofrog commented Apr 23, 2024

dnarayanan commented May 9, 2024

astrofrog left a comment

Choose a reason for hiding this comment

astrofrog May 12, 2024

Choose a reason for hiding this comment

dnarayanan commented May 15, 2024

astrofrog commented May 15, 2024

dnarayanan commented May 15, 2024

astrofrog commented May 15, 2024

dnarayanan commented May 15, 2024

astrofrog commented May 15, 2024 • edited Loading

astrofrog commented May 15, 2024 •

edited

Loading