[mcstas-users] [EXTERNAL] Re: Inconsistent results using MPI and SPLIT

[Granroth, Garrett]

Hi Thierry,
     I took a quick look through your simulation and the wavelength range at the start is quite large especially since you are feeding it though a filter and onto a monochromator. One way to speed things up is to only feed the instrument the wavelengths you need to go through the Monochromator. I have often tied the wavelength band to be some window around the wavelength set for the Monochromator.  Then this window can be adjusted to tradeoff simulation speed for how far from the center wavelength you want to go.
If you need to check lambda/n,  specific different simulations will give you better results.  However if you need all the wavelengths at once you probably need to be running 10^8  neutrons for this wide bandwidth range.
As you discovered, the MPI feature is the only way to run lots of neutrons.  It scales well.  So if you can find 32 or 64 cores or better yet 100s or 1000s, that is the best way to go.

If you use split I would recommend using it once. Probably right after the monochromator.  However, as Peter said, you have to get enough neutrons to the split to help you.  So this may still need 10^8 at the beginning.

Best of Luck!

Garrett


Garrett E. Granroth
Instrument Scientist
Neutron Scattering Division
Oak Ridge National Laboratory
P.O. Box 2008 MS 6475
Oak Ridge TN 37831
Tel: 865-241-8241
Fax: 865-574-6080
email:granrothge at ornl.gov<mailto:granrothge at ornl.gov>



From: mcstas-users <mcstas-users-bounces at mcstas.org> on behalf of Peter Kjær Willendrup <pkwi at fysik.dtu.dk>
Date: Thursday, June 18, 2020 at 2:08 PM
To: Thierry Bigault <bigault at ill.fr>, McStas <mcstas-users at mcstas.org>
Subject: [EXTERNAL] Re: [mcstas-users] Inconsistent results using MPI and SPLIT

Dear Thierry,

Just a quick response for now, will have a closer look later.

One has to be quite careful with SPLIT, it is bossing, and if the statistics is not “big enough” it will most certainly lead to bogus and unstable results. What big enough means is hard to quantify, but generally speaking one typically needs on the order of 1e5-1e6 events at the point of split at the given ncount.

I didn’t check your instrument file, but one very important detail is that SPLITS must be inserted right before components that apply Monte Carlo / random numbers, otherwise one event will simply become 10 identical and thus drive the errorbars to unphysically low values. And 5 splits (each 10) is 10^5 and sounds like a quite aggressive splitting. 😊

I am sure we will find a good way to fix this, but I don’t have time this side of the weekend.

Best

Peter

Hent Outlook til iOS<https://aka.ms/o0ukef>
________________________________
Fra: mcstas-users <mcstas-users-bounces at mcstas.org> på vegne af Thierry Bigault <bigault at ill.fr>
Sendt: torsdag, juni 18, 2020 7:03 PM
Til: McStas
Emne: [mcstas-users] Inconsistent results using MPI and SPLIT

Dear All,

I built the attached instrument file to try and simulate a reflectometer.
When I saw that I needed 10 minutes of calculation to get 3 neutrons on
my sample, I thought it was time to try and optimize.
After having a look at the manual, I first put some SPLIT keywords
positioned at 5 different components, which I estimate as strategic. I
gained a lot in statistics on the sample.
Then I decided to use the MPI feature, as my laptop has 8 cores. It
nicely reduces the computing time, at first sight it looks really great !
But the results looked strange, so I made some more systematic tests and
the result is on the attached plots. With all SPLIT commented ("no
SPLIT") it looks fine, the calculated intensity on the sample is
consistent within the error-bars, whatever the number of nodes I use.
When combining all SPLITs active and MPI ("5 SPLIT"), the calculation
time and error-bars can be strongly reduced but the result depends
completely on the number of nodes, with differences much larger than the
error-bar...
Either I did something wrong, or there is a bug somewhere. If someone
has an idea about this issue, I would be interested.
I use version 2.6.1 (May 04, 2020) on Windows 10.

Thierry




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman2.mcstas.org/pipermail/mcstas-users/attachments/20200618/1f718935/attachment-0001.html>