Yeah! It feels great when I get the workhorse computer really chugging along. 85% is unusual to see on a normal run, since this beast (a 2010 3.32 Ghz Mac pro quad core) has a lot of computational capacity that you don’t need for most tasks. I’m getting up to 85% because all four cores are crunching away independently on the same problem (written in MatLab) but different parts of it. Each point on the resulting graph will be the result of having computed 2000 points. Each of those 2000 points comes from computing a boundary value problem discretized into a million points. See an earlier post for more about that.
(Update: Now running even more tasks associated with this problem: Up to 96% now:
…with nothing idle at all. This is probably not as efficient any more, but it is for a few hours and then back to 85%. In the meantime, it is amusing how it makes me feel I’m doing more work somehow…)
I assigned different parts of the graph to different processes by hand, not using a fancy parallelization routine since in this case I don’t need that (and don’t know how to). It takes about 24 hours to produce the 2000 set and hence the one point, but the great thing is that I get them in batches of four, because of the parallel run. There’s someting nice about logging on and seeing the little collection of precious, shiny new points that have been mined after all that work. This morning I did the data analysis on the small cluster of them I have so far, and they are already lined up along the curve I expected, so I’m quite pleased. This all relates to an earlier work I talked about, if you’re interested.
More later…
-cvj
Definitely let me know! Enjoy!
-cvj
C – I just retired my vintage (ancient?) dual core Athlon MP 2200 of 2GB fame. Circa 2002. Replaced it with the just out Xeon E3-1245V3 (3.4 Gigaloos) and 32GB core. Quite the upgrade! Primarily doing some Android devel and surfing but I’ll let you know when I finish my hydrodynamic thermonuclear simulations 🙂
I know one can sandbox a given program in other ways (mostly limiting its permissions), so I would guess there might be a way to limit its memory usage. But from what I understand, the Mathematica kernel isn’t able to deal gracefully with memory limits. The best it can do is restart (which of course clears all prior computations). It makes no attempt to manage its own swapping.
The OS of course manages swapping, but this is all done transparently, so that e.g. Mathematica effectively thinks it has 48GB to play with.
I have heard that in the 32-bit days, Mathematica was better behaved, because there was only 4GB of address space. But with 64 bits, the address space is many orders of magnitude larger than the physical memory. So it effectively doesn’t know when to stop.
It is certainly possible to write programs more intelligently. I have a few pieces of graphics software that have configurable memory limits and are written in such a way that jobs do not fail when the limit is reached, but simply take longer to finish. I’m not sure why Mathematica didn’t take this approach. I wonder if Maple has done any better?
The main culprit of high memory use is symbolic computation. It will handle numerical work very easily. Wolfram’s official advice is “break the computation up into smaller bits so that the kernel doesn’t reach the memory limitations of your machine.”
Hmmm. That’s interesting, and indeed strange. I wonder if, instead of a cap, there is a way of somehow fooling particular programs into thinking that the memory of the computer is actually smaller than it actually is? Then they go into disc swapping earlier, not perceiving it as a user-imposed cap?
-cvj
I haven’t yet managed to occupy all of my cores to that degree, but I have easily used up all 16GB of my memory on one Mathematica process.
This is partly Mathematica’s fault. It will happily gobble up any available RAM, and does not have a global option to set a cap. The memory-limiting options it does have simply cause it to throw an error if the cap is reached, rather than using the disk to manage RAM swapping. Kind of strange, really, for a professional tool supposedly intended to be used on large computations.