Christopher Lameter, Linux Foundation, Shoot first and stop the OS noise, here.
The problem was first noticed first in High Performance Computing.2 HPC applications typically go through a highly parallel calculation phase and then a rendezvous phase in which the results of the calculations are ex- changed. It was noted that the calculation phase was rarely performing within the boundaries expected. The problem became worse as the number of processor in- creased. The investigation found that the rendezvous phase will be delayed if any one of the processes is held up due to OS interference (like for example a timer in- terrupt). The more processors exist the more likely the chance that OS interference will cause a delay. This is especially severe on Linux due to the staggering of the timer interrupts over all processors in the system. As a result the timer interrupts will not run all at the same time (which would potentially overload the interconnect between the processors).