This reverts commit 4900bbee17.
The issue that was supposed to fix is now fixed by lazy-loading rpy2 and
makign sure scipy is loaded before that.
That is not quite as nice, but preloading is now causing its own issues
with openblas 0.3.4:
https://github.com/xianyi/OpenBLAS/issues/1936