# Enabling MPI for multi-node **URL:** \ **Category:** Applications\ **Created:** [March 26, 2024, 10:47pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203 "2024-03-26T22:47:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ethan\_Evans](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/ethan_evans/32/141_2.png) [@Ethan\_Evans](https://gpusph.discourse.group/u/Ethan_Evans)\ **Post date:** [March 26, 2024, 10:47pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/1 "2024-03-26T22:47:53Z")
I am attempting to run on multiple nodes in my organizations high performance computing cluster. I know this requires enabling MPI. I have loaded Open MPI version 4.1.4 (ompi/4.1.4) and then running: `make mpi=1` This seems to stall and never progress or error no matter how long I wait. Is it possible this is an issue with the ompi version I use or my approach? The output is shown in the attached image Note: my organizations HPC only has loaded ompi/4.1.0 - 6 I have tried some other versions with no different results ![image](https://global.discourse-cdn.com/free1/uploads/gpusph/original/1X/fe5d6ec7b4619ee4c1ed96958220f23ff4720dec.png) ---
**Author:** ![giuseppe.bilotta](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/giuseppe.bilotta/32/72_2.png) [@giuseppe.bilotta](https://gpusph.discourse.group/u/giuseppe.bilotta)\ **Post date:** [March 27, 2024, 5:47am UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/2 "2024-03-27T05:47:01Z")
Hello @Ethan_Evans I seem to recall I’ve heard of a similar issue in the past but I can’t seem to find a report about it, so I’m afraid we’ll have to do some debugging. The first thing to check is if at least `make show` works, and if so what is the output. If `make show` works, the next thing to try is a `make echo=1 plain=1 -j1` to see where the build stalls. ---
**Author:** ![Ethan\_Evans](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/ethan_evans/32/141_2.png) [@Ethan\_Evans](https://gpusph.discourse.group/u/Ethan_Evans)\ **Post date:** [March 27, 2024, 7:39pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/3 "2024-03-27T19:39:54Z")
Thanks for the help @giuseppe.bilotta `make show` does appear to work and I attached the output below. ![makeshow_out](https://global.discourse-cdn.com/free1/uploads/gpusph/original/1X/9e1a2452817dd9cebbd0e9053270174323353313.png) Then running `make echo=1 plain=1 -j1` has the following output which stalls: ![make_echo_plain_1](https://global.discourse-cdn.com/free1/uploads/gpusph/original/1X/6dd9a9ea7b5b5a1fdf860d17308e53b438e42e5b.png) Appreciate the help and let me know if I can try anything else! P.S. I know @AlirezaZarei will be interested in the results as well ---
**Author:** ![giuseppe.bilotta](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/giuseppe.bilotta/32/72_2.png) [@giuseppe.bilotta](https://gpusph.discourse.group/u/giuseppe.bilotta)\ **Post date:** [March 27, 2024, 7:46pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/4 "2024-03-27T19:46:12Z")
OK one issue I’m seeing is that `CXX` is being set to the MPI compiler. I suspect this is the reason for the stall. Did you set `CXX` manually or was it autodetected that way? ---
**Author:** ![Ethan\_Evans](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/ethan_evans/32/141_2.png) [@Ethan\_Evans](https://gpusph.discourse.group/u/Ethan_Evans)\ **Post date:** [March 27, 2024, 8:00pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/5 "2024-03-27T20:00:42Z")
It must be autodetected that way. I have done nothing to set CXX. Should I set it to something else? ---
**Author:** ![giuseppe.bilotta](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/giuseppe.bilotta/32/72_2.png) [@giuseppe.bilotta](https://gpusph.discourse.group/u/giuseppe.bilotta)\ **Post date:** [March 27, 2024, 8:04pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/6 "2024-03-27T20:04:19Z")
You can override the autodetection adding something like `CXX=g++` to `Makefile.local`, and see if this solves the problem. We should try to understand why it’s being detected this way too. What does `command -v c++` report on that system? ---
**Author:** ![Ethan\_Evans](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/ethan_evans/32/141_2.png) [@Ethan\_Evans](https://gpusph.discourse.group/u/Ethan_Evans)\ **Post date:** [March 27, 2024, 8:17pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/7 "2024-03-27T20:17:08Z")
Here is what `command -v c++` gives: ![image](https://global.discourse-cdn.com/free1/uploads/gpusph/original/1X/5fc5d30f5be25c0e505ccfb942ee9c278eba3dd9.png) If it is helpful, this is what I currently have pertaining to CXX in Makefile.local: ```auto CXXFLAGS=-std=c++14 -march=native CPPFLAGS += -g ``` Also: Adding `CXX=g++` to the Makefile.local is sucessful. It now sucessfully compiles! ---
**Author:** ![giuseppe.bilotta](https://yyz2.discourse-cdn.com/free1/user_avatar/gpusph.discourse.group/giuseppe.bilotta/32/72_2.png) [@giuseppe.bilotta](https://gpusph.discourse.group/u/giuseppe.bilotta)\ **Post date:** [March 27, 2024, 8:28pm UTC](https://gpusph.discourse.group/t/enabling-mpi-for-multi-node/203/8 "2024-03-27T20:28:04Z")
Interesting. So something is setting the default value of `CXX` for `make` to the MPI compiler, probably when the mpi module is loaded.