Note for all users: I would not reuse a /home partition from some other Linux distro or an older version of OM Lx. I can assure all users that OM developers rather strongly do not recommend this practice. Any problems that result have an easy solution. Do not do that.
It is hard to know exactly what you did without screen shots or calamares-installation.log. The calamares-installation.log is in /var/log so if you could attach that here that would be helpful.
The only way I know of to have separate / and /home with OM’s Calamares installer is by using Manual Partitioning. If you are in position to do a reinstall that is described here.
Manual Partitioning would be the custom option you refer to.
For more about OM Lx see this and this. OM Lx documentation is a work in progress at present.