When you add a ground plane you always get the actual geometry. The problem here is the shadow catcher implementation of Cycles which currently has its drawbacks like the ones I mentioned. That is why I suggest using ground plane with material, that gives better lighting and light bounces between object and ground.