A 650W power supply ought to be fine, but to check you can go to www.pcpartpicker.com and “build” a PC with your components: it will tell you what the power requirement is.
Your power supply could be faulty of course, but you might also consider the possibility that the 1080Ti was not properly seated in its slot or that the additional power cable it requires wasn’t fully engaged. Replacing the card in your PC and trying again, whilst boring, might be worth an experiment.