Using COM/ActiveX automation in C++ is not for the faint of heart. But is is possible to get Grasshopper’s IDispatch object and invoke methods on it. What is available to invoke on the object is listed here:
To get Grasshopper’s object, use CRhinoApp::GetPlugInObjectInterface and pass it the UUID of Grasshopper - which you can get from Rhino’s plug-in manager (Tools -> Options -> Plug-ins). CRhinoApp::GetPlugInObjectInterface returns a pointer to a IUnknown object. But you should be able to cast it as an IDispatch pointer.
Once you have this, you can find the DISPID of the OpenDocument using IDispatch::GetIDsOfNames, and then invoke the method using IDispatch::Invoke.
There is a lot of info on C++ COM/ActiveX on MSDN - work doing some reading.