In my option a keyframe is an attribute of an object (or a view, a layer, …) not the other way around. A keyframe exist by the grace of his owner. Delete an object and observe his keyframes being deleted at the same time. So it’s fairly logic one has to select the object in order to change its attribute (just as an object needs to be selected to change its name, display color, texture mapping …).
It must be kept in mind that a keyframe marker on the timeline at a certain tick can be a multiple of keyframes belonging to various objects. It must also be kept in mind that users may want to select various keyframes (even on various objects) eventually using a marquee.
Direct selection would need a rather elaborate (possibly confusing) right mouse button dialog to build the desired selection of object(s). The process would become laborious when multiple keyframes are at stake. I personally feel the current setup – primary select one or more objects, then pick one or more keyframe and edit – is very convenient. I myself often use the Animation manager (giving an overview of animated items) as a selection tool.
The commands BongoMoveKeyframe, BongoCopyKeyframe, Bongo DeleteKeyframe all start with an item-selection session when prior no item is selected.
BTW The technique (‘drill’) selecting all objects I described above is only useful (in complex models) to find out which object is owner of a curious mysterious keyframe one has lost track of (“who the f… do you belong to?!”). In that particular case one wishes for direct selection. The Ctrl-A trick is the solution.