Advice for managing unresponsive RhinoCompute instances

[MonkeyFace]

We have RhinoCompute running under IIS using the recommended install for an Azure VM.

Sometime (not that uncommon) a compute.geometry process will hang when trying to calculate something we’ve sent to it. This results in a timeout being returned to our middleware “management” layer. Here’s a screenshot of such as occasion (on this occasion we think it might be a gh file that is hanging it):

What has happened above is that our “management” layer has daftly reattempted the timed-out call and therefore progressively locked out all the other instances, IIS has then spawned further ones, and they then also get locked out by our poor “management” layer that is reattempting the failed calls! Oops!

Can anyone point me to good practice on how to monitor for unresponsive instances and kill them (c# .NET) ? Perhaps there is an endpoint dedicated for this?

[MonkeyFace]

I share the following DM which explains the above behaviour for the benefit of anyone this is relevant to:

@MonkeyFace

Hi Andy, and apologies for kind of duplication but I’ve had no response for quite some time on these questions ( detecting/killing unresponsive instances, gh docs cached in memory ) and the issue is raising itself again for us…

Our team developing gh scripts makes fixes or improvements to scripts now and then and it is important to us that we can be confident that if a gh script named “A.gh” that has been previously sent to and solved by grasshopper compute does not reevaluate the old/cached version of “A.gh” when we send the fixed version. Given the size of our team it is not practical for each member to log in and restart the VM in order to ensure the document cache is cleared.

It maybe that this is handled already but we don’t know and so I am asking for clarification - or whether there is an option for us to include in the schema when we send a script to be solved that instructs it NOT to use the cache, or better still not to even keep a cache ever.

Regards
Dan.

@AndyPayne

Hi Dan. I understand this frustration and I’m sorry for not getting back to you in the other threads. This is already possible, but it may not be a fun exercise. Let me explain what happens when you send a definition over to Rhino.Compute (from Hops).
First, the Hops component takes the definition you selected and converts it to a Base64 string and it assigns it to the Algo properties in the JSON schema. The request is sent and Rhino.Compute consumes the JSON request and deserializes the Algo property (effectively turning this back into a Grasshopper definition). It goes through this definition, looking for components that will either be inputs or outputs on the Hops component and returns a response with that data. It’s also at this point that the server caches the definition and sets the Pointer property in the JSON schema which is basically an ID value pointing to that specific definition.
The Hops component gets this response and basically builds out the inputs and outputs. But, it also stores the Pointer value for the next time it’s needed.
Next, the Hops component consumes the input data that you might feed it and generates another request, this time filling out the necessary data to be used by the Solve function in Rhino.Compute. It also fills out the Pointer value in the JSON schema so that Rhino.Compute knows which Grasshopper definition to use. Now, the reason we’re using this Pointer in the first place is because the serialized data that gets sent as part of the Algo in the first request can become quite large (depending on the size of the definition). If you were to have to send all of that data every time you want to make a solve, it will slow things down. So, this is done just once and then a pointer is sent so you can simply pass that pointer to Rhino.Compute instead of having to pass the full definition every time you want to make a Solve.
It sounds like the short term fix for you would be to include the base64 string of the Algo in EVERY request that is made to the /solve endpoint (making sure to clear the Pointer property as well). This would effectively make sure to use the exact definition that you just sent every time you want to solve a particular set of inputs. The downside here is that it’s going to be slower because you’re passing the definition every time you make a request and the transport will be slower.
You can actually see exactly where in the code this happens (see this link). You see that it first checks to see if the Pointer value is valid and tries to load the GH definition from there. If that is null, then it will move to the next line and check the Algo property from which it will try to create a definition from that.
I should mention that the cache is only saved in memory for each session the Compute.Geometry project is loaded. So, there is a idleTime value that Rhino.Compute uses to check if you have received a valid request recently (the default is 1 hour). If you haven’t received a request during this time, it will shut down all of the child processes (ie. Compute.Geometry) and thus the cached memory should be freed up. So, this would effectively be like a “reset” for your definitions.
The best solution (although this would take a little time on our part) would be to add a “Clear Cache” endpoint which would effectively delete all of the cached files. There is a problem with this though. Let me try to explain.
Let’s say that you have a user who has sent a request using the typical process… meaning they first sent the request which contained the Algo property and the response back contained the Pointer to the GH definition. From this point, all of their requests will simply use the Pointer value since it’s much faster. Now, let’s say that you come along and send a request to the Clear Cache endpoint to delete all of the existing cached definitions. The next time your user sends a request, it’s going to fail because it’s going to be pointing to a file that doesn’t exist anymore. The code would essentially hit this point and it would return the following error: “Unable to load grasshopper definition”. The only fix would be to resend the full Grasshopper definition via the Algo property to essentially start rebuilding the cache again.
So, I’m a little hesitant to go down that route as I can see it being very error prone.
Ultimately, I think the safest, but slowest method (if you definitely want to ensure that the definition that you’re using is the absolute latest) would be to pass the full serialized GH definition as the Algo property in every request. This would work every time, but it would be slower.
Does this help?

@MonkeyFace

Thanks for that clear walk thru. Yup. In our case we are not using Hops but sending the json schema directly to Compute from our C# server and if i have understood correctly then you have reassured me that in fact because we are indeed including the definition each time (not null) and we never set the pointer - that in fact our scripts are always solved afresh - which is exactly the reassurance i was seeking. It also explains why we see a build up of multiple documents even duplicates of the same definition when we send it more than once.
(We are very happy to miss out on the extra performance that can be obtained from using the Pointer field because we cache the results ourselves anyway after some further processing of our own.)
I also appreciate your comment about the timeout – with that knowledge we can look into adjusting the timeout to prevent memory climbing too high when our process is sending the same definition many times (different inputs) and we are deliberately not making use of the Pointer field .

We will probably play tunes on this going forward to get the benefit of using the Pointer field or not using it when we want to be sure to generate afresh.

Thanks again.
dan