100s Timeout on hops

Hi there,
I just ran into a weird issue with hops where it doesn’t return a result when it runs an external gh-Script via the gh-component but the script itself works fine with the same inputs.

It looks like the request automatically times out after 100 seconds, which would be unfortunate because I’m trying to make some pretty big calculations asynchronous.

If thats the case, can I change this behaviour somehow?

info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint '/grasshopper HTTP: POST'
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMD07MDKOGBI", Request id "0HMD07MDKOGBI:00000002": An unhandled exception was thrown by the application.
      System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
       ---> System.TimeoutException: The operation was canceled.
       ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
       ---> System.IO.IOException: Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..
       ---> System.Net.Sockets.SocketException (995): Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen.
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
         at System.Net.Http.HttpConnection.FillAsync(Boolean async)
         at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at rhino.compute.ReverseProxyModule.SendProxyRequest(HttpRequest initialRequest, HttpMethod method, String baseurl) in D:\BuildAgent\work\4852f584398a78e4\src\rhino.compute\ReverseProxy.cs:line 137
         at rhino.compute.ReverseProxyModule.ReverseProxyGrasshopper(HttpRequest req, HttpResponse res) in D:\BuildAgent\work\4852f584398a78e4\src\rhino.compute\ReverseProxy.cs:line 190
         at Carter.CarterExtensions.<>c__DisplayClass1_0.<<CreateRouteHandler>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Also it would be great to have some feedback on the GH-Component when that happens because it remains stuck in the “solving…” state

The timeout part is nice for some things though. Perhaps manually setting a timeout threshold is beneficial. If you are running async with threaded inputs (in trees) it could be nice to have a status bar saying “solved 5 of 200 branches”

I’ve also experienced fairly simple hops components, where the group of components work fine in grasshopper, but it says “one or more errors” when i put it into hops.

I’ve got the same issue here. @stevebaer Did anyone come up with a solution yet? I just upgraded to Rhino7 to get rid of some weird plugin called GHAsynchronous which was removed from Food for Rhino some months ago. I was hoping that Hops would provide the same functionality. This time-out issue really is a show-stopper. Working in 3d-printing, I have to deal with complex meshes that sometimes have dozens of millions of vertices; they take their time to be processed, sometimes up to hours. Asynchronicity is vital for me. I simply cannot work when Rhino freezes due to long calculations. Please solve this.

@henry.langner I can’t check at the moment but RHINO_COMPUTE_TIMEOUT looks like what could help you (https://discourse.mcneel.com/t/hops-version-0-12-0-available/139243)

Thanks to @Elena4 (Why did you delete your post?) I found that there is the environment variable of RHINO_COMPUTE_TIMEOUT which I just set to 3600. The error message is gone and the Hops component is solved. However, the result is stuck somewhere inside the system. I can only retrieve it when I disable and reenable the Hops component. This, however, works only if the timing is correct. So I have to observe the CPU load of compute.geometry in Windows Task-Manager and I have to wait for the message of rhino.compute that it has a result in the Rhino Compute Console Window:

[10:43:55 INF] HTTP POST /grasshopper responded 200 in 172043.9054 ms

Then I disable and reenable the Hops component and get another message:

[10:44:50 INF] HTTP POST /grasshopper responded 200 in 1148.2222 ms

You can see that there is a significant difference in computation time. First, the result is calculated, and second, the result is delivered.

What is happening there? Is there a way I could fix this myself? I want my Hops component to behave like any other component, except that it would not freeze GH when it is being solved. That is the main reason to me for using it.

If you right-click on the Hops component, you can specify whether you want the Hops call to made asynchronously. This will make sure that Grasshopper does not lock up or freeze during the calculation.
image

Thanks, but I know that. Let me specify the problem: When Hops finishes its internal calculation, the result is missing. It only shows up if I disable Hops and re-enable it again.

Have you considered making two or more calls for a heavy computation? One for creating and running the computation, one for getting the status, and one for getting the result?

This is what I do, when I disenable and enable the component, isn’t it? Otherwise, how would I do it?

For HOPS, I don’t know. But this is one typical way on how you usually deal WebAPIs regarding very long computations.
One POST request for “starting” something and GET for getting the “status” (frequently called) and one GET for returning the result, once it’s done. For Hops implementing this could indeed fail due to the fact that the developers probably never intended this sort of behavior. And probably a RestApi approach is not really suited for this situation. I don’t know. I personally would also never use something like Hops, but instead create my own little TCP server-client connection. Its a bit more to-do, but gives you much more freedom… I don’t know.

We didn’t really consider hour long background computations when designing Hops. We’ll need to think of a possible different approach to take for this type of scenario.

1 Like

:sweat_smile: Touché.

But anyways, I could use it, it works. I just had to do multiple workarounds as described above. Observing the task-manager, observing the Rhino Compute Console Window, and refreshing Hops on time. Convinience is different.
Additionally, if I refresh Hops too early, it starts a second parallel computation, which I can see in the Task-Manager. Then I don’t get any results, even if the first computation is finished. As long as this doesn’t change, I’ll have to use GHAsync again. Sigh.

@henry.langner Strange behaviour of the Hops component after I tried setting RHINO_COMPUTE_TIMEOUT value myself was the reason why I deleted the post. I’m sorry I didn’t mean to confuse anyone but I did it anyway. I restored the post just in case.

2 Likes

@henry.langner Do you have an example that I could look at to replicate this issue?

I don’t believe the RHINO_COMPUTE_TIMEOUT is the entire solution. There is also a default timeout on the request side (hops) that is set to 100 seconds. We would need to allow this to be configured as well.

@stevebaer If you compute only locally (as I do), why would you need these timeouts anyway?

And by the way, I’d like to renew @blind’s feature request, that the component may tell the user that it ran into a timeout. By now I can only notice it when I see that the CPU load of rhino.compute is 0%, or the Rhino.Compute Console Window is open (which is off by default).

@AndyPayne I attached a file with a computation that takes slightly longer than 100s (on my computer about 130s) hops-time-out.gh (319.6 KB)

Hey Henry. I’m looking into your issue and I think we might have a solution… However, I’m trying to test out your file and the rhino.compute server tells me that I’m missing one of the GH components in the Hops file. Are you using any 3rd party plugins that I need to have installed?

@henry.langner I’ve just pushed a new build of Hops (version 0.15.1) to the package manager. If you download this latest version, you should see a new textbox in the Hops preferences dialog (see below) where you can set the http client timeout for Hops. I believe this should fix the issue you’re seeing with having to disable/enable the Hops component. If you increase the timeout time and then run your calculation, I believe the component should update appropriately even on long running calculations.
Hops_0.15.1

3 Likes

:heart_eyes: Thank you very much! You just made my day!

1 Like