HttpClient.Timeout of 100 seconds with Rhino Compute

Rhino Developer compute.rhino3d
, , ,
1

Hello Team,

We are facing the following timeout error. Appreciate any pointers or suggestions. We did set up Rhino Compute as per the deployment guide:- Rhino - Deployment to Production Servers
with Rhino 7. The grasshopper template typically takes more than 1min to resolve for the first time and in look out for options to increase the timeout limit.

2022-03-05 01:41:10.542 +00:00 [INF] Started child process at http://localhost:6001 at 3/5/2022 1:41:10 AM
2022-03-05 01:41:42.305 +00:00 [INF] Max concurrent requests = 1
2022-03-05 01:42:42.335 +00:00 [INF] Max concurrent requests = 1
2022-03-05 01:42:55.400 +00:00 [ERR] HTTP POST /grasshopper responded 500 in 132594.0965 ms
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
—> System.TimeoutException: The operation was canceled.
—> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
—> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request…
—> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Http.HttpConnection.FillAsync(Boolean async)
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
— End of inner exception stack trace —
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
— End of inner exception stack trace —
— End of inner exception stack trace —
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at rhino.compute.ReverseProxyModule.SendProxyRequest(HttpRequest initialRequest, HttpMethod method, String baseurl) in /home/runner/work/compute.rhino3d/compute.rhino3d/src/rhino.compute/ReverseProxy.cs:line 139
at rhino.compute.ReverseProxyModule.ReverseProxyGrasshopper(HttpRequest req, HttpResponse res) in /home/runner/work/compute.rhino3d/compute.rhino3d/src/rhino.compute/ReverseProxy.cs:line 195
at Carter.CarterExtensions.<>c__DisplayClass1_0.<b__0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at rhino.compute.ApiKeyMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/compute.rhino3d/compute.rhino3d/src/rhino.compute/ApiKeyMiddleware.cs:line 34
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
2022-03-05 01:42:55.657 +00:00 [ERR] Connection ID “18230571301796315142”, Request ID “80000007-0002-fd00-b63f-84710c7967bb”: An unhandled exception was thrown by the application.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
—> System.TimeoutException: The operation was canceled.
—> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
—> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request…
—> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Http.HttpConnection.FillAsync(Boolean async)
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
— End of inner exception stack trace —
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
— End of inner exception stack trace —
— End of inner exception stack trace —
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at rhino.compute.ReverseProxyModule.SendProxyRequest(HttpRequest initialRequest, HttpMethod method, String baseurl) in /home/runner/work/compute.rhino3d/compute.rhino3d/src/rhino.compute/ReverseProxy.cs:line 139
at rhino.compute.ReverseProxyModule.ReverseProxyGrasshopper(HttpRequest req, HttpResponse res) in /home/runner/work/compute.rhino3d/compute.rhino3d/src/rhino.compute/ReverseProxy.cs:line 195
at Carter.CarterExtensions.<>c__DisplayClass1_0.<b__0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at rhino.compute.ApiKeyMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/compute.rhino3d/compute.rhino3d/src/rhino.compute/ApiKeyMiddleware.cs:line 34
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

2

It looks like we can increase the timeout limit in rhino.compute. Would 200 seconds be enough?

3

@stevebaer Thanks, for prompt reply. Would prefer 300 sec. Also, appreciate any pointers on how to make the changes to rhino.compute for reflecting desired timeout.

4

Hi @ShivaNaveen_Ravi Please see this post for a new update to Hops (0.12.0). There is a new environment variable called RHINO_COMPUTE_TIMEOUT which can be set to configure the request timeout (in seconds) for the HttpClient in the reverse proxy module. Hopefully this will resolve the timeout issue you were experiencing.

5

Hi,

jumping on this thread here. Just as a clarification, should the environment variable RHINO_COMPUTE_TIMEOUT be set in the Admin user (where the IIS runs) or the RhinoComputeUser (where rhino.geometry runs) if you’ve followed the deployment guide?

Best,
Erik

6

Good question. It should be set in for the Admin user because this is checked in the rhino.compute.exe process (not the compute.geometry.exe).

2 Likes
7

Hey thanks a lot for the quick reply! I’ll try it tomorrow!

Best,
Erik

8

Hello again Andy!

I’ve tried this now and it does not seem to work to set the RHINO_COMPUTE_TIMEOUT variable for me. The server times out after 60 000 ms regardless of what I set for this value. The problem is not the script I’m trying to run most of the time, but rather the startup time for the server since we’ve got quite a few plugins. It takes anywhere from 60-80 seconds before the server starts to work and then we’ve already reached the timeout. However when I send my second request and the server has already started I don’t have the same problem.

We’re not using hops but an API to call on compute. Not sure if this makes any difference in the matter. We used the bootstrap script to install rhino like it is explained in the guide.

Thanks!
Best,
Erik

9

Hmm. Is it possible to trim down your 3rd party plugin list? If not, have you tried beefing up your VM instance to a higher tier so that the performance is better? 60-80 seconds seems like a long time for startup.

10

I think that it’s mostly due to having way to many plugins that all takes some time to load. I’m looking into it right now and I’ll try to get it down from 20-something to about 3 or 4. I’ll update here with the results for future reference once it’s done.

Anyway, is the rhino compute timeout variable only expected to work for hops or should it work the same when calling it from an external API? In that case I’m not sure what I’m doing wrong. I guess the bootstrap script installs the latest rhino release so the updates your shared here

should be in there, right?

11

Well the RHINO_COMPUTE_TIMEOUT would need to be set on the VM side. Once you restart the VM, then the rhino.compute server should use that value as the timeout value for the server. For the client, you’ll need to setup your client-side timeout for whatever HTTP service you’ve configured. I don’t know how you’ve set that side up, so it’s possible that this could be be causing part of your issue.

12

I’ve indeed set the RHINO_COMPUTE_TIMEOUT on the server side as an env variable in windows. This morning I removed all the unnecessary packages and it reduced he warmup significantly as expected. Probably down from 80 seconds to like 10 or 20 seconds so it should solve the imediate problem.

Thanks for taking the time to look at this with me!
Best,
Erik

1 Like