Dynamically updating a 'DisplayBitmap'

Rhino Developer
, ,
1

Hello, I am encountering issues with dynamically drawing images in the viewport using a DisplayConduit.

My goal is to display an OpenStreetMap or Google Satellite Map in the viewport dynamically.

The map is divided into several images (called tiles) that are requested from the tile server via a REST API.

I draw each tile (256x256px) in a DisplayConduit.DrawForeground using the DisplayPipeline.DrawSprite function.

All downloaded images are stored in memory; I never have more than 100 tiles in memory to display the map in full screen.

When I use GDI+, I can successfully draw the images dynamically, keeping about a hundred System.Drawing.Graphics objects open to redraw the tiles. This incurs a slight memory overhead, but it works without leaks or memory corruption.

My issues arise when I need to encapsulate the images (System.Drawing.Bitmap) in a Rhino.Display.DisplayBitmap to use the DisplayPipeline.DrawSprite function.

The DisplayBitmap seems to perform a copy of the image at the time the class is created, therefore any modifications made to the image (System.Drawing.Bitmap) are not reflected in the DisplayBitmap.

In the following video, you can see that the zoom level (the last digit of each tile: 12, 13, or 14) that should change each time the map is zoomed in or out remains the same and corresponds to the zoom level at which the DisplayBitmap was created.

I have tried recreating a new DisplayBitmap on each render pass, but Rhino hangs with a considerable spike on one CPU core. I have tried to clean up the memory properly using DisplayBitmap.Dispose and even force the garbage collector with GC.Collect, but the result is the same.

The question is, how can I dynamically update a DisplayBitmap ?

jmv

2

I’m not the most knowledgeable about all of this. But I’ll do my best.

It seems like this is correct, as per the API docs
https://developer.rhino3d.com/api/rhinocommon/rhino.display.displaybitmap/displaybitmap#(string,bitmap)

the bitmap will be used and it will be added to Rhino’s bitmap cache with the path supplied

And checking source code, yes it looks like it’s copied. And I don’t see any way around that currently. This would mean you can’t update the Bitmap and you’ll need 1 DisplayBitmap per bitmap.

This is not possible in Rhinocommon as far as I can tell.

Are you caching old ones? Or a new one every render pass? They look like they have IDs so caching into a dict should work fine, that would be how I’d approach this.

1 Like
3

Hello @CallumSykes, Thank you for your response!

During my test, I created a new DisplayBitmap for each rendering pass. Currently, I am using a geographical area of about 20 km by 50 km, and I would need to cache nearly 2 million (1,779,032) tiles of 256 pixels by 256 pixels to cover the entire area and different zoom levels.
(This would be like downloading all the images from the Google server and storing them in memory…)

I just tested creating a DisplayBitmap for each rendering pass and storing the previous ones in a dictionary. This seemed to work a bit better for 3000 images, but beyond that, Rhino started to become unstable, black squares appeared on the interface, and then it crashed.
(Surprisingly, there was no noticeable increase in memory or CPU consumption.)

The first test I conducted to assess the feasibility of this plugin was done using GDI+ and the System.Drawing space.
It worked perfectly. Do you think it would be possible for me to use HDC GetDC(IntPtr hWnd) (User32.dll) to retrieve the handle of the Rhino window (as in GHGL) and then pass it to System.Drawing.Graphics.FromHDC to draw directly on the screen?
If so, are there any precautions to take with the Rhino pipeline?

4

Incase you’re not I’d suggest loading them lazily, i.e only when you need them with a max size, say, 100, and when the maximum items of the dictionary overflows, remove the earlier items. The great thing is you know the bitmap sizes so you know exactly how much memory 100 bitmaps is. I’m assuming you’d never need to see all 3000 at once? Maybe I misunderstand.

Anddddd we have reached the limits of my knowledge :slight_smile:
I have no idea honestly. @stevebaer would know.

1 Like
5

I just ran another set of tests, storing the DisplayBitmap in a dictionary and cleaning it periodically as you suggested.
All tests lead to the same conclusion: it is not possible to create DisplayBitmap recursively and intensively.

I tried to retrieve the HDC from Rhino to draw directly on the screen with System.Drawing.Graphics. It works, but but it’s no use since the Rhino pipeline overrides it to display the 3D view.

6

Hello,

I think I’m going to use OpenGL directly, which is quite a blow because I thought displaying a dynamic sprite (like a GIF) with the Rhino API would be something trivial.

This is a real project, not just a personal experiment. We have a model of Paris to create, a 16m by 10m dimensional project that could take us a year.
All existing files, such as satellite point clouds or other techniques like importing OpenStreetMap data, remain unusable for our goals.
For now, the work of modeling the city will occupy 8 people 40 weeks (approximately), and I need to find solutions to facilitate this work.
I saw that Bango 2 allows for the creation of video textures and for this project, I could have purchased the licenses, but unfortunately, I don’t see any usable API.

@nathanletwory, perhaps you have an idea? Even if I have to force the 3D models to render, is there a way to create animated textures that I could generate on the fly for each frame?

Thank you,
jmv.

7

How is the data structured that is coming in from these APIs? Maybe we can eliminate the entire step of making a system.drawing.bitmap.

1 Like
8

Hello @stevebaer ,

This is a flat structure.
Each server call is made with a simple URL and returns an image of 256x256 pixels, for example:
$"https://{Server}/tiles/{Longitude}/{Latitude}/{Zoom}.jpg"

My function that calls the server looks like this:

using SD = System.Drawing;

SD.Bitmap GetRequest (string url) {
    var httpWebRequest = (HttpWebRequest) WebRequest.Create (url);
    using var httpWebResponse = (HttpWebResponse) httpWebRequest.GetResponse ();
    using var stream = httpWebResponse.GetResponseStream ();
    return new SD.Bitmap (stream);
}
9

Ah, so this is a jpg stream so some level of decoding does need to be performed. We probably can’t eliminate that step then.

I’m just finishing up a vacation right now and will try to find some time in the next couple weeks to look closer into this.

1 Like
10

OK, thanks Steve!

I understand that there is a need to decode the JPEGs each time, and if I use OpenGL,
I will also have to create the texture arrays for the graphics card.

The problem with the API is that it seems to have no solution for updating the Rhino cache;
DisplayBitmap does not have any function for that,
and based on all my tests, it cannot be destroyed and recreated intensively.

Have a great end of your vacation!

11

Hello!

I’m wondering about this line of code in the GhGL plugin:

Is creating a new VAO for each frame a necessity and a precaution to take with the Rhino Pipeline?

Thanks.

12

We generate many temporary vertext array objects in the core. They don’t appear to have any impact on performance

1 Like
13

Hi @kitjmv ,
It looks like you are close to finding a solution, but I thought I might mention with Heron in GH, you can use the Slippy Map tools to dynamically display slippy map tiles using a boundary based on a viewport and a zoom level. The SlippyRaster component caches the image tiles to a local location (which can be a shared folder) and uses GH’s AddPreviewItem (Heron/Heron/HeronRasterPreviewComponent.cs at eee9c25c9c70d053da8712f48f0d52d16cae4b82 · blueherongis/Heron · GitHub) to update as you zoom around.

20241104_HeronSlippyMap.gh (21.5 KB)

-Brian

2 Likes
14

Hello @Brian_Washburn ,

Not really… :slight_smile: But it’s progressing.

Heron was the first thing I tried before even thinking about writing a single line of code.
Unfortunately, it didn’t meet certain requirements.
First, I need a map that is limited to a rectangle, and I need to be able to distort the image with this rectangle to adapt it (cheat).
Also, I need it to work all day and across an entire city without too much lag.
For performance and storage reasons, I do not want to use the hard drive.

Having to write all this in OpenGL with C#, well, I don’t want to do this every day, but in the end, it has some advantages.
I could imagine that since it is now a shader with some OpenGL functions, I could apply it as a texture on any type of surface. Even on deformed heightmap surfaces…

So, I was wondering if anyone knows if it’s possible in Rhino to create a material when you have a shader and OpenGL code that updates the texture?
(I am not inquiring about cycle materials, just basic materials for rendering in the viewport.)

Thank you, jmv

1 Like
15

I made some changes to DisplayBitmap today. I can send you an internal build tomorrow if you want to try.

4 Likes
16

I am working on a similar project that streams map tiles and would like to learn more about the correct way of using DisplayBitmaps in materials so I don’t need to save the tiles to disk.

Is there an example of this I can read up on?

17

Hello @Holo

I store all my images in a Dictionary<"zoom-lon-lat", bitmap>. I haven’t had the chance to test it over a long period yet, but for now, I don’t notice any memory increase or particular issues with caching directly in memory. I assume it will be necessary to periodically clear out unused images, but it seems good.

For your information, BlenderGIS apparently uses an SQLLite table the Open Geospatial Consortium seems to indicate that this is a high-performance solution…

18

Thanks, I have all up and running, made a custom tile finder, sizer etc and convert the byte_data to a bitmap that I want to place in a material with out saving the bitmap.

# Download the tile
response = requests.get(url)
if response.status_code == 200:
    # Convert bytes to System.Array[System.Byte]
    byte_data = Array[Byte](response.content)
    stream = MemoryStream(byte_data)
    bitmap = Bitmap(stream)
    # Display bitmap in Rhino
    display_bitmap = Rhino.Display.DisplayBitmap(bitmap)

So where I struggle right now it finding some documentation on how to quickly put this display_bitmap in a rendermaterial that I can assign a mesh for viewing.

19

Hello @stevebaer!

Yes, of course! I’m curious to see what improvements you’ll bring!

For information, the biggest issue I encounter with this implementation is the need to refresh the viewport frequently and randomly when an image is received.

When working with offline images, the performance is excellent, but with images from an HTTP server like OpenStreetMap, silent errors occur, and Rhino eventually freezes after 30-60 seconds of use.

To solve this issue, I have an IdleRedraw class that manages downloads and periodically refreshes the viewport.

Due to these silent errors, I’m not sure if the problem comes from multiple calls to RhinoView.Redraw in asynchronous functions or from multiple HTTP connections (possibly both).

However, now the problem is that it significantly slows down performance, even with offline images; the tiles display one image at a time, as shown in the video above. (but, there’s much smoother interface performance.)

I was wondering if there might be another place to perform these operations?
Even if I understand when RhinoApp.MainLoop events occur,
what is the purpose of using this event in C# plugin development?
And would you recommend using it?

20

Yes!
That’s exactly the same observation that led me to write the first message at the top.
From my tests, it’s not possible to frequently update a DisplayBitmap, but Steve just announced some updates… :slight_smile:

1 Like