Bug: conversion of text to plane fails


text_to_plane_2021_Jul26a.gh (283.5 KB)

@Dani_Abalde Thanks for the explanation. That makes total sense actually that it doesn’t sufficiently describe a plane. I will create my own format, probably just going straight for a transformation matrix.

@andheum cool idea to convert to json format. Took me a while to figure out JSwan and that you have to zoom in to add an input. I have not seen that in any other plugin - maybe you could add a small gif of that in food4rhino. You only show the deserializing part, but not the serializing.

@Joseph_Oster thanks for the super quick way of doing it. I will add some IDs and make sure it’s all in one line per plane for a text file. Right now it would just flatten everything into lines, so you would have 3 lines per plane. So to better simulate what will happen you would have to flatten the text output and then reconstruct from there. But I will do that part.

IDs? Like data tree paths?

I’ve used GH Python to produce (json.dumps()) and read (json.load()) JSON successfully but hesitate to wade into it for this…

https://docs.python.org/3/library/json.html

By the way, working with 14,400 planes as test data is a pain! (SLOW)

P.S. Here is a single line per plane text format using “|” as a separator.:


text_to_plane_2021_Jul26b.gh (287.8 KB)

Thank you. Yes, I know having that large a data set is a pain, but it’s the data set I have to work with, so its nice to keep it realistic and see also which solutions are slower than others.

I am still kind of baffled why Grasshopper is so terrible at dealing with large datasets. Anything above say 10.000 list length and it seems to really get slow.

Try whether disabling the autosave speed up your workflow.

I just disabled it, but it seems just as slow. A lot of things you don’t realise are slow until you really crank the numbers. I will post a separate issue as to why ScaleNU is so much slower than Scale (factor 20x). Understanding these things is the first step in working around it.

Here is a version that includes the original data tree paths as IDs, flattens the “text file” (yellow group), then reconstructs the original data tree. The bottlenecks here are Construct Plane (17.5 secs.) and Replace Paths (6.3 secs.).


text_to_plane_2021_Jul26c.gh (289.0 KB)

1 Like

@andheum I just gave your suggestion a go in the hopes it would be faster/better, but get stuck on getting the C# to work.

I have added the Newtonsoft.Json.dll file to the assemblies and added in the code with “using…”.

But I somehow get lots of error messages:

Any idea what I am doing wrong?

Also what does the “Deserialize” do if not convert it back to its original data format? At least that is what other deserializer do that I have coded myself!?

json_serialize_planes.gh (7.4 KB)

I tried Python and C# versions of Construct Plane and discovered some interesting things:

  1. Python and C# are almost identical in Profiler times.
  2. A substantial part of the time in all cases is converting the X and Y vectors from text, as you can see by passing the text versions through Vector params first (or not).
  3. Python and C# are both slower than the standard GH component. (or nearly the same when vector conversions are included)

text_to_plane_2021_Jul26d
text_to_plane_2021_Jul26d.gh (283.0 KB)

objet to string = SErialize.
string to object = DEserialize.

You are using the wrong method.

@Joseph_Oster I noticed the same actually, in most cases of very slow components with lots of data it is because of the type conversion. Thanks for verifying this on your end as well. Such a shame that it is that slow and kind of not understandable. I use the same kind of datasets in other tools like vvvv and if you convert from string to number there it barely makes a dent in the 60fps it normally runs at.

@Dani_Abalde wait, I am confused now. That is what Andrew shows in his screenshot though!? I want to convert the JSON string into an object again of datatype plane.

Ok then you have to DEserialize but in your screenshot your input is Plane type instead of string type, but even so you are sending a Json Object.

Right click on “x” input > type hint > string and plug the panel to “x”.

1 Like

So I just gave it a go with actually writing and then reading a JSON file and it doesn’t seem to want to work.

In the documentation it says you can just plug a path to a .json file into the deserializer. But it always says “no valid JSON properties found”, even though the file literally just contains the same output as what the serializer outputs.

json_serialize_planes_read.gh (20.4 KB)

So from what I gather the Deserialise compontent of jSwan just splits the JSON data into its groups, but does no data type conversion the way the serializer does!? Just trying to wrap my head around what jSwan does and does not do.

I guess that JSwan component input is asking you for json data, not file path data.


ObjectToStringJSON.gh (8.2 KB)

In C:\Program Files\Rhino 7\System you can find the library Newtonsoft.Json.Rhino.dll to add in the C# Script components if needed.

  private void RunScript(object obj, ref object Json)
  { 
    Json = Newtonsoft.Json.JsonConvert.SerializeObject(obj); 
  }
  private void RunScript(string json, object type, ref object Obj)
  { 
    Obj = JsonConvert.DeserializeObject(json, type.GetType()); 
  }

Thanks I will take a look at your implementation!

Here is the documentation for jSwan. You can apparently either input json data or a file path to a .json file.

JSwan supports a file path as input — it will read it in. The problem is that you’re not writing a valid json file — you’re writing a bunch of individually valid JSON files into one file (without grouping them). This is valid:

{
"property": "value"
}

but this is not:

{
"property": "value"
}
{
"property": "value"
}

To remedy this, right click the “planes” input to your Construct JSON component and set it to “list access” instead of “item access” so that it will produce a single valid JSON from your list.

1 Like

Oh, that makes sense. With item access, it treats every branch as a separate file, right!?

What I noticed though, is that my strategy is actually not fast enough. The back story:

We are rendering animations in Grasshopper using the V-Ray Timeline. The whole patch is quite slow because there are a lot of objects we are moving in every frame (14.400 objects). Since the whole patch takes close to 10 seconds per frame just for the updating, not the rendering, it would be nice to be able to cache the transforms and be able to play them back. So the idea was: save a json file with all transforms for each frame, then have a sort of playback mode, where it just reads the file and applies the transforms without doing all the rest of the patch.

Turns out it’s actually slower that way then just letting everything calculate because the conversion of string to either plane or a matrix transform is way too slow.

Can you think of a quicker way to cache a bunch of solutions and then “play them back”? There must be a better way, although we have been using the safe-each-frame-to-a-text file with other software and that is working really well, not like real-time speed - that is not the goal, but something like 0.5s per frame to load and apply all the transforms.

Alternatively do you know why the conversion is so slow in Grasshopper, I have never experienced it being that slow in anything else. Also is there a way to store it without conversion of datatype?

Too bad Data Output doesn’t have a ‘Destination’ filename input that would allow a different file for each frame. When flattened, your 14,400 planes are saved quickly into a 107 KB file. Grafted makes a 148 KB file.

How many frames in your animation? Maybe if you created a data tree with one branch per frame, each branch containing 14,400 planes…? ~3+ MB per second of video at 30 FPS?

P.S. On my poor little machine with only 16 GB RAM, a data tree with 300 branches containing 14,400 planes per branch (4.32 million planes!) takes 34 seconds for Data Output to save a 32 MB file.

text_to_plane_2021_Jul30a
text_to_plane_2021_Jul30a.gh (151.8 KB)

That’s ten seconds of video at 30 FPS.

2 Likes

I think @Joseph_Oster’s solution is likely to be the most performant.

I experimented a bit with just serializing the matrix components and deserializing those to a custom class, but it’s still much slower than the data output component, which use Grasshopper’s native binary serialization (the same way it stores data in the .gh file when you, for instance, internalize a parameter)

@Joseph_Oster’s solution with Data Output is definitely performant, but on the Data Input side, it takes about as long to bring the data back in as it does to export because you are importing the whole data tree. Hacking a post from @DavidRutten 's post here, I was able to get a user input for the Destination filename working in a C# component (specifically for planes). This component takes about 3 times as long as the native Data Output (likely due to casting), which isn’t great, but you get individual ghdata files for each branch of the tree. Bringing the individual ghdata files back in with a C# component takes a fraction of the time of the full tree Data Input component. So, depending on whether you need to re-import the data multiple times, breaking out the branches of planes into individual ghdata files may save you time.

20210801_ExportData.gh (20.8 KB)

@andheum would your approach from this post work now as part of MetaHopper?