Quicker way of creating specific paths in a data tree

Hi, I want to create a specific data structure consisting of 8,000,000 paths (or more).

The GH Create Path is taking roughly 3 hours to create it… I am wondering if there would be a quicker way to do so using Python or C# ?

I don’t think Grasshopper was designed to process so much data. I don’t know what you do, and I have difficulties to see any real-world use-case which yields something of use. But lets assume it is reasonable. I would then eliminate GH out of the data manipulation as much as possible. Effectively you won’t work with DataTrees/GH_Structures at all. Build a custom graph or use a better suited data-structure and do everything by code. You want to allocate it once and keep in memory until you finished. Ideally you have one big array, and a mapping of indices. Be careful with strings, type conversion, boxing etc. You can quickly increase the data by magnitudes. Be aware, that piping data into a custom script component, will copy it. So you need a lot of understanding about how scripting works beneath the surface, to prevent increasing the data to process.
You should try to reduce the data, that is the only true solution. What is it that you trying to do? Can you elaborate?

Yes probably best to take out GH from this part of the equation and process the data differently.

I am parsing .asc files for high res georeferenced topological models (elevation @1m over 2km x 2km). I was using the tree path as a “coordinate system” as a mean to isolate easily the information I needed for the desired scope (say I’m interested in an area which overlaps over 4 different files, I could pick the domain of those exact coordinates).

It was working fine with lower res as files were 1/5 in data.

this sounds more like a job that is done with Streamreader - try to never load all data (unsorted ) in memory. process it line by line or in packages of fixed size.

google for c# process large (text) files.

if you need more help here, a minimal working example / dataset would help a lot. (…that can be processed max 3 seconds)