Improved performance for private package repositories

Rhino 8.15 included some performance improvements for private package repositories.

The yak.exe tool has a new “cache” command that, when run inside the private package folder, will generate an index of the available packages. When the package manager sees this index file, it will use it instead of traversing the directory. This greatly reduces the time it takes for the Package Manager to load when dealing with private repositories with many packages, large packages, or slow network connections.

$ cd X:\private\repo\directory
$ "C:\Program Files\Rhino 8\System\yak.exe" cache

Building cache for local package repository in X:\private\repo\directory

[...]

Make sure to re-build the cache if package files are added or removed from the directory. Remove the .cache* files from the directory to revert to the old behaviour.

We’ve been testing it internally with our own private package repository. For a repository of 150 package files (~4 GB total) stored on a remote file server, the time to load the Package Manager dropped from 1.5 minutes to less than a second.

Hello @will ,

Thank you for this great improvement. It is very useful !

We may have found a bug, or misalignment in function. By default YAK looks at the root and inside any subdirectory in the registered source. This works perfectly fine without index.

When running yak.exe cache at the root of the source, yak also expectedly scans all subdirectories and indexes each YAK file encountered. However, the resulting index ommits the subdirectories from the path of installation for YAK files that were not directly in the root.

Here’s an example:

Folder structure of the directory in PackageManager.Source =
MyPrivateRepo/
|-- Plugin3.yak
|-- SubDir1/
|---- Plugin1.yak
|-- SubDir2/
|---- Plugin2-alpha.yak
|---- Plugin2-beta.yak
|---- Plugin2.yak

With no index we get this and installation works.

Running yak cache from the root returns:

C:\MyPrivateRepo>yak.exe cache
Building cache for local package repository in C:\MyPrivateRepo

.cache-tmp\Plugin3\manifest.yml
.cache-tmp\Plugin3\meta.json
.cache-tmp\Plugin1\manifest.yml
.cache-tmp\Plugin1\meta.json
.cache-tmp\Plugin2-alpha\manifest.yml
.cache-tmp\Plugin2-alpha\meta.json
.cache-tmp\Plugin2-beta\manifest.yml
.cache-tmp\Plugin2-beta\meta.json
.cache-tmp\Plugin2\manifest.yml
.cache-tmp\Plugin2\meta.json

.cache.v1.zip

Rhino sees the index correctly and uses it correctly (faster load time and information matching the index even if yak files have changed), however the path in the cache is wrong, as for each YAK file in a subfolder Rhino tries to look for it in the root of the source repo.

Installing Plugin3 which is at the root works. Trying to install any other (from subdirectories) gives the following error message:

This has been tested on multiple Windows 11 machines all running Rhino 8.25.25314.11001.

Are we right to think this is a bug, and could this be fixed ?

@will @curtisw any help on this topic ? We’d like to cache a directory with subfolders but it is broken as of 8.25.25314.11001. when using YAK.