IronPython and shell subprocesses? Windows and macOS inconsistencies?

[diff-arch]

Hi,

Why is that on Windows, you can directly use executables - that have been enabled for shell use, with an environment path -, in an IronPython subprocess ( e.g. subprocess.Popen(["mytool", ...]) )?
On macOS, you need to specify the absolute path of the tool ( e.g. subprocess.Popen(["/usr/local/bin/mytool", ...]) ), which in many cases isn’t very handy! For instance, you can’t even check if the tool was installed or not, when you don’t know its path!?
To be clear, the tool also works on macOS, if you call it in bash by name (without the absolute path)!

In principal, I get why this doesn’t work. IronPython is installed as a separate, encapsulated Python installation within Rhino, but why does it work on Windows? And is there a way to fix this on macOS?

[dan]

I don’t know the answer to your question about why this does work, I could only speculate.

Can you subprocess.Popen(["which", "mytool"] on macOS? That should return /usr/local/bin/mytool if that tool is your path (obviously /usr/local/bin should be). A return value from which gives you some assurance that the tool is installed and you can pass that path into your subsequent subprocess calls.

[diff-arch]

Thanks for your reply, Dan! I had nearly given up.

@dan

Can you subprocess.Popen(["which", "mytool"] on macOS?

No, that’s not possible, although the tool is exactly installed there and readily available in macOS shell.

What I believe is happening is that the subprocess module from IronPython doesn’t run a system Unix shell per se (?), but some kind of quarantined (the irony ), emulated (?) shell, that has no access to or knowledge of the bash profile, where the $PATH variables are defined (?).
I’m sure you know this, but the bash profile, which resides in the home directory on Unix based systems, defines where the shell environment should look for tools.

On Windows, the subprocess module seems to handle things differently. When properly installed Command Line tools can be called by name. Maybe the roots of IronPython run deeper on its maker’s territory, or it even runs natively (since Windows doesn’t come with a default CPython installation, like macOS or Linux distros do)?

Man, I really wish McNeel would turn its back on .Net and IronPython, but I guess at this state we are at a point of no return.

Thanks again, Dan! Muy apreciado.

[dan]

@diff-arch

No, that’s not possible, although the tool is exactly installed there and readily available in macOS shell.

Why not? I do it all the time from “the real” Python. Does it return null or something and just not work?

[diff-arch]

@dan

Why not? I do it all the time from “the real” Python. Does it return null or something and just not work?

So I took a look at your code and I do practically the same, and stdin, stdout, as well as stderr all return empty strings.

[dan]

@diff-arch

So I took a look at your code and I do practically the same, and stdin , stdout , as well as stderr all return empty strings.

Damn. It was worth a shot. I’m looking for the :sad_trombone: emoji here, but not finding it. Well, I’m out-of-ideas on this one. Perhaps @stevebaer can think of a way around it.

[stevebaer]

Have you tried using the shell in POpen?
subprocess.Popen(["mytool", ...], shell=True)

[nathanletwory (Nathan 'jesterKing' Letwory)]

using shell=True won’t work either.

If you do in an EditPythonScript

import os
print(os.environ)

you’ll find that PATH is very, very minimal when Rhino is started from the launcher.

If on the other hand you start Rhino from a terminal (that has PATH already correctly set up), you’ll find that you’ll get a whole lot more environment variables - including the PATH you need.

I don’t know if it is possible to have Rhino start up with the environment from the default shell set up, but currently iit doesn’t get that from the launcher.

[nathanletwory (Nathan 'jesterKing' Letwory)]

Apart from starting on a properly primed shell (export PATH=...) the only thing I have found to work is:

sudo launchctl config user path "/path1:/path2", then restart your machine.

I have compiled a Blender version in a non-standard location, this is what I did to have Blender --version work with subprocess.Popen()

1. sudo launchctl config user path "/usr/bin:/bin:/usr/sbin:/sbin:/Users/Shared/dev/build_darwin/bin/Blender.app/Contents/MacOS"
2. restart OSX
3. start Rhino
4. run script

import subprocess
import os

print(os.environ)

try:
    print("*"*20)
    p = subprocess.Popen(["Blender", "--version"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out = p.communicate()
    print(out[0])
    print("-"*20)
except OSError as r:
    print(r)

This gives me as output in the command history panel:

Command: EditPythonScript
{'SHELL': '/bin/bash', 'MONO_CURRENTUSER_REGISTRY_PATH': '/Users/nathan/Library/Application Support/McNeel/Rhinoceros/registry', 'TMPDIR': '/var/folders/4l/0bk7djg14m99zn60vr_1bl480000gn/T/', 'USER': 'nathan', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin:/Users/Shared/dev/build_darwin/bin/Blender.app/Contents/MacOS', '__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x0', 'XPC_FLAGS': '0x0', 'MONO_REGISTRY_PATH': '/Library/Application Support/McNeel/Rhinoceros/registry', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.35VHLTy7gA/Listeners', 'LOGNAME': 'nathan', 'HOME': '/Users/nathan', 'XPC_SERVICE_NAME': 'com.mcneel.rhinoceros.15348', 'MONO_GAC_PREFIX': '/Applications/Rhinoceros.app/Contents/Frameworks/RhCore.framework/Resources'}
********************
Blender 2.83.0
	build date: 2020-06-03
	build time: 12:26:08
	build commit date: 2020-06-03
	build commit time: 12:23
	build hash: 7fc0053c27e5
	build platform: Darwin
	build type: Release
	build c flags:  -Wall -Werror=implicit-function-declaration -Werror=return-type -Wno-tautological-compare -Wno-unknown-pragmas -Wno-char-subscripts -Wstrict-prototypes -Wmissing-prototypes -Wunused-parameter  -mmacosx-version-min=10.11 -Xclang -fopenmp -I'/Users/Shared/dev/blender/../lib/darwin/openmp/include' -std=gnu11   -msse -pipe -funsigned-char -msse2
	build c++ flags:  -Wall -Wno-tautological-compare -Wno-unknown-pragmas -Wno-char-subscripts -Wno-overloaded-virtual -Wno-sign-compare -Wno-invalid-offsetof  -mmacosx-version-min=10.11 -ftemplate-depth=1024 -stdlib=libc++ -Xclang -fopenmp -I'/Users/Shared/dev/blender/../lib/darwin/openmp/include' -std=c++11   -msse -pipe -funsigned-char -msse2
	build link flags: -fexceptions -framework CoreServices -framework Foundation -framework IOKit -framework AppKit -framework Cocoa -framework Carbon -framework AudioUnit -framework AudioToolbox -framework CoreAudio -framework Metal -framework QuartzCore -framework ForceFeedback -liconv -Xlinker -unexported_symbols_list -Xlinker '/Users/Shared/dev/blender/source/creator/osx_locals.map' -stdlib=libc++
	build system: CMake

--------------------

Not very ideal. But should show that it isn’t Rhino nor IronPython doing anything wrong.

[diff-arch]

@stevebaer

Have you tried using the shell in POpen?

Yes, I’d already tried that, but it did not make a relevant difference. When reading through the documentation of the subprocess module, I also found a note indicating that using a shell is a potential security risk, because malicious code could be injected between processes.
Thanks for your reply, Steve!

@nathanletwory

using shell=True won’t work either.

You’re absolutely right!

@nathanletwory

you’ll find that PATH is very, very minimal when Rhino is started from the launcher.

If on the other hand you start Rhino from a terminal (that has PATH already correctly set up), you’ll find that you’ll get a whole lot more environment variables - including the PATH you need.

Ah, I recognise this from my Blender scripting workflow, where I needed to make an alias and start Blender from a shell to get a console output for debugging.
(On Windows, you can open a Command Line window from Blender, but on macOS you can’t.)

@nathanletwory

I don’t know if it is possible to have Rhino start up with the environment from the default shell set up, but currently iit doesn’t get that from the launcher.

I can confirm that it’s possible! You just have to add an alias to your bash profile:
alias rhino="/Applications/Rhinoceros.app/Contents/MacOS/Rhinoceros"
Rhino crashes though if you close the shell before exiting the app.

The EditPythonScript and RunPythonScript commands seem to only output to the Rhino command history (not the bash shell that’s still running), but you’re right the path information extracted from os.environ is much more detailed now.

@nathanletwory

I have compiled a Blender version in a non-standard location, this is what I did to have Blender --version work with subprocess.Popen()

1. sudo launchctl config user path "/usr/bin:/bin:/usr/sbin:/sbin:/Users/Shared/dev/build_darwin/bin/Blender.app/Contents/MacOS"
2. restart OSX
3. start Rhino
4. run script

Thanks for sharing this! Great stuff! Much appreciated!

I don’t know if it’s useful for this case though, since I’m developing a Grasshopper component and I can’t really ask of futur users to go through all of this.
Great information though thank you very much!!

I guess I have to come up with something entirely different.

Dan, Steve & Nathan! Thanks.

[nathanletwory (Nathan 'jesterKing' Letwory)]

@diff-arch

since I’m developing a Grasshopper component and I can’t really ask of futur users to go through all of this

The easiest way would be to provide your tool executable alongside the gha. Then you can figure out the location of the gha and use that to set up the absolute path on both Windows and MacOS.

[diff-arch]

@nathanletwory

The easiest way would be to provide your tool executable alongside the gha. Then you can figure out the location of the gha and use that to set up the absolute path on both Windows and MacOS.

Yes, it think I’ll do something like that. Thanks again!