Rhino8 python subprocess.check_output() error

Hi There,

In rhino7 I was used to running a piece of code that runs an external instance of python3.11, to perform functionality that wasn’t available in Ironpython2.7.

If I run the same piece of code in Rhino8 I get an error, likely related to the PYTHONHOME variable judging by this post.
To me it seems that Rhino8 has set the PYTHONHOME variable and I need to make sure that the check_output() call does not use this variable

Does anyone know how to solve this issue?

Thanks,
Tim

Status : CalledProcessError 1
Python path configuration:
  PYTHONHOME = '\\?\C:\Users\info\.rhinocode\py39-rh8'
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C:\Users\info\.rhinocode\py39-rh8\Lib'
  sys._base_executable = 'C:\\Program Files\\Python311\\python.exe'
  sys.base_prefix = '\\\\?\\C:\\Users\\info\\.rhinocode\\py39-rh8'
  sys.base_exec_prefix = '\\\\?\\C:\\Users\\info\\.rhinocode\\py39-rh8'
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\Program Files\\Python311\\python.exe'
  sys.prefix = '\\\\?\\C:\\Users\\info\\.rhinocode\\py39-rh8'
  sys.exec_prefix = '\\\\?\\C:\\Users\\info\\.rhinocode\\py39-rh8'
  sys.path = [
    'C:\\Program Files\\Python311\\python311.zip',
    'C:\\Users\\info\\.rhinocode\\py39-rh8\\Lib',
    'C:\\Users\\info\\.rhinocode\\py39-rh8\\DLLs',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ImportError: bad magic number in 'encodings': b'a\r\r\n'

Current thread 0x0000435c (most recent call first):
  <no Python frame>
#! python 2
# example_test_file.py
import json
import sys
from subprocess import check_output, STDOUT, CalledProcessError

def main():
    try:
        # example payload 
        input = { 'a':'my_value' }
        json_data = json.dumps(input, ensure_ascii=False).encode('utf-8')

        # Call external process in python3, send payload as json string
        filename = 'RunTestFile.py'
        output = check_output(['python', filename, '-i', json_data], stderr=STDOUT, shell=False, universal_newlines=True, creationflags = 0x08000000)

        # Temp disabled, try with exact executable path
        # path = r'C:\Program Files\Python311\python.exe'
        # output = check_output([path, filename, '-i', json_data], stderr=STDOUT, shell=False, universal_newlines=True, creationflags = 0x08000000)

        # process the output
        lines = output.split('\n')
        for line in lines:
            try:
                # try to capture any line that looks like a response in json
                response = json.loads(line)
                if response['status'] != None:
                    return response
            except Exception as exc:
                # print any other output
                print('py3 output : %s' % (line))

    except CalledProcessError as exc:
        # capture timeouts
        if exc.returncode == -1073741510:
            return {'status':False, 'msg':'API call timeout'}

        # print the external process error
        print("Status : CalledProcessError", exc.returncode)
        lines = exc.output.split('\n')
        for line in lines:
            print(line)

    except Exception as exc:
        print(exc)

if __name__ == '__main__':
    main()
#! python3
# example_file_to_execute.py
import sys
print('version from instance:', sys.version)

UPDATE 2023/12/13

I was able to bypass the issue by deleting the “PYTHONHOME” variable just before calling subprocess. I will further investigate if this affects the workings of Phino8 python engine

        # Remove PYTHONHOME variable
        if os.environ.get('PYTHONHOME'):
            del os.environ['PYTHONHOME']

        # Call external process in python3, send payload as json string
        filename = 'RunTestFile.py'
        output = check_output(['python', filename, '-i', json_data], stderr=STDOUT, shell=False, universal_newlines=True, creationflags = 0x08000000)

1 Like

Has this solution consistently solved the problem for you?

I have some scripts such as rhinopython/spb_Brep_filletEdges_in_FreeCAD.py at master · CADacombs/rhinopython · GitHub that subprocesses to FreeCADcmd.exe. From Rhino 8, some/most of the time an error like that below occurs. Only sometimes I have found PYTHONHOME set to the RhinoCode library. Even when it is not, the same error may occur.

stderr from Popen.communicate: Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "\\?\C:\Users\spb\.rhinocode\py39-rh8\lib\site.py", line 73, in <module>
    import os
  File "\\?\C:\Users\spb\.rhinocode\py39-rh8\lib\os.py", line 29, in <module>
    from _collections_abc import _check_methods
  File "\\?\C:\Users\spb\.rhinocode\py39-rh8\lib\_collections_abc.py", line 12, in <module>
    GenericAlias = type(list[int])
TypeError: 'type' object is not subscriptable

@timcastelijn

Yes that is correct. Rhino sets up PYTHONHOME so it can load pytohn properly. If you have a separate python.exe that you would like to launch as a subprocess, the PYTHONHOME needs to be reset for that since it will be automatically inherited from parent process (Rhino)

I will make sure this is documented.

1 Like

The value of PYTHONHOME doesn’t seem to explain all occurrences of this problem.

After running the script referenced in my previous post, and receiving the error shown in the same post, in the _EditPythonScript editor, I then run:

from __future__ import absolute_import, division, print_function, unicode_literals

import os, sys
from subprocess import Popen, PIPE

def main():
    
    sEval="os.environ.get('PYTHONHOME')"; print(sEval+':',eval(sEval))
    sEval="os.environ.get('PYTHONPATH')"; print(sEval+':',eval(sEval))
    print("sys.path:")
    print(*sys.path, sep='\n')

    sForConsole = [
        "C:/Program Files/FreeCAD 0.21/bin/FreeCADcmd.exe",
        "-c",
        "print(f\"{os.environ.get('PYTHONHOME') = }\"); print(f\"{os.environ.get('PYTHONPATH') = }\"); print(\"sys.path:\"); print(*sys.path, sep='\\n')",
        ]


    p = Popen(sForConsole, stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True) # shell=True hides console window.

    try:
        stdout, stderr = p.communicate()
    except:
        pass
    else:
        print("\nstdout from Popen.communicate: {}".format(stdout))
        print("\nstderr from Popen.communicate: {}".format(stderr))
        print("\nreturncode from Popen: {}".format(p.returncode))

if __name__ == '__main__': main()

and receive:

os.environ.get('PYTHONHOME'): None
os.environ.get('PYTHONPATH'): None
sys.path:
C:\My\Rhino\PythonScript\Study\subprocess
C:\Program Files\Rhino 8\Plug-ins\IronPython\Lib
C:\Users\spb\AppData\Roaming\McNeel\Rhinoceros\8.0\Plug-ins\IronPython (814d908a-e25c-493d-97e9-ee3861957f49)\settings\lib
C:\Users\spb\AppData\Roaming\McNeel\Rhinoceros\8.0\scripts
C:\My\Rhino\PythonScript

stdout from Popen.communicate: 

stderr from Popen.communicate: Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "\\?\C:\Users\spb\.rhinocode\py39-rh8\lib\site.py", line 73, in <module>
    import os
  File "\\?\C:\Users\spb\.rhinocode\py39-rh8\lib\os.py", line 29, in <module>
    from _collections_abc import _check_methods
  File "\\?\C:\Users\spb\.rhinocode\py39-rh8\lib\_collections_abc.py", line 12, in <module>
    GenericAlias = type(list[int])
TypeError: 'type' object is not subscriptable


returncode from Popen: 1

8.7.24138.15431, 2024-05-17

I am not sure if I am getting this correctly.

  • Is FreeCADcmd.exe a python executive?
  • Are you setting the PYTHONHOME and PYTHONPATH for the subprocess?

Yes

No

I modified the above script so it includes sys.path for both Rhino and FreeCAD. Of course, the FreeCAD output is unknown when the rhinocode referencing problem occurs. Here is the output of the script above after restarting Rhino and before running my fillet script.

os.environ.get('PYTHONHOME'): None
os.environ.get('PYTHONPATH'): None
sys.path:
C:\My\Rhino\PythonScript\Study\subprocess
C:\Program Files\Rhino 8\Plug-ins\IronPython\Lib
C:\Users\spb\AppData\Roaming\McNeel\Rhinoceros\8.0\Plug-ins\IronPython (814d908a-e25c-493d-97e9-ee3861957f49)\settings\lib
C:\Users\spb\AppData\Roaming\McNeel\Rhinoceros\8.0\scripts
C:\My\Rhino\PythonScript

stdout from Popen.communicate: 
os.environ.get('PYTHONHOME') = None
os.environ.get('PYTHONPATH') = None
sys.path:
C:\Program Files\FreeCAD 0.21\Mod\Web
C:\Program Files\FreeCAD 0.21\Mod\Tux
C:\Program Files\FreeCAD 0.21\Mod\Test
C:\Program Files\FreeCAD 0.21\Mod\TechDraw
C:\Program Files\FreeCAD 0.21\Mod\Surface
C:\Program Files\FreeCAD 0.21\Mod\Start
C:\Program Files\FreeCAD 0.21\Mod\Spreadsheet
C:\Program Files\FreeCAD 0.21\Mod\Sketcher
C:\Program Files\FreeCAD 0.21\Mod\Show
C:\Program Files\FreeCAD 0.21\Mod\Robot
C:\Program Files\FreeCAD 0.21\Mod\ReverseEngineering
C:\Program Files\FreeCAD 0.21\Mod\Points
C:\Program Files\FreeCAD 0.21\Mod\Plot
C:\Program Files\FreeCAD 0.21\Mod\Path
C:\Program Files\FreeCAD 0.21\Mod\PartDesign
C:\Program Files\FreeCAD 0.21\Mod\Part
C:\Program Files\FreeCAD 0.21\Mod\OpenSCAD
C:\Program Files\FreeCAD 0.21\Mod\MeshPart
C:\Program Files\FreeCAD 0.21\Mod\Mesh
C:\Program Files\FreeCAD 0.21\Mod\Measure
C:\Program Files\FreeCAD 0.21\Mod\Material
C:\Program Files\FreeCAD 0.21\Mod\Inspection
C:\Program Files\FreeCAD 0.21\Mod\Import
C:\Program Files\FreeCAD 0.21\Mod\Idf
C:\Program Files\FreeCAD 0.21\Mod\Fem
C:\Program Files\FreeCAD 0.21\Mod\Draft
C:\Program Files\FreeCAD 0.21\Mod\Arch
C:\Program Files\FreeCAD 0.21\Mod\AddonManager
C:\Program Files\FreeCAD 0.21\Mod
C:\Program Files\FreeCAD 0.21\lib
C:\Program Files\FreeCAD 0.21\Ext
C:\Program Files\FreeCAD 0.21\bin
C:\Program Files\FreeCAD 0.21\bin\python38.zip
C:\Program Files\FreeCAD 0.21\bin\DLLs
C:\Program Files\FreeCAD 0.21\bin\lib
C:\Program Files\FreeCAD 0.21\bin
C:\Program Files\FreeCAD 0.21\bin\lib\site-packages
C:\Users\spb\AppData\Roaming\FreeCAD\Macro\
C:/Users/spb/AppData/Roaming/FreeCAD/Macro
C:\Program Files\FreeCAD 0.21\Macro


stderr from Popen.communicate: 

returncode from Popen: 0