Pythonception!

Posted on Thu 15 November 2018 in blog

Even though serious Maya tools development has been done in Python for almost a decade now, Python still doesn’t get a lot of love from Maya when the application starts up.

You can run a userSetup.py, which isn’t bad — but it isn’t very flexible: you’re going to get one, and only one chance with userSetup, which makes things a lot more complicated if you have support a bunch of possible configurations, like different projects or different tool setups for different specialists; you don’t want to either physically swap userSetup files all the time, or to try to make one that’s complex and flexible enough to handle all your needs. And, of course, a lot of users have their own userSetups and they get mad when you want to take them away.

Unsurprisingly. many teams end up with dedicated launchers — either dedicated applications or just .BAT files — whose job it is to configure maya on startup. For example you might keep parallel userSetup/code combinations in different locations and then launch Maya with an .BAT that manipulates sys.path or PYTHONPATH to get the right version to the top of the list. These kinds of systems work reasonably well, although they also complicate your support structure — when you’re trying to add a new artists to the team you need to train them out of just launching Maya from the applications menu, and if you’re supporting outsources or partner studios you need to distribute the launcher infrastructure as well as the code that does the actual work.

If you stick with the simple stuff, on the other hand… well, BAT files are just nasty — they’re the computing equivalent of truckstop corn dogs that have been rolling under the heatlamp since 1984.
In the words of the great Raymond Hettinger, there’s got to be a better way. A way you can distribute a toolkit that’s:

Easy to activate — It won’t require me to fire up a different program just to use the one I want to use
A good citizen — It doesn’t force me to reorganize my Maya install (particularly if I’m an outsource who might only be using this toolkit for a few weeks!)
Install Free — It doesn’t demand a separate installer or a plugin or anything else I don’t already have.

… but at the same time it’s

Fully functional — you can’t compromise the depth of what you’re delivering just in order to make it easy to deliver.

The first three are actually surprisingly easy. After all, Maya has always has it’s own dedicated script file type — you can always double-click or drag-and-drop a mel file and it will just work. No installation, not configuration, not even any special rules about where the file has to be. You can mail somebody a MEL file and they can just run it from their downloads folder.
Alas, it works in MEL. Which is to say, it’s going to be an uphill battle to make it hit that last bullet point and be fully functional. And how much power can you really pack into a single MEL file?

But, as Ray would say, there is a better way.

If you’ve ever had to write one of those BAT file launchers, you may have seen something like this:

maya.exe -c "python(\"import sys; sys.path.insert(0, 'c:\\path\\to\\mayatools'); import mayatools; maytools.run()\")"

That’s telling Maya to run a MEL command that in turn tells Python to add a folder to the path and then run. Conceptually simple, if ugly as sin; and fine enough for a one liner. Not, however, the stuff of which great programs are made — those string escapes alone are enough to make one want to take a hot shower.

You can make things a bit better if you delegate the work to an actual Python file; that gets you down to something like:

maya.exe -c "python(\"execfile 'c:\\path\\to\\mayatools\\mayatools.py')\")"

One easy thing to do from here is simply to convert these BATs to MEL files — after all, they’re really just calling MEL anyway. That gets you doen to one less layer of string escapes (you’re still stuck quoting all the python, alas) and it will let you run Python from anywhere. In many scenarios this is a good, low-maintenance way forward. You can create dedicated MELlaunchers that can fire up Python as needed without actually mucking around in the depths of your Maya install directory, without dedicated launchers, and without even writing more than the teensiest smidgen of MEL. Heck, this even works with drag-and-drop!

However it’s not perfect. The main weakness of something like this — and with BAT files, for that matter — comes when you start to wonder where, exactly, the relevant python files are. As soon as the logic gets more complex than a hard-coded path name you’re suddenly back in the business of trying to write a complex algorithm in MEL without all of the nice tools you get in Python. Sometimes you need the power of Python to unleash the power of Python… it’s a sort of snake-eating-its-own-tail situation.

To escape from that conundrum you really just need to figure out a way to squeeze the Python logic into thae executable shell of that MEL file. Unfortunately, with all those stupid string escape codes, that’s not a trivial task. Unless — let’s get really meta here for a moment — you use the power of Python to get MEL to help you unleash the power of Python.

Waaaaat?

Using Python to make MEL to make Python…?” Pythonception!

The problem with jamming complex code into your MEL is all the string escaping. But Python comes with built-in tools for dealing with string-safe encoding of arbitrary data — including strings. The base64 moduie is used to encode aribtrary data for transmission over the internet as blocks of characters. For example you can take a string like

example = r'This is a long string, including punctuation and "difficult" characters like quotes\slashes'

and turn it into something that will not require special escapes:

import base64
print base64.b64_urlsafeencode(example)
#  VGhpcyBpcyBhIGxvbmcgc3RyaW5nLCBpbmNsdWRpbmcgcHVuY3R1YXRpb24gYW5kICJkaWZmaWN1bHQiIGNoYXJhY3RlcnMgbGlrZSBxdW90ZXNcc2xhc2hlcw==

Base64encoding doesn’t use the full ascii character set, and the b64_urlsafeencode in particular avoids slashes — so you can be sure that your big blob or random characters will not cause MEL to fritz out.
This kind of packing gives you the means to take a whole Python file — as much code as you need to find your install directory, or download updates from the network, or sync up to perforce — and jam it right into your MEL file. Here’s about all you need to compile a Python file to MEL:

def python_to_mel(pythonfile, melfile):
    with open(pythonfile, 'rt') as reader:
        py_code = reader.read()
    encoded = base64.b64_urlsafeencode(py_code)
    py_to_mel = "import base64; _py_code = b64_urlsafedecode('%s'); exec py_code"  % encoded 
    with open(melfile, 'wt') as writer:
        writer.write('python("%s");' % py_to_mel

Now, when you double-click that Mel file the original code — safely stashed away inside that block of gibberish — will be turned back into code and run as if you’d typed it into the script listener.

Incidentally if something goes wrong you can see the actual text fo the Python code in Maya — in the example above it’s stashed in variable called _py_code, which will be in the global namespace (again exactly as if you had typed it in the listener).

While you can certainly achieve the same end in different ways, this method as a lot of appeal, particularly because it gives you a maximal amount of power without any infrastructure on the receiving end. You could email a MEL file like that to anybody and they could run it as is. Anything in the standard library — including important things like the ability to download an update from an http server or to call out to a source control program — can now be done from a vanilla MEL file. It’s almost enough to make you like MEL again… it’s like the good old days of 2005.

Of course you’ll quickly run into the other big hassle of Maya startup, which is setting up the rest of your Python environment: the endless old struggle of PATH and PYTHONPATH and the rest. But don’t worry! MEL has an answer for that as well… but it’s an answer that will wait until next time.