Add a test for SetMemoryLimit

The test allocates a buffer with a number of different memory limits. It should pass (the first three limits are too low), but currently fails since the first two limits return a nil result and no error rather than reporting an "out of memory" error:

$ go test --run ExampleVM_SetMemoryLimit
--- FAIL: ExampleVM_SetMemoryLimit (0.35s)
got:
<nil> null
<nil> null
<nil> InternalError: out of memory
65280000 <nil>
want:
<nil> InternalError: out of memory
<nil> InternalError: out of memory
<nil> InternalError: out of memory
652800000 <nil>
FAIL
exit status 1