vm: stop executing bytecode on return of nested calls

When a managed function is indirectly invoked during bytecode execution,
e.g. when calling the tostring() method of an object prototype during
string concatenation, the invoked function must stop executing bytecode
upon return to hand control back to caller.

Extend `uc_vm_execute_chunk()` to track the amount of nested function
calls it performs and hand back control to the caller once the toplevel
callframe returns. Also bubble unhandled exceptions only as far as up
to the original caller.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>

1 files changed, 52 insertions, 0 deletions

diff --git a/tests/custom/04_bugs/36_vm_nested_call_return b/tests/custom/04_bugs/36_vm_nested_call_return
new file mode 100644
index 0000000..6a52b78
--- /dev/null
+++ b/tests/custom/04_bugs/36_vm_nested_call_return
@@ -0,0 +1,52 @@
+When indirectly invoking a managed function from manged code, e.g.
+on stringifying an object using it's tostring() prototype method
+during string concatenation, bytecode execution of the nested managed
+function call did not stop and return to the caller, but continued
+past the return of the invoked function, clobbering the VM context.
+
+
+-- Testcase --
+{%
+	let o = proto(
+		{ color: "red" },
+		{ tostring: function() { return "I am a " + this.color + " object" } }
+	);
+
+	print("Result: " + o + ".\n");
+%}
+-- End --
+
+-- Expect stdout --
+Result: I am a red object.
+-- End --
+
+
+-- Testcase --
+{%
+	let o = proto(
+		{ color: "red" },
+		{ tostring: function() { die("Exception while stringifying") } }
+	);
+
+	function t() {
+		try {
+			print("Result: " + o + ".\n");
+		}
+		catch (e) {
+			warn("Caught exception: " + e.stacktrace[0].context + "\n");
+		}
+	}
+
+	t();
+%}
+-- End --
+
+-- Expect stderr --
+Caught exception: In [anonymous function](), line 4, byte 62:
+  called from function t ([stdin]:9:23)
+  called from anonymous function ([stdin]:16:4)
+
+ `        { tostring: function() { die("Exception while stringifying") } }`
+  Near here ---------------------------------------------------------^
+
+-- End --