How Exactly Does a Script Language Work?

Perhaps you have already program in Python, Ruby or JavaScript. You know how to use the language. But have you ever wondered how they work fundamentally?

That is what I intend to show you, using the Lua programming language as an example? Why Lua and not a better known language such a Python?

Python has more complexity which means the fundamental operation of the language gets obscured by complexity. It is kind of like how you learn about how a computer works at university. You don’t start by looking at a modern PC. It is such a complex piece of hardware. Instead you look at a micro controller. You can do that because a micro controller like a PC, consists of all the same fundamental components such as a microprocessor, memory, IO, etc.

Lua is similar in that it has all the same basic building blocks as other script languages, and unlike Python and Ruby it exposes these pieces clearly. For instance Lua does not have any keywords for defining classes, inheritance or methods, but it gives you all the building blocks you need to build an object oriented system yourself.

That is what I am going to do here. I am going to build an OOP system with you from scratch, to demonstrate how inheritance, instantiation, polymorphism and duck typing works.

Lua Quick Intro

But before I show you that, let me just give you a quick introduction to the language itself, how you define variables, functions and do a for loop.

Define two local variables. A number and a string

local number = 20 
local str = "Hello world"

Globally defined numbers array

numbers = {2, 4, 6, 8}
second_number = numbers[2]

A hash table

person = {name = "Erik", phone = 9364, company = "roxar"}

Change a member

person["phone"] = 543

Define a struct

person = {}
person.name = "Erik"
person.phone = 9364
person.company = "roxar"

Control structure

if person.name == "Erik" then
print("That is me")
else
print("I don't know who "..person.name.." is")
end

Define a function

function square(x)
return x*x
end

Standard for loop

for index, value in pairs(numbers) do
print("square of "..value.." is "..square(value))
end

The allow us to concatenate strings and interpolate non-string values into a string.

Hash Table Magic

The beauty of Lua is that they managed to do almost anything by using just one data structure, the hash table. You might know it as a dictionary. Many other languages prefer to say dictionary as that does not say how the data structure is implemented but rather what it is for.

It is a bit like how the whole world is just made up atoms. The only computer science equivalent I can think of is LISP where everything is a list or Tcl where everything is a string.

You want an array in Lua, just use a hash table with integer indices as keys. Want a C style struct, just use hash table, where the keys are strings with the same names as the fields. Want a class or an object, use a dictionary! Well we are getting ahead of ourselves. Lets look at some code examples first.

rectangle = {"height" = 10, "width" = 20 }

However as long as string keys don’t contain spaces and begin with a letter, we can write it as:

rectangle = {height = 10, width = 20 }

We can change the height later by using common hash table syntax for looking up a value by key:

rectangle["height"] = 150

However as long as the string doesn’t contain spaces and begins with a letter, we can write the equivalent form:

rectangle.height = 150

So now we got something that looks like member variables on an object in an OOP language. However we are not quite there yet, let is make a little detour and talk about functions.

Functions as Objects

Functions are first class objects in Lua. That used to be more cool, now I think it is highly unusual for languages to not make functions first class objects. What this means is that you can pass around and store functions the same way as you pass around numbers and strings.

function square(x)
return x*x
end

This is a definition of a function, squaring a number. However as with many things in Lua, it is simply syntactic sugar. Remember this is a dynamic language and not a statically compiled language. That means a function definition is just like regular code which gets run when the interpreter gets to that line in the source code.

That means that the line above is really short for code which creates a function object and stores it in the variable .

square = function(x)
return x*x
end

Turning Functions into Methods

Now lets look at how this ability lets us build an OOP system. Since functions are objects, we can store them on keys in hash tables.

local rectangle = {width = 10, height = 20}

rectangle["area"] = function(rect)
return rect.width * rect.height
end

Or as we have just learned we can shorthand this using the fact that strings without spaces can use some syntax sugar.

rectangle.area = function(rect)
return rect.width * rect.height
end

Except we are not done yet. We can write this much the same way as the original syntax I showed you for defining a function.

function rectangle.area(rect)
return rect.width * rect.height
end

Since the rectangle object now has a function object on the key we can call the function like this:

rectangle.area(rectangle)

You notice there is an annoying repetition here. We got a function object stored on which takes as an argument the very same rectangle argument. But Lua has more syntax sugar up its sleve to remove this duplication.

rectangle.area(rectangle)
rectangle:area()

These two forms are equivalent. The nice things about the operator is that is has another naturally related usage.

function rectangle:area()
return self.width * self.height
end

You see by using we create an automatic first argument to the function named . You don't have to do this, but it makes it more natural to write code that looks object oriented. It looks more like you are writing a method than a function. However the nice thing about Lua's approach is that there is no fundamental difference between methods and functions apart from a light sprinkle of syntax sugar.

Duck Typing

We don’t have full OOP programming yet, but with just what we got thus far, we can achieve duck typing. We can store different function objects on the same key on different objects.

E.g. we can define an function for many different shapes.

local circle = {radius = 5}

function circle:area()
return pi * self.radius * self.radius
end

We can then utilize duck typing in other functions to calculate the total area of multiple geometric objects without having to know exactly what they are. We only need to know that they have a function object stored on the key.

function total_area(shapes)
total = 0
for sh in shapes do
total += sh:area()
end
return total
end

total_area({circle, rectangle})

Thus far, the solution works but is rather clumsy. For every object we create we need to make sure it has the appropriate function object stored on the appropriate key. What we really want is a common mold to stamp out identical objects. Objects which have the same set of functions attached to it. What we need are classes.

Creating Inheritance With Meta tables

While Lua has no class concept, we can easily create it ourselves, because of one significant detail which is different from regular hash tables and Lua hash tables.

Lua tables, may have meta tables. A meta table is just like a regular table. The difference is really just in its usage.

If you try to lookup a key on a table and it doesn’t exist, then the table will continue searching for that key in the table which has been assigned as its meta table. You can set it up so that it continues searching up a long chain of meta tables until it finds the key and returns the corresponding value.

The specific way this works is a little bit more involved in Lua, so the code I will show you now is not valid, Lua code, we will have to modify it a bit more. This is just to show the principle.

Rectangle = {}
function Rectangle:area()
return self.width * self.height
end

Above we essentially define a class, which is why I chose to capitalize the name. But really this is just a regular table which will be used as another tables meta table

We can no create an instance of a rectangle, we call it

local r = {width = 10, height = 20}
setmetatable(r, Rectangle)

Now if we call it won't exist on but Lua will find it on the meta table and pass as the first argument which gets named .

r:area()

We can make it look nicer to create instance by defining and instance creating function like this:

function Rectangle:new(w, h)
local r = {width = w, height = h}
setmetatable(r, Rectangle)
return r
end

Now we can create and use rectangles easily:

local r = Rectangle:new(4, 5)
local s = Rectangle:new(3, 2)
local total = r:area() + s:area()

The __index Key

Okay so we got almost working Lua code. I left out a detail, which I need to explain. Lua does in fact not look directly for the missing key in its meta table. It takes a detour through the key. Lua tables have several keys with special meaning. They are all prefixed with to not mix them up with your regular keys.

Let me explain the motivation for having these keys in the first place.

local r = {width = 3, height = 2}
local s = {width = 5, height = 4}

What happens if we do this:

local t = r + s

We’ll get an error because there is no defined behavior for adding two hash tables. However we can define a behavior for this using Lua special keys. When Lua encounters operators such as , and , it will lookup function objects on the special keys and run those functions. These are some of the special keys

  • For the operator
  • For operator
  • Function on this key gets called whenever a key is not found.

So Lua does in fact not go to the meta table immediately to look for an unknown key.

We could add e.g. something like this to the rectangle object:

function r.__index(table, key)
if key == "typename"
return "Rectangle"
elseif key == "area"
return table.width * table.height
else
return nil
end
end

What Lua does, is that it goes to the meta table when special keys like and are not found.

So lets experiment with that, making a function for adding rectangles which will return a rectangle encompassing both rectangles added together.

local Rectangle = {}
function Rectangle.__add(a, b)
local result = {}
result.width = math.max(a.width , b.width)
result.height = math.max(a.height, b.height)
return result
end

Unlike , we can now use the operator defined on the meta table.

local r = Rectangle:new(4, 5)
local s = Rectangle:new(3, 2)
local big_rect = r + s

However we don’t need to make a big change to be able to use .

function Rectangle.__index(table, key)
return table[k]
end

Implementing the function to make a table lookup like this, is so common, that Lua allows you the shorthand of simply assigning a table to , to perform a lookup. We can then implement real inheritance for our class like this:

Rectangle = {}
Rectangle.__index = Rectangle

function Rectangle:new(w, h)
local r = {width = w, height = h}
setmetatable(r, self)
return r
end

in this case will be the same as .

Inheritance

Now we got all the pieces needed for object oriented programming, in the style of Python and Ruby. We got classes, duck typing and polymorphism. What I have not showed you yet is inheritance although you’ve probably guessed how to do that by now.

Say I want to create a subclass called which has a thickness, and mass density so I can calculate its mass. We just need to make sure the meta table of our new class is the same as the table representing the super class .

SolidRectangle {}
SolidRectangle.__index = SolidRectangle
setmetatable(SolidRectangle, Rectangle)

Now we can create an instantiation function and methods in exactly the same was as with our "class".

function SolidRectangle:new(w, h, d)
local r = {width = w, height = h, density = d}
setmetatable(r, self)
return r
end

function SolidRectangle:mass(thickness)
self:area() * self.thickness * self.density
end

Comparison with Python

So now you’ve seen how you build an OOP system in Lua. Lets look at how this is different from say Python. Let’s start by creating a similar class.

class Rectangle:   
def __init__(self, w, h):
self.width = w
self.height = h

def area(self):
return self.width * self.height

I do this in a Python REPL. As I’ve written about earlier I am not a fan of the built in Python REPL. So I advice you to get either PtPython or bpython. I am using PtPyhon in this examples.

Here is a normal interaction on the Python REPL creating a rectangle object and using the method.

>>> r = Rectangle(20, 30)
>>> r.width
20

>>> r.height
30

>>> r.area()
600

However we can peek under the hood and see that Python has many similarities with Lua. I can access member variables as keys in a dictionary:

>>> r.__dict__['width']
20

>>> r.__dict__['height']
30

However unlike Lua, Python will not default to looking at its class, for entries not on the object, such as the method:

>>> r.__dict__['area']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'area'
'area'

We have to go to the class itself.

>>> Rectangle.__dict__['area']
<function area at 0x1051bdc08>

To prove that this is in fact the function, we can try to fetch the function object and call it with the rectangle object.

>>> f = Rectangle.__dict__['area']

>>> f(r)
600

Comparison with Objective-C

While not strictly a script language, Objective-C is a very dynamic language with a lot of similarities to Python and Ruby. Most closely it resembles Smalltalk, but few people have heard about that today. With Objective-C 2, things have changed a bit, so I am comparing with the old Objective-C here. A rectangle object in Objective-C would essentially look like this in memory

struct objc_object {
Class isa;
int width;
int height;
};

So each object is basically a C style struct pointing to another struct of type which contains information about its class.

A class essentially looks like this. So just like Lua has tables pointing to tables, we got class objects pointing to class objects.

struct objc_class {
Class isa;

Class super_class;
const char *name;
long instance_size;
struct objc_ivar_list *ivars;
struct objc_method_list **methodLists;
struct objc_cache *cache;
};

The in the class object unlike for objects, points to a meta class. The super class is a separate entry.

In Objective-C if I make a method call like this:

[rectangle area];

That is essentially syntax sugar for the C function call:

objc_msgSend(rectangle, "area");

Objective-C will use the pointer to find the class object and then go through the linked list to find a function pointer to the method. In practice it will look in the first to see if if method which has recently been used. This is to speed up lookup times in right loops calling the same method multiple times.

Just as with a script language, classes are objects created at runtime. They are not like in C++ defined at compile time. Instead they are registered with the function:

void objc_addClass(Class myClass);

An Objective-C programmer of course does not see this. They just write a class definition and the compiler will turn this into code that at runtime will register the class object. This is somewhat similar to how a Python developer doesn’t have to chain up tables to meta tables himself, the way a Lua programmer has to.

Comparison with Julia

My favorite script language Julia works a bit different because it uses multiple dispatch, rather than single dispatch like most OOP script language. I’ll try to explain this in Lua terms. Say I got two objects representing shapes such as a rectangle and a circle, and want to calculate their intersection. The algorithm used will depend on both types. In Julia this works because it always considers the types of all arguments.

Say we got this line:

intersect(circle, rect)

You can think of this a syntax sugar for the following code:

specialized_functions = generic_functions["intersect"]
f = specialized_functions[{typeof(circle), typeof(rect)}]
f(circle, rect)

So in Julia we essentially do two hash table lookups for functions, not just one like in Lua or Python.

  1. There is a global dictionary (hash table) over all functions. Where we locate the function.
  2. Each function comes in multiple flavors or specialized functions depending on the number and type of arguments it takes. We use a tuple of all the argument types to look up the function taking exactly those arguments.

I used a hash table here, since Lua doesn’t have tuple types. It doesn’t really have a function either, so you would have to use meta tables instead. So the lookup key would be more like:

{getmetatable(circle), getmetatable(rect)}

And to be more precise it is not really a hash table lookup but a search through a list, looking for the best matching entry. The types of the arguments don’t need an exact match. E.g. an argument of type would match a function with an argument.

So Julia code isn’t really about defining relationships between tables and meta tables. Instead each function definition causes a specialized function with those particular argument to be placed in the right entry over generic functions and specialized functions.

Written by

Geek dad, living in Oslo, Norway with passion for UX, Julia programming, science, teaching, reading and writing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store