How Exactly Does a Script Language Work?

Perhaps you have already program in Python, Ruby or JavaScript. You know how to use the language. But have you ever wondered how they work fundamentally?

That is what I intend to show you, using the Lua programming language as an example? Why Lua and not a better known language such a Python?

Python has more complexity which means the fundamental operation of the language gets obscured by complexity. It is kind of like how you learn about how a computer works at university. You don’t start by looking at a modern PC. It is such a complex piece of hardware. Instead you look at a micro controller. You can do that because a micro controller like a PC, consists of all the same fundamental components such as a microprocessor, memory, IO, etc.

Lua is similar in that it has all the same basic building blocks as other script languages, and unlike Python and Ruby it exposes these pieces clearly. For instance Lua does not have any keywords for defining classes, inheritance or methods, but it gives you all the building blocks you need to build an object oriented system yourself.

That is what I am going to do here. I am going to build an OOP system with you from scratch, to demonstrate how inheritance, instantiation, polymorphism and duck typing works.

Lua Quick Intro

But before I show you that, let me just give you a quick introduction to the language itself, how you define variables, functions and do a for loop.

Define two local variables. A number and a string

Globally defined numbers array

A hash table

Change a member

Define a struct

Control structure

Define a function

Standard for loop

The .. allow us to concatenate strings and interpolate non-string values into a string.

Hash Table Magic

The beauty of Lua is that they managed to do almost anything by using just one data structure, the hash table. You might know it as a dictionary. Many other languages prefer to say dictionary as that does not say how the data structure is implemented but rather what it is for.

It is a bit like how the whole world is just made up atoms. The only computer science equivalent I can think of is LISP where everything is a list or Tcl where everything is a string.

You want an array in Lua, just use a hash table with integer indices as keys. Want a C style struct, just use hash table, where the keys are strings with the same names as the fields. Want a class or an object, use a dictionary! Well we are getting ahead of ourselves. Lets look at some code examples first.

However as long as string keys don’t contain spaces and begin with a letter, we can write it as:

We can change the height later by using common hash table syntax for looking up a value by key:

However as long as the string doesn’t contain spaces and begins with a letter, we can write the equivalent form:

So now we got something that looks like member variables on an object in an OOP language. However we are not quite there yet, let is make a little detour and talk about functions.

Functions as Objects

Functions are first class objects in Lua. That used to be more cool, now I think it is highly unusual for languages to not make functions first class objects. What this means is that you can pass around and store functions the same way as you pass around numbers and strings.

This is a definition of a function, squaring a number. However as with many things in Lua, it is simply syntactic sugar. Remember this is a dynamic language and not a statically compiled language. That means a function definition is just like regular code which gets run when the interpreter gets to that line in the source code.

That means that the line above is really short for code which creates a function object and stores it in the variable square.

Turning Functions into Methods

Now lets look at how this ability lets us build an OOP system. Since functions are objects, we can store them on keys in hash tables.

Or as we have just learned we can shorthand this using the fact that strings without spaces can use some syntax sugar.

Except we are not done yet. We can write this much the same way as the original syntax I showed you for defining a function.

Since the rectangle object now has a function object on the area key we can call the function like this:

You notice there is an annoying repetition here. We got a function object stored on area which takes as an argument the very same rectangle argument. But Lua has more syntax sugar up its sleve to remove this duplication.

These two forms are equivalent. The nice things about the : operator is that is has another naturally related usage.

You see by using : we create an automatic first argument to the function named self. You don't have to do this, but it makes it more natural to write code that looks object oriented. It looks more like you are writing a method than a function. However the nice thing about Lua's approach is that there is no fundamental difference between methods and functions apart from a light sprinkle of syntax sugar.

Duck Typing

We don’t have full OOP programming yet, but with just what we got thus far, we can achieve duck typing. We can store different function objects on the same key on different objects.

E.g. we can define an area function for many different shapes.

We can then utilize duck typing in other functions to calculate the total area of multiple geometric objects without having to know exactly what they are. We only need to know that they have a function object stored on the area key.

Thus far, the solution works but is rather clumsy. For every object we create we need to make sure it has the appropriate function object stored on the appropriate key. What we really want is a common mold to stamp out identical objects. Objects which have the same set of functions attached to it. What we need are classes.

Creating Inheritance With Meta tables

While Lua has no class concept, we can easily create it ourselves, because of one significant detail which is different from regular hash tables and Lua hash tables.

Lua tables, may have meta tables. A meta table is just like a regular table. The difference is really just in its usage.

If you try to lookup a key on a table and it doesn’t exist, then the table will continue searching for that key in the table which has been assigned as its meta table. You can set it up so that it continues searching up a long chain of meta tables until it finds the key and returns the corresponding value.

The specific way this works is a little bit more involved in Lua, so the code I will show you now is not valid, Lua code, we will have to modify it a bit more. This is just to show the principle.

Above we essentially define a class, which is why I chose to capitalize the name. But really this is just a regular table which will be used as another tables meta table

We can no create an instance of a rectangle, we call it r

Now if we call area() it won't exist on r but Lua will find it on the meta table Rectangle and pass r as the first argument which gets named self.

We can make it look nicer to create instance by defining and instance creating function like this:

Now we can create and use rectangles easily:

The __index Key

Okay so we got almost working Lua code. I left out a detail, which I need to explain. Lua does in fact not look directly for the missing key in its meta table. It takes a detour through the __index key. Lua tables have several keys with special meaning. They are all prefixed with __ to not mix them up with your regular keys.

Let me explain the motivation for having these keys in the first place.

What happens if we do this:

We’ll get an error because there is no defined behavior for adding two hash tables. However we can define a behavior for this using Lua special keys. When Lua encounters operators such as +, - and ==, it will lookup function objects on the special keys and run those functions. These are some of the special keys

  • __add For the + operator
  • __eq For == operator
  • __index Function on this key gets called whenever a key is not found.

So Lua does in fact not go to the meta table immediately to look for an unknown key.

We could add e.g. something like this to the rectangle object:

What Lua does, is that it goes to the meta table when special keys like __add and __index are not found.

So lets experiment with that, making a function for adding rectangles which will return a rectangle encompassing both rectangles added together.

Unlike area(), we can now use the + operator defined on the meta table.

However we don’t need to make a big change to be able to use area().

Implementing the __index function to make a table lookup like this, is so common, that Lua allows you the shorthand of simply assigning a table to __index, to perform a lookup. We can then implement real inheritance for our Rectangle class like this:

self in this case will be the same as Rectangle.

Inheritance

Now we got all the pieces needed for object oriented programming, in the style of Python and Ruby. We got classes, duck typing and polymorphism. What I have not showed you yet is inheritance although you’ve probably guessed how to do that by now.

Say I want to create a subclass called SolidRectangle which has a thickness, and mass density so I can calculate its mass. We just need to make sure the meta table of our new class is the same as the table representing the super class Rectangle.

Now we can create an instantiation function and methods in exactly the same was as with our Rectangle "class".

Comparison with Python

So now you’ve seen how you build an OOP system in Lua. Lets look at how this is different from say Python. Let’s start by creating a similar class.

I do this in a Python REPL. As I’ve written about earlier I am not a fan of the built in Python REPL. So I advice you to get either PtPython or bpython. I am using PtPyhon in this examples.

Here is a normal interaction on the Python REPL creating a rectangle object and using the area() method.

However we can peek under the hood and see that Python has many similarities with Lua. I can access member variables as keys in a dictionary:

However unlike Lua, Python will not default to looking at its class, for entries not on the object, such as the area() method:

We have to go to the class itself.

To prove that this is in fact the area() function, we can try to fetch the function object and call it with the rectangle object.

Comparison with Objective-C

While not strictly a script language, Objective-C is a very dynamic language with a lot of similarities to Python and Ruby. Most closely it resembles Smalltalk, but few people have heard about that today. With Objective-C 2, things have changed a bit, so I am comparing with the old Objective-C here. A rectangle object in Objective-C would essentially look like this in memory

So each object is basically a C style struct pointing to another struct of type Class which contains information about its class.

A class essentially looks like this. So just like Lua has tables pointing to tables, we got class objects pointing to class objects.

The isa in the class object unlike for objects, points to a meta class. The super class is a separate entry.

In Objective-C if I make a method call like this:

That is essentially syntax sugar for the C function call:

Objective-C will use the isa pointer to find the class object and then go through the linked list methodLists to find a function pointer to the area method. In practice it will look in the cache first to see if area if method which has recently been used. This is to speed up lookup times in right loops calling the same method multiple times.

Just as with a script language, classes are objects created at runtime. They are not like in C++ defined at compile time. Instead they are registered with the function:

An Objective-C programmer of course does not see this. They just write a class definition and the compiler will turn this into code that at runtime will register the class object. This is somewhat similar to how a Python developer doesn’t have to chain up tables to meta tables himself, the way a Lua programmer has to.

Comparison with Julia

My favorite script language Julia works a bit different because it uses multiple dispatch, rather than single dispatch like most OOP script language. I’ll try to explain this in Lua terms. Say I got two objects representing shapes such as a rectangle and a circle, and want to calculate their intersection. The algorithm used will depend on both types. In Julia this works because it always considers the types of all arguments.

Say we got this line:

You can think of this a syntax sugar for the following code:

So in Julia we essentially do two hash table lookups for functions, not just one like in Lua or Python.

  1. There is a global dictionary (hash table) over all functions. Where we locate the intersect function.
  2. Each function comes in multiple flavors or specialized functions depending on the number and type of arguments it takes. We use a tuple {typeof(circle), typeof(rect)} of all the argument types to look up the function taking exactly those arguments.

I used a hash table here, since Lua doesn’t have tuple types. It doesn’t really have a typeof function either, so you would have to use meta tables instead. So the lookup key would be more like:

And to be more precise it is not really a hash table lookup but a search through a list, looking for the best matching entry. The types of the arguments don’t need an exact match. E.g. an argument of type Int64 would match a function with an Integer argument.

So Julia code isn’t really about defining relationships between tables and meta tables. Instead each function definition causes a specialized function with those particular argument to be placed in the right entry over generic functions and specialized functions.

Geek dad, living in Oslo, Norway with passion for UX, Julia programming, science, teaching, reading and writing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store