The maturity of visual programming

by Rémi Dehouck | Sep 29, 2015 | Tech | AI  Behavior Trees  Visual Programming 

The maturity of visual programming

Visual programming has come a long way since the first attempts in the 1970s, but it still has an aura of unfulfilled promises. In this article we will have a look at today’s main types of visual programming languages, and where they can be useful and why.

In the above image are pictured, clockwise from top left: Scratch, CryEngine Flow Graph, craft ai, Kodu

Visual programming languages

Box and Arrows on a whiteboard

When programmers explain to someone what a program should do, they often end up using a whiteboard and a graphical representation of a control flow, with boxes and arrows. If this kind of representation is so convenient, why couldn’t we write programs with them? Visual programming languages (VPLs) let users create programs through the manipulation of graphical elements, as opposed to the text edition of source code.

The idea is very tempting, and in some areas VPLs have already made a name for themselves. In particular one of the first fields where they met success is teaching, because they can help users to learn how to make a computer program.

Make programming more accessible

The objective of many VPLs is to make programming more accessible, in particular to reduce the difficulties that beginners face when they start programming.

The Scratch Programming Language and Environment

Scratch syntax examples. Source: Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010). The Scratch Programming Language and Environment.

One major point where they help is syntax discovery. Even when only using valid words, only an extremely small portion of what can be written in a text file is valid source code for a given language, and an even smaller portion makes up a program that produces any output. In VPLs, basic programming elements are generally represented as blocks, with visual cues about how they can be used and linked with each other. So to create programs, the user can assemble blocks like Legos or electronic components, by arranging them in a way that “fits”. Therefore invalid expressions may simply be impossible to assemble. Furthermore blocks are generally presented from a central catalog of existing ones, so the user does not need to read documentation or guess which functions are available. These two points practically remove the main problems beginners face when learning a programming language’s syntax, allowing them to focus on the logic of their program.

The development environment of VPLs is often coupled with a simplified (or specialized) execution environment, so users can quickly and easily run their program and see results. This is not necessarily directly related to the VPL itself, but it is an important factor in making it easier for beginners to get their first program running without having to worry about technicalities.

These points are not exclusive to VPLs. More abstract or simplified programming languages and IDEs have used similar ideas to make syntax discovery easier. But it is an area where the specificities of VPLs make them shine.

Another point is that visual programming can make it easier to see the big picture. The program is defined by its shape, so beginners and advanced users alike can get an idea of what the program does at a glance. Shapes and colors also take advantage of the user’s visual abilities to a greater extent than code indentation and coloring ever could, to make the source more readable. This helps describe the program more easily: You can present it to others and explain its main points, even to people who are not familiar with the VPL that was used, by showing the graphical source code directly instead of drawing an abstract diagram; what you show is the source directly. This also help maintainability: when you look at VPL code that is not familiar to you, you can more easily see what it does, and how complex elements come together to make up the program. A block structure also means that it is often easy to rearrange the graphical code, as opposed to the tedious refactoring of text-based source code.

Visual, but still programming

Beyond some specific fields however, so far VPLs are nowhere near the popularity of classical programming languages.

One reason why they are sometimes perceived as a disappointment is that unlike hand-drawn “boxes and arrows”, they still require a precise, unambiguous definition of the control flow. In short, visual programming is still programming. Even precise UML-like diagrams describing processes often rely on context and assumptions, and cannot be interpreted directly to make a program.

And even though VPLs can make learning easier and reduce syntax difficulties, every type of programming requires users to become familiar with general as well as language-specific programming concepts (for example the concept of variables, and the generalities of imperative programming). So to be able to write a program with a VPL, you still need to think like a programmer.

No silver bullet

Other factors can explain the bad reputation that VPLs have with many developers.

When we look at professional developers who write programs every day, visual programming may not seem to bring much to the table. When the user is familiar with a programming language’s syntax and needs to work with many heterogeneous components or libraries, the advantages of VPLs for syntax discovery are not very interesting. Then in terms of representation of expressions, text is both very compact and open: Many possible words can fit in the space that a VPL’s “block” would take while still being readable and memorizable, and you can easily refer to external functions and extend the language by adding words where in VPLs there are only so many easily recognizable shapes and colors that a user can remember. Even though some VPLs combine shapes and text, their visual representations generally cannot compete with the information density of text. So even though the advantages of VPLs in terms of code readability still hold with developers, their advantages for accessibility are not a selling point.

Also, VPL development environments are sometimes specialized, applied to a reduced domain (for example game design). This way programs can be executed directly in an integrated execution environments, and the commonly used logical blocks are not so numerous that syntax discovery becomes difficult. But these points are hard to achieve with a general-purpose VPL.

Another factor is that even some “serious” VPLs have had technical shortcomings, in particular in their speed of execution.

Types of VPLs

Now that we looked at the general pros and cons of VPLs, here is a tentative categorization of the ones that seem to be used the most at the moment. We will try to identify their strong and weak points in terms of expressivity and ease of use.

We will provide examples of implementations for each category, but these are not exhaustive lists, and the categorization is of course debatable.

Scratch & friends

The Scratch Programming Language and Environment

Scratch and its derivatives are possibly what most people think of when they think of VPLs right now. Its grammar is actually essentially the grammar of a classical imperative programming language, but graphical cues help make it apparent how to combine basic elements like variables, conditions or flow controllers. Inside blocks, text is used to describe blocks, and text fields are used to define variable names and values. It is possible to extend the language with customized blocks, which can also themselves be defined with Scratch. Beyond the visual grammar, the original Scratch itself includes an execution environment with a graphical rendering to easily create basic animations or games. There are Scratch “users” that directly use Scratch’s format like one would a scripting language, as well as derivatives that follow the same principle.

Compared to other VPLs, this format is very close to source code for an imperative language, but uses visual blocks in a way that helps syntax discovery and prevents mistakes by making it obvious how to construct a working instruction and how elements relate to each other. Actually the core of the visual language can be seen as merely a syntactic guide for a language that resembles C or Java.

Also, it is very colorful, and Scratch’s execution environment is endearing and more likely to interest someone who would be put off by a wall of text.

Examples

And many, many others…

Flowcharts-inspired

Flowgorithm

Stemming directly from the idea of using “boxes and arrows” to describe programs, these VPLs use a visual representation close to that of flowcharts to describe the main control flow. They represent the directed sequence of execution between blocks, with the flow passing through a block on to the next, often with “forks” that use the result/output of a block to choose which block to execute next.

This focus on execution and simple visual grammar means that the format is simple to understand, so syntax discovery is not a problem. But the logical constructs that can be created directly through the VPL are limited, much depends on what is inside the blocks, which is often a given and cannot be altered from the graphical interface.

Some formats similar to flowcharts have an extended grammar that gives the ability to write more complex instructions directly from the graphical interface. In this article we will look at some of them: data flow programming, finite-state machines and behavior trees.

Examples

And many, many others…

Data flow programming

CryEngine Flow Graph

This format is one of the most used VPLs for professional applications, aimed at designers rather than end users or programmation beginners.

With data flow programming, blocks represent functions, and they are linked with each other along the flow of data, along input and output. By creating a link between one block’s output marker and another block’s input marker, you define the flow of execution of the program through the flow of data.

So the visual grammar is quite simple, you mostly choose which blocks to use and how to link them. This means that much of what defines the program is actually inside the blocks, which are usually written with a classical programming language.

The format may be simple, but it requires an understanding of programming notions around data flow, and it requires existing blocks. So its target is often users with a technical thought process, with an emphasis on showing the main structure of the program and the flow of data around blocks whose behavior is a given. It is adapted to applications where high-level logical components are often the same.

Examples

And many, many others…

Finite-state machines (FSM)

Unity3d Mecanim

FSMs define states, and transitions between states that are triggered by conditions. When the state changes, instructions are triggered. When FSMs are represented graphically, blocks represent states and links between them represent transitions. The user designs the flow of execution as with flowcharts by linking states, and by defining the triggers of state changes. So the logic of the choice among the possible next states is controlled more directly through transitions and their conditions, instead of being inside the blocks.This difference gives the user the ability to create more complex instructions while still using the encapsulated behavior of blocks.

However this requires the user to understand and manipulate state change conditions, generally with text expressions. So compared to data flow programming and flowcharts, this format is more complex to understand and with more text, but more expressive. You still get a “big picture” view of the structure of the program and a very simple visual grammar, but the creation of programs requires a better technical understanding.

Examples

Behavior trees (BT)

craft ai

Similar in spirit to FSMs, behavior trees have seen a growing interest lately. BTs are the format behind the VPL at the core of craft ai.

In BTs, the visual grammar is made of blocks and links between blocks that define a flow of execution. However unlike formats directly inspired from flowcharts, blocks are evaluated and return a status (success or failure) back to the previous block, along a tree structure of parents and children. So a block can have a behavior that depends on that of its children, and control the flow of execution accordingly. This gives the ability to define the logic of the program from the graphical interface without adding more to the visual grammar nor editing text, by using blocks dedicated to control flow. For example the “sequence” block evaluates its children in order and stops if one of them fails.

So compared to FSMs, the edition of the program from the graphical interface is more direct and more can be done just by rearranging links between blocks. The program is also more directly readable because less of the logic is in text fields. The simplicity of the visual grammar makes the discovery of the basic syntax very easy, but BTs require the user to understand how trees are executed, which is similar to understanding the concept of stack of execution in classical programming. The user also needs to become familiar with the behavior of control blocks to be able to design programs.

Therefore BTs have a steeper learning curve than most flowcharts-inspired formats, but give more control to the users form the graphical side.They are a compromise between a format with a very simple visual grammar but limited expressiveness from the interface like flowcharts, and a format that has the complexities of classical programming but gives more low-level control like Scratch.

The reason why we use BTs in craft ai is that we think that this compromise between expressivity and ease of use is at the right level for modern applications where the positive points of VPLs bring real value in a production environment.

Examples

Event-based rules

kodu

This is the simplest type of VPL we will see here. The user defines rules like “if this happens, do that”. A rule is triggered whenever a condition is met, and launches specific instructions. The visual elements are often quite basic, mainly blocks that can be used either on the condition or instruction side to be combined. The visual aspect helps discover what can be done, and recognize existing rules at a glance.

The advantage and disadvantage of this format are the same: it is simple. The language is so reduced that you could argue that this is not programming anymore, but this makes it easily understood even by users who have never developed a program. Generally this type of representation is aimed at end users so they can customize their applications with personalized rules. So the programs that can be written this way are quite limited, but they are a way to let end users take control.

Examples

Coming of age

Although we have not yet seen a generalized professional use of VPLs, there are areas where they gained ground, to a point where you could now say that they are mature. In these areas they are efficient tools adapted to their purpose, because their advantages over classical programming make them truly useful.

Teachers use VPLs as a more accessible way to learn the basics of programming. Scratch and its derivatives seem to be used widely, but other flowcharts-inspired formats have been used as well, for example in educational games.

In some areas related to media, designers seem to use VPLs professionally, especially those that let them structure programs around often-reused blocks in a readable format that helps them manage complexity. For example animation engines often use FSM-like representations; 2D, 3D or music composition software use data flow programming VPLs; designers of non-playing characters in games use various types of VPLs.

In other areas, VPLs are more directly aimed at end users, in particular to customize applications or create rules. Their format is often event-based rules for its ease of use, but we see more and more attempts at using more expressive formats. For example this is the case in Internet of Things dashboards or the customization of devices.

In applications aimed at ‘makers’, VPLs are also used more directly for programming, for example applications for robotics or game-making use formats similar to Scratch because they are more accessible than classical programming languages.

So VPLs are being used more and more in a “serious” setting, and here at craft ai we talk to people every day who see real value in using a VPL for their applications, either for its accessibility or because it helps them manage the complexity of their system in a maintainable way.

We believe that this trend is here to stay!

Share