Custom Transformers and Loops

Loops are a way to carry out a process that repeats a section of transformers.

What is a Loop?

A loop is a programming structure that allows an action to be repeatedly executed.

Often this is used to carry out iteration; where a process repeats to gradually narrow down the process to a desired result. Usually a loop is linked to a condition; i.e. the action continues until a certain condition is met.

In FME, loops are only permitted inside a custom transformer.

Why Use a Loop?

As you know, FME processes one feature at a time. Therefore, when you create a loop, each feature is being sent round the loop individually.

However, the essence of a loop is that each iteration is slightly different. Why else would you want to repeat the same action again and again, if the result was going to be no different?

So, to be worthwhile, each iteration of the loop must either fetch new data (for example, it reads another entry from a list) or it repeats the same process but using the results of the previous loop.

NB: In the following screenshots, the processing section is a single transformer labelled "ProcessData", in order to give a generic view of how a loop is created. In reality, this process must be doing something to the data in order to be worthwhile carrying out at all.

First Officer Transformer says…

It's stating the obvious I know, but you only use a loop to repeat an action inside the custom transformer. One-off actions should take place outside of a loop. For example, here is my landing sequence: The "wheels down" part comes outside the circling loop, because I don't want to raise/lower the wheels every time I circled the airport. That would be very inefficient.

Setting up a Custom Transformer Loop

A loop in a custom transformer requires two components: the start and end point of the loop.

Loop Start Points

The start of the loop is identified by an Input port object. Although it can be the same input port as used for features to enter by, this does not have to be the case. For example here there is an input port for features to arrive into, and another one for the start of the loop:

This allows the loop point to be other than the beginning of the custom transformer.

By default, this second input port also appears on the transformer itself, like this:

If you don't require this, then you simply have to open the Input port’s parameters and ‘unpublish’ it:

Loop End Points

The end of a loop is identified by a Loop object. You can insert one by selecting it from the canvas context menu in a custom transformer:

When you place a loop object you are asked which Input object is to be looped to:

And then the loop is complete:

Of course, this example is an infinite loop. The action is repeated but there is no condition being tested to stop it. FME won't let an infinite loop run forever - it will recognize the problem and stop it - but we should set up something to force an ending.

Loop Conditions

There are two general types of condition we can test for. Firstly we can loop a set number of times. Secondly we can loop until a specific condition is met.

Here is a custom transformer that loops a set number of times:

Notice that we have an attribute that is a counter for the number of times we have looped, and an attribute that tells us the maximum number of loops to carry out. In each loop the counter attribute is incremented by 1. When LoopCounter < MaxLoops, then we loop back and process the data again. When LoopCounter = MaxLoops, then we exit the transformer.

Instead of a simple count, another method is to replace the counter with a measure of data quality.

Loops and Transformer Types

As you should already know, transformers that operate on one feature at a time are called Feature-Based, and transformers that operate on multiple features at a time are called Group-Based.

We can also call a loop "Feature-Based" because it only processes one feature at a time. Unfortunately, that means there are issues around using a group-based transformer inside a (feature-based) custom transformer loop.

If you attempt to create a loop inside an embedded custom transformer, when it includes a group-based FME transformer, then you will receive an error message. Group-based transformers are only permitted inside a loop in a linked custom transformer. There are technical reasons for this that we won't go into right now.

This is the error message you will get:

So, inside a linked custom transformer definition, you'll see a particular parameter (in the Navigator window) called Enable Blocked Looping:

When set to Yes then other parameters are exposed to set the number of iterations and an attribute that will hold that value. Notice how parallel processing is turned off (the parameters are removed) for custom transformers that are being looped, and the Insert Mode is automatically changed to "Linked Only".