Translation and Transformation Statements

The main body of the FME log concerns the translation and transformation of data. This section often appears confusing to new users, until they understand how FME operates.

Firstly, there can be several parallel streams of processing in a workspace. If you do not explicitly set the order of these streams, the order of the log will be similarly indecisive.

Secondly, there is the issue of transformer type.

As you might know, some transformers operate on a feature-by-feature basis, while others work on a group of features at a time.

When a series of transformers are all feature-based, FME works by pushing features through on an individual basis, not as a group. For example, the first feature is read and then processed by each transformer in turn. Then the next feature is read and processed by each transformer, and so on.

This makes for a log file that is harder to understand because you don’t see one specific entry for each reader or transformer in turn. Nor do you see a message for each feature. Instead, a cumulative time is calculated and output at regular time-based intervals.

Feature Tables

Feature Tables are a new technology happening behind the scenes in FME. They are for vector or tabular data but resemble raster or point cloud data in how they operate. Instead of reading and processing one feature at a time, as before, data is handled more in "chunks", as a form of bulk processing.

This bulk processing makes interpreting the log window a little more difficult. So far only a handful of formats use this technology (CSV, FFS, MapInfo Extended, SQLite Non-Spatial); but expect this number to increase with future releases.

Key Messages

There are several key messages that may appear in a log that signify specific events in a translation.

Emptying factory pipeline

This message signifies that reading of data is at an end. Because of how FME processes data, this message may appear near the beginning of a translation or near the end. However, after this point, the source datasets are closed and only transformation and/or writing take place.

By checking the timestamp for this message, you can get an approximate value for how long the workspace took to read the source data.

Unexpected Input Remover

This message denotes where features read by a reader are tested against the available feature types (and merge filters) to see if they are allowed to pass into the workspace.

This message may indicate a performance issue as it highlights potentially unnecessary work, where features are being read but not used in the translation.

ResourceManager: Optimizing Memory Usage. Please wait...

This message alerts you to the fact that some reorganization of memory usage is going on. For example, maybe the memory management limits mentioned in the configuration part of the log have been reached, and FME is having to work around the issue.

If you see this message frequently, then it’s time to either redesign your translation or switch to a 64-bit setup. It is a key indicator that a lack of memory is affecting performance.