Transformation Optimization

As with Readers and Writers, you can’t make performance improvements unless you can accurately interpret a log file and measure a baseline performance for your translations.

There are two aspects to transformation performance: the time taken and the amount of memory used.

Transformation Length

Assessing the time taken in transformation requires a two-step process.

First disable Writers and run the translation, taking a note of the elasped time:

Then disable the transformers too and run the workspace again, to calculate the time taken to read the data only. The difference in elapsed time between reading the data and reading/transforming the data, is the elapsed transformation type.

It's important not to add an Inspector or Logger transformer to the end to see what is happening to the output. This will only slow the translation down and give you a false measure. You must also be sure to disable the actual Writer, and not just the feature types or connections to them.

The only Writer that is useful in this scenario is the Null format Writer. This causes a Writer to be present, but it does nothing except to log features and then discard them. The benefit is improved logging of feature counts, but without any data having to be written.

With the above workspace, I now know it took 5.4 seconds to read the data, and the whole process (excluding writing) took 28.2 seconds, so I can infer that the transformation part takes 22.8 seconds.

With the Reader and Writer figures as well, I now have a complete breakdown of how long each section of my workspace takes.

Transformation Memory Use

The time taken to carry out a translation is only one aspect of performance. Another important aspect is the amount of memory used during processing. You can't tell how much memory each individual transformer used, but you can see the maximum memory used during a translation by examining the very foot of the log file:

INFORM| Translation was SUCCESSFUL with 0 warning(s) (13597 feature(s) output)
INFORM| FME Session Duration: 28.3 seconds. (CPU: 27.3s user, 0.6s system)
INFORM| END - ProcessID: 28336, peak process memory usage: 178388 kb

For performance tuning, the idea is to reduce this number, as the more memory used the more chance there is of laborious disk caching taking place.