Additional Procedures
While many datasets will work well using FME defaults, some formats require additional steps to control how they are read or analyzed. In this section we will walk through a few procedures that can help when working with some of the open Vancouver data. You can skip this section and go straight into your analysis if you want, referring back to this section if you run into any roadblocks.
Reading (X, Y) Coordinates as Points
If you add any tabular readers (.csv, .xlsx, etc.), you might notice they are imported without geometry. For example, adding the DrinkingFountains.csv file with default settings just adds features with string attributes. In order to be read as point geometry, FME must be told which columns contain the x and y coordinates of the data. There are two ways to do this:
When creating the reader, click Parameters and look at the Attributes table at the bottom of the window. The x_coord and y_coord columns are just being read as strings, not coordinates. To change this, change the Attribute Definition button from Automatic to Manual. Then use the dropdown in the Type column of the Attributes table to change x_coord to x_coordinate and y_coord to y_coordinate.
You could instead add the reader with default parameters and then attach a VertexCreator transformer to the reader. This creates point geometry from x and y coordinate attributes. For DrinkingFountains.csv you would open the VertexCreator parameters and for X Value select x_coord and for Y Value select y_coord.
Note: even if you have point geometry, some file formats like CSV do not store coordinate system information. You can refer to the City of Vancouver metadata to look up the coordinate system, or chances are it is stored as either latitude and longitude (LL84), UTM Zone 10N (UTM83-10), or BC Albers (BCALB-83 or NAD83.BC/Albers). Once you have identified the correct system you can set it under Navigator > Reader > Coordinate System, or under Coordinate System in the dialog box when you are adding a Reader, or by using the CoordinateSystemSetter transformer. Refer to Exercise 9 for more information.
Individual Versus Merged Feature Types
Some datasets contain more than one feature type. For example, if you add CommunityMap.gdb with the default settings you will be presented with a Feature Types To Read dialog. Here you can select which layers from the geodatabase you wish to read in as feature types.
When you have data like this you can choose to read the data as individual or merged feature types. This option appears only in the Add Reader dialog when you initially add a reader. It cannot be changed after the fact without removing feature types.
If you choose individual feature types each layer (or level, table, etc. depending on the format) will be added as a separate feature type. If you choose merged feature types the reader will combine all the layers into one feature type named <All>. This can be useful if you don't care about the distinction between layers and just want all the features grouped together.
Reading Attributes in Special Formats
Some formats do not store attributes in a straight-forward way. One example is CAD data. Because this format is designed for creating digital technical drawings, the features are not by default stored with attributes as in a GIS. However, they can store attributes as "extended entity data."
This has implications for reading CAD data. For example, if you add an Autodesk AutoCAD DWG/DXF reader and choose C:\FMEData2018\Data\Transportation\CompleteRoads.dwg, by default you will only read geometry. However, if you select Parameters > Group Entities By > Attribute Schema in the reader dialog, FME will read the geometry and associate it with each feature's extended entity data. This allows you to use these attributes in your analysis. You can see this in action if you inspect the reader with the default (Parameters > Group Entities By > Layer Name) and compare it to a reader with Parameters > Group Entities By > Attribute Schema. The former will have an empty column <no schema> in Table View and the latter will have a series of attributes.
There are other complicating factors in reading AutoCAD data. You can check out our Getting Started with AutoCAD series if you want to learn more.
Joining Features
By default, bringing two data streams together in FME does not merge data. Instead, data is combined or accumulated. You may know this as a union of data.
In order to merge data you have to carry out a join. There are two major kinds of joins in FME: key-based and spatially-based joins. Key-based joins connect two data sets based on a shared unique key, e.g. an ID number, a census tract ID, or a street address. Spatially-based joins connect two data sets based on spatial location, e.g. giving the attributes of census tract polygons to point locations of businesses that are contained by the tracts.
There are many transformers you can use in FME to carry out these operations. Here we will just show two of the most common join operations. For more information, you can refer to this joiner transformer article and flowchart.
Key-based joins with the FeatureJoiner
The FeatureJoiner transformer uses SQL join terminology to combine two data sets based on a shared unique key. The default Join Mode is an Inner join. For an inner join the data will be combined based on a common identifier, or attribute, from the left table and the right table.
Spatially-based joins with Overlayer transformers
FME comes with many Overlayer transformers:
- AreaOnAreaOverlayer
- LineOnAreaOverlayer
- LineOnLineOverlayer
- PointOnAreaOverlayer
- PointOnLineOverlayer
- PointOnPointOverlayer
- SurfaceOnSurfaceOverlayer
- VectorOnRasterOverlayer
These transformers overlay two data streams based on geometry. By default they simply count the number of overlaps in each stream and add a new attribute to both layers called _overlaps. However, if you change the settings under Attribute Accumulation > Merge Attributes you can combine attributes to join the data. For example, if you use VancouverNeighborhoods.kml as the polygon and DrinkingFountains.csv as the points (making sure they share the same coordinate system), and you check Merge Attributes, the polygons will get the attributes of the first point they overlap and the points will get the attributes of the neighborhood polygon they overlap. With an operation like this you can integrate data between layers.
Creating Charts and Reports
One easy way to show the results of your data integration is to create a visualization. FME has several built-in transformers that can help here. ChartGenerator can create charts. Simply add a ChartGenerator, pick your chart type, and select an attribute to analyze.
You can also easily create HTML reports using the HTMLReportGenerator. You can add elements to your page by clicking in the Page Contents table and adding a new piece of content from the dropdown menu. Each content type has its own options for you to fill in and can be dynamically created by using attributes. For example, if you have joined some data sets together using an Overlayer transformer and then calculated some average values using StatisticsCalculator, you can create a table of those values here. You can use Group By to split up your analysis by groups. Finally, don't forget to write the content of the HTMLReportGenerator out to an HTML writer. Once you do that you can open the resulting HTML file in a web browser.
Creating Raster Maps using MapnikRasterizer
Check out this blog post for FME cartography tips or this article on the Mapnik Rasterizer.
Creating Labels using LabelPointReplacer
We covered creating labels in Exercise 7. Refer to that exercise if you run into problems creating labels for your map. There are a few key points to remember:
- You can create labels combining multiple attributes using the StringConcatenator transformer.
- The label height is in the units of your data, so it depends on the coordinate system. You can check the linear units of your data by opening it in FME Data Inspector, selecting a feature, and looking in the Feature Information window. One of the attributes displayed is Coordinate System. Click on the link there (e.g. UTM83-10) and a Coordinate Systems Properties window with more information about the system opens. Here you can find the UNIT information, e.g. for UTM83-10, UNIT: METER.
Rejected Features?
You might encounter this error message in your log:
ERROR |{TRANSFORMER NAME}_<Rejected>: Termination Message: '{TRANSFORMER NAME} output a <Rejected> feature. To continue translation when features are rejected, change 'Workspace Parameters' > Translation > 'Rejected Feature Handling' to 'Continue Translation''
This means that one of your transformers rejected a feature (the message above was edited to use a placeholder {TRANSFORMER NAME}). Rejected features can sometimes mean something is wrong with your workspace, especially if a transformer is outputting only rejected features. However, in some cases rejected features are to be expected. For this reason FME gives you the option to decide how to handle rejected features. See this section of our training manual for more information.