Data Integration Scenario | |
Data | City of Vancouver Open Data (see table below) |
Overall Goal | Use your FME skills to integrate data from several sources to answer a question. |
Demonstrates | Content Transformation, Structural Transformation, Schema Editing |
Start Workspace | C:\FMEData2018\Workspaces\DataIntegration\ScenarioBegin.fmw |
End Workspace | Instructors can request access to the final workspaces here. |
Introduction
In the previous exercises you have learned how to translate and transform data using FME. Now let's put that knowledge to use! As we covered in the Lecture section, data integration is a key requirement for modern organizations. In this exercise you will develop a workspace that solves a data integration problem in a city government scenario.
You will:
- inspect your data
- combine your datasets
- query your data to answer a question or solve a problem
- answer lab questions about the process of integrating data
This exercise will provide an example of integrating three datasets into a central database. After that you can follow similar steps to choose one additional dataset to answer another question. If you have trouble carrying out the steps you can refer to the Additional Procedures section, FME Workbench Help (also available online in our Documentation) or the community Knowledge Center.
Please continue to refer to the Lab Questions throughout, as some of the questions refer to the scenario.
Exercise Goals
- Integrate data into a single database using multiple feature types on a single writer
- Use transformers to create the desired schema in the destination database.
- Merge data using an Overlayer transformer
- Use ChartGenerator to make analyze integrated data
- Select another dataset and add it to your workspace to answer a question.
Inspect Data
To start off, inspect the data you have been given. There is a wide variety of topics and formats available. Following the instructions from Exercise 3, open some of these files in the FME Data Inspector to see what attributes they have.
For the walkthrough we will be using the following datasets:
- Bike Paths
- Neighborhoods
- Public Art
More data is available from the City of Vancouver Open Data catalogue.
Topic | Format | Location (FMEData2018\Data...) | |
---|---|---|---|
Addresses | Esri Geodatabase | Addresses\Addresses.gdb | |
Building Footprints | SpatialLite Database or AutoCAD DWG | Engineering\BuildingFootprints\building_footprints.sl3 or C:\FMEData2018\Data\Parcels\BuildingFootprints.dwg | |
Bike Paths | Esri Shapefile | Transportation\Cycling\BikePaths*.shp | |
Business Licenses | Excel Spreadsheet | Planning\BusinessLicenses.xlsx | |
City Grid | Geography Markup Language | Boundaries\CityGrid.gml | |
City-owned Properties | Esri Shapefile | Parcels\CityProperties\CityProperties.shp | |
Coastal Zones | Geography Markup Language | Boundaries\CoastalZones\CoastalZones.gml | |
Community Resources | Esri Geodatabase | CommunityMapping\CommunityMap.gdb | |
Crime | Comma Separated Values | Emergency\Crime.csv | |
Drinking Fountains | Comma-separated Values | Engineering\DrinkingFountains.csv | |
Election Results | Excel Spreadsheet | Elections\ElectionResults.xls | |
Election Voting Districts | Geography Markup Language | Elections\ElectionVoting.gml | |
Elevation | Digital Elevation Model | ElevationModel\DEM-Full.dem | |
Firehalls with Zones | Geography Markup Language | Emergency\FireHallsWithZones.gml | |
Forward Sortation Areas (for postcode) | Esri Shapefile | Addresses\ForwardSortationAreas.shp | |
Land Boundary | Esri Shapefile | Boundaries\LandBoundary\VancouverLandBoundary.shp | |
Neighborhoods | Google Keyhole Markup Language | Boundaries\VancouverNeighborhoods.kml | |
Orthophotos (scale corrected aerial photographs) | TIFF | Orthophotos*.tif | |
Parcels | AutoCAD DWG | Parcels\Parcels.dwg | |
Parking Meters | MapInfo TAB | Transportation\Parking\Meters.tab | |
Parking Tickets | Comma-separated Values | Transportation\Parking\ParkingTickets.csv | |
Parks | MapInfo TAB | Parks\Parks.tab | |
Planning Restrictions (Business Improvement Areas, Historic Areas, Noise Control Areas, and View Cones) | OCG Geopackage | Planning\PlanningRestrictions.gpkg | |
Point Cloud (LiDAR) | LAS | PointClouds\CoV*.laz | |
Public Art | Excel Spreadsheet | Culture\PublicArt.xlsx | |
Public Trees | Comma-separated Values | Parks\Parks.tab | |
Roads | AutoCAD DWG or Microstation DGN | Transportation\CompleteRoads.dwg or RoadsDGN.dgn | |
Street Lighting | AutoCAD DWG and Microstation DGN | Engineering\StreetLighting.dwg and StreetLightingDGN.dgn | |
Traffic Signals | AutoCAD DWG and Microstation DGN | Engineering\TrafficSignals.dwg and TrafficSignalsDGN.dgn | |
Zoning | MapInfo TAB | Zoning\Zones.tab | |
Zoning Descriptions | Adobe PDF | Zoning\ZoneDescriptions.pdf |