Web Based Datasets

There can be no doubt that there is a trend towards data being stored in the cloud, including spatial data. For that reason FME has comprehensive tools for reading datasets that are web-based.

Simple URL Selection

The easiest way to read a web-based dataset is to simply paste the URL into the source dataset parameter.

Here the workspace author is adding a reader to read a Shapefile directly from an open data ftp site.

The URL entered into a source dataset field may also be a reference to a shared resource on a web-based file storage system. For example, here a user is reading a MapInfo TAB dataset directly from a Dropbox link into the FME Data Inspector:

The key is to force the web service to provide a direct link to the file, rather than to their own web interface. For example, to cause Dropbox to render data you should set the dl query parameter to 1 in the URL, as in the above screenshot, instead of the default value of 0 (zero).

However, there are better ways to read data from a web service...

TIP

If the web site data is being read from needs authentication, most readers have parameters to enter such information:

Web Services

Besides being able to read from a URL, FME can also directly access certain web services to read data. This is done with functionality inside FME called Web Connections.

Web Connections are created by selecting Tools > FME Options > Web Connections on the FME Workbench menubar and clicking the plus button in the Web Connections dialog:

There are a large number of web services to which a connection can be made - and the Manage Services button allows you to integrate any other web service you require - but the key ones are:

• Autodesk A360
• Box
• Dropbox
• FME Server
• Google Drive
• Microsoft OneDrive

These are key types because they are capable of storing data in a way that can be accessed directly from the Add Reader dialogs:

In the above screenshot an author is adding a CSV dataset from Google Drive. This action opens a dialog in which they can browse Google Drive for the required file:

The reader is then added to the workspace and functions just as any other.

Note that if you wish to use a FeatureReader transformer instead of a reader, then the same "Select File From Web" option is available.

Connector Transformers

Besides being able to direct a reader to a web service, various transformers exist that can read files. These relate to the same list supported by readers, so we have:

• AutodeskA360Connector
• BoxConnector
• DropboxConnector
• FMEServerResourceConnector
• GoogleDriveConnector
• OneDriveConnector

These transformers don't read data in the traditional FME sense. Instead they read a selected file and either add the contents to an attribute or download the file to the local filesystem.

Here, for example, an author is using a Creator transformer to trigger the reading of a file from Google Drive:

The contents of the file have been added to an attribute that can then be processed as required. For example, maybe it is a snippet of XML that can be decoded with an XML transformer.

Given the "Select File From Web" tool on readers, Connector transformers are not generally intended for reading source data; although they could be used to download a file that is subsequently read using a FeatureReader.

Another use would be to retrieve a list of files that are then read directly using a FeatureReader.

But perhaps the more likely use is to transfer written data to the web. Writers don't have an equivalent "Select File From Web" option, so the best alternative is to write data with a FeatureWriter transformer and then use a Connector transformer to transfer that data to a web service of choice.