cURLing in Claris FileMaker

At LuminFire, we’re very excited about enhanced support for cURL added in FileMaker 16. And no, we’re not talking about curling the sport, though it is very popular here in Minnesota and we recommend you try it too if you never have. We’re talking about cURL the free and open source project consisting of two products, a command line utility called curl and a portable library called libcurl, designed for transferring data with URLs. cURL stands for “Client for URLs” or “Client for URL Request Library” and it includes the URL spelled in uppercase to emphasize it deals with URLs. Practically speaking, cURL is a swiss army knife of tools used by programmers to exchange information between systems using dozens of supported protocols. The project name is conveniently/appropriately pronounced “see URL” while the command line utility name is pronounced just like the sport. cURL is the most popular file transfer library on the planet–it is used by billions of devices every day. Enhanced cURL options in FileMaker make it much easier for FileMaker solutions to interact with other systems, opening doors for what is possible within FileMaker.

In order to appreciate the new cURL support in FileMaker, it is important to understand a little bit about the anatomy of a HTTP request. In other words, what happens when you type a URL into a web browser?

What happens when you type a URL into a web browser?

• URL is resolved to an IP address using Domain Name System
• TCP connection initiated
• Security stuff
• HTTP request sent. It includes:
  • http method
  • url
  • http headers
  • http body
  • accept header
  • (content negotiation)
  • user agent
• Server handles request. Replies with:
  • status code
    • 1xx: not done yet, but still here
    • 2xx: success!
    • 3xx: go look somewhere else
    • 4xx: error, your fault
    • 5xx: error, server’s fault
  • headers
  • body
  • content length
  • content type
• Browser receives response, renders body

As you can see, there’s actually a lot going on even with a basic webpage (HTTP) request. When you’re simply typing a URL into a web browser, the browser handles everything else for you behind the scenes. Now as programmer you can see that a lot more can be gained by understanding and exploiting components of HTTP requests above and beyond just the URL itself. In fact as you start to interact with a variety of web services in FileMaker scripts, doing so becomes a necessity. For example, many APIs require custom headers to be sent such as API key(s), language preference, or content type. Additionally, when scripting in FileMaker, you won’t have a browser to elegantly handle 500 Internal Server or other errors, doing so will be your responsibility. cURL provides everything you need to configure requests of almost all imaginable types.

cURL in FileMaker

FileMaker 16 supports only a fraction of the protocols and options included with cURL. FileMaker 16 gets rid of the somewhat unconventional httppost/httpspost protocol that exists in earlier versions and now supports http, https, ftp, ftps, and file protocols. It has been possible to use cURL in versions of FileMaker prior to 16 with the help of plugins like MBS and BaseElements, and you may still need to look to these plugins in 16 for advanced use cases requiring options that aren’t supported natively. Additionally, rudimentary cURL has been possible natively with Insert From URL script step since it was introduced in FileMaker 12. There are two big changes in version 16 that really increase its capabilities:

1. Variables. It now possible to set the response of a URL request to variable. Previously, “Insert From URL” only allowed a field to be set which so often required defining unnecessary schema and user interface manipulation for something that really should just happen in a script. Allowing a variable to be defined and within a single “Insert From URL” script will reduce overhead while improving efficiency, readability, and interoperability. Variables can also be used to refer to files stored in container fields or filesystems–more on that in a bit.

The following example is a subscript I use all the time that attempts to determine device location but when that fails (i.e. on Desktop) it looks up approximate location using cURL and two web services, one to fetch the user’s public IP address and the other to geocode it. You can see in this example just how convenient and clean it is to use Insert From URL with variables.
Example:

# Purpose: Use device location or IP geocoding to determine current location
# Returns: e.g. {"latitude":45.1578,"longitude":-93.3205}

# Try iOS device location first
Set Variable [ $location; Value:LocationValues ( 9 ; 5 ) ]

If [ IsEmpty ( $location ) ]
# No location found? (i.e. using desktop) Geocode IP address instead

Insert from URL [ $ip; "http://ipecho.net/plain" ; Do not automatically encode URL; Select; No dialog ]
Insert from URL [ $res; "http://freegeoip.net/json/" & $ip ; Do not automatically encode URL; Select; No dialog ]

Set Variable [ $location; Value:List ( JSONGetElement ( $res ; "latitude" ) ; JSONGetElement ( $res ; "longitude" ) ) ]
End If

# Build JSON object to return
Set Variable [ $res; Value:JSONSetElement ( "{}" ; ["latitude" ; GetValue ( $location ; 1 ) ; JSONNumber ] ; ["longitude" ; GetValue ( $location ; 2 ) ; JSONNumber ] )]
Exit Script [ Result: $res ]

2. cURL options. The last parameter supplied to the new and improved Insert From URL, Specify cURL options, is where the magic happens. If you’re familiar with cURL on the command line you’ll quickly grasp how this works but if you’re new to cURL there’s a bit of a learning curve. In short, Specify cURL options allows you to supply a string of options (and sometimes values) that provide instructions as to how to construct a URL request, what to include in the data payload, and how to handle results and errors. Unfortunately FileMaker supports only a subset of libcurl options so be sure to refer to FileMaker’s Supported cURL options help page for details (unsupported options are ignored). Several cURL options worth noting:

• –FM-text-encoding. A huge improvement in 16 is that cURL options can be encoded in character sets other than UTF-8 using the –FM-text-encoding option.
• –data. Instead of using the somewhat obscure httppost protocol data for POST, requests can simply be supplied using the –data or -d option set to key-value pairs as text or set to a container field whose content is a text file containing key-value pairs. Read more about POST vs GET requests.
• –user. It is now possible to supply a username and password for server authentication using this option. The username and password are split on the first colon, which makes it impossible to use a colon in the username; however, the password still can.
• –header. Extra header(s) can now be included when sending HTTP requests. Many APIs do not allow incoming requests that lack headers, such as API key or data type, so this option allows FileMaker to communicate natively with countless new web services.
• –upload-file $file and –form file=@$file. Variables can now be used to reference a file in a container field or file system and the variables can be included within cURL option strings to allow uploading to ftp servers, posting to http forms, etc..

Let’s take a look at how we use cURL options. National Renewable Energy Labs provides an API that returns a list of pv solar installation costs and counts by state. The API requires a key to be sent via custom header and parameters to be sent specifying which state and county you’re interested in. Here’s how we build a working “Insert from URL” script step from its defaults to integrate with this API:

1. Determine if SSL (https) is required and check “Verify SSL Certificates” accordingly. If so, decide if you want to verify the certificate and present warning to user in cases where verification fails. Read more in FileMaker 16 Help – Insert from URL.
2. Leave “Select entire contents” checked unless you want to concatenate or insert response into existing data/value(s).
3. Check “Target:” and Specify $Response… You can use a field or variable. Important nuance here is that if the variable doesn’t exist, this script step creates it.
4. Specify URL “https://developer.nrel.gov/api/solar/open_pv/installs/rankings
   • If issuing a request of type GET, include parameters as a string on the url like this: “https://developer.nrel.gov/api/solar/open_pv/installs/rankings?state=TN&county=Hamilton“
   • If issuing a POST request, eliminate the ? and everything after it in the url and include the parameters as part of the cURL options using –data like this: “–data state=TN&county=Hamilton” or “-d state=TN -d county=Hamilton” or using @$postdata where the $postdata is set to key-value pairs as text or set to a container field whose content is a text file containing key-value pairs.

5. Specify cURL options. Here we need to build a string of all the options we need separated by spaces. Options take the form of double hyphens followed by complete words or single hyphen followed by letter abbreviation. Values, where required must be quoted using \” or Quote(). The options –show-error and –dump-header $variable are good to include on every request–they are essential to debugging. The string we would need for this request, if it supported POST and our data was stored in fields, might look like this:

"-H \"X-Api-Key:" & prefs:api_key & "\" --data state=" & table:state & "&county=" & table:county & " --show-error --dump-header $dump"

The complete Insert from URL script steps:

For GET:
Insert from URL [ Select ; With dialog: Off ; $Response ; https://developer.nrel.gov/api/solar/open_pv/installs/rankings?state=MN&county=Aitkin ; cURL options: -H "X-Api-Key: RzPZJyKOZs1e7xw689Obhv9SVc6MvZItTkvhGFxt" --show-error --dump-header $dump]

For POST:
Insert from URL [ Select ; With dialog: Off ; $Response ; https://developer.nrel.gov/api/solar/open_pv/installs/rankings ; cURL options: -H "X-Api-Key: RzPZJyKOZs1e7xw689Obhv9SVc6MvZItTkvhGFxt" --data state=MN&county=Aitkin --show-error --dump-header $dump]

Transferring files

We’ve covered GET and POST but another powerful feature supported is PUT or the transferring of files. There are a multitude of ways that files can be transferred, typically dictated by the receiving end, that can be a bit confusing. “Insert from URL” can issue a POST request containing a file option just like countless forms on the web do, e.g. upload a resume when filling out a job application, attach a profile picture when signing up for a new account, etc. Mike Duncan describes this method in detail in this blog post so I won’t go into it here. FTP and FTPS work differently and admittedly took me a while to figure out. I was originally mislead by Example 4, in FileMaker’s Insert from URL help page which states “The filename on the server will be the same as the filename in the container field.” This applies only to HTTP PUT requests, not FTP. For FTP it is essential that the destination filename be included on the end of the url or the request will fail. DLPTest.com provides an FTP site that can be used for testing, files are stored for 30 minutes, then deleted. Here’s a working FTP PUT request, again note the power of using variables within an options string to indicate the file to be transferred directly from a container field:
Set Variable [ $file ; Value: Request::File ]

Insert from URL [ Select ; With dialog: Off ; $Response ; ftp://ftp.dlptest.com/Lego.png ; cURL options: --upload-file $file -H "Content-type: image/png" --show-error --dump-header $dump --user [email protected]:fwRhzAnR1vgig8s]

Try it Yourself

We’ve only scratched the surface of what you can do with the new native cURL functionality in FileMaker 16. Provided is a demo file that is completely unlocked that you can download and use to demo and learn how to build cURL request. Each record represents a single request and the user interface allows you to easily build/manipulate options, send requests, and explore headers/results. It is preloading with examples, but you can use it to explore APIs you work with to extend your FileMaker solutions. It supports GET, POST, and PUT methods in the user interface; however, others (e.g. DELETE) are possible once you get into using cURL options. Look under the hood at the scripts for details. Feel free to post any questions or comments about cURL in FileMaker 16 or the demo file… and happy cURLing!

Download the LuminFire cURL demo file:

Andy – Software Engineer, GIS Specialist

Tim Cimbura – CEO, CFO and Software Engineer

Tim is an expert in creating custom business solutions that make businesses more effective, productive, and profitable. He specializes in rapid application development with the Claris platform including FileMaker, Laravel, and WordPress. He also knows Apple macOS technology inside and out.