FileMaker

9 Steps to Sync your FileMaker data with Drupal

Mediacurrent Team — Mon, 18 Mar 2013 13:27:13 +0000

9 Steps to Sync your FileMaker data with Drupal

If you read my first blogpost on using WSClient and Rules to push your Drupal data to FileMaker, then you're ready for the rest of the story: How to get your FileMaker data into Drupal. Well, sir or madam, there are two ways: write FileMaker scripts to push data to Drupal, or use Drupal to pull data from FileMaker. We suggest the latter, and here’s why.

FileMaker scripting is a major PITA

Pushing field-level data to Drupal with the OnObjectSave trigger sounds like such a great idea at first blush, but major blockers surface pretty quick. Guess what happens every time you modify any FileMaker field: OnObjectSave calls your custom script to push a REST request to Drupal services, and the FileMaker client interface locks up for the duration of the request! That's about a three second "freeze" every time your client edits a field. “Well, let’s simply log the change for later processing by a server-side FileMaker script.” Great idea. But there still remains the impossibility of maintaining object-level triggers on each your 50 fields × 10 layouts = 500 point-and-click-operations = “Wait, did I miss one? How the heck do I track that down.”

For the FileMaker scripting gurus who hope to avoid the pain of PHP coding, there is still hope. You might try pushing record-level data to Drupal with the OnRecordCommit trigger. Since this is a layout-level trigger, maintenance is feasible, and your Drupal configuration is all point-and-click: set up the Drupal Services module to receive your REST requests. To send those REST requests in FileMaker scripting, you’ll need a plugin that provides HTTP request functions. The Troi URL Plugin costs $300 but provides everything you need out of the box. The 360works ScriptMaster plugin is free but only provides HTTP GET and POST; you can write the necessary HTTP PUT and DELETE functions via the plugin interface in a language called Groovy (=Java).

The best approach: skip Drupal Services entirely.

I know, it’s a bold idea. And it was also rejected by our lead developer when I first proposed it. But here’s where we ended up: skip the FileMaker “push” entirely and pull all FileMaker changes via Drupal Web Services Client! This approach has the added benefit of reducing our dependency on FileMaker scripts that reside outside our project codebase. Here’s the basic idea:

FileMaker work

Add a “Deletions Log” table to keep track of any records that were deleted in FileMaker.
Create a custom script “Log Deletion” that will add a row to the “Deletions Log” table when records are deleted.
Use FileMaker Pro Advanced (yes, you need the “Advanced” version) to modify the default “Menu Actions” for your FileMaker file. Have the “Delete” menu action call a custom script to add the ID of the deleted row to your new “Deletions Log”.
Add a “modification timestamp” field to all your FileMaker tables.

Drupal work

Modify your WSClient LIST operations to filter on that “modification timestamp” field.
Add new LIST and DELETE operations to act on your new “Deletions Log” layout.
Add a “Skip FileMaker CRUD” field to all your to-be-sync’d entities.
Create a new function in your existing custom module to perform the sync operations.
In this sync function, call each of your WSClient LIST operations, and pull all the FileMaker records where “modification_timestamp > $timestamp_of_last_sync”.
Process the records returned by those LIST operations into their corresponding Drupal entities.

System work

Use cron (whether Drupal or system) to hit your custom module’s “sync” function on a regular basis.

Admittedly, there are more than a few gotchas for each of the steps in this recommended approach. But fortunately, yours truly has tripped over all of them and found solutions to each. Here’s the walk-through.

Step 1. Add a Deletions Log to your FileMaker file

This log table needs two fields: the Drupal Entity ID of the record which was deleted, and the Drupal Entity Type to which it belongs.

Step 2. Write the Log Deletion custom FileMaker script.

This custom FileMaker script will create a new row in the Deletions Log table and set it with the Drupal Entity Type and Drupal Entity ID of the record being deleted.

To get the Drupal Entity Type, you might use the TableName of the deleted record to lookup the corresponding type. Or you might use a global variable of the current table to store the Drupal Entity Type. The Drupal Entity ID is available as a field of the record being deleted. Next, add script steps to switch to your “Deletions Log” layout and create a “New Record/Request”. After setting the Drupal Entity Type and Drupal Entity ID into the new record, you’re all done!

Note that if the Drupal Entity ID of the deleted record is not set, we may assume that our Drupal Sync Script has not yet pulled the new record into Drupal, and the deletion need not be logged.

Step 3. Modify your FileMaker file’s Menu Actions.

Tools → Custom Menus → “Custom Menus” Tab → Create → and choose “Records”. From there you can edit the “Delete Record” Menu Item and add an action to perform the “Log Deletion” script outlined above. Don’t forget to create a “Custom Menu Set” and mark it as the default for your FileMaker file.

Step 4. Add a “modification timestamp” field to all your FileMaker tables.

Add this “calculation” field to each of the FileMaker tables you want to sync:

Let ( ~trigger = GetField ( "" ) ; 
  Round ( Get ( UTCmSecs ) / 1000 ; 0 ) - 
  Timestamp ("01/01/1970" ; "00:00:00")
)

The first line triggers our field to update every time the record is modified. The second line gets the current UTC (~GMT) time in seconds since 00:00 on 1/1/0000. The third line converts that to epoch/unix time (seconds since 00:00 on 1/1/1970).

Step 5. Add a filter to your WSClient LIST operations

If you haven’t yet created your WSClient LIST operations, see part one of this blogpost. Also, make sure you’ve reviewed the RESTfm manualwhich describes how to add filters to your LIST requests. Filters can be added to any LIST request with a combination of two URL parameters: one to specify the field, and one to specify the value. For example, if your field were named “modification_timestamp”, your WSClient LIST operation URL might look like:

http://your-server/RESTfm/your-database/layout/your-layout?
  RFMsF1=modification_timestamp&RFMsV1=>@timestamp_of_last_sync

Note the greater-than sign before @timestamp_of_last_sync. This tells RESTfm that we want any record whose field value is greater than the value we pass to it. Also note that I suggest using the RFMmax URL parameter to explicitly define how many records your server can realistically handle (probably a few hundred). Without specifying a value, RESTfm defaults to only 25, which may cause you to miss some records.

Step 6. Write LIST and DELETE operations for your “Deletions Log”

Follow the steps outlined in part one to write additional LIST and DELETE operations for WSClient. These will act on your “Deletions Log” layout.

Step 7. Add a “Skip FileMaker CRUD” field to all your entities

Choose "boolean" for the field type.

Step 8. Update your Rules to respect the "Skip FileMaker CRUD" field

To each UPDATE Rule created in part one, add two "Data Comparison" conditions. The first condition acts on "your-entity:field_skip_filemaker_crud" and the other acts on "your-entity-unchanged:field_skip_filemaker_crud" (the entity data before it was modified). Also add the first condition to each of your CREATE and DELETE rules. Be sure to check the "Negate" option, because we only want our rules to execute when this field is FALSE or missing.

Step 9. Pull and process FileMaker data!

You might keep this sync code in a custom script that manually bootstraps a Drupal environment, or you might store it in an include file of the custom module you already created (in part one of this tutorial). Note, the code that follows is just a starting point and, with the exception of Deletion Log processing, syncs only one entity type.

Setup.

// Set the new timestamp before sync to err on the side of overlap.
$lastsync = variable_get('_your_module_last_sync_timestamp', 0);
variable_set('_your_module_last_sync_timestamp', time());

// Load the WSClient service. In part one, we named it “restfmclient”.
$service = wsclient_service_load('restfmclient');

// At minimum, we need to know the entity type and bundle.
$basetype = ‘your_entity_type’;
$bundle = ‘your_bundle’;

// Now use entity_get_info to set the other parameters we’ll need later on.
$entityinfo = entity_get_info($basetype);
$entityid_key = $entityinfo['entity keys']['id'];
$typefield = $entityinfo['entity keys']['bundle'];

Pull FileMaker records that were modified since last sync. If you’ve followed along faithfully to this point, you should be able to call your WSClient operation like so.

// Get all records modified in FileMaker since this script last ran.
// In part one of the post, we used “list” as the operation name.
$response = $service->your-wsclient-list-operation($lastsync);

But! RESTfm likes to return a 500 error when no records are found. So let’s catch it.

  try {
    // Get all records modified in FileMaker since this script last ran.
    $response = $service->your-wsclient-list-operation($lastsync); 
  } catch (WSClientException $e) {
    $last_response = $service->endpoint()->client()->lastResponse;
    $body = $last_response->body;
    if (500 == $last_response->responseCode &&
        FALSE !== strpos($body, '401: No records match the request')) {
      // OK to ignore error - RESTfm returns 500 error on no records found.
    } else {
      // Continue error propogation.
      throw $e;
    }
  }

Loop through the returned records Get the corresponding Drupal entity. Set the new values. Save.

foreach($response['data'] as $fmrecord) {
  // Check whether FileMaker already had an entity id for this record.
  if (empty($fmrecord[$entityid_key])) { 
    // Since FileMaker did not send the entity id, this entity does not
    // yet exist in Drupal and must be created (specifying the bundle property).
    $entity = new Entity(array($typefield => $bundle), $basetype);
  } else { 
    // FileMaker had entity id, so the entity already exists in Drupal.
    $entities = entity_load($basetype, array($fmrecord[$entityid_key]));
    $entity = array_pop($entities);
  }
  // We must get our field list from someplace other than the data sent to us by
  // RESTfm, because RESTfm leaves empty fields out of the response.
  $fields = array_keys(field_info_instances($basetype, $bundle));
  // Update entity data with values from FileMaker.
  foreach ($fields as $field) { 
    // Check whether RESTfm sent us data for this field.
    if(empty($fmrecord[$field])) { 
      // FileMaker sent no data. If the field previously had data, it was
      // deleted in FileMaker, so delete it from Drupal, too.
      unset($entity->$field);
      // Move on to next field.
      continue;
    } 
    // Get the value sent by RESTfm for this record.
    $value = $fmrecord[$field];
    // Non-property fields use a multi-lingual and multi-value array structure.
    if(substr($field, 0, 6) == 'field_') { 
      // Set the field's value.
      $entity->{$field}[LANGUAGE_NONE][0]['value'] = $value;
    } else { 
      // Assume no value array structure to deal with.
      $entity->$field = $value;
    } 
  } 
  // We use a flag to keep Rules from executing WSClient calls to RESTfm on
  // save, which would cause an infinite loop of sync updates.
  $entity->field_skip_filemaker_crud[LANGUAGE_NONE][0]['value'] = 1;
  $saveresult = entity_save($basetype, $entity);
  // Unset the flag and re-save the entity. For this to work, our Rules
  // conditions must check for the flag in both the "unchanged" (i.e.,
  // pre-update) entity data and the new entity data.
  unset($entity->field_skip_filemaker_crud);
  entity_save($basetype, $entity);
  // If the entity was just created in Drupal, we want to send the entity id
  // back to FileMaker, so we'll call entity_save to trigger an update operation.
  if (SAVED_NEW == $saveresult) {
    entity_save($basetype, $entity);
  }
}

Phew. If you made it through that, you’re doin' swell! Now, let’s process our Deletions Log. Start by calling the WSClient LIST operation we created for our Deletions Log. Remember to catch the 500 error returned by RESTfm on no-records-found! LIST records in the FileMaker Deletions Log

try {
  $response = $service->your-wsclient-deletion-log-list-operation(); 
} catch (WSClientException $e) {
  // @see Above.
}

Loop and Delete!

foreach ($response['data'] as $index => $deletionlog_entry) {
  // Get the entity type.
  $entity_type = $deletionlog_entry['entity type'];
  // Get the entity id.
  $entity_id = $deletionlog_entry['entity id'];
  // Delete!
  $deleteresult = entity_delete($entity_type, $entity_id);
  if (FALSE !== $deleteresult) {
    // Success, so clear the entry from the FileMaker Deletions Log.
    $recordID = $response['meta'][$index]['recordID'];
    $service->your_wsclient_deletion_log_delete_operation($recordID);
  }
}

To learn about syncing FileMaker join tables with multi-value entity reference fields, contact me on d.o.

This post was made possible with support and guidance of lead project developer James Rutherford, trusty project manager Donna Merit, and tireless FileMaker guru Dale Long. I also thank the Mediacurrent partners and marketing team for making it a priority to promote community contributions like this one.

Additional Resources

Inline WYSIWYG Editing in Drupal

Responsive Design: Mobile Menu Options

Display Suite - A Themers Perspective

Mediacurrent Team Mon, 03/18/2013 - 09:27

Featured Recipe: FileMaker and Drupal

Mediacurrent Team — Thu, 21 Feb 2013 16:52:30 +0000

Featured Recipe: FileMaker and Drupal

Do you love FileMaker? Do you work with (or for!) someone who does? I’m now a Drupal Developer at Mediacurrent, but my last position (at Georgia Tech’s Office of International Education) involved a lot of FileMaker work, and this FileMaker integration would have been a lifesaver, so here goes! First, a bit of background on FileMaker for the unindoctrinated: “Tackle any business task... Powerful, easy-to-use database software for yourself or a team. Create forms, reports and labels. Share on a network or over the web.” Lots of non-technical folks love to use FileMaker; it's a cheap and intuitive alternative to more expensive visual database solutions, like Microsoft Access. It's the #1 best-selling standalone database for both Macintosh and Windows and has won dozens of awards in the industry. (Source: ScottWorld.com). At Mediacurrent, we recently integrated a client's existing FileMaker database with a freshly-launched Drupal 7 website. Here’s an overview of the workable solutions we explored:

Drupal → Filemaker
push data to RESTfm (a REST server for FileMaker) via the WSClient and Rules modules.
Filemaker → Drupal
send REST requests from FileMaker to Drupal Services via the Troi URL plugin.
Drupal ← Filemaker
pull data into Drupal via a LIST response from RESTfm.

Ingredients

FileMaker Server ($3k)
FileMaker Pro Advanced (yes, you need the “Advanced” version) ($500)
RESTfm ($400)
Troi URL plugin (optional, for pushing directly Filemaker to Drupal) (web server license - $300).
Drupal ($Priceless.00)

Baking Instructions, Drupal → Filemaker

Enable PHP Web Publishing (details) on your FileMaker server. (Documentation v10 and v12)
On your FileMaker database (aka your FileMaker “file”), go to the Privileges settings and enable “[Full Access]” for PHP Web Publishing. (details, and documentation)
Install and configure RESTfm on the Web Server of your choice. Check out the RESTfm manualfor instructions. I chose to follow the manual's suggestion and keep-it-simple by installing RESTfm on IIS (Windows web server) on the same server hosting FileMaker Server.
Test your RESTfm configuration by browsing the RESTfm HTML interface: "http://your-server/RESTfm".
Test creation, deletion, and modification of records via the RESTfm demo page, "http://your-server/RESTfm/demo.html". Note that RESTfm returns a 500 error if any fields on your layout are set to both (1) Auto-Enter: Prohibit modification of value during data entry, and (2) Validation: Not empty. The assumption here is that FileMaker processes the validators before calculating the Auto-Enter fields. You will also see this error if you leave empty any field set to Validation: Not empty.
Test CRUDL (CREATE, RETRIEVE, UPDATE, DELETE, LIST) access to RESTfm from your browser with the Postman extension for Chrome or Poster add-on for FireFox. If none of that made sense, you may want to do some reading up on REST servers.

If you’ve made it this far, congratulations! Those sweat and tears were worthwhile. Let’s look at Drupal configuration now. Start by installing the WSClient and Rules modules. (How to install modules). WSClient doesn’t support configuration of a full REST client via its user interface. In order to configure a WSClient service to consume RESTfm, you’ll have to write it in code with a custom module (for the unfamiliar, review this module creation tutorial (general) and this module developer's guide (technical)). All the code that follows was adapted from the "WSClient Examples" module (included with the WSClient module), and I suggest reviewing it. Note that the useful example will not appear unless you have the RESTful Web Services module enabled. Your code might look like this inside hook_default_wsclient_servicein your custom module:

service = new WSClientServiceDescription();
$service->name = 'restfmclient';
$service->label = 'RESTfm Client';
// Replace "server" with the hostname of your RESTfm web server.
$service->url = url('http://server/RESTfm', array('absolute' => TRUE));
$service->type = 'rest';

// Add HTTP Basic Authorization headers.
// Or you can look into using RESTfm's support for authentication via API key.
// @see http://www.restfm.com/manual_v1/authentication
$curl_options[CURLOPT_HTTPAUTH] = CURLAUTH_BASIC;
// Replace "filemakeruser" and "password" with your FileMaker file login.
$curl_options[CURLOPT_USERPWD] = ‘filemakeruser:password';
$service->settings['curl options'] = $curl_options;

But we’re not done yet! RESTfm is persnickety. It wants you to send data to it in just the right structure: an array called 'data', pointing to another array called '0' (zero), pointing to a third array of fields and values. In PHP code, it looks like "array('data' => array(0 => array('field_name' => $value)))". In order to formulate our data into this structure, we must create three separate data structures for WSClient to pass on to Rules. (some info on Rules data types)

// Data structure to send on UPDATE or CREATE:
//   array('data' => array(0 => array('field_name' => $value)))
$service->datatypes = array(
  'request_data_wrapper_outer' => array(
    'label' => 'request data - outer wrapper',
    'property info' => array(
      'data' => array(
        'type' => 'request_data_wrapper_inner',
        'label' => 'request data - inner wrapper',
      ),
    ),
  ),
  'request_data_wrapper_inner' => array(
    'label' => 'request data - inner wrapper',
    'property info' => array(
      '0' => array(
        'type' => 'request_data',
        'label' => 'request data',
      ),
    ),
  ),
  'request_data' => array(
    'label' => 'request data',
    // HERE: LIST THE FIELDS YOU WANT PUSHED TO FILEMAKER
    'property info' => array(
      'title' => 'text',
      'nid' => 'integer',
      // 'field_your_field' => 'text',
      // ...
    ),
  ),
);

Phew. Glad that’s out of the way. Now, let’s setup the data structures to receive data back from a RESTfm CREATE operation.The data we receive looks like "array('meta' => array(0 => array('recordID' => $fm_id, 'href' => $uri)))".

// Create the data structures needed to receive responses from RESTfm CREATE.
//   array('meta' => array(0 => array('recordID' => $fm_id, 'href' => $uri)))
$service->datatypes = array_merge($service->datatypes, array(
  'create_result_wrapper_outer' => array(
    'label' => 'response from CREATE operation - outer wrapper',
    'property info' => array(
      'meta' => array(
        'type' => 'create_result_wrapper_inner',
        'label' => 'response from CREATE operation - inner wrapper',
      ),
    ),
  ),
  'create_result_wrapper_inner' => array(
    'label' => 'response from CREATE operation - inner wrapper',
    'property info' => array(
      '0' => array(
        'type' => 'create_result_data',
        'label' => 'response from CREATE operation - data',
      ),
    ),
  ),
  'create_result_data' => array(
    'label' => 'response from CREATE operation - data',
    'property info' => array(
      'recordID' => array(
        'type' => 'integer',
        'label' => 'FileMaker Record ID',
      ),
      'href' => array(
        'type' => 'text',
        'label' => 'Resource request URI',
      ),
    ),
  ),
);

Now we’re ready to create our CRUDL operations! It is important to remember that RESTfm performs these operations against FileMaker layouts and not against the actual underlying tables.Note that RESTfm needs a FileMaker Record ID for RETRIEVE, UPDATE, and DELETE operations. This Record ID is an internal FileMaker identifier and can be found by creating a calculated field in your FileMaker table set to "Get ( RecordID )". Also note that there is no way to set or modify a record's Record ID (FileMaker assigns it automatically).

// Replace "your-FileMaker-dabase" and "your-FileMaker-layout" here:
$url = 'your-FileMaker-database/layout/your-FileMaker-layout’;
$targeted_url = $url . ‘/@recordID’;

// CREATE operation.
$operation = array();
$operation['label'] = 'CREATE';
$operation['url'] = $url;
$operation['type'] = 'POST';
$operation['data'] = 'restfm_data';
$operation['parameter']['restfm_data'] = array(
  'type' => 'request_data_wrapper_outer',
  'label' => 'request data - outer wrapper');
$operation['result'] = array('type' =>
  'create_result_wrapper_outer', 'label' => 'response');
$service->operations['create'] = $operation;

// RETRIEVE operation.
$operation = array();
$operation['label'] = 'RETRIEVE';
$operation['url'] = $targeted_url;
$operation['parameter'][‘recordID’] = array(
  'type' => 'integer',
  'label' => 'FileMaker Record ID');
$operation['result'] = array('type' => 'text', 'label' => 'response');
$service->operations['retrieve'] = $operation;

// UPDATE operation.
$operation = array();
$operation['label'] = 'UPDATE';
$operation['url'] = $targeted_url;
$operation['type'] = 'PUT';
$operation['data'] = 'restfm_data';
$operation['parameter']['restfm_data'] = array(
  'type' => 'request_data_wrapper_outer',
  'label' => 'request data - outer wrapper');
$operation['parameter'][‘recordID’] = array(
  'type' => 'integer',
  'label' => 'FileMaker Record ID');
$operation['result'] = array('type' => 'text', 'label' => 'response');
$service->operations['update'] = $operation;

// DELETE operation.
$operation = array();
$operation['label'] = 'DELETE';
$operation['url'] = $targeted_url;
$operation['type'] = 'DELETE';
$operation['parameter'][‘recordID’] = array(
  'type' => 'integer',
  'label' => 'FileMaker Record ID');
$operation['result'] = array('type' => 'text', 'label' => 'response');
$service->operations['delete'] = $operation;

// LIST operation.
// You may add parameters to the URL to filter the result set.
// @see http://www.restfm.com/manual_v1/uri-restfmdatabaselayoutlayout
$operation = array();
$operation['label'] = 'LIST';
$operation['url'] = $url;
$operation['result'] = array('type' => 'text', 'label' => 'response');
$service->operations['list'] = $operation;

And don’t forget to return the service when you’re done.

// All done.
$services[$service->name] = $service;
return $services;

Okay, folks, the hard part is out of the way; pat yourselves on the back! All that’s left is Rules configuration. Head to the Rules Configuration page and set up

a “RESTfm Client: CREATE” action on event “After saving new content” - Include three "Create data structure" actions to set up the RESTfm request data: - (1) request data, (2) inner wrapper, (3) outer wrapper - Include a "Set a data value" action to save the returned FM recordID into your entity. - The "Set a data value" value would look like "response:meta:0:recordID".
a “RESTfm Client: UPDATE” action on event “After updating existing content” - Again, include three "Create data structure" actions to set up the RESTfm request data.
a “RESTfm Client: DELETE” action on event “After deleting content”

That takes care pushing Drupal data to FileMaker. Stay tuned for part deux of this blog post for a walk-through of FileMaker → Drupal solutions. Would you like to see this topic presented at DrupalCon Portland? +1 the session proposal and leave a comment for the selection committee! Link to DrupalCon Session Proposal.

Additional Resources

Inline WYSIWYG Editing in Drupal

Responsive Design: Mobile Menu Options

Display Suite - A Themers Perspective

Mediacurrent Team Thu, 02/21/2013 - 11:52