What Are Data Pulls?
Data pulls enable data that an external system exposes to be brought into Gladly, e.g., pulling order data in from an OMS, like order status or the products in an order.
Most external systems have complex data models that are comprised of multiple related data types. An order, for example, may reference products and shipments, which are all stored as separate objects in an external system. In order for this data to be useful in Gladly, all related order data must be retrieved and made available (consistently) within Gladly. Data pull retrieves all related data in one operation and guarantees that the data is stored consistently and can be accessed consistently within Gladly.
Important
Data brought into Gladly from external systems must always be associated with a Customer Profile.
Configuring Data Pulls
An App can be configured to pull data from external systems. This data can be used throughout Gladly to assist Heroes and for consumers to self-serve via Sidekick threads.
Some external systems have simple data models a single API call can fill. Other systems are inherently more complex and may require several API calls to fill out their data models. For example, to find all the details for an order, it might be necessary to run a query to get the order, another to get the line items, another to get the shipping status, and even more to get product details.
Data pulls can support both types of systems. The complexity of the configuration greatly depends on the REST APIs defined by the external systems.
High-Level Approach
Identify the inputs and outputs of the REST APIs for the data you wish to bring into Gladly.
Use the appcfg tool tool to create the request URL, headers, and/or body go templates.
Determine whether or not the response data is in an acceptable format for use in Gladly.
If it’s not in an acceptable format, you can create a template to transform the data into the desired format.
The format of the transformed data should be based on what parts of the output data will be useful in Gladly (Sidekick, etc.).
Create the GraphQL schema that corresponds to the transformed data. The schema defines the structure of the data as well as the data types of the individual fields.
Add a field to the GraphQL
Query
object in the schema that will be used to query the data.
Data pull folder structure
Configuration for each individual REST call made to the external system will be located in a sub-folder of the data/pull
folder.
The documentation below will touch on the files depicted in the folder structure above.
Template Data Available to URL, header, and body templates
Use the appcfg tool tool to create the data for the request URL, header, and body go templates.
Integration data
The following data is available to the request URL, header, and body go templates.
This is integration-specific data entered in the Gladly admin UI when adding the application to Gladly. This usually consists of the account and credential information needed to execute REST calls against the external system.
For example, authentication based on shared secrets usually expects these secrets to be provided via an HTTP header. This implies that your data pull configuration will have a header template that references the attribute that stores the secret in the secrets
object.
For example, an API that expects the shared secret to be specified in an X-API-Key header would have a corresponding X-API-Key.gtpl
template file where the template would be something like the following:
Gladly Customer Profile data
Data brought into Gladly from external systems is always associated with a Customer Profile. Customer Profile data is used in HTTP URL or body templates to retrieve data associated with a given Customer Profile:
For example, an OMS may support retrievingCustomer orders by their email address:
Attribute | Type | Description |
---|---|---|
String | Full name | |
customer.address | String | Complete address |
customer.primaryEmailAddress | String | The email address marked as primary in Gladly’s customer profile, often used to uniquely identify a Customer. |
customer.emailAddresses | [String] | All email addresses configured for the Customer in Gladly. This also includes the customer.primaryEmailAddress. Email addresses are in lower case. |
customer.primaryPhoneNumber.number | String in E.164 format | Customer’s phone number marked as primary in the Gladly customer profile. |
customer.primaryPhoneNumber.type | String | The value will be one of HOME, MOBILE, OFFICE or OTHER. |
customer.phoneNumbers | [{”number”: “”, “type”: “”}] | All phone numbers configured for the Customer in Gladly. This also includes the customer.primaryPhoneNumber. number is in E.164 format and the type is one of HOME, MOBILE, OFFICE, or OTHER. |
Create the folder for the data pull configuration
All configurations associated with a given data pull will reside in the same sub-folder of the data/pull
folder within the app configuration. The name of the folder is also the name of the data pull.
For example, the folder for a data pull that retrieves Customer orders:
Create the JSON configuration file for the data pull
The config.json
file contains all the configuration information that is not template-related. Each template is stored in its own file for readability and testing.
Attribute | Type | Description |
---|---|---|
String | Data from an external system is identified by a data type name in Gladly. Each unique type of data will have a unique data type name. It is generally good practice to prefix the data type name with the data source. For example, a Shopify order object should be named something like shopify_order. Data type names are globally unique, and data of the same type should use the same data type name. | |
dataType.Version | String | Gladly uses semantic versioning to support the evolution of data types and backward compatibility. Only add new attributes to a given data type whenever possible to guarantee backward compatibility. This data will be used throughout Gladly, and making data enhancements backward compatible ensures that all existing functionality continues to work. |
httpMethod | String | The HTTP method used to submit the request. E.g., GET, POST. |
contentType | String | The Content-Type if the request has a body. |
Create the request URL template
This template specifies the fully qualified URL for the REST call. The HTTP method used to submit the request is specified in the config.json
file. The template can access the data outlined in the Template Data available to URL, header and body templates section above.
For example, request URL template:
Depending on the HTTP method, create the request body template
This template specifies the body’s content for HTTP methods that require a body. The content type associated with the body is specified in the config.json
file.
For example, a request body template that is in application/x-www-form-urlencoded
format (i.e., Content-Type: application/x-www-form-urlencoded
)
Request headers
Configuration for any required request headers is located in the data pull’s headers
folder.
Each header is configured via a template that provides the template value. Each header template file is named after the header.
Create the response data transformation template
In many cases, the data returned as part of the response can contain some information irrelevant to Sidekick and/or not in an optimal format. The response transformation template is used to extract relevant data from the response as well as converting it into a more user-friendly format. The output of the template MUST be in JSON format.
The response data transformation template has access to all of the data provided to the URL, body and header templates and, in addition, the associated HTTP request and response data. Currently, response data formatted as JSON (application/json) or XML is supported. The response data is accessed using the rawData
attribute of the template data. In most cases, the response template will just be transforming the rawData
.
Defining the corresponding GraphQL object type is generally good practice for helping visualize the desired output format. Use that as the blueprint for the transformation template.
Template data
Attribute | Type | Description |
---|---|---|
request.url | String | The fully qualified URL of the request. |
request.method | String | HTTP method for the request. |
request.headers | Object | Each attribute corresponds to a request header. The value associated with each header is an array of String values representing one or more header values. |
request.body | String | The raw request body text. |
response.status | String | The HTTP status of the response. |
response.statusCode | Int | The HTTP status code of the response. |
response.headers | Object | Each attribute corresponds to a response header. The value associated with each header is an array of String values representing one or more header values. |
response.body | String | The raw response text. |
rawData | Object | The object representation of the JSON or XML response data. |
See the accompanying go template documentation for the syntax of referencing this data in your templates.
The entire response template data looks as follows.
For example, hypothetical response data:
Hypothetical response transformation template (output MUST be in JSON format):
Create the ID template
This template extracts the unique ID from the data.
External data pulled into Gladly must have a unique ID that identifies the data in the external system. This ID uniquely identifies the data in Gladly, and any subsequent data retrieved from the external system with the same ID is deemed the same. Newly pulled data with an existing ID in Gladly will replace the existing data; i.e., it will update the data with the newly retrieved values.
Most external systems also require these IDs when executing actions on the data. For example, an external system will need the order ID to delete an order.
The ID template is executed against the transformed data and NOT the original data returned by the external system. That means in the example above, the ID is specified via the id
attribute and NOT the original orderId
attribute.
The ID template file:
The content of the external_id.gtpl
file for the above example:
Create the 'Updated At' template
This template extracts the “updated at” timestamp from the data.
If the external system provides an “updated at” timestamp for the data, make sure to include that as part of the data transformation. This timestamp will be the most accurate indication of the age of the data. Gladly will check the “updated at” timestamp when updating existing data, and retain the newest version.
The “updated at” timestamp is also used when retrieving data. External data is accessed from newest to oldest, to ensure you always have access to the most recent data.
The “updated at” template is executed against the transformed data, and NOT the original data returned by the external system.
You can omit this template if the external system does not provide an “updated at” timestamp, in which case Gladly will use the timestamp of when the data is persisted in Gladly.
The “updated at” template file:
The content of the external_updated_at.gtpl
file for the above example:
Create the GraphQL schema for the data being pulled
The outputs associated with each configured data pull are defined using a single GraphQL schema.
Each data pull configuration has an associated type defined in the GraphQL schema. Each field in the GraphQL type definition corresponds to one of the attributes of the transformed response data. Ensure that the GraphQL field types match the attribute types from the response template JSON; i.e., the JSON attribute name and type in the response template MUST match the corresponding GraphQL field type.
Important
Any external data JSON attribute value that does not match the corresponding GraphQL field definition will cause the GraphQL query for that data to fail.
Ensuring that attribute values match their corresponding GraphQL type makes it possible for Sidekick to apply appropriate rules based on the data type. E.g., a DateTime field can have a “no older than one week” rule applied to it, because Sidekick can treat the value as a valid timestamp.
GraphQL types are associated with external data pull configuration via the @dataType
directive. Each config.json
file for a given data pull configuration specifies the data type and version associated with the data being pulled. The data type applies to the transformed version of the data.
@dataType
directive definition:
Add fields to the GraphQL Query
type definition to let Sidekick know what external data can be queried.
For example, using the above sample orders
data pull:
Gladly supports the standard GraphQL data types (Int, Float, String, Boolean, ID) and the following additional types.
Data Type | Format | Description |
---|---|---|
DateTime | String value in ISO8601 date format | DateTime fields can be used in conditionals and can be manipulated. |
Currency | String representation of currency value (real number) | Representing currency values as a string ensures no loss of precision. Gladly ensures that no precision is lost when manipulating Currency values. |