Data Requirements Specification

Introduction

This document outlines data requirements for Da Vinci email orchestration and personalization.

We request that you adhere to all aspects of this specification. If you’re unable to meet the requirements in this specification, or if you must deviate from these requirements, notify your Implementation Architect (for onboarding clients) or your Client Experience Manager (for existing clients).

Once you’ve deployed your first production email campaign through the Da Vinci platform, you should consider all data feeds listed in this document to be critical production assets. If, for any reason, you must modify these data feeds (whether it be the file name patterns, file extensions, file types, field names, contents, or delivery timing/cadence), alert your Implementation Architect (for onboarding clients) or your Client Experience Manager (for existing clients) at least 5 business days prior to making the changes to ensure there is no impact to email campaign production.

General Requirements

  • All data feeds should be compressed or encrypted
    • Movable Ink Da Vinci supports compression (gzip and bzip) as well as PGP encryption
    • Note that compression should not be used if the file(s) are PGP encrypted; PGP encryption already effectively compresses the data
  • For all data feeds, use the following file extensions:
    • For uncompressed, unencrypted data (not recommended): .txt
    • For gzip compressed data: .txt.gz
    • For pgp encrypted data: .txt.pgp or .txt.gpg
  • Any practical column delimiter can be used

Save time by avoiding this common data problem!

Check your initial data hand-offs to ensure that column delimiters are functioning as expected and that parsing errors will not be encountered.
  • Da Vinci supports Boolean, Date, Date & Time, Float, Integer, String, and Categorical data types
  • All date and datetime fields must be in ISO 8601 format
  • All data provided to Movable Ink Da Vinci must be UTF-8 encoded
  • All data feed files should contain a header row with field names
  • Column field names must not contain any spaces or any of the following special characters:
    • ;
    • {
    • }
    • (
    • )
    • \\n
    • \\t
    • =

Policy on Sensitive Personal Data

Movable Ink Da Vinci does not permit Sensitive Personal Data in any of the data feeds that it processes.  Sensitive Personal Data consists of:

  • (a) all government-issued identification numbers, including US Social Security numbers, driver's license numbers, and passport numbers; 
  • (b) all financial account numbers, including bank account numbers, credit/debit card numbers, passwords and other information if that information would permit access to a financial or other account; 
  • (c) all information relating to racial or ethnic origin, political opinions, religious or philosophical beliefs, trade-union membership, health, sex life, criminal matters or offenses, biometrics, and physical, physiological, genetic, mental, economic, cultural or social identity of a natural person or any minor under the age of thirteen that would be subject to the Children Online Privacy Protection Act or similar applicable laws; and 
  • (d) any other personal data designated by applicable law as "sensitive personal data," "special categories of data" or similar designation.

File Transfer Mechanisms

Data can be transferred to Movable Ink Da Vinci through one or more of the four supported Data Connection types listed below.

  • SFTP Hosted - A SFTP server hosted by Movable Ink
    • Note that authentication using a SSH keypair is required when using SFTP Hosted Data Connections (password authentication is not supported).
  • GCS Hosted - A Google Cloud Storage bucket hosted by Movable Ink
  • SFTP External - A SFTP server hosted by our client or another third party
    • Note that both Password and SSH Keypair authentication is supported for SFTP External Data Connections.
  • GCS External - A Google Cloud Storage bucket hosted by our client or another third party
  • Amazon S3 External - An Amazon S3 bucket hosted by our client or another third party

Data Connections can be configured and managed in the Da Vinci web application (navigate to Data, then Connections).

Data Overview

To deploy personalized email campaigns, Da Vinci requires three types of data: Contact Data, Transaction Data, and Product Data.  Your Da Vinci account team must understand the “join relationships” between each data feed.

  • It’s typical for all Contact Data to be keyed on either email address or an alternative reference id that can map to an email address in the ESP.
  • It’s typical for Transactions Data to contain an email address or an alternative id (such as a customer id) that is represented in the Contacts Data.  Da Vinci supports one-to-many, many-to-one, and many-to-many mappings for email address and customer id.
  • It’s typical for Products Data to be keyed on sku or a similar product identifier that is represented in the Transaction Data.

It’s not necessary for your data schema to match this example exactly.  Discuss with your Da Vinci account team how your data schema may differ from this example.

 

Contacts Data

Contacts Data Delivery

  • Cadence:
    • Delivery once daily in the early morning (prior to business hours) is recommended.
    • Option A: Full Table Replacements
      • Each delivery of the Contacts Data Feed includes your full table of emailable contacts.
    • Option B: Delta Files
      • Each delivery of the Contacts Data Feed includes records that have changed
      • Note: If this option is chosen, then we recommend you maintain the ability to easily and quickly provide a "full table replacement" in the event that a data integrity problem is identified.
  • File Name Pattern:  DV_contacts_{clientname}_{label}_{yyyyMMddHHmmss}.{extension}
    • Replace {label} with a short description of the feed; this is useful when multiple file feeds are needed to assemble your contact table.
    • Examples:
      • DV_contacts_inkredibleRetail_attributes_20240215062355.txt
      • DV_contacts_inkredibleRetail_loyalty_20240215155502.txt
      • DV_contacts_inkredibleRetail_segments_20240215120906.txt.pgp

Contacts Data Contents

  • Includes all emailable contacts, including those who might be contacted less commonly (weekly, monthly, etc).
  • Seed records must be included in the Contacts data feed.  This allows Da Vinci to construct seed emails using the same data workflow as other contacts’ emails.  This makes the seed emails you receive as representative as possible of your real customer's experience.
  • Required Fields
    • Data must be keyed on email address or a “reference id” field that maps one-to-one with an email address within your Email Service Provider (Subscriber_ID, Email_ID, RIID, etc.).
    • Includes a "customer id" or a similar identifier that transactions are keyed on.
    • Includes one or more fields that can be queried to determine email opt-in status.
    • Includes all necessary fields to power components in your email template (first name, postal code, etc.).
    • Includes data that can be queried to determine a contact’s membership in various audiences/segments.
  • All date and datetime fields must be in ISO 8601 format.

Example:

email_address customer_id opt_in_status first_name postal_code loyalty_tier store_shopper
a@example.com 8223344 True Susan 17563 Gold True
b@example.com 8226060 False   30022   False
c@example.com   True Joe   Platinum False

 

🎯 Ready to share Contacts Data with Movable Ink?

Click here, then follow the instructions to share sample data with your account team.

In this schema document, only provide sample contact list/profile attribute datapoints - not real customer data.

 

Transactions Data

Recurring Transactions Feed

Frequency: 1x Daily (at any consistent time)

File Name Pattern:  DV_transactions_{clientname}_{yyyyMMddHHmmss}.{extension}

Examples:

  • DV_transactions_inkredibleRetail_20230211030255.txt.gpg
  • DV_transactions_inkredibleRetail_20230702034311.txt
  • DV_transactions_inkredibleRetail_20230615051522.txt.gz

Required Fields:

  • Unique customer identifier (email address, Customer ID, etc.)
  • Order/transaction ID
  • SKU or other product identifier
  • Date and time of transaction in ISO 8601 format
  • Revenue amount for this sku (excluding shipping and tax, in dollars not cents)
  • Quantity (clarify if the “amount” provided in the data is per unit or total)

Optional Fields:

  • Source channel for the transaction (in-store, online, phone, catalog, etc.)

Example:

customer_id order_id source_channel sku amount quantity timestamp
26932005 1635665 online 5780118 12 1 2024-08-27T12:34:00
26932005 1635665 online 5780164 12 1 2024-08-27T12:34:00
14522585 1635666 store 4655467 16 2 2024-08-27T12:34:00

 

Additional Notes & Requirements:

  • The customer identifier used in the transactions data must be present in the Contacts Data feed
  • Each delivery should contain data for transactions occurring within a reasonable look-back window (1 to 5 days is typical).
  • Gaps in provided transaction history should be avoided.
  • Transactions data should include all relevant sales channels (online, in-store, catalog, phone, etc.).

Provide documentation on the following:

  • How returns are treated in the provided transaction data.
  • What zero-revenue transactions represent (if relevant).
  • Documentation on “special case” transactions, such as a guest customer transactions.

Historic Transactions - 1 to 5 Year(s) of history

Frequency: One-time handoff

File Name Pattern:  DV_historic_transactions_{clientname}_{yyyyMMddHHmmss}.{extension}

Examples:

  • DV_historic_transactions_inkredibleRetail_20230211030255.txt.gpg
  • DV_historic_transactions_inkredibleRetail_20230702034311.txt
  • DV_historic_transactions_inkredibleRetail_20230615051522.txt.gz

Required Fields:

  • Fields should match the schema of the recurring transactions feed

Additional Notes & Requirements:

  • Da Vinci requires at least 1 year of historic transaction data.  Providing additional history would further improve our ability to effectively personalize content.
  • Provide the recurring feed first. Once that feed is reviewed and approved by your Da Vinci team, then the one-time historic file can be sent.

🎯 Ready to share Transactions Data with Movable Ink?

Click here, then follow the instructions to share sample data with your account team.

In this schema document, only provide sample contact list/profile attribute datapoints - not real customer data.

 

Products Data

Recurring Products Feed

Frequency: At least once per week (on a consistent day and time)

File Name Pattern:  DV_products_{clientname}_{yyyyMMddHHmmss}.{extension}

Examples:

  • DV_products_inkredibleRetail_20230211030255.txt.gpg
  • DV_products_inkredibleRetail_20230702034311.txt
  • DV_products_inkredibleRetail_20230615051522.txt.gz

Required Fields:

  • SKU or another product identifier - This should be the Primary Key with no Duplicates
  • Product ID (required if SKU is at the variant level (color/size))
  • Division, if relevant (for example, Mens, Womens, Kids, Home, etc.)
  • Category (for example, Jackets)
  • Section (for example, Denim Jackets)
  • Product title

Optional Fields:

  • Additional product attributes (brand, season, color, pattern, etc.)

Example:

sku product_id division category section product_title
4770925 489021 Women Cardigans Cotton Cardigans Women's Supima Cotton Cardigan
6048318 516594 Women Jeans Straight Leg Jeans Women's Plus Straight Leg Jeans, Mid Rise
5705487 476517 Women Tops & T-shirts Long Sleeve T-Shirts Women's Supima Long Sleeve Crew Neck T-shirt

Additional Notes & Requirements:

  • If the product identifier (SKU) relates to a specific color and/or size product, then a higher-level product identifier that relates to the family of products should also be provided.
  • Clean data leads to the best personalization performance!  For example, if the Women’s division is represented as 'Women', 'women', 'Womens', and 'woman' in the products data, then our AI model may treat them as separate divisions, which would not be ideal for performance.

Historic Products - 1 to 5 Year(s) of history

Frequency: One-time handoff

File Name Pattern: DV_historic_products_{clientname}_{yyyyMMddHHmmss}.{extension}

  • Examples:
    • DV_historic_products_inkredibleRetail_20230211030255.txt.gpg
    • DV_historic_products_inkredibleRetail_20230702034311.txt
    • DV_historic_products_inkredibleRetail_20230615051522.txt.gz

Required Fields:

  • Fields should match the schema of the recurring products feed

Additional Notes & Requirements:

  • Historic product data is needed to properly process historic transactions data.  The amount of history provided for products data should correspond with the historic transaction data provided.
  • Provide the recurring products feed first. Once this feed is reviewed and approved by your Da Vinci team, then the one-time historic file can be sent.

🎯 Ready to share Products Data with Movable Ink?

Click here, then follow the instructions to share sample data with your account team.

Was this article helpful?

/