Data Prep and Ingestion into Adobe’s Experience Platform (AEP) Webinar Re-Cap

In this webinar, our AEP experts concentrate on Customer Data Platforms (CDPs), specifically on the Adobe Experience Platform (AEP). At Softcrylic, we have been doing a lot of work within the AEP space and see that it is an area many clients need help uncovering the mystery that can be planning for using AEP. Our Solution Architecture Group, Jerry Helou (VP Martech Strategies) and Natts Sivaramakrishnan (VP of Architecture) and our Director of Application Engineering, Bhupendra Kumar join to discuss the considerations you will need to consider for data prep ingestion in AEP.

What is AEP 6:48

We believe, delivering good customer experiences depends on having a full and accurate profile of your customer, this is what AEP will help you accomplish. Data fuels your AEP, it considers any data you bring to it (analytics data, email data, in-store data etc.) the more you can bring in, the stronger your platform and view of your customer becomes. AEP offers a data foundation; it should not be your only data lake, but it should be your main data lake to fuel activation. AEP provides nearly a 360-degree view of your customer, which is almost as good as it gets. Especially when it comes to knowing customers, AEP is a great fit to come and activate on. The ML based behavioral analysis is used to segment and target your audience, taking a step beyond with your data it allows you to run models and personas that let you activate on AEP.

Why do you need AEP? 9:11

Improve end-to-end customer experience, reduce cost & technical debt, enhance digital transformation, respect privacy/enforce data governance & secure data, enterprise data collection infrastructure & ingestion, enterprise advanced analytics & data science for better customer experience, actionable view of the customer for enhanced enterprise-wide personalization, enforce data governance and usage policies.

AEP Architecture: 12:37

It is all about the data, data sources, bringing in data, building segments, building profiles then activating on them.

Data sources can be in real-time (behavioral, interaction) they don’t have to be only Adobe sources, if you are using Google analytics you can bring this data with you, AEP is very API friendly, your data can come in streaming or batch. As the data comes in, and is built within a schema, the schema will be defined to connect the different data sources calling for discussions and implementation of identity. There are some limitations within AEP, but they are on track to be resolved within the next year.

Data Ingestion in Adobe Experience Platform: Prep & Setup 14:44

Finalize marketing use cases with your business team before you can prepare any data. You must define your digital experience blueprint “what are my repetitive processes, what will my segmentation goals be”

1. XDM Schema
   1. This is your blueprint for organizing your data. XDM will help put your data in the proper form to prepare your data to be ingested.
2. Identity Strategy
   1. Helps combine data from multiple channels to develop 360-degree view of the customer.
3. Datasets
   1. Focuses on the storage and management of your data. This data can be in your data lakes and contains the metadata of various aspects of your data you are restoring.
4. Workflows
   1. This is a step-by-step process of ingesting data once your data set and schema are defined. AEP provides many out of the box workflows, but you can have them available from other companies as well.

Finalizing your Marketing Use Cases 18:35 *important to know*

First and foremost, you need to document your business use cases. Based on what you define, these will drive your technical implementation, be sure to also review these with your stakeholders and agree on your segmentation goals. Second, know your constraints about your Platform and Platform license. You need to understand when you are ingesting the data, that you will have limitations you need to follow to get quality results from your data. You need to identify which data sources you will be ingesting into AEP. Your primary source should be AEP, then you need to know what data in your data lake is needed and what data will be required for RTCDP. Fourth and finally, is to have your centric ERD diagram for data sources. You must create the diagram by defining your data sources’ current states and specify regarding AEP.

XDM Schemas: Schema Composition 24:09

Schemas are used in AEP to describe the structure of your data consistently across systems. These schemas need to be defined prior to you ingesting your data into AEP. The schemas will start at a base class and then go into field level groups (see Adobe’s breakdown here).

Schema Considerations 36:55

When thinking of your schemas, you need to know they will be hierarchical. Adobe has a recommended profile and size to make sure you do not exceed, and once the data is loaded into the dataset representing a schema it is not possible to remove or modify the schema structure. There will be no more than 10 items in array objects and there will be a max of 5-dimension entity lookups with the recommended for United Profile. AEP only supports star schema, and your hierarchy should have no more than 4 levels.

Data Ingestion Overview and Options 43:27

After the above are complete, you are ready to bring your data into AEP! You can bring your data in from other Adobe apps, pretty much out of the box, but you can bring data in from anywhere. If you are using your enterprise data, Adobe has over 40 connectors to bring data from other sources. AEP also can gather client-side data from Web and Mobile SDK as well as with a batch and streaming API. To remind yourself what your team will have to keep in mind when ingesting data, fast-forward to 49:49 in the video.

53:47 Monitoring

AEP offers 2 types of monitoring for your choosing, aggregated and end-to-end. Aggregated monitoring will give you a top-down view from data ingestion status by Source broken down by workflow. Whereas end-to-end monitoring will view the status across data ingestion, into data lake, Unified Profile, Identity Service and Activation!

55:58 Data Quality

AEP enforces this through validation checks and XDM schema compliance validations. When checking your data quality, be sure to mark down what is not ingested when the validation rules are not met. Some validation checks are supported in batch ingestion, and some are not. The downfall is that there is no QA tool available to validate data for consistency and completeness once the data is ingested.

57:10 Datafacts

Datafacts is an Adobe Experience Platform Data Quality Management solution that Softcrylic has developed over the past year. Softcrylic used Adobe’s Firefly framework and built Datafacts on top of it! This tool detects and alerts data anomalies at record and column level for all AEP data sources including Adobe, CRM, eCommerce and more. Datafacts has a data quality dashboard that gives you the ability to gauge the quality of customer data, identify and resolve data quality issues. Using Datafacts you can better target your audience, increase customer engagement, satisfaction and retention all while reducing costs and improving your campaign effectiveness!