Trifacta: A Tool for the Modern-Day Data Analyst

Challenges of achieving Business Intelligence

More than ever before organizations are utilizing business intelligence to help form strategy and make informed decisions. However, the biggest barrier to achieving actionable business intelligence is often curating clean and organized data, since most BI tools are ineffective when sitting on top of raw data where formats and terminology are different across sources. In the data analytics world, the data preparation and enrichment steps necessary to create clean data have been coined “data wrangling”, which as the term wrangle implies, represents the battle a business user fights to produce usable datasets for exploration, insights, and reporting.

In the past, Data Analysts have typically been responsible for the full end to end process of gathering raw data, turning it into useful information through this process of data wrangling, and then creating business intelligence out of it to deliver to decision makers within their organization. Despite that last step being the most important, it tends not to be where Data Analysts are able to focus their energy. Instead, they spend most of their time wrangling data, which traditionally has been a very manual and complex process that requires deep understanding of the data.

Over time, businesses started to recognize that having Data Analysts bogged down with data wrangling duties was expensive, inefficient, and prevented them from delivering high quality business intelligence. Therefore, there was a natural shift in the data analytics industry to push most of these data collection and data wrangling duties onto IT teams, who build data pipelines to curate the data and push it into BI tools for further analysis and insights by the Data Analyst.

Although this new strategy has its advantages, it also creates a bottleneck with an IT team who normally requires constant direction from business users. Even in the best-case scenario, it can typically takes weeks for IT to develop custom scripts to transform the raw data into the different audience and objective specific structured datasets required by the Data Analysts. Additionally, there are usually constant changes in raw data formats, sources that need to be considered, and functional integration requirements as objectives and data structures evolve. Therefore, these rigid solutions require frequent updating that must be done by IT, requiring Data Analysts to constantly communicate change requests and wait for weeks for updated solutions.

In an ideal world, Data Analysts need to be able to wrangle the data themselves, rather than depending on IT, but without the huge time commitment. Luckily, in the past decade, the development of data wrangling tools, such as Trifacta, have allowed Analysts to once again be in complete control of their data.

Trifacta, a Data Wrangling tool designed for Data Analysts

Featuring a leading-edge interface, powerful machine intelligence, and advanced distributed processing, Trifacta renders the time-consuming, complex, and error-prone process of preparing datasets of any volume into a point-and-click exercise. What used to take weeks in a traditional IT curated data engineering development lifecycle can be done in less than few hours at the Data Analyst’s desk.

Flows and Recipes

In Trifacta, a user will create a “Flow” by first selecting their source datasets, which can be imported from a file or a connected datastore (typically a cloud storage location). They can then create a “recipe” by defining a sequence of

Transformation steps that clean, enrich, and merge source datasets into something new based on their needs. Finally, the user will be able to publish the final dataset in their Flow to an output location.

Above is an example of a simple Flow with a single data source and recipe, and below is an expanded list of the transformation steps in that recipe.

Visual Profiling

When creating recipes, data quality information via histograms is immediately available at the top of each column and can help identify potential issues. Trifacta will even provide recommended transformations to fix the issues identified, and allow the user to see all the rows that would be affected by the change. This Visual Profiling allows Analysts efficiently fix potential problems as they are closest to the data and can determine the right fix as they are moving through the transformation logic.

Sampling

When developing transformations, one limiting factor is often dataset size, because larger datasets can quickly overwhelm the desktop through which they are being viewed.

Trifacta overcomes these desktop limitations by sampling, which means it will only download up to 10MBs of data from each source dataset using either random or filter-based sampling techniques.

This allows for a smooth experience and reduces the overall time required to build recipes since users can quickly and easily see how each transformation step changes their sample of data, without needing to wait for the entire dataset to be processed before moving on to the next step.

Predictive Transformation

Another helpful technique that speeds up development efforts is Trifacta’s  Predictive Transformation feature, which provides context-specific transformation recommendations to the user when they select a piece of data or column through the interface. Trifacta will attempt to find similar matching items and make suggestions about what the user could do next to clean or modify the data, often including several pattern-based variants.

Machine Learning

More impressive still, as a user makes selections within their Flow, Trifacta’s predictions become smarter and will begin to adapt to the types of transformations that the user has employed elsewhere.

These Predictive Transformation and Machine Learning features are especially helpful for when Analysts do not yet have defined business logic for how to prepare the data and are more interested in exploring the dataset to find cleaning procedures.

A Final Evaluation

Trifacta is a powerful tool that empowers users to transform raw data into actionable information. It allows those users that are closest to the data to have the ability manage their own datasets and reduces or completely removes the dependency on IT to prepare data, thereby bypassing many of the common bottlenecks seen in traditional data platform solutions.

However, there are also some challenges with Trifacta that should be addressed.

First, Trifacta is designed to be a data wrangling tool not a full-blown Extract, Transform, and Load (ETL) tool, so while it handles the transformation piece very well there is limited functionality related to extracting data from your raw data sources and loading final prepared datasets into a datastore.

Second, Trifacta is a no-code/low-code solution designed for Analysts, which can be a huge benefit for those users, but at the same time it does not have the same breath of capabilities that a traditional DB programming language has. For example, Data Manipulation Language (DML) statements such as insert, delete, and update, cannot be directly added into Trifacta recipes. Instead, these actions must be achieved using the built-in transformation options, which can often turn something that was done with a single line of SQL code into a multi-step process in a Trifacta flow.

Third, despite the generally intuitive UI, Trifacta should be thought of as almost a new programming language that will need to be mastered before it can be used effectively within an organization, especially when transformation requirements get complicated.

For enterprise level data platforms that will manage many complex ETL pipelines with a lot of intricate transformation logic, Trifacta may not be the best option for an organization. However, Trifacta excels at being leveraged by Data Analysts and other citizen developers to quickly and efficiently deploy data transformation workflows to prepare data for reporting and further analysis in BI tools.

Data wrangling is a tough job any way you slice it, so the best tools and type of resources used to tackle the job will depend on your use case and the objectives you are trying to achieve. Regardless if Trifacta is perfect for your organization, it is clear that these types of data wrangling tools have made a huge impact on the data analytics landscape.