Welcome back to another session of Power to the Power Users! In this series, we explore some common questions that pop up when taking the next step beyond becoming a Marketo Certified Expert and becoming a seasoned architect.
While pretty much every Marketo user uses Smart Lists to some degree, power users often find themselves tasked with creating complex Smart Lists that can take a lot of time to generate—and can cause headaches both in terms of building and waiting for them to select the records you're looking for.
Today's question is:
"What is the logic behind the Smart List filters/trigger orders? How long is too long for a smart list? In other words, how do you know when it’s time to use another channel for what you’re trying to achieve with a Smart List, like segmentation?"
While there is no bulletproof answer to these questions that apply to everyone, all of these questions are variants of the same problem: how do you effectively create Smart Lists? After all, having fast and accurate Smart Lists takes care of filter order, length and using segmentations vs. searches.
On the consulting side of Marketo, we talk a lot about creating effective Smart Lists because we've seen cases where slow Smart Lists can hamper sends or cause Smart Campaigns to take longer than they normally should. However, that doesn't mean these things don't happen to everyday users, too.
In my first couple of years using Marketo, I used to get frustrated because I would have stakeholders ask for very complex lists of people to target email messages to, and the sheer amount of rules would sometimes cause Marketo to time out and for emails not to send at all! To be fair, Marketo has dramatically improved Smart List performance since then, but the pain of needing complex lists won't be changing for companies any time soon. As a Marketo power user, you should be equipped to create efficient lists, too. Let's break down how Smart Lists work to figure out how to optimize them.
Using effective filter choices
Marketo Smart Lists are a graphical frontend for SQL queries, as former Marketo product designer Glen Lipka describes in his archives. As such, it's important to understand that the Smart Lists you create have a lot of the same rules and concepts behind making efficient SQL queries. To start with something straightforward, take a look at this basic Smart List. I've put in one filter for email and set up custom columns for display on my Smart List results:
These two screenshots are powered by one SQL query:
SELECT Id, FullName, Title, Company, Email from `my_marketo_instance.leads` WHERE Email = 'email@example.com';
Marketo simply acts as a graphical interface to make queries more digestible. Therefore, we can translate a lot of the best practices around SQL queries back into Marketo. A big idea from the world of SQL is to make sure, whenever possible, you create queries that are SARGable, or Able to respond to Standard Arguments.
To break this down, think of having a Rolodex full of cards as your list of records. When you put together a Rolodex, you have some sort of way of sorting them in your file. For example, in this image we have Rolodex cards sorted by last name, with "Dietrich, Marlene" as the index or key value of the card. Therefore, if we flip to the D tab and go in alphabetical order, we'll find this card.
The index allows us to quickly search for records if we know what the index is, but only in certain cases. If we say:
- Find the card that says "Dietrich, Marlene" on it
- Find cards that start with "Diet"
- Find cards that are alphabetically before or after "Dietrich"
All we need to know is to find what's on the index and then we can grab all the appropriate cards in one go--whether that's Marlene Dietrich's card, the ones before hers, the ones after hers, or just all the ones near hers that start with "Diet". These are SARGable searches–we have everything we need to determine our data just based on one search.
But what happens if you want to find all the cards that have "rich" somewhere in the index? In order to solve this question, you would first have to grab every card in the Rolodex, then evaluate the text on the index to see if it includes rich, then set it aside. Once you're done, you'd have your list. In other words, you have to pull, then analyze—rather than just pull. If you were going through a Rolodex by hand to do this, this would take longer than the first three examples—and as such, these sorts of searches take longer since they're not SARGable.
When we apply this same concept of SARGable searches to Smart Lists, we find that our filters fall into two categories:
- Value is (single or multiple values)
- Value starts with
- Value is greater than
- Value is less than
- Value is at least
- Value is at most
- Value is in past/in future
- Value is in past before/in future before
- Value is empty
- Value is between two dates/numbers/etc.
- Value is not (single or multiple values)
- Value contains
- Value not starts with
- Value is not empty
- Value does not contain
Therefore, for best performance, you want to rely on filters from the faster set and limit your use from the slower set when possible.
A BIG ASTERISK AROUND SEARCHING FOR EMAIL DOMAINS
The Rolodex analogy of databases breaks down a bit when you realize that all fields have indices. If you wanted to query street addresses, they'd still be indexed in the same way that names are, and the same filter rules apply. However, in a few select cases, there can be multiple indices for one field! This is used in cases where you may want to analyze the field values in a few different ways, as it allows for better performance.
Most notably, when storing email addresses, Marketo makes two indices: one for the overall email address itself, and one for everything after the @ symbol. To access this second index, you can use a filter such as:
Email Address contains @demandlab.com
Using the @ symbol at the start of the contains query tells the system to run a query that's like "Value is" or "Value starts with" on that second index of email domains. By contrast, if you use a filter like:
Email Address contains demandlab.com
Marketo will try to do a non-SARGable query on the first Email Address index and thus take longer.
COMBINE FILTERS INTO SEGMENTATIONS
(or create Marketo indexing fields of your own!)
Segmentations are like filters on steroids. Because they evaluate a Smart List, then assign an index value, they can speed up slower queries into fast, indexed ones. For example, when creating a Smart List to send an email, it's not uncommon to see people create Smart Lists that have:
Email Unsubscribed is false
Email Invalid is false
Email Address is not empty
Black Listed is false
Marketing Suspended is false
However, rather than having multiple filters determine who you can send an email to, consolidating these into one segmentation and pulling
Mailable Segment is Mailable
avoids having to use extra processing time to go through each filter step in your Smart List.
However, it may not always make sense to use segmentation, as they are mutually exclusive, collectively exhaustive–and as such, a record can only fit in one segment at a time. That's fine for discrete items (the record can receive email or it can't, the record is in a given region or it's not, and so on), it gets trickier when people can qualify for multiple things. As an example, think of a field that lists all the products someone has purchased from you. Let's say you have five products:
- ABC Apples
- New ABC Apples
- DEF Exhaust Fluid
- Ghee from GHI
- GHI Bananas
If someone can have any combination of these items, this means that you may be stuck with some fields that say "ABC Apples, DEF Exhaust Fluid, GHI Bananas" and others that say "New ABC Apples, Ghee from GHI, GHI Bananas", or even "Ghee from GHI, ABC Apples". If you're looking to find everyone who has some sort of ABC product for a weekly email send, querying your Product field for "contains ABC" can add up in Smart List processing time costs.
In cases like this, you can speed up processing time by creating a Marketo-only field. Set up a recurring nightly batch to populate/depopulate a boolean field called "Has ABC product(s)" and you'll have a SARGable field to check against for your sends. This solution can help when you have complex queries that you need to reuse over and over without being constrained by the exclusivity of segmentations.
Understanding the foundational blocks of searching with Smart Lists will set you up for success—and speed—when it comes to creating criteria for your campaigns. However, how do you take the ideas presented here and turn them into actionable results? After all, it's pretty rare to only need one or two filters, and anyone who's used Marketo long enough will get asked to create some intricate Smart Lists for qualifying records.
In Part 2, we'll take a look at some real-world examples of criteria and how to best optimize your Smart List—and even show how some filter order, query techniques, and data hygiene can reduce Smart List processing times by 85 percent.
Until then, if you have any questions you'd like us to tackle (or just want to nerd out together about Smart Lists and Marketo performance), feel free to drop us a line by emailing firstname.lastname@example.org or contact us here.
About the AuthorFollow on Linkedin More Content by Courtney Grimes