Watch the episode or tune in on Ops Cast to find out what the mistakes were and how (if possible) they were resolved.

 

Transcript:

Episode 8 – Weisi Kang

Hi, everyone. Thanks so much for joining us for another episode of moves TV. I’m joined today by Weisi Kang. Okay. Um, I’m excited for you all to hear from her. Uh, without further ado, let’s go ahead and just jump right in and Weisi. If you wanted to start with some introductions, uh, who are you and where do you currently work?

Awesome. Hi, I’m Weisi. Um, I currently work in marketing operations in-house, um, and I focus specifically on data and reporting. So all the ways that data come in and how it gets transformed, it’s how, it’s, how it’s used specifically for marketing team reporting. Awesome. Um, a little bit more about your background.

How long have you been in the marketing revenue ops space? I’ve been in marketing operations for almost eight years. Um, it feels both like a lot of time and not a lot of time. So, uh, yeah. Awesome. Eight years, uh, going off script here, but probably enough time to work on some of the. Tools before, like the big four right now.

So like pre Marketo, Eloqua, Pardot, HubSpot, yes or no?

No, no. I started on Marketo and I’ve consistently been on Marketo ever said, so we’ll see, we’ll see what our fit. Perfect. I was going to say you bleed purple, but I can’t really say that. Uh, 10 months when it’s no longer purple.

At what point in your career did the mistake we’re going to chat about today happened. Um, so the moves I made was very, very early on in my career. I was probably about a month into, um, my role there, um, at the time. And it was also about a month into just marketing operations more generally. So it was a very like, uh, startling and traumatic way to get started.

Um, but definitely a lot of learning. So, uh, I love it. Um, nothing like something. Kind of horrifying right at the beginning of a career transition to make you question like your whole reality. Yup. Awesome. So kind of getting into the meat, um, what were you trying to do? So, um, I was trying to set up an AB tests, um, for an email.

And this was right around the time, or maybe right before the time, um, that email send programs could do AB tests on their own. Like you just said it in the control panel, basically. Um, I was doing it as a Marketo and so there were a lot of articles already out that were teaching you basically how to set up, um, an AB test using different smart campaign flows and setting up two different emails, all that.

So, um, I was basically trying to break up our email, send lists into, to kind of randomize static lists to use that in the email send, um, smart campaign later on. Awesome. So you’ve already answered my next question, which is what system. Um, so Marketo, and I love the nuggets that it was like pre native AB test functionality, um, and having to do it all via smart campaigns.

Cause I have some ideas of where this was going based on that. Um, and I’m really curious to see if I’m right. Um, did you have to do, I guess for my own like random sample in the. List split campaign or could they random sample in the email side at that point? Or why were you trying to break them into lists?

Um, I don’t remember, uh, why we wanted to just like break them up into two is I think it was probably just like. The like extra caution of seeing the two different lists with two different numbers of people and, you know, half of them go here and half of them go here. Um, and just to double check that, I’m pretty sure.

Um, and so, uh, you know, at the time, like neither of us trusted random sample, we didn’t know what that was. You know, we didn’t know that, you know, maybe at the time, uh, AB testing and email programs, was a thing, but like not too many people were using it. Um, and so we were just like, okay, this is our first like big AB test.

Like let’s just, let’s just be extra careful and try to split things up visually. Perfect. Yeah. I’ve I think I’ve seen work with some clients who, the same thing. They like to see the number or the count, or, you know, have a good sense of kind of what’s happening before they hit the button. Um, yep. So you get in your smart campaign and you split everyone up until this year, you’re doing your AB test.

What exactly happened? Yeah. So, um, we were basically, so again, month into the job, um, my manager at the time, he, um, knows who he is. He was, um, going to, uh, Tahoe, which at the time, I guess I’d really bad or wherever he was going, had really bad internet. So, um, you know, in order to check that this smart campaign.

Ranch, just like these people up into the list. Um, he graciously was able to log on while he was driving up there or maybe tethered to his phone or something like that. So came back, looked at the logic, you know, was looking at okay, random sample, 50% go to this list, the rest, go to this other. And ironically, we were looking at it.

We’re like, wow, we did a really good job. Like, so detail oriented, like very, um, uh, you know, set up a lot of like different types of logic than either of us had done before. It was great. Um, and so from there we were like, okay, this looks good. Let’s just run it. Um, the only thing that we were trying to do was split.

This list into two lists. Um, what neither of us remembered, uh, or what we both forgot was that I’m adding someone to a list at the time in that system was a really big deal. It was like one of those, like our target account list situations. Um, if it wasn’t named. It would basically send out an email alert to, um, uh, you know, people on sales, ops, people on marketing ops, people like the head of marketing at the time.

Um, and so because this list wasn’t like any other lists that we had ever uploaded or ever attempted to add people to. Um, our Marketo instance thought that this was a big deal and started firing off email alerts for every single person that was added to a list. Oh. And, uh, it was, it was very, very chaotic.

How many people were on your list or list was, uh, yeah, it was, uh, the total list was about 60,000 people. And so it was 60,000 email alerts, basically going out to this list of people. Um,

I’ve heard some alert, horror stories, um, 60,000 might be the new record number of internal that I step on.

My next question is going to notice it. And it sounds like pretty fun. Check up, um, noticed it. Yeah. So basically I think it’s funny, basically everyone except for me and my manager. So because I was blissfully unaware because I was not on email alerts. I had no idea what I was doing. I had no idea that this had just set off a chain reaction of things.

Um, my manager was probably without email turned off his email. What have you? And so maybe like 30 or 40 minutes later, I had a sales ops colleague walk over to my desk and she goes, Hey, um, You know, I’m getting all of these emails from Marketo like, is this, did you guys like send something today? Like, I, we didn’t really know what was going on.

And I was like, that’s weird. Like there were no emails sentence for today. Like, I dunno what that is. Like maybe, maybe let’s contact Marketo support. I don’t know what’s going on. Um, so then finally, maybe like 10, 15 minutes later, it comes back to my desk and it’s like, Hey. These emails are coming from Marketo.

Can you please look into this and like, figure out what’s going on? And then suddenly you can hear the head of marketing from her office is like, why am I getting so many email alerts? I can’t even access my inbox anymore because more email alerts just start coming. Um, and so no. So I go from blissfully unaware to, uh, like sinking feeling in your chest and in the pit of your stomach, just like.

What just happened with a manager who sounds like he’s out of pocket at this point. Yeah. Oh man. I feel for you. Um, so it sounds like obviously at this point, the situation is pretty public, right? It involves, uh, sales ops is involved. The head of marketing is involved. Um, You’re maybe realizing at this point that it wasn’t, it Marketo support related, but, um, what kind of steps did you take at this point to remedy and diagnose what had actually happened?

Yeah, so, um, I, what I ended up doing, I didn’t know what to do. I ended up just calling my manager. I’m like, this feels like an emergency that like I can call you for. And so he basically, uh, took one, look at the email, knew immediately what was happening because. You know, obviously in the system for awhile, he knew immediately what was happening.

And just like in the, I will never forget in the calmest voice. And the calmness demeanor was like, okay, like we let’s pull the kill switch, we’ll figure it out. Like, he’s probably like, you know, just got to his cabin and Tahoes, like tethered to his phone, trying to, you know, uh, stop the, stop, the campaign, aboard the campaign, do all of this stuff.

Um, and. I cannot even express to you in the most calm manner ever walked me through the different ways. You know, here’s how you aboard a campaign. This is, this is why this happened. This is what we need to look for. This is actually what this alert is for. So let’s turn this off too, and do all of these.

Um, and was gracious enough to basically come out of PTO to help, um, kind of resolve this issue and explained to me along the way, like this is exactly what happened, um, and how we could prevent it in the future. Um, love a level head in what feels like a panic moment at that point. Um, yeah, I think that’s a really impressive, impressive skillset.

Um, we’re kind of had a board campaign eight years ago. I think so. Yeah. I remember, um, it was either, either they had it or it was like a treasure chest type of thing. Um, but yeah, he was like, th this is, this is the kill switch. Like, this is what we do. Okay. Um, I love it. I think it, it sounds like he was able to a step in and help a ton, but also be really educational while doing it.

I think that. Awesome manager. So you’ve got the kill switch. You stop the alerts at this point. Um, but it sounds like a decent number had gone out. Um, what did you guys, or what did you do at some point to acknowledge with the team that had received the alert? Yeah. So from that point forward, we were just kind of evaluating like, okay, are these email alerts actually necessary to be, you know, do we need to be alerted every time someone gets added to this type of list?

Um, you know, what do we want to do there? And I think, um, just simplifying a lot of those email alerts more broadly was, uh, was great for us. Um, and I think. Eh, you know, moving forward, you know, as we work together to, um, you know, revamp, lead funnel and things like that, and people wanted all different types of alerts, you know, stopping to say like, Hey, do, do we actually need this?

Or can you run a report? Do we actually need an alert every single time come one someone comes in or can you refer to something else? Um, so it just really, really helped us pause and, and be able to, uh, to think about, you know, It’s something actually necessary or can we, can we figure out a different way to do it?

Um, uh, yeah, I love that. I think that’s such a powerful learning, um, alerts seem like an easy default for a lot of things, but I think being able to take a step back, having had that experience and asking. Stakeholders. Like, what are you actually trying to get out of this? Um, can I deliver that in a more succinct manner is a really good, uh, hallmark of a really good ops person in my opinion.

Yeah. And it turns out people don’t like reading email alerts, like they’ll read the first couple and then afterwards it’s like, oh, great. Another email alert. Like, did we need that? Probably not. Surprise. Everyone already has too much email. Um, and they just get lost in the shuffle. Um, I never would’ve guessed it.

Uh, so obviously some learnings around, uh, better ways to provide information to stakeholders. Um, anything else you learned in the remediation process that you want to share? Yeah. Um, I think just the idea of like a campaign execution. Some sort of abort process or anything like that. Um, it’s something that you can just turn off to prevent any sort of like future damage that I do.

Um, but also I think just, uh, I become ever so sensitive to, um, you know, fields that are used by certain processes or what, what actions will kick off other processes. And so even to this day, you know, We’re running a process that I know very well, but okay. Let’s just double check. Where’s this field being used, what sort of campaign is running?

What’s going to be the next few steps. How can I make sure that this is actually what we want to do? Um, and so, you know, in that situation, maybe add to lists, I would have been a little bit harder to figure out like where exactly add to list would be. Um, but just, you know, uh, coming into new Marketo instances, thinking about, you know, okay, let’s just poke around in each of the operational folders.

Let’s see what’s going on. Let’s, you know, follow a typical lead and see, you know, what values change, what lists they get ads added to. Um, and just make sure that you have a really clear understanding of what’s going on in the background, so that if something does happen, you can kind of backtrack to where and why and figure it out that way versus kind of just being like, I don’t know what’s going on.

I love that. I think. My team thinks I’m crazy because I mean, I’ve been in the current instance I’m in for a year now, but even if I’m doing something in that instance that I haven’t done before, I’m like, let’s just like, let’s do five people. Let’s do 10 people. Like that’s yeah. It’s going to be a larger batch type of thing.

You know, I think recently we had a kind of import situation where I was like, okay, Here’s our cleanup steps. Let’s try it with, you know, 1% of the list and then something that all goes well, we can kind of go from there. So I’m a big fan of like, obviously understanding the incidents and documenting and under having a good idea of what kicks off, what but something you’re doing for the first time.

Uh, running a test batch.

So, uh, like your somebodies fairy, godmother or mentor, and they, they make a similar mistake. Um, what advice would you give to them? Uh, they call you out on the phone and kind of same thing. What’s, what’s your advice there? Um, you know, the, the manager that I had and, uh, I stress that he was so calm in this situation because I think in a situation like that, the only.

Sometimes, the only thing you can do is just to take a deep breath and kind of use the like adrenaline or whatever is rushing through your body at the time to kind of make very, uh, like clear and deliberate choices. Like. To either, you know, be calm about the situation. He, he could have also been angry that, you know, neither of us had thought to look up something like this.

Um, but instead he chose to be very level-headed and just to be very calm about the situation, I think that ultimately passes on to, um, to other people who maybe are feeling, you know, differently about the situation. Keeps an even temper in the room. Um, and ultimately I think, you know, being a little bit more, even tempered, especially in an emergency situation makes for a little bit better.

Decision-making um, you know, making very deliberate choices, you know, doing one step instead of five to get something remediated. Um, so that’s, that’s definitely really, really important. I love. I, uh, if those of you watching, haven’t watched Mike’s episode, he panic reacted to a MOOCs and cause another mopes, um, in his episode sent out the wrong email and then the oops email had the wrong link.

So I, I think like, as terrifying as it sounds though, and then like trying to be calm can really help this situation. Um, Last question here. Um, any other advice kind of mistake related or otherwise that you have for your fellow marketing and revenue ops professionals? Um, I think, uh, you know, this serious has generally just made me think more about all of the things.

Um, it made me appreciate a lot of the things that we do and a lot of the things, things that we can automate, but yeah. So kind of highlights a lot of places where, um, there are potentially unintended consequences. And so I’m thinking about ways to alert ourselves or run reports to give ourselves indications of when something could be going wrong before someone points it out to you or, um, you know, reports or alerts that help you prevent these kinds of issues in the future.

Um, Obviously things happen, stuff happens all the time and all of our systems. Um, but anything that we can do to kind of stay ahead of it, I’ll just, just a little bit even. Um, it’s definitely useful. So I, yeah, I, I have a handful of those in my system too, though. Like if something mismatched here type of reports on alerts, So I try and check, um, so that somebody else isn’t the one pointing it out.

Um, I think that’s a really good advice. Piece of advice. Yeah. Awesome. Well, anything else you want to share with our audience? Um, this is kind of like a tangent to the. Uh, that day. Um, but that day, right, it was either right before, or right after kind of the whole situation was finally like, you know, things had settled, email alerts stopped coming in, um, all of that stuff.

Um,

Oh, maybe like 30 minutes to an hour. It was just like, they just kept coming in. That’s honestly, the last time that I thought you were going to say, you know, it, it might’ve been longer itself longer. Maybe it was, maybe it was. Um, but the, uh, kind of just in, in the mid slash at the end of all of that chaos, it happened to him.

The, um, my, my near my birthday. And so I had a lot of coworkers come to my desk with a cake. I’d like to surprise me, like, oh, happy birthday. As I’m in this middle of this, like meltdown of like, am I going to lose my job? Like, everyone’s so mad at me. I don’t understand what’s going on. And like, suddenly there’s like sensory overload.

Wow. Yeah. It’s one of those days you go home with the item, you just are like, I’m going to bed now. I’m done. Uh, well, wow. Uh, I feel like we went on a journey with that one was very chaotic. Uh, yeah. I’m I, uh, I’ve been there not to the, uh, 60,000 mine was like a couple thousand. Um, and I know some other, we’ve got some other recorded weapons and some people have shared similar stories with me.

So, uh, This is, I think probably one of the most common scenarios I’ve seen is like, obviously the email you said is totally messed up is a big one, but the eye in intent on an advertently alerted a whole bunch of people internally is, uh, yeah, you’re in good company. Oh my gosh. Well, thank you so much for coming on in and chatting with us.

I’m uh, I learned a lot. I had a great time and chatting with you and appreciate it. Of course, this is so fun.