Watch the episode or tune in on Ops Cast to find out what the mistakes were and how (if possible) they were resolved.

 

Transcript:

Episode 9 – Andy Caron

Mike Rizzo: Andy. Thank you so much for getting on to share, an episode of what we fondly refer to as moops TV. I don’t know if that’ll stick forever, but for right now it’s holding true. Um, and we are really excited to learn about. The marketing ops, oops. That you might have encountered in your career, but before we get going, we would love to learn a little bit more about who you are and where you’re currently working.

Obviously we see a nice little background there, so that’s awesome. And then just a little bit about your

Andy Caron: background. Tell us about yourself. My background, not my background, right? So I’m Andy Carone. I am the VP of consulting at revenue Paul, and I’ve been there for about two and a half years now. Uh, I started out actually more on the sales ops side of things, uh, own Salesforce back in the day, uh, and became a marketing automation.

And then Marketo a customer in 2012. Okay. Uh, yeah,

Mike Rizzo: so you can on the sales side of things, which is great.

Andy Caron: I started out in Salesforce and then you got to own both systems at my first company, which was, I was so spoiled. Yeah. Yeah, for sure.

Mike Rizzo: Yeah. I was going to say arguably spoiled. Right? Uh, we don’t always, we don’t always like on both systems, but sometimes it’s helpful.

So, where were you in that track record of, you know, in your tenure of being in kind of a sales ops, marketing ops role? Um, what point in your career did, did this mistake? You’re going to share with us, uh, happen.

Andy Caron: This particular error occurred, um, several years in probably would have been after I’d been in Marketo.

Easily five years at that point, um, I owned a system that was leveraged globally, uh, by multiple teams, a lot of end users, a lot of casual users, um, and despite many discussions about a potential center of excellence model, the best that I was able to manage was a. Loosely police system with some approval processes in place, and a lot of programs templating to get people to follow.

Just say, ABCB staff within the system on deploying their programs.

Mike Rizzo: Gotcha. Gotcha. Okay. So, so this, so this environment, uh, lack of controls, you know, people can get in there. Um, and to sounds like you, you tried to get some organization around it struggled a little bit. Um, so what was it that, that the system was trying to do for you or this program that was in place?

Like what was trying to be accomplished?

Andy Caron: Yeah. So in participation with one of the marketing teams, that was not a common user. Of Marketo. They were actually more on the partnership side. There was a monthly newsletter that needed to go out and there was a lot of back and forth on how the internal teams would know that that email had gone.

They want. I see it, they, they essentially wanted to be participative in it, but when you’re sending a newsletter email and trying to gauge engagement, and then you’re sending to a bunch of internal stakeholders that well, that’s going to, you know, massively in place, uh, your, your potential, right? So. Yeah.

And so this, like, I want to TC, I want to BCC sort of conversation was, was very, um, uh, debated. And of course at that time, that functionality did not yet exist in any incarnation inside of Marketo. So it was, we can send you the test, we can send you, you know, something. Um, but we recommend not having you be part of the audience for this particular email deployment.

Mike Rizzo: Totally. I, you know what, that, even in my most recent roles, we would just copy paste the email into like a G suite, right. And just hit send, or like send a test email to one person and that person would forward it internally

Andy Caron: or publish, you know, the code out and then put that on a page where people could go play with it or whatever.

Right. There’s like a bunch of different options, but they were just dead set. Like we want to get the email at the same time, same experience, like we want it. And so. In an effort to be accommodating the individual who was scheduling this email to go out, had member of lists. Right? So there’s my Smartlist as a member of smart lists, um, and three footsteps, which was.

Send email change program status, send alert now in your brain, if you don’t really drink things through it, you’re thinking, okay. Send alert. That’s going to send an email to an individual. Cool like this, basically faking a BCC and in a scenario where maybe you’re doing that with a demo form request or something like that, that might actually work.

Right. Potentially. I could see that being. Something you might do. Right. The challenge with this particular setup was that because it was a batch that you never wanted to schedule an alert against, or put that in your flow, right? Like that’s probably not a great idea. Um, and also that of the three emails contained in the alert sun, two, one was an internal distribution list.

Mike Rizzo: Oh,

Andy Caron: yeah. So what that meant was some quiet, rainy Thursday morning, um, that, uh, hit it scheduled batch time. And I was opening up my email, just getting to my desk and went, oh, and then my phone rang and I went triple up. Uh, and next thing I go in. And it’s play with stop the execution of that batch. Like there’s no calling it back.

These are out the door, like it’s a done deal. So yeah. So the output of this was that for every individual who received the newsletter and alert of that same newsletter was also sent to the individuals in the. Alert plus step, because one of those was a distribution list that went over 2000 employees.

And because the sun list for the newsletter was 8,434 people. We ended up with this scenario where the company’s server got shut down because over 2000 people got the same email over 8,000 times.

No, I know plenty of choice words actually came out of my mouth at that point, but also you could see the intention you could see. Someone trying to do something in this system to accommodate a request that never should have been made of that system, that there were other ways to account for.

Mike Rizzo: So who, who on the light? You said you got a phone call who actually noticed it first, was it you or

Andy Caron: someone? I was noticing it as someone else was calling me to say, why is my inbox slowing up right now? And then very quickly, you know, it was slipped in. Everybody was looped in, um, you know, the CEO of this company was on that distribution list, right?

Like this was front and center. No way to do anything other than be like, Hey guys, this is a very shameless to me, but I don’t care. Hey guys, this is why we don’t do CC or BCC on emails going out in Marketo. We can send you. Yeah, that’s all hard

Mike Rizzo: lesson learned. That’s for sure.

Andy Caron: I will say it had the outcome of definitely stopping any future requests to be CC’d on the marketing.

Um, not necessarily the way I would have liked to have gotten to that resolution point on that particular, uh, topic, but, um, effective.

Mike Rizzo: Yeah. Wow. I, well, Hey, you got there, right? So, so in terms of remedying this, like, is it at this point it’s just a fall on your sword moment, right? Where you go to the team or the stakeholders that received all this stuff. Do you, first of all, your servers are shut down. So you’ve got to boot that back up and then what you send

Andy Caron: an apology.

The inbox is, has to be cleaned out, right? Like there’s a whole process of remediation organizationally that has nothing to do with Marketo or w or even marketing at that point. Um, and it’s essentially a process of. Thing, we know why this happens and it will not happen again. And these are the measures that we’re taking to ensure that this does not occur, right.

Ownership of the newsletter and execution of that newsletter shifted. And then an approval processes went into place that had not been there previously. That said, there’s now two different levels of people that need to look at this from both an email approval, which is already in place. Um, and in automation approval.

Mike Rizzo: Right. Yeah, that makes a ton of sense. So, so during this remediation process, just unpacking that for just a second. Um, we’re like who got involved cause this, this, like you said, it falls outside of justice system now. So did you end up calling a handful of meetings real quickly with, was it you or just like some other team members?

Like how did that?

Andy Caron: So I had my, um, The direct manager who owned marketing operations as a whole schedule with teams to fortnight on that end, so that I could focus on doing what I need to do on my end, which was wonderful. Um, and we did sit down and basically have a post-mortem about what had happened, why it had happened and how we would avoid it future state.

But at the end of the day, it really was. Someone trying to accommodate a request that should never have been accommodated in the first place. Right? Like this is not. Yeah, 2000 people inside the company probably don’t need to get the partner newsletter actually. Totally. Right. That that’s probably not something to be putting in their inbox either.

Um, either they’re not going to digest it or it’s eating up inbox space or their time, or what have you. Right. Like,

Mike Rizzo: and, and deliverability and like, um, sender, credibility, right? Like your, your domains, like, uh, Right. Is, is tremendously important. So like, if you just keep sending people emails that they never opened, like,

Andy Caron: right, right.

And I mean, this is an alert, so it was a little bit, but it’s still, I mean, it’s yet again, proof point of why we didn’t want to be sending those emails as part of the newsletter blast. Right. Right.

Mike Rizzo: So, so the, you put in a process, you guys put in a new checks and balances. Um, and I assume obviously that was taken, never happened again in the future. Um, so

Andy Caron: that’s not on my watch anyway,

Mike Rizzo: I’ve washed my hands of this now,

Andy Caron: so that. Well,

Mike Rizzo: cool. So, all right. So final parting thoughts. What would you say to someone who’s made a similar mistake is maybe about to make a similar steak or just like, kind of broadly, like your advice for marketing and revenue, operations professionals out there around, you know, avoiding the moves moment.

Like. Yeah.

Andy Caron: Um, you know, at some point in your career, someone’s going to ask you to do something and you’re going to be rushing, or you’re going to be trying to be accommodating, or you’re gonna you’re, you’re going to be going out of your way to do something that maybe isn’t something that you would normally.

Do whether it’s because you’re rushed, whether it’s because you’re new in a role, whatever that might be new boss, you’re trying to, you know, impress. Um, and I think it’s just a, it’s imperative to have the pieces during your career, because that’s how we learn that till we grow trying new things and failing at least means that you’re actually growing as an individual.

Right. If you just, okay. Stick on the straight and narrow and you never make a mistake, then that means you never tried anything either, in my opinion. Um, so sometimes, you know, you fail, but if you’re trying something new and out of the box, maybe try it with a smaller audience first or not a distribution list, or what have you to see how that plays out right test before you deploy, especially new things or things that are a little bit different, but also be prepared to own your mistakes.

Yeah, right to go in front of your boss or your boss’s boss or your boss’s boss’s boss and be like, these things occasionally do happen in systems like this, because they need to be this powerful to execute and do the job they’re intended for. This proves point a B or C potentially of something of an advocating for in my case center of excellence, right.

This reinforced that recommendation. Right. Unfortunate way to have that occur, but it did. Right. So let’s talk about not just from a system or user perspective on how we can ensure this doesn’t occur again, but if need be organizationally to support that this isn’t even a potential. Right.

Mike Rizzo: Yep, absolutely.

Get it outside. And those conversations are tremendous, right. Getting outside of the tool itself and it’s kind of capabilities, whether they’re limited or robust. Um, it’s taking it beyond that and saying, how do we ensure that across the organization we’re aligned. So that in your example, That request.

Everyone understands that that request isn’t really a sound request. And here’s why right. And create that, that alignment across

Andy Caron: the markets. Exactly. And I think the other thing there is, there’s a lot of power in being able to say to your boss, I know what happened. I know what happened and it’ll never happen again.

Right. Being proactive about coming to them with, Hey, this thing just went down and you need to know about it. Um, has, uh, it’s invaluable, right? If you wait to be called into the principal’s office and you’re all Towery in there, which is the natural state, right? Like we don’t really want to necessarily be in the spotlight on things in general and certainly not like this.

Right. But, but at the same time, back to life, it happened, we can’t go back and change. So what can we do? Um, yeah,

Mike Rizzo: absolutely. No, all, all great advice. And I think, um, the more professional we are about handling, excuse me, those situations, the better. Right. And it just, um, we all want to crawl under a rock when stuff like that happens, but that the reality is is that hopefully.

With a boss or have a manager who’s either already managed someone like you who’s probably made mistakes too, or has literally been in your shoes. So, or you just sit like, you know, say, I promise this happens all the time. Shameless plug, go watch movies, TV. Right. This is

Andy Caron: to you literally that this is so common.

A online segment about it. Here you go. Yeah.

Mike Rizzo: Anyway. Well, Andy, thank you so much for getting on and sharing your moves moment with us. Uh, and, you know, thank you for letting us learn from you. And hopefully others who are watching this will, um, create a center of excellence and maybe avoid those costs of mistakes.

So really appreciate it.

Andy Caron: Thanks for having me.