Inside the Des Moines Public Schools Ransomware Attack: Q&A with Director of Technology, Lisa Irey

Q: Give us an idea for what your critical applications are, what does it take to run a school system, and which applications hurt the most when they go down?

A: Our two biggest application priorities are our ERP system and our student information system. The ERP system is how we pay our employees, how we pay our vendors, and is critical to our business operations. Our student information system houses all the contact information, including emergency contacts, for our students, their individual education plans, and different accommodations that they may have for their education. That system is key for our teachers and support staff in our buildings to be able to serve our students.

Additionally, there are several other suites of services we support every day, like our food service operations and transportation services. We joke that we’re the largest restaurant in the state of Iowa because we serve meals to 30,000 students each day. Our transportation service, as far as routing, dispatching, etc., is crucial, and the information needs to be accurate and available in real-time so that we know where our students are within the district.

Q: Clearly it is a very diverse environment, and very complex with lots of stakeholders, lots of critical applications that can be difficult to manage. Let’s talk about the cyberattack, I know there are certain aspects you can’t speak publicly about, but if you talk about how that day started, how did you initially learn about the attack?

A: Okay, it was January 9th, 2023 — my wife, who was seven months pregnant at the time, was home sick that day. I got a call from one of our network engineers and he said “Hey, just a heads up: we think something might be going on.” My wife suggested I take our new car into work; she said, “You deserve it, it’s going to be a long day for you.” Little did she know …

When I arrived at the office, I met up with our network architect, who was already on site with several of our engineers. They basically confirmed that there was something happening, but no one was quite sure what it was, but as the day progressed, we learned it was not a rinky-dink little thing, it wasn't one of our students trying to pull a prank or get out of finals.

In fact, what we came to learn was that this was a massive, orchestrated, tactically engineered cyberattack against Des Moines Public Schools, one that sent shockwaves through the threat intelligence community due to the size and magnitude of the attack. When we asked the incident response team to give us an idea of how bad this is on a scale from 1-10, they said it was a nine. So, we knew that we were not dealing with something minor, this was extremely sophisticated; these bad actors did this for a living, and it wasn’t just a fly-by-night attack. That makes you feel a little bit better because you’re just racking your brain, trying to figure out how this could have happened, but to know that this was THAT severe of an attack to bring down an entire school district does change the way you feel a little bit.

Q: So, you’ve received a notification from your network administrator that something suspicious is happening … you’re driving in to work in your nice new car, hoping that this issue will be resolved quickly. You arrive at the office, and you just start peeling back the layers of this attack, talk to us a little bit about how that unfolded.

A: We use a team channel for emergencies, which includes various members from my internal team. We affectionately call this the "situation room." In there, we were actively gathering information. I was making phone calls while the team worked diligently to understand the issue, peeling back the layers of the attack. I contacted the interim superintendent, who is my boss, and he’s rallying the executive team to come on site to the building where our IT team is staged. I also called the state CSO, Shane Dwyer, and he arrived on site within hours.

We staged a temporary “war room” in one of our computer labs, which, oddly enough, had no computers in it at the time. My IT team was there on one side of the room, trying to get to the bottom of the situation and on the other side the executive team of Des Moines Public Schools was set up. My job was moving between the two groups, providing hourly updates to the executive team.

It wasn't long before I reached out to our insurance company to start the claim process and start managing this crisis, and that’s exactly what this was, a crisis. Sometimes, you're in the middle of a crisis, but you don’t realize it, but once you start digging deeper, the feeling is just gut-wrenching; your heart sinks, you feel sick to your stomach, your head is pounding — physically, your system feels compromised.

Q: Have you ever seen a situation that called together the executive team and the IT team like this cyberattack did?

A: I would say a little thing called COVID. When the pandemic hit, there was a rapid response at the beginning and as things unfolded there was a more measured response. But with this incident, it was the COVID response, but on steroids. We had no idea how much danger we were in, we didn’t even know the extent of the attack.

It was shaping up as one of the worst days ever, so your mind starts building these worst-case scenarios and you start going to some pretty dark places. I’m thinking, “Am I about to be fired? I’m not going to be able to support my family, we have a baby on the way — and why did I just buy this new car?!”

But to answer the question: no, nothing had ever brought the executive team and IT together quite like this before.

I will say that prior to this attack, we had a very responsive relationship, our IT team to the rest of the organization. For example, when the teachers would implement a new curriculum that required a certain application, they would be far down the implementation path and they would finally come to IT and say, “Oh hey, we’re introducing this new thing, we need you to install it, and support it, and update it and everything.” What this event did for our organization was catapult our IT team from a responsive relationship with the rest of the organization but to a relationship of being right at the front. IT now has representation on the executive cabinet, our voices, our concerns, and our ideas are being weighed at the same level as all the other organizational stakeholders.

This event really impressed upon the entire system how vital IT is to the success of our school district.

Q: How long from the moment when you received notification of the attack and the moment you decided to “pull the plug” and contain everything?

A: It wasn’t very long. Within an hour to 90 minutes from the initial discovery, the decision was made by our network architect to pull the plug. So of course, that means the network goes down, the internet goes down, and all access to any of our services is all gone.

Q: Did the students stay in school the rest of the day, or did they get sent home?

A: The kids did stay in school for the remainder of the day. This happened on a Monday, and we ended up canceling classes for both Tuesday and Wednesday. Thankfully, by Thursday the students were back in school.

Q: Could you tell us a little bit about the process of involving your insurance broker?

A: I made the call to our insurance company, and just like any insurance claim you get a claim number. I started to receive notifications from folks at a security company that our insurance company had contracted as an incident response team. That team was deployed and was tasked with deploying security forces in our environment, conducting a forensic analysis, and finding out what was happening.

It was a tedious process, especially since we had pulled the plug, we weren’t able to just plug everything back in. We had to be very orchestrated, and we relied heavily on Shane Dwyer’s team. His team acted as consultants while the members of our team were the “hands on the keyboards.” We had never dealt with anything like this before, our team is focused on the daily maintenance of the network and the systems our district relies on for daily operations; so, we needed that coaching, that guidance from Shane’s team.

One other key learning point from working with cyber insurance that I will never forget is the absolute necessity to get explicit approval – IN WRITING – before you do anything. We planted a scribe in both our executive team camp and our IT team camp. Their only job was to document every decision that was made, because if legal action is ever required, you need that documentation.

In the midst of all of this, I also put in a call to Heartland Business Systems, and I promise this is not a paid promotion, but HBS has been a partner of ours for many, many years and they were able to help us interpret what the incident response team was asking us to do since, again, these were not the skills we had. HBS was able to say, “Okay, here’s what we need to do, and here’s how we get you up and going again.”

Q: Did you have anybody on your team that was specifically focused on cybersecurity on a day-to-day basis, or was it more of combination of network and server administrators that had knowledge of cybersecurity?

A: Certainly more of the latter. For an organization our size with 24 or so technical people split into service, support, and networking teams, we don’t have the luxury of having a dedicated position for cybersecurity.

Our approach was, and continues to be, that cybersecurity is part of everyone’s job, whatever your position is. We expect our entire team to approach the system they support with security in mind. And that approach worked really well for us. We’ve had several security assessments working with Microsoft, and in 2021 they came in and did a security optimization assessment in our environment and they gave us some really positive feedback, saying, “We don’t see other K-12s doing what you’re doing, implementing what you’re implementing.”

Q: So now that we’ve contained it, you’re starting to bring systems back online, talk a little bit about the recovery, going from ground zero with all networks offline to having kids back in classes three days later.

A: Each of our departments enacted its business continuity plan, and despite not having certain IT applications, they worked to figure out how they could still run their portion of the organization. For example, our facilities personnel were literally employing folks to physically lock and unlock doors for schools every morning and every night.

Our executive cabinet handled everything beautifully. I didn’t have to worry about anything other than what was happening in our IT department, how we were going to navigate this road ahead.

Building from ground zero involved a lot of long days and late nights. We worked 12-16 hours per day for 12 consecutive days, just trying to get the lay of the land, figure out our priorities, and how we were going to move forward. After that 12-day slog, I sent everyone home.

When we came back that next Monday I said, “This is a marathon, not a sprint, we’re not going to burn ourselves out. We have to keep our physical, mental, and emotional aspects in check if we have any chance of getting through this and coming out the other side.”

We shifted from “hair on fire,” working as fast and as hard as we could all day and all night to a more strategic approach. We also shifted from temporary half-measures and fixes to focusing on building a stronger and more resilient environment, one that we can all feel confident in withstanding another attack as severe as this one. To be honest, that was a very difficult conversation to have with our CFO and director of HR and say, “No, I’m sorry, we’re not going to do that. We are implementing a game plan for the long term, otherwise we’re going to die a death of a thousand papercuts.”

All the data tells you that after an attack like we experienced, you are 80% more likely to be re-attacked within six months. I wasn’t going to go through that again — we’re going to build back a better system so that when we turn that switch on, we can be confident that it was secure.

Q: Can you give us an idea of what the attackers were after?

A: Money. And as the state’s largest school district, we are rich in data. Student data is some of the most valuable data on the market because they have no credit histories, they have no histories at all, so their identities are very valuable.

This was a ransomware attack, as we know, and their demand was money. They left their telephone number for us to call — their “customer service” number as they referred to it. The incident response team handled that portion of the attack, dealing with the attackers. There was a specific person from the IR team that was trained to negotiate with bad actors.

The funny thing was, when we called that phone number it was out of order. It was probably two weeks straight that we tried calling to get ahold of them, and their “customer service” website was also offline, so we were joking that maybe the hackers themselves were hacked.

I had never thought that there was an entire community out there dedicated to threat intelligence, and when an FBI agent shows up on the first day I’m thinking, “Am I in trouble?” But this is something that the FBI is tracking constantly, something the US government is looking into. We later learned that this particular threat actor used this attack as their swan song, and their group broke up soon after this attack.

Q: Paint a picture for us, what does the district environment look like now?

A: Our design philosophy has completely changed. It’s not every day that you have the opportunity — to put a silver lining on something like this — but to be able to completely redesign your entire network and build it from the ground up. As many of you know, you start with your network foundation and then as you evolve and grow it becomes like a house with a bunch of additions that weren’t exactly built to code.

What we had to do was search through the rubble of our crumbled house, figure out what was salvageable and then build another structure right next to it with a completely different model. We moved to a segmentation model where we could have more control over how traffic moves in our environment. But, by nature, if you make it more difficult for a threat actor to infiltrate and move around your network, it also makes it more difficult for your own IT administrators to move around in your network. There was a little bit of insult added to injury there after suffering an attack like that, we also made some things 10-times harder to do than they were before.

Another thing we are doing with our environment is offloading risk. If there is an option to move one of our critical application systems to a vendor, we’re taking it. If there isn’t that option, and an application has been deemed a critical application, instead of hosting on-prem in one of our physical environments, we’re moving it to our cloud environment with Microsoft Azure.

Q: If you were to share the three biggest lessons you learned from this experience, what would those be?

A: My biggest message is that the technology part is easy compared to the emotional, human component of navigating a crisis of this scale. You can have the best talent on your team, the best partners, and the best service providers, but the hard part is navigating the grief that comes from experiencing an attack like this. You have to have a culture where you can come together like a family and grieve together and support one another. Everyone is going to bounce between denial, anger, bargaining, and depression — as many different people you have on your team, that is as many different experiences that people are going to have.

If you’re the person leading the IT organization, you’ve got a lot on your shoulders, you have to be willing to roll up your sleeves and do the dirty work with them and for them. You’re caring for humans that are experiencing a traumatic group event.

These bad actors didn’t just infiltrate our networks and compromise our data, they actually breached our spirits and they got into our heads, and some of us are still feeling the repercussions of that.

The second lesson I would pass along is document everything. It isn’t sexy, and it is probably the worst part of any job, but without proper documentation, you are powerless to respond to an event like this. If you don’t know what is in your environment, you don’t know what data is on your server, that renders you even further behind. Just having key contacts for all your applications is key so that you can reach out and connect with them for support.

Lastly, your IT team’s reputation and relationship with the rest of your organization is critical. We were fortunate to have built an already strong relationship with the rest of our organization, so when this happened nobody said, “Well this is just an IT problem,” but, rather, everyone realized that this was everyone’s problem.

We had built a reputation of excellence, so our executive cabinet, who was leading the other factions of the organization, could trust that we were doing what was in the best interest of the entire school district. There was no blame, no finger points, none of that.

Don't wait for a cyberattack to jeopardize your organization's security and reputation. Take proactive steps now. Contact HBS today for a comprehensive cybersecurity assessment and ensure your institution's defenses are prepared for potential threats — because prevention is always better than a cure.