Our IT Provider's Remote Management Tool Was Hacked and What We Do Now

On Tuesday, November 10, a software company called N-able shipped an emergency fix for N-central, a tool that many outside IT companies use to look after their clients' computers. By Friday morning, a security firm named Huntback reported that attackers had taken over 21 N-central servers anyway. Nineteen of those 21 were already running the fix N-able had just told everyone to install.

If you own or run a business, you have probably never heard of N-central, and you should not have to. But if your IT company uses it, that one server is effectively the master key to every computer in your office. This post is written for the client side of that relationship. It is for the owner, office manager, or controller in McKinney, Plano, or Frisco who pays an outside IT company every month and wants to know three things: what to ask on Monday morning, what a good answer sounds like, and what happens if the answer is bad.

This is a response post, not a prevention post. The attack is already happening to real servers, so the useful question is not how to avoid it in theory. It is how to find out whether it reached you, and what to do in the right order if it did. The full technical detail is in our N-central advisory from November 13.

What a Remote Management Tool Is and Why It Matters to You

Most outside IT companies, often called MSPs, short for managed service providers, do not drive to your office every time a computer needs an update. They use a remote monitoring and management tool, usually shortened to RMM. An RMM is a central console that lets your IT provider install updates, run scripts, and open a remote session on any computer they look after, from wherever they are. A small program called an agent sits on each of your computers and takes instructions from that console.

That arrangement is efficient, and it is the reason your provider can patch fifty laptops overnight without anyone noticing. It is also the reason this week matters. The same channel that installs your Monday updates will run whatever its operator tells it to run, and every computer on the other end is set up to trust it without question. Whoever controls the console controls the agents. In the words of our advisory, taking over the N-central server puts an attacker one step from full control of every client machine behind it.

Earlier N-central attacks show what that looks like for a client business. In August, a security firm called Huntress investigated one self-hosted N-central server through which nine separate organizations were reachable, and the attackers touched one computer at each of them. Also in August, Microsoft tied attacks on earlier N-central flaws to a criminal group it calls Storm-1175, which used compromised N-central servers to push ransomware onto the computers those servers managed. Ransomware is software that locks your files and demands payment to unlock them. For a client business, that means downtime you did not cause, through a tool you did not choose, on a schedule set by someone else.

So far, the attacker behind this week's wave has used the stolen servers to run a cryptominer, which is a program that quietly uses a computer's processing power to generate cryptocurrency for the attacker. That sounds mild, but the same attacker already has the highest level of access on those servers and has been collecting information about the machines. Huntback said any confirmed hit should be treated as "a full compromise, not a cleanup job."

The Questions to Ask Your IT Provider Today

You do not need to understand how the attack works to ask good questions about it. You need four answers, and you are entitled to all of them in writing.

The first question is simple. Do you use N-central to manage our computers, and if so, is it hosted by N-able or running on a server you operate yourselves? This matters because N-able has said that customers on its hosted version are not affected by the server-side attack. The servers taken over this week were ones that IT companies run on their own. If your provider does not use N-central at all, that is a fine answer, and you can still ask what remote management tool they do use and how they track emergency fixes for it.

The second question is about something called the Legacy API. An API, short for application programming interface, is a door that lets other software talk to a program automatically. The Legacy API is an older version of that door in N-central, kept switched on in some setups because an old integration still depends on it. The attack this week came in through that door. N-able says it is off by default and not recommended, and it has urged every customer running N-central on its own servers to confirm it is disabled. Ask your provider to confirm it is off, or, if something still needs it, that it is blocked from the open internet.

The third question is whether they have looked for signs of the attack since November 8. Our advisory lists what to search for. That includes any connection to or from the attacker's internet address, which is the numeric location a computer uses on the internet, published as 157.20.241[.]16. It also includes any administrator accounts, scripts, scheduled tasks, or remote sessions that nobody on the provider's own team created. You are not asking whether they think they are fine. You are asking whether they looked, when they looked, and what they found.

The fourth question is what they will tell you if they find something, and how fast. You want a name, a phone number, and a commitment to notify you the same day.

A good answer is specific. It names the version, the date the fix went in, the Legacy API setting, and the results of a search that has already been done. A vague answer, such as "we are monitoring the situation" or "we are fully patched," is not necessarily a sign of a problem, but it is not an answer to the question you asked. Ask again, politely, and ask for the specifics by a set time.

Why "We Installed the Patch" Is Not the Answer

A patch, sometimes called a hotfix when it is rushed out for an emergency, is a software update that closes a known hole. It is necessary, but this week shows it is not sufficient.

N-able shipped its second emergency hotfix on Tuesday to close the flaws a researcher had published over the weekend. By Thursday, that same researcher had released new attacks that worked against the hotfix itself. Nineteen of the 21 servers taken over were running it. Huntress also warned that a server with the fix applied may not show a new version number, so even "we checked the version" needs to be measured against N-able's own guidance, not the number on the screen.

Our advisory notes that this is the fourth emergency response N-able has run in about 15 months. In August, N-able found that its first fix for an earlier flaw was incomplete, and that customers who had patched on the original guidance were still exposed. In September, the company shipped four hotfixes in five weeks, and when Huntress found a compromised N-central server in the middle of that run, the logs had already been overwritten, so nobody could say for certain which flaw the attacker had used to get in.

There are two business lessons in that history. First, a patch closes the door, but it does not tell you who already walked through it. If an attacker got in before the fix, the fix does nothing to remove them. Second, the evidence that would tell you what happened is perishable. Server logs are routinely overwritten after a set period, and once they are gone, the question of whether your company was reached may never get a clean answer. That uncertainty is expensive. It affects what you tell your clients, what your insurer will accept, and whether you have to treat every computer as suspect. We covered this same gap from the patching side in we installed the security update and whether we were already hacked.

If They Find Something, the Order Matters

If your provider's search turns up anything unexplained, the order of the next steps decides how much the incident costs you. The instinct in most offices is to clean up fast and get back to work. In an RMM compromise, that instinct destroys the evidence you will need later.

The first step is to preserve logs and a snapshot before anyone touches anything. A snapshot is a frozen copy of the server exactly as it is right now, including whatever the attacker left behind. Preserving it is what lets a forensic team, meaning investigators who reconstruct what happened from the digital record, answer the questions that carry legal and financial weight. Which client networks did the attacker reach? When did access start? Was data taken, or only computing power? Those answers drive whether you owe anyone a notice, what your insurer will cover, and how much of your network has to be rebuilt.

The second step is to rebuild the server rather than clean it. Cleaning means removing the parts of the attack you can see. Rebuilding means starting from a known-good installation and restoring only what you trust. When an attacker has had the highest level of access, you cannot prove you found everything, so the advisory's guidance is to rebuild. That depends on having good copies to rebuild from, which is why tested data backups that live outside the reach of the management tool matter so much in this kind of event.

The third step is to change every key the attacker may have seen. The advisory names administrator and technician passwords, agent registration tokens, which are the codes that let a new computer join the management tool, and integration keys, which are the passwords other software uses to connect to it. If the attacker copied any of these, rebuilding the server without changing them simply hands the new server back.

The fourth step is to check every computer the server manages, including yours. This is where client businesses get hurt. In the August attacks, the intruders set up hidden tunnels, which are quiet outbound connections back to the attacker, on the computers they reached, and those tunnels kept working after the attackers lost access to the N-central console itself. In September, investigators found fake console accounts created with the odd ending ".invalid" on the email address. Shutting the front door does not evict someone who has already built a back door into your office. Your provider should tell you, in writing, what they checked on your computers and not only on their own server.

Who Owns Which Step When Your IT Provider and a Security Team Share the Response

Here is the awkward part. When the compromised tool belongs to your IT provider, the company you rely on to fix the problem is also the company whose system was the way in. That is not an accusation, and many providers handle these events well. It is a conflict of roles, and it is easier to manage on paper.

The arrangement where an outside security team works alongside your existing IT provider is called co-managed IT. In an RMM compromise, a sensible split looks like this. Your IT provider owns the facts about its own server: the version, the Legacy API setting, the account list, the logs, and the list of client networks that server manages. It also owns the rebuild and the credential changes, because it holds the keys and knows the setup. The outside security team owns preservation of the evidence, the search across your own computers, and an independent check of what the provider reports. It produces the written record you will hand to your insurer, your lawyer, or a client who asks what happened.

You own the decisions that only a business owner can make. Do you accept a few hours of downtime to disconnect the management agents while the search runs? When do you call your insurer? Do you bring in legal counsel? What do you tell your own clients, and when?

The time to write that split down is before you need it. If you already have a co-managed arrangement, check whether it says who leads when the provider's own tools are involved. If you do not, our earlier post on who acts at 2 AM in a co-managed arrangement walks through the general boundary, and this week is a good reason to put a specific line in for remote management tools.

Getting Help From a Team in McKinney

Innovation Network Design is based in McKinney and works with businesses across Collin County and the rest of North Texas. If your IT provider uses N-central and cannot give you specific answers to the questions above, or if their search has found something they cannot explain, we can preserve the evidence, check your computers independently, and work alongside your provider rather than replacing them.

If you are an IT provider reading this and you run N-central for your own clients, we also partner with MSPs on exactly this kind of response.

You can call us directly at 512-518-4408, send a note through our contact page, or start with a security assessment if you want a clear picture of what your provider's tools can reach before the next emergency fix ships.

Frequently Asked Questions

What is an RMM tool and does my business use one?

An RMM, short for remote monitoring and management tool, is the console an IT provider uses to update, monitor, and remotely control the computers it looks after. If an outside IT company manages your computers, they almost certainly use one. Ask them which product it is and whether they run it on their own server or use the vendor's hosted version.

If our IT provider installed the latest N-central fix, are we safe?

Not necessarily. Nineteen of the 21 N-central servers taken over this week were already running the latest hotfix, and a fix does not remove an attacker who got in before it was applied. Ask whether the Legacy API is off and whether they have searched for signs of the attack since November 8.

How do we know if our computers were reached through our IT provider's tool?

Your provider needs to check its server for unexplained accounts, scripts, scheduled tasks, and remote sessions, and then check your computers for anything those could have left behind, such as hidden remote access tools. If they find anything, the logs and a snapshot of the server should be preserved before any cleanup so a forensic team can trace which computers were touched.

Should we disconnect our computers from our IT provider's management tool?

Not on your own and not in a panic, because pulling agents without a plan can destroy evidence and leave computers unpatched. If your provider finds signs of compromise, decide together, with an outside security team involved, whether to cut the connection while the search runs. That trade between a few hours of downtime and the risk of further access is a business decision that belongs to you.