[QNAP users are currently victims of the DeadBolt ransomware – I didn’t have it in the blog, but within a week there were more than 3 of them.600 victims. The NAS manufacturer is now resorting to drastic means and trying to forcibly update the firmware of affected devices. But this causes malfunction on some devices (iSCSI devices do not work anymore).
Attacks on QNAP systems
I covered the topic of ransomware attacks on QNAP devices a few days ago in the article QNAP drives targeted by ransomware attacks in January 2022. In the last 14 days, I have come across repeated reports of problems and vulnerabilities with QNAP firmware. And owners of QNAP NAS drives became victims of various ransomware families. Below is a typical cry for help from a sufferer in a Facebook QNAP group.
The advice was to make sure that the devices are not accessible via internet and that the firmware and the used QNAP software is up to date. The colleagues at Bleeping Computer also warned about a new wave of attacks on QNAP devices the other day in the post New DeadBolt ransomware targets QNAP devices, asks 50 BTC for master key.
The files of the encrypted devices are ransomed with the extension .deadbolt provided. The ransomware also replaces the regular HTML login page with a message containing 0.03 bitcoins worth about 1.100 US dollars are demanded to get a decryption key and restore data. Threat actor claims to use zero-day vulnerability to hack QNAP devices and encrypt files with DeadBolt ransomware.
QNAP forced update after infection wave
Now the colleagues from Bleeping Computer report in the article QNAP force-installs update after DeadBolt ransomware hits 3,600 devices, that QNAP resorts to more drastic means after 3.600 QNAP units were encrypted by the DeadBolt ransomware (see also this tweet). QNAP forces a firmware update to version 5 for all customers’ NAS devices.0.0.1891. This is the latest firmware released on 23. December 2021.
QNAP owners and IT administrators told BleepingComputer that QNAP forced this firmware update on the devices even when automatic updates were disabled. That’s not so easy, QNAP already states in the release notes that the support for USB printers has been removed by the firmware. In addition, the description of the firmware changes lists some known issues.
And collateral damage occurs. However, some owners found that iSCSI connections to the devices did not work after the update. On reddit.com there is a post QNAP ISCSI Failed after update (FIX), where a user warns about problems.
Just thought I would give everyone a heads-up. A couple of our QNAPS lost ISCSI connection last night. After a day of tinkering and pulling our hair out we finally found it was because of the update.
It has not done it for all of the QNAPs we manage but we finally found the resolution.
In "Storage& Snapshots> ISCSI& Fiber Channel" right-click on your Alias (IQN) select "Modify> Network Portal" and select the adapter you utilize for ISCSI.
All fixed. I hope this helps someone.
All his iSCSI connections stopped working after the upgrade. But this could be fixed by reassigning the adapter via the settings as described above. In response to numerous complaints about a firmware update forced by QNAP, a QNAP support representative replied that it was to protect users from the current DeadBolt ransomware attacks.
"Qnap must let us know how and why they did this. People have so many different configurations on their machines, that may or may not function well on new updates. That Qnap needs to explain why the forced update was necessary and how they did it."
We are trying to increase protection against deadbolt. If recommended update is enabled under auto-update, then as soon as we have a security patch, it can be applied right away.
Back in the time of Qlocker, many people got infected after we had patched the vulnerability. In fact, that whole outbreak was after the patch was released. But many people don’t apply a security patch on the same day or even the same week it is released. And that makes it much harder to stop a ransomware campaign. We will work on patches/security enhancements against deadbolt and we hope they get applied right away.
I know there are arguments both ways as to whether or not we should do this. It is a hard decision to make. But it is because of deadbolt and our desire to stop this attack as soon as possible that we did this.
In summary: They are trying to increase the protection against DeadBolt. Qlocker infections would also have left their mark on the users. Currently it’s not clear to me how the December 2021 firmware update will help there. Someone affected by this forced update?
26 Responses to QNAP rehearses forced update after 3.600 DeadBolt ransomware infections
Moin,
I have since this morning several customers NAS that got the forced update. Some I can’t even reach anymore, I don’t have details yet. Seems QTS 4.5 systems with early December patch level to be affected.
You would always have had the option to patch firmware version 4 and not roll out a major version update just like that.
Especially since in the past such version jumps almost always led to some kind of problems.
for the QNAP TS-431P2 I just checked:
QTS 5.0.0.1891 build 20211221 – from 23.12.2021
QTS 4.5.4.1892 build 20211223 – from 24.12.2021
Thus, for the 431P2 the 5.0 the first new QTS OS, the 4.5.4 but it is still maintained – 1 day later!
These are also the only December- issues for this device.
If you take a look at the many APPS in the store – especially the outdated ones, then the attack surface also increases there, if the device is reachable via internet as well..
You have to think about it if it is a good idea to make the NAS reachable in the internet and for what purpose/for whom.
Are the forcibly patched devices logged into QNAP with an account?
No, the devices were not logged in and most of them were not reachable from the Internet.
But good, now there is QTS 5 everywhere, even if it needed in some cases another reboot because of the volume encryption, which was not available during the automatic reboot.
If you look at "frickelbuden" for the thinks the NAS iSCSI targets on their ESX as storage and of course have only one All/All Allowed rule on the firewall, they will be happy about such an update 
My NAS is not affected, but is neither externally reachable, nor can it reach externally without me releasing it temporarily, I never liked this constant calling home.
Hello,
yes also with me and customers there was forced update.
The devices that only had services like SMB were reachable again.
Those that work as ISCSI targets were there but the hard disks at the guest were not present, in most cases only "offline" in the disk management.
So in this case a "wanted" one Supply chain attack…worrisome that they can just force install an update.
Jo, no sympathy to those where QNAP managed to roll out the forced update, resp. why the NAS has so much internet access that QNAP gets so far… Because if QNAP manages it, it shows that the manufacturer also has a backdoor, which offers enough attack surface for manipulated firmware.
The devices should be completely disconnected from the www, especially if volumes are connected via iSCSI, that shocks me the most.
That means, if the affected people had already installed the updates in December, the attacks in January could have been avoided? My sympathy for the currently affected people is limited.
even if the current 4 release was installed, still the forced update to v5 happened – even without the QNAP being reachable from the internet and not connected to a QNap account.
They must have some kind of "backdoor to perform such forced updates – especially if automatic installation is disabled.
The most interesting question here:
> … That Qnap needs to explain … how they did it."
So the manufacturer of these things can install updates** from the outside at any time – no matter what the update setting in the device says?
** Updates could be beside firmware also other data, which then "suddenly" changed are on the device or not anymore. A not really welcome idea, would scrap the things.
I think the update mechanism looks for an xml control file or something similar when how and where it can load an update. In this file could be a flag for override local settings (or similar) and it will be installed even if the user has turned off updates. There is no need for a backdoor. Microsoft does the same. You can only prevent this by running the device 100% offline.
> In this file there could be a flag for override local settings (or similar) and it will be installed even if the user has disabled updates.
This is also called a backdoor.
Also: If updates are disabled in the device, loading any external xml control files is not necessary at all.
From the reddit thread:
> … I specifically had auto-updates disabled. What this means is that QNAP has some kind of backdoor into MY system.
>
> To me, this is a vector for another party to gain uncontrolled access to my system. ..
Facts would be even more interesting than guesses, let’s see if/what comes out of it.
When I log DNS requests, those seem to be the primary connection targets:
update.qnap.com
download.qnap.com
forum.qnap.com
edge.api.myqnapcloud.com
auth.api.myqnapcloud.com
core2.api.myqnapcloud.com
IdR hardcoded IPs are additionally recorded. As fallback.
On Android/Chrome these are e.g. the NS from Google (8.8.8.8, etc.)
I just had today with a TS332X… I had the updates on "notify only" and still suddenly QTS5 was installed.
Apart from the actual problem, this topic deals with a very exciting issue, which in my opinion will become more and more important in the coming time. There is a clear tension between the individual freedom to operate systems on the Internet and the dangers or risks of doing so. Damage that could be caused to third parties.
In this case, QNAP has probably come to the decision that a forced update of the devices is the lesser evil. QNAP surely knew that this process will cause problems for some customers, complaining included. On the other hand, damage was also averted by not encrypting data. Not an easy consideration, which fortunately I didn’t have to make either.
Now when users complain that their systems are being "supply chain attacked", so to speak or "backdoor" If I have taken over the responsibility for the update of myqapcloud, I have to ask myself why I have not already installed the update? Checking the logs on my private QNAP, I find:
Information – 2021-12-25 – 08:36:29 – [Firmware Update] Updated system from version 5.0.0.1870(20211201) to 5.0.0.1891(20211221)
This was the 1. Christmas day and is now over a month ago. Let me put it diplomatically: You might have managed to install the update yourself by now. This info about this came from the NAS via mail on Christmas Eve:
Severity: Info
Date/Time: 2021/12/24 21:31:02
App name: Firmware update
Category: Firmware update
Message: [Firmware update] The latest firmware version (5.0.0.1891 Build 20211221) is now available for download.
So you can get informed if you want to. I also don’t see why the general public has to suffer from botnets of routers and insecure exchange servers, just because the responsible operators don’t manage to organize their patch management properly. I had written a mail to Gunter about this update and he was kind enough to put it here in the blog together with the CVE numbers:
Who privately wants to run a NAS in his own LAN and has no confidence in the manufacturer, should take the thing from the Internet. With this I don’t only mean "no open ports from the internet", but put a parental control and prevent all traffic from the box to the internet. Updates can still be manually downloaded from the website and uploaded for installation at. From outside you can access the NAS via VPN.
Furthermore I think the idea of some users that you can set up your own NAS in a way that it is secured against the manufacturer under all eventualities and is exclusively under your own control is naive. This is especially true if the NAS is somehow connected to the Internet. Each software update, the update of each App could implement a function, which data ausleitet. This can be passwords, the keys of the drive encryption or the data itself.
But this is not a problem of NAS devices alone, but a general problem of IT in connection with the Internet. It is not possible for the user to test complex software in a depth that one could get absolute certainty about its security and function. In this respect you should be honest that there is always a residual risk and if this risk is too high compared to the benefit, then you should leave it alone. But that would also apply to PCs, laptops and smartphones.
> If now users complain, that their systems are attacked by a "supply chain attack", it’s not nice or "backdoor If I have taken over the responsibility for the update, then they must at least ask themselves why they as responsible operators have not already installed the update?
Apples, oranges… one has nothing to do with the other.
> Every software update, the update of every app could implement a function, which leaks data.
Correct. And that’s why some people set updates to "notify only" or to or "off or on "manually or "no, because they do not want automatic updates. And if the manufacturer bypasses this or. can circumvent, then may or. everyone must be allowed to complain as loud as they can.
That’s what I’m talking about, because that the manufacturer can do it if he really wants to is almost out of the question for me. Whether QNAP was allowed to do that and whether it was ethical, that is exactly the point.
Simply saying that you have to have the absolute control, is not enough for me, if third parties are endangered by your own system and you could reduce this danger by an update or an offline operation.
Of course this is a heavy intervention and just the fact that many were not aware of this possibility doesn’t make it better. Nevertheless, operators also have a responsibility, and in my view, they often fail to do so adequately.
The duties of an "operator and what a manufacturer can and may do are completely different things.
It’s hard to imagine if the forced update happens to a user at the most stupid moment possible. The recourse claims could then also become ugly for QNAP.
Of course a QNAP is nothing for really important environments, but I also had a really expensive HA environment that was hanging by a thread because of a bad bug. First, second and third level support from the HW and SW supplier have never found the error. The system hung in limbo for several weeks. A QNAP was there within a day delivered to write away the data (which is no fun with 60 TB) – if with such a thing then so a "forced update" comes in between – then good night!
No – if firmware updates are set to manual, then any intervention from QNAP that overrides that is absolutely out of the question!
Thank you for your opinion on the subject. In the meantime, there are hints that a previous update might have changed the automatic update setting. So the whole thing could have been done without intention. Let’s see what else can be found out about the topic.
I can also confirm this for my TS-231P2.
Unfortunately the forced update came in last night.
Autoupdate was turned off and the previous version was the latest 4 version (4.5.4.1892). Also the apps were previously all up to date. At the moment everything is running without problems. But I still find it questionable that QNAP can just do such a forced update..
Also here this morning at 7:30 o’clock from the latest 4er version to the 5er Zwangsgeupdated … in the middle of the backup process, not nice.
It also destroyed the SSD cache (but it was already disabled before, because there were already problems under the 4s).
Autoupdate was switched off, an absurdity. Are still checking for collateral damage.
I also registered that one of my NAS systems automatically upgraded to v5.0.Updated 0-1891 (went well, thankfully). In itself we have disabled the automatic updates, also because we had several weeks of problems after the firmware update in the summer. At that time it was not even possible to install the next version without problems.
Then when Christmas quasi simultaneously v4.5.4-1892 and v4.5.4-1891 came out, we have switched to 4.5.4-1892 updated. On weekends 8./9.1. the NAS hung up in such a way that it was only accessible again after switching it off and on again. In the QNAP forum there were a lot of people describing the same thing. I had another brand new QNAP NAS under my wing first thing on v5.0.0-1891 brought. This one had hung up the same way and had to be turned off and on.
There was some discussion on the QNAP forum that the Christmas update v4.5.4-1892 simply overwrote the settings for the update and set it to automatic.
QNAP is really not doing a good job in the last months. We also have three QNAP devices in locations and more than 10 others from other manufacturers.
Two of the QNAP devices we had to set up completely new after firmware updates in the last 12 months, data loss included. Was not too bad, because really critical things do not come on such devices. Rather liberating, but that’s another topic.
In a total of 5 cases QNAP devices had to be hard powered off because they stopped responding. Only the same two devices affected.
The third one is completely cut off from the internet from the beginning and will be updated manually when the others are also updated. We had zero problems with this one.
So QNAP devices don’t seem to get it when you give them access to the internet and thus apparently not only to updates.
All other NAS in our locations require much less attention. And even looking back at the recording history at our site, there have never been workhorses like the QNAPs that have been granted internet access. And there are times included when the things were still bought from the educator.
We went through this yesterday morning, because the day before yesterday another device hung up (on current patch level) and we want to get rid of the stuff now. Then the message read here about the forced update solidified that. The two could also be well related..
I don’t know at which update or updates – could have been at several – but the automatic update was activated again. Had to remove this again on several devices, even though this feature was always already disabled. I check the settings after each update, as QNAP is not the only manufacturer that occasionally reactivates the auto-update function again.
Those affected should check the settings as they are now and not what they were originally set to. In any case, my devices have been spared from the unexpected update, despite Internet access, but not open on the Internet.
Write a comment Cancel reply
Note: Please please note the rules for commenting in the blog (first comments and linked stuff ends up in moderation, I release every few hours, SEO posts/SPAM I delete rigorously). Comments off topic please under discussion.