And it failed spectacularly.
We only needed a simple form, but we wanted to be fancy, so we used “nextcloud forms”.
The docker image automatically updated the install to nextcloud 30, but the forms app requires nextcloud 29 or lower. No warning whatsoever. It’s an official app, couldn’t they wait that it was ready for NC 30 before launching it? The newsletter boasts “NC hub 9 is the best thing after sliced bread” yet i don’t see any difference both in visual or performance compared to NC hub 2
Conclusion: we made our business to rely on nextcloud forms as a signup form, but the only reason we were using it was disabled who knows how many weeks ago.
You must log in or # to comment.
Wait, you update productions systems without running a staging environment? Or even checking the update notes and your installed apps? Also no backups? What kind of business are you running over there?
Oh, Nextcloud docker is a joke. They follow no standards or best practices when it comes to docker. They keep the entire app directory mounted as a volume, which means it does upgrade you without you “needing” to upgrade the docker image. They have volumes within volumes they need to mount. Their configs can (and do) override environment variables. Most actions that need to be taken require running an
occcommand which can only be done by exec’ing into the container.Nextcloud docker is honestly just such a joke. They should have rethought their application from a docker sense and they didn’t. God just number one - Docker images should never update. It’s a freaking pinned version for a reason. If I want to update, it should be as simple as upping the version tag, and it does any upgrades in place when I do that.
I honestly steer people away from Nextcloud now because of how mismanaged their images are.
Yep, and I’d guess there’s probably a huge component of “it must be as easy as possible” because the primary target is selfhosters that don’t really even want to learn how to set up Docker containers properly.
The AIO Docker image is an abomination. The other ones are slightly more sane but they still fundamentally mix code and data in the same folder so it’s not trivial to just replace the app.
In Docker, the auto updater should be completely neutered, it’s the wrong way to update the app.
The packages in the Arch repo are legit saner than the Docker version.
I had to learn how to mount subpaths for their terrible container, and god just the updater is mind boggling. And I have to store their code in a volume, because of course I have to, why would code and configuration ever need to be… configurable? I actually just tried to put their
config.phpinto a ConfigMap just to try, and of course PHP doesn’t allow that - not that I blame PHP for it - but ffs it’s been years, it’s time to allow config to also come from a yaml or something.OwnCloud rewrite in Go is way better
I’m attracted to it because of the posix backend. Did anyone try it? Is it stable?
For reference, https://owncloud.dev/architecture/posixfs-storage-driver/
I’m testing it now. Seems way faster and more stable.
I’m just trying to get the oauth login to work but the actual file sync works great.
Yeah I’ve thought about migrating, but I have a few users on it who use nextcloud regularly now, so I’m forced to support it - unless there’s an easy migration path
Is this compatible with existing (Android) clients? I need offline file support for KeePass.
Yes it works with the android app
Having the web server be able to overwrite its own app code is such a good feature for security. Very safe. Only need a path traversal exploit to backdoor
config.php!
I wiped a whole drive (luckily it was filled with a redundant backup) with the docker image, as the behavior was (or still is, don’t know if it was fixed) to
rm -rf .and replace with fresh stuff ifoccisn’t found. So in the docker compose I accidentally mistyped the wrong volume as /mnt/disk2 instead of /mnt/disk3 and it erased itOh yeah, if you’re in a professional environment, I’m sorry but that’s just not great. The only way I’d consider running Nextcloud professionally would be on a VM of it’s own with nightly disk backups, with blob storage as the backing - and even then with the cloud costs really how close are you to just paying for an enterprise license to Google or Microsoft? Plus the headache of not having to worry about it yourself
What’s the better way of hosting it?
I do it in docker at home, for myself, in an environment I am okay with accidentally destroying - and even then I have nightly backups of the volumes.
In a professional system, as mentioned in my other comment, I would simply just do it in a VM with the disk scheduled also for nightly backups. Nextcloud just hardcoded too many things dependent on thinking the underlying system was mutable. Unfortuantely that’s just the easiest way to handle it.
However, also as mentioned, if I were in a professional environment, I’d have to really look at the cost for all of that infrastructure and my time to run it - and decide if I really thought I could run it myself with all of that overhead, and that it would still make sense compared to just doing google docs or something. Remember it’d be my ass on the line, as OP is learning
The images work fine for me. The problem is that Nextcloud is a complex app that doesn’t really work with the design of one container to do one job. It is pretty much a regular application that uses docker for packaging.
That doesn’t make up for bad container decisions. I run much more complex containers both that split out responsibilities and that contain everything as one container. The size and complexity is irrelevant to the bad design decisions. You can have an image that eats up gigabytes of space that runs off of proper environment/config variables with properly mounted volumes.
Again there docker image is just a packaging format and a health check. I very much wish it were better but for now it works
Just because it works doesn’t mean it follows best practices.
https://docs.docker.com/build/building/best-practices/#create-ephemeral-containers
To be fair a certain security company was in global news for exactly that same send it behavior. Why waste precious resources on multiple instances? Investors hate waste. 😅
The world is your
oystertest envIt worked on my box!
If I understand correctly, nextcloud automatically updated … which I didn’t think it would, normally. Maybe it’s a “feature” of the AIO docker image?
One that lacks a good IT department apparently
Yes no staging because it’s something used at most by 2 concurrent users, we were ok with 95% reliability (we discovered it was disabled after at least two weeks lol)
Otherwise we would just have signed up to one of the many cloud forms sites at $100/year
Backups daily but it’s unthinkable to revert something like nextcloud to a months old one
Subscribed to both newsletter and RSS feed to know about issues (the command to update the docker images isn’t automated but manually issued). The maintainer of the forms app is nextcloud itself so any incompatibility should have been written in red bold characters in the blog posts and newsletter.
Why are your backups so out of date? Just setup daily snapshots and call it a day if it isn’t critical. You never want to update major versions first thing. Wait 3 months and then update.
This smells like shadow IT
I have daily backups and hourly zfs snapshot. The problem is that, because nobody used the useless survey plugin, I have no idea when it broke. It could have been yesterday or it could have been 4 months ago
Backups and rollbacks should be your next endeavor.
Seems easier to blame Nextcloud
I have daily Borg backups held for at least one year but the problem is that the issue came out at least two weeks ago and nobody noticed. It’s better to have nothing (customer gets error page when viewing useless survey that nobody is watching) rather to restore such a old backup (everyone loses 2-4 weeks of data)
The docker image automatically updated the install to nextcloud 30, but the forms app requires nextcloud 29 or lower.
Lol. Do not blame others for your incompetence. If you have automatically updates enabled then that is your fault when it breaks things. Just pin the major version with a tag like nextcloud:29 or something. Upgrading major versions automatically in production is a terrible decision.
Docker images should never self update - that’s an anti pattern. They should be static code. The only time I would expect a docker image to “auto update” is if I was using the “latest” or “stable” tag and Compose/Kubernetes/I repull the image - but the image should never update itself.
Yes, OP bit off more than they could chew. Nextcloud, however, is breaking the entire purpose of Docker images by having an auto-updater at all.
What are you talking about? If you are not manual (or by something like watchtower) pull the newest image it will not update by itself.
I have never seen an auto-update feature by nextcloud itself, can you pls link to it?
I don’t have the link here, but essentially yes, nextcloud can update it’s own app code in it’s image because you have to mount the code to your own filesystem. This means that between docker images you can have a mismatch of the code that you have stored and the code that the image is expecting, which frequently causes mismatches for me. This is an antipattern. The code should be stored in the image, not as a volume mount. There should never be a mismatch of code in a docker image - that’s the whole point. The configuration could be out of date sure, or if there’s a data file that’s needed, that’s expected. The actual running code thought, that should never be on a mountable volume.
Next time you update the image you will probably be greeted with a “Nextcloud needs to update”. That should not exist. You already pulled the image, that should be everything you need to do. The caveats are extensions, kind of a grey area in my book, but I know it’s not a clean pattern with those either. (The best one I’ve seen lets you pin the extension version with environment variables or a config file, and then once again you are in control of when they update, and no running code is stored outside of the image.)
You can disable the web updater in the config which is the default when deploying via docker. The only time i had a mismatch is when i migrated from a nativ debian installation to a docker one and fucked up some permissions. And that was during tinkering while migrating it. Its solid for me ever since.
Again, there is no official nextcloud auto updater, OP chose to use an auto updater which bricked OPs setup (a plugin was disabled).
Thanks, I’ll disable that. I’m extra salty right now because I had to rollback a bad version and had to rebuild some of my config over the last week. I got into version hell because the code on the volume (which is why I’m pissed at it) said that I couldn’t run the image that I had set. So I pinned an earlier version, but then there were extensions that were pinned to a later one and said that I couldn’t rollback and didn’t start. I had to end up redoing the whole drive manually, forcing specific versions in the version.php and the config.php to finally make it work (Why is it in two places). Then after all that I had to run the upgrade command. Extremely annoying, and a waste of time for me. Other docker containers if I need to pin a version? I just… pin the version. Nextcloud is the only one I’ve seen where they store code on my volume and then pin specific versions to it.
They’re releasing a new version every two month or so and dropping them rapidly from support, pinning it with a tag means that in 12 months the install would be exploitable.
Now, I did directly to production because this is low priority stuff, but it would have happened even with a testing stage. I would have never noticed that the forms apps was disabled, the system disabled it without any notification.
You would expect that an official app supports the latest release, no?
This wasn’t an app released by a nobody in their free time, this is a main feature heavily advertised in their blog. Look by yourself:
https://nextcloud.com/blog/nextcloud-forms-to-keep-your-surveys-private/
It’s not unreasonable to get pissed when 6 months after that blog post it doesn’t support the latest release anymore.
They’re releasing a new version every two month or so and dropping them rapidly from support, pinning it with a tag means that in 12 months the install would be exploitable.
The lifecycle can be found with a single online search. Here https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule
Releases are maintained for roughly a year.
Set yourself a notification if you forget it otherwise.
Exactly, they have a release schedule, why their own plugin, that they’re heavily promoting as a feature, isn’t following that? If for some reason the forms app isn’t ready for that date, why not postponing the launch instead of having it broken for who know how many months?
It’s not a plugin made by someone else in their free time. They knew that by updating to NC 30 that feature that was marketed just 6 months ago would be disabled, so at least have the decency to write it in the release notes. I subscribe to the newsletter and the RSS for what, just enjoy the marketing buzzwords?
It’s like if Microsoft releases an operating system with a buggy and broken taskbar because of a rushed self imposed deadline and fixes it one year later.
Okay, let’s be angry at the company and frown a lot at what happened. Gurr, bad company, evil.
And now think of what you’d rather have - a working system, or a reason to be angry? If you have something that integrated with something else, lock it down at a specific version so you control the upgrade and know those versions work 100% of the time together. “Latest” is just asking for trouble - be it in a docker image, in dependencies or elsewhere. It’s absolutely not a “best practice” if it isn’t even a code smell or an outright bug. You could’ve had a slightly outdated version, which won’t be “exploitable” - you wouldn’t have enough time to exploit anything in that time, especially with smaller companies and obscure exploits.
Instead of putting out the fire, you could’ve been now looking into the upgrade, seeing on UAT or Test or whatever that forms aren’t supported, chilling till they are supported or complaining that they aren’t.
Upgrades breaking shit is like programming / devops 101, and a huge reason for technical debt in very old projects. Leaving all that to chance is just irressponsible.
the upgrade command is sent manually, it’s not automated and it’s not unattended. It would have made ZERO difference if i tagged 29 and then in test tagged 30. The upgrade would have not failed, it would have given ZERO warnings, I would have seen that everything worked as expected (because who tests the useless survey that is filled once a semester?) and I would have pushed the update to production.
Who tests the useless survey? Everyone with regression tests. Like dude, everything you talk about has been written “in blood” from years of hosting production systems. If the useless survey is needed, then write a test for it, or a testcase to manually try it. Don’t just upgrade, see that the app is up and push to prod, that’s not testing, that’s asking for trouble.
I’m not arguing that my procedure is right (it is not), I’m arguing that nextcloud rushed the release in an incredibly unprofessional way.
When it’s the last time that Canonical, in the hurry to release Ubuntu xx.yy broke snapd without?
When it’s the last time that Apple, in the hurry to release iOS xx, made it totally incompatible with iMovie and had no timeline on when it would be fixed?
When it’s the last time that Microsoft pushed a Windows update that disabled Microsoft office and left it unfixed for months?
Nextcloud decided to take care of the forms app. They decided to promote it as a selling point. They decided when to release the update. They decided to still push the update even if their own form app didn’t get any new release in 4 months and isn’t compatible.
They aren’t contractually obligated to release a new, indistinguishable version (except some new bits that make it incompatible) every quarter.
Not ready? Delay one week. Still not ready? Delay an additional week. It came out that it needs a complete rewrite and it will take months? Write a note in the blog saying that feature is no more a selling point but now it is deprecated/unmaintained/unsupported and it will be automatically disabled without confirmation.
As someone said, a good app is only late until it ships, a bad app is bad until it’s patched. Why being perpetually bad with updates that nobody is asking?
Specify a Version Tag in docker compose and update nextcloud deliberately through the webapp, that way it doesn’t update automatically on a pull
You can’t update it in the web app. You need to do a docker-compose pull followed by docker-compose up -d
No, I think if you’re using the nextcloud all in one image, then the management image connects to the docker socket and deploys nextcloud using that. The you could be able to update nextcloud via the web ui.
https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-update-the-containers
I’m talking about docker compose
Im using docker compose. Cant remember the image RN, but something like nextcloud 26. AS long AS you specify a full Tag and use volumes, you can upgrade in the webapp.
That’s makes sense
Just a side note but I think 26 is EOL. Upgrade to 29 or 30
Docker automatically upgrades if you tell it to by specifying “latest” or not specifying a version number. But it only upgrades if you issue the pull command or the compose up command. There are ways to start without a pull like using start or restart. So yes, there was warning and something you did actively told it to upgrade.
And it’s really bad practice to update any software without testing, especially between breaking/major version numbers.
Finally, it’s not uncommon for a platform to release its update and then the plugins or addons to follow. Especially with major updates that require lots of testing before release. This allows plugin/add-on makers to fully test their software with the release version of the platform rather than all of the plugin makers having to wait for one that may be lagging behind.
You again commented on the docker upgrade comment that I said it’s irrelevant. It’s really like saying “if you wore pants that day, they wouldn’t have done that, it’s your fault”. It doesn’t make sense to spend $1000 in operating costs to host a useless survey that gets 3 responses a year. If it breaks for a week nobody dies.
Focus on how they’re moving fast and breaking things, ok? It’s not normal that official plugins don’t support the latest stable release. It’s not an alpha, it’s not a beta, it’s stable. Stable means everything needs to work. Official plugins need to support the latest stable release. It’s acceptable only if this was a third party plugin made by a hobbyist in their free time
WordPress updates also break many plugins but it never happened that a stable release blocked official ones like woocommerce.
By the way, now I have learned that the latest version of nextcloud is a public beta and it’s better to always stay one version behind. So why don’t they call it public beta?
Why did you do automatic updates without testing? That is the real issue.
Honestly your IT department sounds like it could use some help
Manual docker upgrade issued my me after reading the official blog and newsletter. The upgrade notes described the new version as the best thing ever and didn’t mention that one of their selling points would be disabled without any notice.
I’m starting to see a pattern in those comments like “why did you wear a skirt that night? It looks like you asked for it…”
Whoah, dude.
Not only are you being told what could have and will ward off unplanned breakage, but you have somehow characterised yourself as an unsuspecting victim here? Inaccurate and really inappropriate comparison.
You knew enough to take on deploying a service, now comes the grown-up part where you hedge against broken updates.
I don’t know if maybe it’s my bad english in explaining it or it’s your comprehension skills that lack something.
I write it again for the 10th time: I’m 100% ok having 1-2 weeks of downtime, and this is why i do it live. It simply doesn’t make sense to dedicate several hours every month on testing if all i need is getting 3 useless surveys filled per year. If it was essential for work and i needed 99.99999% uptime i would directly subscribe typeform or surveymonkey. If tomorrow my install completely bricks and disappears in thin air, i would have lost 30 minutes of time and no valuable data. I literally spent more time designing the logo for the instance than managing it. This is just to state how unimportant the data stored on it.
This post wasn’t made about “oh no i lost millions and all my irreplaceable data thanks to nextcloud stupid updates” but how stupid is to release something that breaks features that they’re using as selling point.
ok, now that we established that my IT skills are lacking and i should be fired because one single survey couldn’t be filled, this is the release notes: https://docs.nextcloud.com/server/latest/admin_manual/release_notes/upgrade_to_30.html
Please tell me where they say that this feature is automatically disabled and also tell me why you think that this is acceptable.
I don’t understand why you think that is acceptable.
I even can’t find other examples where a release is so rushed, that selling points are disabled without ETA. I never saw for Libreoffice 24 dropping support for opendocument files for a couple months just because they had to meet a self imposed deadline
I’m starting to see a pattern in those comments like “why did you wear a skirt that night? It looks like you asked for it…”
Cute victim mentality, but gross and insanely wrong comparison
Learn from your mistake and don’t update without testing next time, it’s 100% on whoever updates the production environment to make sure that shit isn’t broken for whatever reason before pushing it customer-side
It’s more like you bought a random white powder from your dealer without asking what it was and are now upset you almost died
ok, please tell me where in the release notes they say that the forms app will be automatically disabled without warning after update, thanks https://docs.nextcloud.com/server/latest/admin_manual/release_notes/upgrade_to_30.html
Literally just googled “nextcloud forms” and looked at their supported versions and whaddya know, it says right on that webpage that there’s no stable version for 30 yet, so safe bet would be that it wouldn’t properly work when upgrading:
There is a supported nightly build, though, so you could probably have tried that
It’s on you to look up what will break when you update, or to test and see what happens when you do. A major update page isn’t going to list all of the things that rely on it that break because that’s fucking unreasonable
go to watch who is the maintainer of nextcloud forms, then see if they could have known that NC 30 was about to go out or not
It’s definitely not unreasonable that if I make product X and I make product Y, and they’re not compatible, then a bit of warning is suggested.
Again, wordpress updates break plugins all the time, but automattic plugins (same people of wordpress) never break. Coincidence? They just launch a new wordpress without checking if woocommerce or jetpack don’t work?
then a bit of warning is suggested
Which was given by the app that gets broken by the update
Windows doesn’t tell you that upgrading to 11 will break x, y, and z that you have installed, you’re expected to go to the sites for those programs and check if they work. Same exact idea
The same company making both apps is never a guarantee that they’ll play nice day 1, for many reasons
I’ll repeat: learn from your mistake instead of blaming other people for your naivete. If an app is important and might break during an update of something: check the apps documentation to see if it supports said update
Ok i get it, it’s best practice to do rushed releases without QA because users are the free testers.
They definitely had no way to know that their own app was incompatible, this is definitely a problem of the stupid user. Idiot user who believed their newsletter “update now, hub 9 is the best thing ever”. The user should have known that stable = untested beta
Also, this issue happened exclusively to me in the whole world, because everyone else isn’t an idiot like me and checks 30+ release notes scattered in 30 different repositories to guess any incompatibility. I was lazy and only checked the main notes! Such an idiot! Why I didn’t check every single installed app? It’s just 30! Nextcloud devs couldn’t have known that nextcloud devs didn’t update the manifest of the forms app! I should have checked before! Completely my fault!
Now if you excuse me I got an update to the Windows nextcloud desktop app and it must reboot after update because reasons even if there’s a GitHub issue with 200 angry comments about that. No wait! Stupid me! First I have to fire a VM and use a whole week to write automated tests that account for every possible combination of settings, language, power management, installed apps and so on. Otherwise I could lose a worthless survey that nobody reads and that will definitely get me fired!
There was a recent related discussion on Hacker News and the top comment discusses why this sort of solution is not likely to be the best fit for smaller organizations. In short, doing it well requires time and effort from someone technically sophisticated, who must do more than the bare minimum for good results, as you just learned.
Even then, it’s likely to be less reliable than solutions hosted by big corporations and when there’s a problem, it’s your problem. I don’t want to discourage you, but understand what you’re committing to and make sure you have adequate buy-in in your organization.
That reminds me of work. I’m old, young me has been through the mistakes and the pain of wanting to control and self-host everything.
Now I manage a team of young idealists who have not yet been burned sufficiently hard by reality and I feel like I spend half of my time denying them permission to add new self-hosted services to our stack.
Just last month a young padawan was pissed at the spent on an external auth service and had been pushing hard for a self hosted OSS solution which he was convinced he could handle by himself (which was most likely true, from a purely technical standpoint).
Since he wouldn’t let it go, I “punished” him by having him spend one day in excel and powerpoint to prepare a cost benefit analysis to present to the architecture review board, including server cost, backups, redundancy, security, monitoring, pen-testing, auditing, his time and all the bells and whistles we needed to be compliant with all the ISO-x we have to be. (we’re in a banking related field).
Our estimated internal cost ended up about 6x the one of the SASS solutions and still wasn’t as reliable.
Most people don’t understand the amount of effort it requires to run a secure & reliable system and if I had a dollar for everytime I heard it’s as simple as “docker run”, I could retire early.
Never upgrade to the latest and greatest of … anything really, especially in production. Let others test it first, or as suggested already, have a staging environment where you test the upgrade first. I guess you can still downgrade nextcloud though, especially if you have a backup.
Are you using the AIO image? I don’t know how well that works, but yeah, I absolutely hate automatic updates like that. I tried it once and I decided to use the plain “official but not supported” docker image instead, where I manage things myself. Never had an issue, and I can control which version I’m running, I can backup to wherever I want, using whichever system I want, etc.
AIO has a updater but it is manual by default You need to enable automatic updates yourself, which… Is done through a bash script you need to add yourself into the system crontab
And not only that, the instructions do say things could break and even suggests setting up backups for such
Docker is kind of a giant mess in my experience. The trick to it is creating backup plans to recover your data when it fails. As such, I don’t really recommend it to anyone at all.
Docker is kind of a giant mess in my experience. The trick to it is creating backup plans to recover your data when it fails.
Thats the trick for any production service. Especially when you do an update.
I wouldn’t recommend Docker for a production environment either, but there are plenty of container-based solutions that use OCI compatible images just fine and they are very widely used in production. Having said that, plenty of people run docker images in a homelab setting and they work fine. I don’t like running rootful containers under a system daemon, but calling it a giant mess doesn’t seem fair in my experience.
Honestly it is fine assuming you don’t need 24/7 uptime. Just make a compose file and verify you have a working health check
Sounds like you are not using docker correctly.
You can still choose to installt he old version in NC30 and it will do so. and I upgraded to NC30 and my forms app continues to be functional. you can still give it a try.
take a VM snapshot, upgrade the app, validate it still works as intended, if not, revert from snapshot
Not to flame you, but really just an HTML form was all you needed? It’s a super simple feature…
sure, but why solve problems in 10 minutes when i can do it way more sophisticated using 10x more time and resources?
(at the moment reverted to the easy html form + php send mail)
The forms app is useless. It’s basically for surveys. I can’t see how you’d use it for signups.
I wrote signups but I meant survey (in another comment I wrote “I would have never checked the useless survey”)
deleted by creator
I disagree. I use and depend on the apps including things like calendar and talk.
deleted by creator
YOU CAN’T DO THAT
Us too, we only use it as a filelink provider for thunderbird and to host a useless survey that’s going to get filled once a quarter. That’s why nobody noticed the survey was disabled and that’s why we’re not doing multistage testing in multiple virtual machines. We are a super small company and ok with something that one day can be 3 days offline. Otherwise it would be cheaper to pay $100 to Surveymonkey and $100 to Dropbox
No offence, but is Docker really the best way of running NC in a professional environment? Also, if you don’t want Docker to upgrade to latest image, don’t use the “latest” tag in your configuration.
Docker is probably the simplest way to get a working deployment, since there’s a lot of moving pieces in a Nextcloud install.
Though, it’s not going to automatically update itself unless you’ve made a poor choice for a production environment configuration, which sounds like what happened here.
(Even using a latest tag isn’t really a problem until/unless you re-pull the image to do the upgrade. And/or have configured something to automatically update your shit, but again, don’t do that in production.)
Nextcloud is also annoying in that updating the base won’t pull all the apps to a current version, so you have to know what’s going to break before you update the base so you can then update the apps as needed. Which, again, can’t just be left up to automatic updates.
Exactly. I don’t know if the AIO image was used and how that all works (I stay away from that and the snap which is just an abomination) but no one should try to selfhost anything for prod unless they know exactly how it works. That and have a staging env. If you’re not up to the task then just pay for some commercial hosting (even if it’s just Nextcloud that is hosted elsewhere.)
I’ve run the nextcloud image (just docker.io/nextcloud IIRC) pinned for years with k8s and it’s durable and fine. It stays put and I just take the time to update my testing instance, make sure it all works with some cheap smoke tests, then upgrade prod.
Yes, docker is the best way. Anything else is hell. It is still painful with docker but at least it is manageable
