Can't load dashboard (Internal Server Error) after attempted object storage backend migration

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 29.0.5): 28.0.7
Operating system and version (eg, Ubuntu 24.04): Container image from Docker Hub running under Kubernetes 1.28.9 - cluster nodes run Talos Linux 1.7.2
Apache or nginx version (eg, Apache 2.4.25): ingress-nginx 1.10.0
PHP version (eg, 8.3): whatever comes with the docker container

The issue you are facing:
I use minio S3 storage hosted locally. I needed to move to a new version of minio, which required a whole new server and manual data migration. I’ve replicated credentials etc on the new server, that does not appear to be the issue.
With Nextcloud shut down, I migrated the objects from the old Minio instance to the new one using rclone. This appeared to complete successfully. However when I point the Nextcloud instance at the new server and start it up, I log in and then immediately get the ‘Internal Server Error’ plain-text error page.
Looking at the logs, I see it’s trying to access an object that does not exist on the old or new Minio instance. Toward the end of the logs below, you’ll see the reference to

urn:oid:149645

I’ve tried pointing it back at the old minio, same result - Internal Server Error - with logs trying to access the same non-existent object.
Pretty frustrating - I’ve got all my data, but no way to access or export it.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Not sure this is replicable outside my install.

The output of your Nextcloud log in Admin > Logging:


The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):


The output of your Apache/nginx/system log in /var/log/____:


Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

{"reqId":"dMEVQXexHZhiJZpEwaiF","level":3,"time":"2024-07-12T15:10:45+00:00","remoteAddr":"xxxx::8888::89f4","user":"drew","app":"objectstore","method":"GET","url":"/apps/dashboard/","message":"Could not get object urn:oid:149645 for file appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0","version":"28.0.7.4","exception":{"Exception":"Exception","Message":"Failed to read object urn:oid:149645","Code":0,"Trace":[{"file":"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":336,"function":"readObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:149645"]},{"file":"/var/www/html/lib/private/Files/Storage/Common.php","line":200,"function":"fopen","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":253,"function":"file_get_contents","class":"OC\\Files\\Storage\\Common","type":"->","args":["appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps"]},{"file":"/var/www/html/lib/private/Files/View.php","line":1167,"function":"file_get_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps"]},{"file":"/var/www/html/lib/private/Files/View.php","line":582,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["file_get_contents","/appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps",["read"]]},{"file":"/var/www/html/lib/private/Files/Node/File.php","line":54,"function":"file_get_contents","class":"OC\\Files\\View","type":"->","args":["/appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps"]},{"file":"/var/www/html/lib/private/Files/SimpleFS/SimpleFile.php","line":74,"function":"getContent","class":"OC\\Files\\Node\\File","type":"->","args":[]},{"file":"/var/www/html/lib/private/Template/JSCombiner.php","line":118,"function":"getContent","class":"OC\\Files\\SimpleFS\\SimpleFile","type":"->","args":[]},{"file":"/var/www/html/lib/private/Template/JSCombiner.php","line":93,"function":"isCached","class":"OC\\Template\\JSCombiner","type":"->","args":["merged-template-prepend.js.deps",["OC\\Files\\SimpleFS\\SimpleFolder"]]},{"file":"/var/www/html/lib/private/Template/JSResourceLocator.php","line":132,"function":"process","class":"OC\\Template\\JSCombiner","type":"->","args":["/var/www/html","core/js/merged-template-prepend.json","core"]},{"file":"/var/www/html/lib/private/Template/JSResourceLocator.php","line":91,"function":"cacheAndAppendCombineJsonIfExist","class":"OC\\Template\\JSResourceLocator","type":"->","args":["/var/www/html","core/js/merged-template-prepend.json"]},{"file":"/var/www/html/lib/private/Template/ResourceLocator.php","line":73,"function":"doFind","class":"OC\\Template\\JSResourceLocator","type":"->","args":["core/js/merged-template-prepend"]},{"file":"/var/www/html/lib/private/TemplateLayout.php","line":403,"function":"find","class":"OC\\Template\\ResourceLocator","type":"->","args":[["core/js/common","core/js/main","activity/l10n/en","activity/js/activity-api","dashboard/l10n/en","And 53 more entries, set log level to debug to see all entries"]]},{"file":"/var/www/html/lib/private/TemplateLayout.php","line":247,"function":"findJavascriptFiles","class":"OC\\TemplateLayout","type":"::","args":[["core/js/common","core/js/main","activity/l10n/en","activity/js/activity-api","dashboard/l10n/en","And 53 more entries, set log level to debug to see all entries"]]},{"file":"/var/www/html/lib/private/legacy/OC_Template.php","line":142,"function":"__construct","class":"OC\\TemplateLayout","type":"->","args":["error",""]},{"file":"/var/www/html/lib/private/Template/Base.php","line":132,"function":"fetchPage","class":"OC_Template","type":"->","args":[]},{"file":"/var/www/html/lib/private/legacy/OC_Template.php","line":291,"function":"printPage","class":"OC\\Template\\Base","type":"->","args":[]},{"file":"/var/www/html/index.php","line":104,"function":"printExceptionErrorPage","class":"OC_Template","type":"::","args":[["OCP\\Files\\GenericFileException"],500]}],"File":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":93,"message":"Could not get object urn:oid:149645 for file appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps","CustomMessage":"Could not get object urn:oid:149645 for file appdata_oc3fn6i26acm/js/core/merged-template-prepend.js.deps"}}

Are you also using custom SSL/TLS certificates?

There’s some special handling for the bundle involving Primary Storage to deal with the bundle file being stored in S3 to avoid a chicken & egg problem. I don’t recall how it interacts with migrations.

Thanks for your reply!

I think I’ve resolved it in a testing instance by upgrading to 29. At any rate, things seem to be working again, accessing files in the new minio instance. Wish I understood what happened, but I’ll take it :slight_smile:

I’m using automated letsEncrypt certs managed by k8s certmanager. As far as I can remember, tls is all isolated to the k8s ingress, and the nextcloud container is not involved. There is a bit of custom nginx config embedded in the ingress resource though to make the generated reverse proxy config suitable for Nextcloud.

Thanks again!