Unlike most of the information I was able to find, this error is not always related to permissions.
Some history first…
I am running a web farm consisting of 3 servers behind a load balancing firewall. The servers are identical.
Installed Debian Stretch (stable) with some packages from Buster (testing) due to needing newer versions.
I have all 3 machines running gluster with a replicated brick on each machine. Overall performance is okay but not the greatest. It is about where I expected it to be for the hardware used.
SELinux is not in use.
While doing some year end clean up (and new year prep), I found I was behind on updates. So I fetched and installed the latest updates and rebooted just in case. Step through my restart procedure, no errors jump out. Later I found out that none of the web sites were working, they all said “Forbidden”.
So check the logs and the first thing I find is
AH00112 DocumentRoot [xxx] does not exist. for every virtual host. WTF???
The only time I’ve seen this before is when I forgot to mount the gluster volumes.
As root I can clearly see the files and folders. I can even see them as www-data. I can browse the entire file system right back to / as both users. So it is not permissions.
I checked the change logs (apache and Debian versions) and could find nothing that jumped out.
apache2ctl configtest everything was OK. I even put in a bad vhost directory just to test the output from configtest where it showed the truely missing directory.
Everything worked before the upgrades.
Setting apache2’s loglevel to trace1 (also tried 2 through 8) showed nothing useful.
Turns out to be some interaction between apache2 and gluster.
After I upgraded gluster from 3.8.8 (stable) to 3.13.0 (testing) the sites started working again.
So something Apache changed between 2.4.25 and 2.4.29 looked at the file system a bit differently.
Or Gluster between 3.8.8 and 3.13.0 fixed an issue with file access by apache.
Even now I can not find anything in the log files to indicate what apache was choking on when trying to access the files in the gluster storage.
But I got it to work again so I am happy once more.