[packman] PMBS is back up again

[Hans-Peter Jansen]

Dear Stefan,

Am Dienstag, 3. März 2020, 11:32:43 CET schrieb Stefan Botter:
> Hi all,
> 
> PMBS is back up again.
> If you see any strange behavior, please reply here, I will than have a
> look at it.

I'm getting back the "all PMBS packages in broken state" again:
 
Packman:Essentials/package: remote error: Service Unavailable

Waited another day for settling, since this issue was transient last time as
well.

Since the feedback for this issue on the buildservice ML was "manageable", 
not sure, how to precede.

The magic spell, that comes closest seems:

obs_admin --recheck-dod https://pmbs-api.links2linux.de/public

to me, but this just creates a hashed .recheck file in /srv/obs/dods.
The probably related service obsdodup.service is up and running, logs to 
/srv/obs/log/dodup.log, that contains *many* repeats of:

scanning doddatas directory...
checking state of dod entries...

What a beauty. It seems to operate on elements in /src/obs/projects and 
calls into localhost:5232, bs_srcserver, makes sense.

/srv/obs/log/src_server.log contains:

502 remote error: Proxy Error
502 remote error: Proxy Error

Another beauty.

Finally /srv/obs/log/scheduler_x86_64.log shows:
response from watcher for https://pmbs-api.links2linux.de/public
502 remote error: remote error  Proxy Error
retrying in 60 seconds

It's a different error from last time, which was:
400 remote error: document element must be 'events', was 'html'

Still something is borken, question is, what?

Thanks,
Pete