[packman] [opensuse-buildservice] broken builds due to "interconnect error: rpc timeout" with PMBS

[Hans-Peter Jansen]

Hi Stefan,

Am Donnerstag, 26. Dezember 2019, 20:04:03 CET schrieb Stefan Botter:
> Hi Pete,
> 
> On Thursday, December 26, 2019 7:14:27 PM CET Hans-Peter Jansen wrote:
> > Am Mittwoch, 25. Dezember 2019, 18:40:21 CET schrieb Hans-Peter
> 
> Jansen:
> > > [A request on the OBS ML kept unanswered for two days, hence trying
> > > my luck here..]
> > > 
> > > Hi,
> > > 
> > > I stumbled across broken builds in a local OBS due to "interconnect
> > > error: rpc timeout" for packages, that link to PMBS.
> > > 
> > > PMBS is setup as:
> > > 
> > > <project name="Packman">
> > > 
> > >   <title>Packman Build Service PMBS</title>
> > >   <description>This instance can be used to access resources from
> > >   Packman.</>
> > > 
> > > description>
> > > 
> > >   <remoteurl>https://pmbs-api.links2linux.de/public</remoteurl>
> > > 
> > > </project>
> > > 
> > > My OBS itself is fully updated from OBS:/Server:/2.10/openSUSE_15.1.
> > > 
> > > The magic spell, that comes closest to mitigate the issue *ought* to
> 
> be:
> > >     obs_admin --recheck-dod https://pmbs-api.links2linux.de
> > > 
> > > or
> > > 
> > >     obs_admin --recheck-dod https://pmbs-api.links2linux.de/public
> > > 
> > > but even after creating the missing /srv/obs/dods directory, things
> > > still misbehave.
> > > 
> > > Is this a PMBS issue, a local configuration issue, or something
> > > completely different?!?
> > 
> > /srv/obs/log/scheduler_x86_64.log shows this:
> > 
> > 2019-12-26 18:07:17: [2762]  waiting for an event...
> > response from watcher for https://pmbs-api.links2linux.de/public
> > 400 remote error: document element must be 'events', was 'html'
> > retrying in 60 seconds
> 
> I am not 100% sure, PMBS is working correct, but I have two OBS
> instances connected to PMBS without issue. One is running on the same
> subnet as PMBS, even on the same ESXi, the other is running at a
> different location and has to go through my reverse proxy - like your
> instance.

Interesting. How does your Packman interconnect look like on that system?

E.g. cat /srv/obs/projects/Packman.xml

While at it, does systemctl --failed shows any disturbing issues?

> I grepped through a month's worth of log files and on both instances
> there are no mentions of error messages like yours, not even any mention
> of 'pmbs'.
> 
> Did this work before? Up until when?

Yes, it worked nice for a long time (~08/2017) since about a month ago.
I first assumed a transient issue, similar to the OBS outages/issues in 
December (which were transient..).

Later, I updated my obs server with zypper dup from exactly these repos:

# | Alias                     | Name                            | Enabled | 
GPG Check | Refresh
--+---------------------------+---------------------------------+---------
+-----------+--------
1 | OBS_Server_2.10           | OBS Server 2.10 (openSUSE_15.1) | Yes     | (r 
) Yes  | Yes    
2 | openSUSE:Leap:15.1        | openSUSE:Leap:15.1              | Yes     | (r 
) Yes  | Yes    
3 | openSUSE:Leap:15.1:Update | openSUSE:Leap:15.1:Update       | Yes     | (r 
) Yes  | Yes    

Rebooted it several times, etc. All seems well, but still no cigar.

> PMBS and both my other instances are on 2.10 on 15.1 since July.
> PMBS and my close instance are on the ESXi behind the reverse proxy
> since end of August, since then there are just the occasional reboots
> for security updates.

I upgraded to 2.10 a little before you (IIRC, since we had similar systemd 
related issues at that time..). 

> I can see at least 8 other IP addresses accessing the public interface
> today, so to all: Speak up if you too have problems.

Another interesting observation: it indeed makes a difference, if I access the 
public path of OBS and PMBS with curl or wget. 

(All done from the local obs server):

$ wget -qO- https://pmbs-api.links2linux.de/public
<about>
  <title>Open Build Service API</title>
  <description>API to the Open Build Service</description>
  <revision>2.10</revision>
  <last_deployment>2019-07-05 14:35:37 +0200</last_deployment>
  <commit>1b66a6c499ef52f61406c333d92731323d852fbc</commit>
</about>

$ curl -o- https://pmbs-api.links2linux.de/public
<html><body>You are being <a href="https://pmbs.links2linux.de/
about">redirected</a>.</body></html>

but the behavior is consistent with OBS.

I can provide many more details, but still have no idea, where these issues 
stem from. It just shows any package build as broken, that links to PMBS.

It's "zum Mäusemelken"..

Cheers,
Pete