[packman] Migration OBS 2.1

Sun Feb 20 14:35:16 CET 2011

FOSDEM is finally done, now I have time for Packman again ;)

As you might already have noticed, our current OBS instance is
broken, it has severe issues with noarch packages, amongst other
things.
Fixing it is very difficult, I tried to diagnose the problem but
it partly comes from the OBS itself, and partly from my script
that inserts incoming packages into our website's database.
And it's a very old OBS instance (1.7) anyway...

Detlef already took care of almost everything (*huge* thanks to
Detlef!):
* we have an OBS 2.1 instance that is properly configured, and
  up and running on one of the worker hosts
* we discussed the new repository layout a while ago, and came
  to an agreement (see below)
* the new OBS already exports repositories to our FTP server and
  on mirrors

What is merely missing now is
1) implement the glue between the repositories/packages and our
   website
2) move all our packages from the old OBS to the new OBS

THE NEW OBS INSTANCE
~~~~~~~~~~~~~~~~~~~~
It is currently running in an openSUSE 11.3 chroot on one of the
worker hosts, which is only temporary. Once everything is
properly set up, we will scrap the old OBS on
pmbs.links2linux.org and move it there.

It already implements part of the new repository layout (see
below), namely the "Essentials" repository, which is exported to
our FTP site:
ftp://ftp.links2linux.de/pub/packman/suse/

The difference here is that it exports to the subdirectories
"openSUSE_11.2", "openSUSE_11.3" and "Factory" (instead of
"11.1", "11.2", "11.3" and "factory), to adhere to the
conventions of build.opensuse.org.

Those directories are also on our mirrors.

THE NEW REPOSITORY LAYOUT
~~~~~~~~~~~~~~~~~~~~~~~~~
We discussed several options on this mailing-list in December
[1], and came up with the following solution:

[1]http://lists.links2linux.de/pipermail/packman/2010-December/009213.html

* we will split Packman into four repositories:
  * Essentials
  * Multimedia
  * Games
  * Extra (or "Other", I don't remember :))

* we will also, using a little trick, provide one "big"
  repository that aggregates the content of all that

The idea is that by having more fine-grained repositories,
people can decide to only add the packages that are only
provided on Packman and nowhere else (Essentials), without
having duplicates. It makes it also easier for us, because we
currently cannot build everything for Factory and SLE11 (and
Tumbleweed, which is slowly starting to get ready), so we can
limit it to building Essentials only.

The "big repository" will not cause any duplicate efforts, as we
simply use "createrepo" to generate the RPM-MD metadata for all
the packages in all the subdirectories (Essentials, Multimedia,
Games, Extra) into the top-level directory.

The directory layout on the FTP server and our mirrors will
hence be as follows:

/packman
  /suse
    /openSUSE_11.3
      packman.repo ............ (1)
      /repodata ............... (2)
      /Essentials
        /i586
        /x86_64
        /noarch
        /src
        /repodata ............. (3)
        Essentials.repo ....... (4)
      /Multimedia
        /i586
        ...
        /repodata

(1) add this .repo file for everything
(2) contains the RPM-MD metadata for everything
(3) contains the RPM-MD metadata for Essentials
(4) add this .repo for Essentials

THE WEBSITE
~~~~~~~~~~~
I managed to hack all night and implement a script that stores
the repository information into our database.
It does have some major differences with the old approach, which
also makes it slightly simpler:
* the source is the repository on the filesystem itself, merely
  using the RPM-MD metadata (repomd.xml and primary.xml.gz)
  without having to analyse the RPM files
* it inserts missing "packages", "releases" and "files_bin" into
  the database
* it removes "packages", "releases" and "files_bin" that don't
  exist in the metadata from the database

In order for the website to work without modifications, it will
always run on the RPM-MD of "everything", without taking care of
the different repositories (Essentials, ...).
If desired, that's still something we could implement in the
future as well (e.g. show in which Packman repositories the
package is present), but that will require changes to the
database layout as well as to the website itself.

The script is in Perl, on gitorious:
http://gitorious.org/packman/rpmmd2pmdb

TODO
~~~~
So, now is the time to assess what we still need to do.
Off the top of my head:
* minor details in the above mentioned script, such as updating
  the description of "packages" as well as their "last_update"
  timestamp (will be done today)
* migrate our packages into the new OBS instance
* kill the old OBS instance
* move the new OBS instance to pmbs.links2linux.org
* migrate our packager accounts

MIGRATION
~~~~~~~~~
Migrating the packages is obviously the most tedious part of the
job and, as far as I know, Detlef already did everything for the
"Essentials" repository (yay!).

Detlef, can you give us an update on what is already done and
what still needs to be done ?

cheers,
-- 
  -o) Pascal Bleser <pascal.bleser at opensuse.org>
  /\\ http://opensuse.org -- I took the green pill
 _\_v FOSDEM XI: 5 + 6 Feb 2011, http://fosdem.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: not available
URL: <http://lists.links2linux.de/pipermail/packman/attachments/20110220/45860f77/attachment.sig>