[packman] Migration OBS 2.1

[todd rme]

On Sun, Feb 20, 2011 at 8:35 AM, Pascal Bleser
<pascal.bleser at opensuse.org> wrote:
> FOSDEM is finally done, now I have time for Packman again ;)
>
> As you might already have noticed, our current OBS instance is
> broken, it has severe issues with noarch packages, amongst other
> things.
> Fixing it is very difficult, I tried to diagnose the problem but
> it partly comes from the OBS itself, and partly from my script
> that inserts incoming packages into our website's database.
> And it's a very old OBS instance (1.7) anyway...
>
> Detlef already took care of almost everything (*huge* thanks to
> Detlef!):
> * we have an OBS 2.1 instance that is properly configured, and
>  up and running on one of the worker hosts
> * we discussed the new repository layout a while ago, and came
>  to an agreement (see below)
> * the new OBS already exports repositories to our FTP server and
>  on mirrors
>
> What is merely missing now is
> 1) implement the glue between the repositories/packages and our
>   website
> 2) move all our packages from the old OBS to the new OBS
>
> THE NEW OBS INSTANCE
> ~~~~~~~~~~~~~~~~~~~~
> It is currently running in an openSUSE 11.3 chroot on one of the
> worker hosts, which is only temporary. Once everything is
> properly set up, we will scrap the old OBS on
> pmbs.links2linux.org and move it there.
>
> It already implements part of the new repository layout (see
> below), namely the "Essentials" repository, which is exported to
> our FTP site:
> ftp://ftp.links2linux.de/pub/packman/suse/
>
> The difference here is that it exports to the subdirectories
> "openSUSE_11.2", "openSUSE_11.3" and "Factory" (instead of
> "11.1", "11.2", "11.3" and "factory), to adhere to the
> conventions of build.opensuse.org.
>
> Those directories are also on our mirrors.
>
> THE NEW REPOSITORY LAYOUT
> ~~~~~~~~~~~~~~~~~~~~~~~~~
> We discussed several options on this mailing-list in December
> [1], and came up with the following solution:
>
> [1]http://lists.links2linux.de/pipermail/packman/2010-December/009213.html
>
> * we will split Packman into four repositories:
>  * Essentials
>  * Multimedia
>  * Games
>  * Extra (or "Other", I don't remember :))
>
> * we will also, using a little trick, provide one "big"
>  repository that aggregates the content of all that
>
> The idea is that by having more fine-grained repositories,
> people can decide to only add the packages that are only
> provided on Packman and nowhere else (Essentials), without
> having duplicates. It makes it also easier for us, because we
> currently cannot build everything for Factory and SLE11 (and
> Tumbleweed, which is slowly starting to get ready), so we can
> limit it to building Essentials only.
>
> The "big repository" will not cause any duplicate efforts, as we
> simply use "createrepo" to generate the RPM-MD metadata for all
> the packages in all the subdirectories (Essentials, Multimedia,
> Games, Extra) into the top-level directory.
>
> The directory layout on the FTP server and our mirrors will
> hence be as follows:
>
> /packman
>  /suse
>    /openSUSE_11.3
>      packman.repo ............ (1)
>      /repodata ............... (2)
>      /Essentials
>        /i586
>        /x86_64
>        /noarch
>        /src
>        /repodata ............. (3)
>        Essentials.repo ....... (4)
>      /Multimedia
>        /i586
>        ...
>        /repodata
>
> (1) add this .repo file for everything
> (2) contains the RPM-MD metadata for everything
> (3) contains the RPM-MD metadata for Essentials
> (4) add this .repo for Essentials
>
> THE WEBSITE
> ~~~~~~~~~~~
> I managed to hack all night and implement a script that stores
> the repository information into our database.
> It does have some major differences with the old approach, which
> also makes it slightly simpler:
> * the source is the repository on the filesystem itself, merely
>  using the RPM-MD metadata (repomd.xml and primary.xml.gz)
>  without having to analyse the RPM files
> * it inserts missing "packages", "releases" and "files_bin" into
>  the database
> * it removes "packages", "releases" and "files_bin" that don't
>  exist in the metadata from the database
>
> In order for the website to work without modifications, it will
> always run on the RPM-MD of "everything", without taking care of
> the different repositories (Essentials, ...).
> If desired, that's still something we could implement in the
> future as well (e.g. show in which Packman repositories the
> package is present), but that will require changes to the
> database layout as well as to the website itself.
>
> The script is in Perl, on gitorious:
> http://gitorious.org/packman/rpmmd2pmdb
>
> TODO
> ~~~~
> So, now is the time to assess what we still need to do.
> Off the top of my head:
> * minor details in the above mentioned script, such as updating
>  the description of "packages" as well as their "last_update"
>  timestamp (will be done today)
> * migrate our packages into the new OBS instance
> * kill the old OBS instance
> * move the new OBS instance to pmbs.links2linux.org
> * migrate our packager accounts
>
> MIGRATION
> ~~~~~~~~~
> Migrating the packages is obviously the most tedious part of the
> job and, as far as I know, Detlef already did everything for the
> "Essentials" repository (yay!).
>
> Detlef, can you give us an update on what is already done and
> what still needs to be done ?
>
> cheers,

Just out of curiosity, what is happening with the opensuse 11.4 repos
besides essentials?  It looks like they are building successfully and
are being published, but I don't see them anywhere.  Are they
available somewhere?

-Todd