Mass Build: Missing Sources

Hi All,
I have been analysing the mass build failures and have discovered that there
is a large group of packages which have missing sources.
I have attached the list and if a senior dev would like to cast their eye over
it and comment. Then it may be possible to start fixing these simple problems
with the help of the QA team. This could be seen as a training excercise to
get those who wish to learn a bit more about the dev process familiar with the
tools that are used. I am happy to supervise this process if it’s thought to
be worthwhile.
Colin

(Attachment Missing_sources is missing)

My attachment was rejected hopefully renaming it will get it through.

Missing_sources.txt (12.5 KB)

In some cases it seems that .abf.yml is missing or empty.

If @Colin or some senior developer will write a step by step how to do this I will put some time in. And if I learn something in the process so much the better. We maybe might even get this done if more folks pitch in and help?

:hear_no_evil::see_no_evil::speak_no_evil:

1 Like

To fix this issue, one should upload missing files to file-store.

To check which files are missing, go through the files mentions in Source* tags of the spec file. If you don’t see one of these files in the current folder and it is not mentioned in .abf.yml (or the latter is missing at all), then the file should be first downloaded from Internet and then uploaded to file-store.openmandriva.org. Hash sums should be added to .abf.yml.

As an example, here is a fix which I made for the first texlive package from the list:

https://github.com/OpenMandrivaAssociation/texlive-babel-vietnamese/commit/f61156299ed4cc8662afe4e65c5d1c558df6117b

I use abf console client, so I did everything from command line, step by step:

$ abf get texlive-babel-vietnamese
$ cd texlive-babel-vietnamese
$ grep Source *spec
Source0:        http://mirrors.ctan.org/systems/texlive/tlnet/archive/bohr.tar.xz
Source1:        http://mirrors.ctan.org/systems/texlive/tlnet/archive/bohr.doc.tar.xz
$ wget http://mirrors.ctan.org/systems/texlive/tlnet/archive/bohr.tar.xz
$ wget http://mirrors.ctan.org/systems/texlive/tlnet/archive/bohr.doc.tar.xz
$ rm -f .abf.yml
$ abf put -m "Upload files to file-store"
1 Like

Btw, I’ll take care of all texlive packages mentioned in the list.

1 Like

Some time ago, when ABF was down, most of my packages lost their source, but I’ve uploaded them again.

Nevertheless, I have observed that the loss of sources happens quite often. Two weeks ago I uploaded the source to the server, and a few days ago it was already deleted…

a lot of the time the source is not actually missing from abf.yml but has been removed during the build, this appears to be a bug.

the scripts will first download all the sources, then check the sha1sum of the downloaded file, if it doesnt match it will delete the file, but doesn’t stop the build (not all files mentioned in the yml are actually used so this is probably sensible). this then leads to the build failing because the main file is missing.

i’ve no idea why it happens, presumably sometimes filestore returns an error (build scripts dont show any output, but in the example below it doesnt seem to have downloaded the whole file - the filesize is too small). maybe the solution is to try more than once.

here’s an example:

http://file-store.openmandriva.org/api/v1/file_stores/b6793c00b730695a1236844a7f774e7c7e7ac521.log?show=true

you’ll see:

Checking file sha1sum
sha1sum INCORRECT! Removing file chromium-70.0.3538.77.tar.xz

later the build fails with:

shell False
BUILDSTDERR: Unsharing. Flags: 134217728
BUILDSTDERR: error: Bad source: /builddir/build/SOURCES/chromium-70.0.3538.77.tar.xz: No such file or directory
Building target platforms: znver1

attempting the build again without changing anything usually.

I’ve taken the first 10 off the list and will fix. If anyone fixes anything in the list post info to this thread…

It looks like Denis has answered your plea : )

Sorry Colin Close, I am completely in the dark here :). I’ve never recieved any emails of this type. Are you asking for help for is this just an email to keep us informed ?

Graham,
This is an informative email. If you want to learn a bit about packaging for the OpenMandriva distribution I am suggesting that adding back the missing sources is a way that people could make a start on learning how the build systems works and how to update the source code for a particular software package. If you would like to help with this work I would be happy to guide you through the process.
Colin Close

I’m missing something. In @Colin’s list we are pointed to somewhere in ABF say for xcreensaver here. There is no .spec file listed. So am I supposed to add one from Internet? Where on the Internet? And then add that file to here? Just put the file there? How would file store know what to do with the file? And then the example of a successful fix in an entirely different location on Github. I need this broken down more to understand what to do.

:speak_no_evil::hear_no_evil::see_no_evil:

Maybe we are supposed to have extrapolated form the explanation to go to Github and look for a .spec file which for xscreensaver looks to be present with apparent sources listed. And there is an abf.yml file.
Post-edit: But xscreensaver should be xscreensaver-5.40.tar.gz not 5.38. So how to fix that?

The next package in the list ucarp also has a .spec file with apparent sources listed. And there is an abf.yml file.

So I’m feeling kinda stoopid here. Perhaps those where already fixed?

If I ever do find a package that does not have a .spec file where do I go to find one? Or am I completely missing what’s going on here?

Post-edit: The list @Colin posted isn’t very long so these are probably already all fixed and I’m late to the party?

:monkey:

I will answer from what I know, or from what I think I know. If I’m wrong, please correct me someone :penguin:

On the @Colin list, there is no missing .spec file, but only missing sources. Here:

error: Bad source: /builddir/build/SOURCES/xscreensaver-5.38.tar.gz: No such file or directory

This file need to be downloaded (in best case from its developers website) and then uploaded to file-store.openmandriva

What to .spec file. I think almost all packages have it. It is necessary to build an RPM package. It may be missing when you import a new package into openmandriva, then spec must be done from scratch or you can “sneak” from Fedora, OpenSUSE, Mageia, PCLinuxOS, ALT or Rosa. Of course, modifications will have to be made, but most things should be good. File .spec should be in GitHub not in file-storage.

It seems that crisp has already fixed it.

But if you see any other package that need to be updated (some from main and many from contrib :disappointed_relieved: ) or just is without source like on colin list then, download latest source, upload it to file-storage [you need log in - same login as in ABF) and then go to github/openmandriva, find this package, open file .abf.yml and paste here full source name for example YXZ-1.2.tar.gz and after “:” paste hash generated by file store.
Look at this:
file-sorage
As you see, I just uploaded the evince source. You have here full name of source evince-3.30.2.tar.xz and also generated hash.
Paste this both to .abf.yml file in GitHub.

Next, you need edit .spec file. If you only upload the missing source then you do not need to edit the spec - usually.
If you are updating the package, for example from version 1.2 to version 1.3, then you must edit it in the .spec file.
In the simplest version, you only change the package “version” from 1.2 to 1.3 and if the release is higher than 1, then change it to 1 - as first release. If you build this same version again and fixing somethings then change it to 2 and if another time then to 3 etc…
Look here, this is my quick update for mutter:

Now save .spec and now you can go to ABF and try to build a package.

in the case of xscreensaver the source was not actually missing, the filestore failed to download it correctly and the local file was deleted at the start of the build. this is an issue with filestore (see my earlier post https://forum3.openmandriva.org/t/mass-build-missing-sources/2228/8?u=crisb)