My previous post turned out to be very messy with updations because of my confusions. Hence I am doing a new post.
Metalink is a protocol which allows to embed several links for a given file. The links can be bittorent, ftp, http. Metalink enabled clients can use all these links to download segments of files from different sources.
aria2 is one such client. It is a special download utility which is really fast. Its power lies in ability to handle multiple connections from multiple sources simultaneously.
Opensuse has adopted such goodness already. All its iso's and rpm's also come with metalinks, which means people can download opensuse faster.
This sweetness is also being adopted in libzypp as well by providing a download handler using aria. However the default handler is curl for opensuse 11. zypper can be made to use aria by setting the environment variable as follows
ZYPP_ARIA2C=1
Now here comes the part, where I am confused.
Looking at the invocation of aria by the zypper (ps aux |grep aria) it seems libzypp instructs aria to download from direct rpm link and not its metalink. The metalink for each rpm file can be seen in html pages of repos (look to right of each rpm link in http://download.opensuse.org/distribution/11.1/repo/oss/suse/noarch/)
According to my testing, zypper is quite slow even while using aria for downloading which suggests its not using metalink.
So, somebody please confirm whether libzypp-aria integration uses metalinks to download rpm files or not.
Meanwhile I tried a manual download of several rpm metalinks and this was really fast. I could get a stable download rate of 80+kbps.
So I decided to work out some python magic to extract metalinks of packages to be downloaded from zypper output and then feed those metalinks to aria2c. The script is messy, but it did increase download speed. More importantly it served my purpose of maximizing bandwidth exploiting in the free download hours.
It certainly is a good idea for zypper to use metalinks rather than direct rpm links if its using aria. However this might already be the case as I am unsure. If it isn't, may be it can be done as summer of code project as well :)
So, I hope to have cleared up the confusion I set up in my previous post.
I will end with a question. Is there a way to get openSuSE factory KDE packages as an ISO which is available through torrents/metalink ?
Saturday, 10 January 2009
Subscribe to:
Post Comments (Atom)
3 comments:
I believe download.opensuse.org will serve you a metalink based on your preferred mimetypes in the HTTP ACCEPT header
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Try it yourself:
curl -vv -H "Accept: application/metalink+xml" http://download.opensuse.org/distribution/11.1/repo/oss/suse/noarch/ant-1.7.0-199.15.noarch.rpm
Thanks for the info benji. You are right. I did verify the download invoked by zypper using aria and it did use the metalink.
So, now my only request for the libzypp-aria developers is to provide a mechanism to specify number of concurrent downloads.
(actually I use update/install with --dry-run so that I download all packages first and only then I invoke the actual update - hence concurrent downloads benefits my type of usecase)
The ps ...| grep you mentioned doesn't show any --no-conf argument. So I guess you can try setting other aria2c params (like number of concurrent downloads) to its configuration file ~/.aria2/aria2.conf.
Refer manpages for more info.
Post a Comment