Distfile Survey ToDo
Checking Files
Check MASTER_SITE_BACKUP to see if the file is there -> is it *completely* unfetchable or only unfetchable from all MASTER_SITES?
implemented, not in cron yet, first check run going 200510132357
- Randomize recheck for known-good / known-bad urls, for a given site, so that we don't get synchronized and check millions every 14 days and few the rest of the time.
- Try GET after a failed HEAD, to see if a server (against the standard, but who cares about the standard?) replies differently?
Try GET after a failed LIST, in case an ftp server doesn't permit LIST. See, e.g., http://www.freebsd.org/cgi/query-pr.cgi?q=90751
- Still some weirdness with timeouts vs. other errors
- Don't check any ports that:
- set FETCH_ENV
- set FETCH_CMD
- have a do-fetch: target
- SecureIIS returns 406 error
- handle HTTP redirects to ftp: URL's
- make 421 There are too many anonymous users currently logged in.
- a permanently transient error.
- Chase down LWP saying that IO::Socket::INET says that the error is (Timeout) when connecting to a system that's not running an HTTP server.
- Found it, but I don't like it. This is IO::Socket's connect() having an implementation of non-blocking connect for timeout that's incompatible with FreeBSD. Upgrading IO::Socket::INET at least got it to change to "connect: Invalid argument", but it'd be nice to get to actually get "Connection refused".
Output format
- Output DEPRECATED and EXPIRATION_DATE so people don't waste time on ports that will go away (or are motivated to fix before they go away)
- Say whether the file is on MASTER_SITE_BACKUP
- Levels of badness: right now it's just 100% unfetchable, what about levels at 25%, 50%, 75%?
- That would help point out impending problems, like the current ruby/contrib situation
- How many ports depend on this one, if it's unfetchable?
- Always output maintainer even if it's all fetchable?
- Suggestion from aDe:
- augmenting the ok/bad/skipped with a +/-/? character in the order they were checked (the default case for most people) so you'd have something like: 3 ok, 2 bad, 1 skipped: +-?-++
- Identify groups of bad sites that are some MASTER_SITE_FOO and say "MASTER_SITE_FOO is all bad" instead of listing all sites.
- Want to have a lighter weight format for first viewing, then "drill down" if you're interested.
e.g., this is completely gratuitous:
File: arts-1.4.2.tar.bz2 has 106 possible URLs: 105 OK, 1 bad, 0 skipped Port maintainer: kde@FreeBSD.org ftp://ftp.scarlet.be/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.scarlet.be/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://mirrors.isc.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.gtlib.cc.gatech.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.gtlib.cc.gatech.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.oregonstate.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.eu.uu.net/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.tiscali.nl/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.tiscali.nl/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.du.se/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.du.se/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.solnet.ch/mirror/KDE/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp-stud.fht-esslingen.de/Mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.gwdg.de/pub/x11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://kde.mirrors.hoobly.com/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.informatik.uni-hamburg.de/pub/soft/X/contrib/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.rediris.es/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.rediris.es/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.mirrors.net.ar/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.man.szczecin.pl/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://gd.tuwien.ac.at/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://gd.tuwien.ac.at/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.informatik.hu-berlin.de/pub/Mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://mirrors.dotsrc.org/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://mirrors.dotsrc.org/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://mirrors.ibiblio.org/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ibiblio.org/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: 500 read timeout (Last OK result Sat Oct 8 01:07:02 2005 UTC) ftp://ftp.roedu.net/pub/mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.fi.muni.cz/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.fh-heilbronn.de/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.fi.muni.cz/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.tuniv.szczecin.pl/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.tuniv.szczecin.pl/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://sunsite.icm.edu.pl/pub/unix/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://mirror.switch.ch/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://mirror.switch.ch/ftp/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.sunet.se/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.sunet.se/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.rz.uni-wuerzburg.de/pub/unix/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.dataplus.se/pub/linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.duth.gr/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.duth.gr/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.unina.it/pub/Linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.unina.it/pub/Linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.gwdg.de/pub/x11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.kde.org.yu/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.kde.org.yu/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.tu-clausthal.de/pub/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.belnet.be/packages/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.belnet.be/packages/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.lip6.fr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://www-ftp.lip6.fr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.tu-chemnitz.de/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.uvsq.fr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.rhnet.is/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.rhnet.is/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.esat.net/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.esat.net/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.mirror.ac.uk/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://download.mirror.ac.uk/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://kde.paralax.org/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.cronyx.ru/pub/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.fu-berlin.de/pub/unix/X11/gui/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://sunsite.sut.ac.jp/pub/archives/linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://sunsite.tus.ac.jp/pub/archives/linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://linux.cis.nctu.edu.tw/X/wm/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.na.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.sun.ac.za/mirrorsites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.sun.ac.za/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.estpak.ee/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.ntua.gr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.ntua.gr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.hol.gr/pub/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.hol.gr/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.pbone.net/mirror/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.heanet.ie/mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.heanet.ie/mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://linux3.cc.ntu.edu.tw/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://linux3.cc.ntu.edu.tw/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://mi.mirror.garr.it/pub/mirrors/KDE/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://mi.mirror.garr.it/mirrors/KDE/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://mirror.xmission.com/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://mirror.xmission.com/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.iway.fr/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.kddlabs.co.jp/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.kddlabs.co.jp/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.no.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.yz.yamagata-u.ac.jp/pub/X11/wm/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.yz.yamagata-u.ac.jp/pub/X11/wm/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.funet.fi/pub/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.funet.fi/pub/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.planetmirror.com/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://public.planetmirror.com/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://ftp.ussg.iu.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.ussg.iu.edu/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://carroll.aset.psu.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://chernabog.cc.vt.edu/pub/projects/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://chernabog.cc.vt.edu/pub/projects/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://mirror.cc.columbia.edu/pub/software/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://ftp.chg.ru/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://www.mirrorservice.org/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok ftp://www.t.ring.gr.jp/pub/X/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok http://www.t.ring.gr.jp/archives/X/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok