NetBSD Mirror FAQ

FTP and ISO Mirror FAQ

BitTorrent Mirror FAQ

WWW Mirror FAQ

CVSweb Mirror FAQ

FTP and ISO Mirror FAQ

Overview for setting up a NetBSD FTP/ISO Mirror (top)

The basic steps for setting up a NetBSD FTP or ISO Mirror are:

  1. Prepare disk space. See "How much disk space is needed for an FTP/ISO mirror" entry below for information on amount of space required.
  2. Retrieve a copy of the NetBSD FTP site contents
  3. Set things up to synchronize the contents of your mirror with ftp.NetBSD.org.
  4. Send in the information about your mirror to . See "Contact information format for FTP/ISO Mirrors" for details on that.

How much disk space is needed for an FTP/ISO mirror? (top)

The amount of disk space needed fluctuates due to things such as pre-built packages being added, binary snapshots being deprecated, and new formal releases of NetBSD. At this time (2005-11) a complete mirror of NetBSD needs about 220 GB and a new release of NetBSD adds between 10 and 25 GB (including binary packages and ISO images).

Space requirements for ISO files are dependent on how many binary releases are supported at the time. The size may be severely decreased when old branches becomes obsolete and would grew when new versions of NetBSD will be released. Each release takes from 7 up to 9 GB, and currently (2005-11) the complete iso area needs around 36 GB space.

How much bandwidth does offering an FTP/ISO mirror require? (top)

At the moment, we do not have any detailed numbers on bandwidth usage from our mirrors. The actual usage of any mirror depends too heavily on a variety of factors such as already available bandwidth, geographical location relative to other mirrors and amount of data mirrored for us to provide you with accurate numbers.

If you have been running a full mirror for a while and have collected bandwidth statistics, please let us know.

Retrieving/Syncing a copy of the NetBSD FTP site and ISO images area (top)

There are several ways that you can retrieve and sync the files. Depending on your personal preference, your network connection, the packages installed on your systems etc. you may choose one over the other. Either way, it's probably a good idea to first run the relevant command by hand and then, when you have confirmed that everything is working as it should, create a script to be run by cron(8).

All ISO images of the NetBSD distributions are stored on the FTP site inside iso/ subdirectory. Thus, mirroring of ISO images is very similar to mirroring the entire FTP site.

Below you can find some examples of how to keep your local copy in sync. Simply replace "<FTPROOT>" with the actual root of your ftp server.

Notes about a NetBSD FTP/ISO mirror (top)

As a NetBSD FTP/ISO mirror, it's important that you synchronize your copy of the NetBSD FTP tree and ISO images area regularly. Otherwise, old and/or obsolete files may confuse users.

Also, it might happen that due to an e.g. security issue, file removal (or another administrative task) may be requested from the NetBSD Project.

Where should an outage be announced (top)

If your FTP/ISO mirror experiences any significant system outage, you should notify as soon as possible so that we can temporarily redirect your CNAME (if there is one) to a nearby alternative. Also, you may wish to send a note about the outage to the netbsd-users mailing list.

If the CNAME for your mirror has been redirected, be sure to let us know when the situation is rectified, so we can revert the change.

Contact information format for FTP/ISO mirrors (top)

Contact information needs to be submitted to .

This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD FTP/ISO mirrors should be sent in the following format:

mirror          *machine name*
country         *location (country) of host*
location        *more specific location (e.g, ``University of Foo'')*
ftp		*url to NetBSD ftp mirror area*
ftp-host	*fqdn of the host providing ftp mirror service*
ftp-releng	*url to NetBSD release engineering files, aka NetBSD-daily*
iso		*url to NetBSD iso images area*
iso-host	*fqdn of the host providing iso images mirror service*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          netbsd.foo.com
country         Japan
location        Department of Computer Science, Foo University
ftp             ftp://ftp.foo-univ.ac.jp/pub/NetBSD/
ftp-host        ftp.foo-univ.ac.jp
iso		ftp://iso.foo-univ.ac.jp/pub/NetBSD/iso/
iso-host	iso.foo-univ.ac.jp
organisation    Foo University
contactperson   Foo Bar
contactemail    Bar@foo-univ.ac.jp
contactkeyid	GA23FG12
contactphone    +81 3 xxxx yyyy
contactaddress  9876 Foo City, Tokyo 1234567 Japan

Please use ftp* and iso* fileds in case only if you providing appropriate mirror services. For example, omit ftp-releng field if there are no releng files on your server, or don't use any of ftp* fields if you running ISO mirror only.

If the contact information for your mirror ever needs to be changed, please feel free to let us know at .

How to get (ftp|iso).country-code.NetBSD.org (top)

If you would like a ftp.country-code or iso.country-code (e.g. ftp.jp) CNAME for your mirror, please ask . We do not normally do subdomain delegation, so you cannot run a nameserver for "country-code.NetBSD.org".

If you can no longer provide a FTP/ISO mirror (top)

Please let us know at so that we can remove your site from our list and redirect the DNS name to an alternative site.

BitTorrent Mirror FAQ

Bit-what? (top)

BitTorrent is a tool for distributing files. It's extremely easy to use - downloads are started by clicking on hyperlinks. Whenever more than one person is downloading at once they send pieces of the file(s) to each other, thus relieving the central server's bandwidth burden. Even with many simultaneous downloads, the upload burden on the central server remains quite small, since each new downloader introduces new upload capacity.

Overview for setting up a NetBSD Bittorrent Mirror (top)

The basic steps for setting up a NetBSD BitTorrent Mirror are:

  1. Install net/bittorrent
  2. Fetch the .torrent files from our FTP site. The files are in the /pub/NetBSD/iso/<release> subdirectory, so if you already are mirroring this directory, you're all set. For more information see this page.
  3. Start BitTorrent: 
    cd /ftp/pub/NetBSD/iso/<release>; (btlaunchmany.py .) >/dev/null 2>/dev/null &
    . If you didn't download the ISO images already, they will be downloaded now. You can use the --max_upload_rate arg and --max_download_rate arg options to limit the transfer rates.

Contact information format for BitTorrent mirrors (top)

Contact information needs to be submitted to . This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD BitTorrent mirrors should be sent in the following format:

mirror          *machine name*
country         *location (country) of host*
location        *more specific location (e.g, ``University of Foo'')*
bittorrent	*BitTorrent URL / URL with additional Information*
ipv6		*yes or no, if your server supports IPv6*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror           ftp.NetBSD.org
country         USA
location        Silicon Valley, California
bittorrent	http://www.NetBSD.org/mirrors/torrents/
ipv6            yes
organisation    The NetBSD Foundation, Inc.
contactperson   Admins
contactemail    <email-here>@NetBSD.org
contactkeyid	AF226A4C
contactphone    555-123-1234
contactaddress 	Somwhere in Silicon Valley, California

WWW Mirror FAQ

Overview for setting up a NetBSD WWW Mirror (top)

The following is an overview of the recommended procedure for setting up a NetBSD WWW mirror:

  1. For setting up a mirror with this procedure, you'll need cvs, ssh, and wget. Both cvs and ssh ship with NetBSD versions after 1.5, and wget is available from the NetBSD packages collection (see net/wget).Of course you'll also need a web server, such as apache.
  2. Retrieve a copy of the NetBSD WWW site. See "Retrieving a copy of the NetBSD WWW site" for instructions on how to do so with anoncvs.
  3. Set things up to synchronize the contents of your mirror with www.NetBSD.org. See "Syncing the NetBSD WWW site contents" for an example of how to do so using a cron job.
  4. Send in the information about your mirror to . See "Contact information format for WWW Mirrors" for details on that.

How much disk space is needed for a WWW mirror? (top)

The amount of diskspace needed for a complete copy of the htdocs repository is currently (2005-11) around 96 MB. The total amount should not vary too much, but will obviously grow with time.

Retrieving a copy of the NetBSD WWW site (top)

To retrieve a copy of www.NetBSD.org via anoncvs, you should do: 

# su - nbwww
# cd /path/to/htdocs
# env CVS_RSH=ssh CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot \
  cvs checkout -P htdocs

Note: "/path/to/htdocs" should be where you want the mirror of www.NetBSD.org to reside. The directory should be owned by an e.g. "nbwww" user.

Syncing the NetBSD WWW site contents (top)

The best way to do this is to set up a cron job for your e.g. nbwww user to update the web site at a regular interval using anoncvs and wget (more on the wget usage below): 

# crontab -u nbwww -e
SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin
MAILTO=whoever@gets.the.mirror.mail
CVS_RSH=ssh
CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot
0 1 * * * cd /path/to/htdocs && cvs update -dP && wget -nv -xnH -N -Fi mirrors/fetch.html && wget -nv -xnH -N -r -np http://www.NetBSD.org/Gnats/

Note: In the suggested contents, you'll want to change "MAILTO" to reflect who should get the mirror-related mail. Also, "/path/to/htdocs" should be changed to where your www.NetBSD.org mirror resides.

What is the purpose of htdocs/mirrors/fetch.html (top)

The fetch.html file links to the handful of documents on www.NetBSD.org that are generated by www.NetBSD.org. These files get generated daily from content in the doc module of the NetBSD CVS repository.

In the example cron tab above, you'll see "wget -nv -xnH -NFi mirrors/fetch.html". This means that daily, through wget, these www.NetBSD.org created files will be updated in your copy of the web site contents.

Ensuring HTTP/1.0 clients get the right page (top)

It is much preferred that NetBSD WWW mirror sites run a separate virtual web server for a www.NetBSD.org mirror on a separate IP address. This is so that clients using HTTP/1.0 still get to the correct web site.

If that's not possible in your set up, then the other solution for HTTP/1.0 clients would be putting the NetBSD mirror as the primary name-based virtual host.

Contact information format for WWW mirrors (top)

Contact information needs to be submitted to . This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD WWW mirrors should be sent in the following format:

mirror          *machine name*
country         *location (country) of host*
location        *more specific location (e.g, ``University of Foo'')*
www             *WWW URL*
www-host        *external WWW name*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          netbsd.foo-univ.ac.jp
country         Japan
location        Department of Computer Science, Foo University
www             http://netbsd.foo-univ.ac.jp/
www-host        netbsd.foo-univ.ac.jp
organisation    Foo University
contactperson   Foo Bar
contactemail    Bar@foo-univ.ac.jp
contactkeyid	GA23FG12
contactphone    +81 3 xxxx yyyy
contactaddress  9876 Foo City, Tokyo 1234567 Japan

If the contact information for your mirror ever needs to be changed, please feel free to let us know at .

How to get www.country-code.NetBSD.org (top)

If you would like a www.country-code (e.g. www.jp) CNAME for your mirror, please ask . We do not normally do subdomain delegation, so you cannot run a nameserver for "country-code.NetBSD.org".

If you can no longer provide a WWW mirror (top)

Please let us know at so that we can remove your site from our list and redirect the DNS name to an alternative site.

CVSweb Mirror FAQ

Overview for setting up a NetBSD CVSweb Mirror (top)

The following is an overview of the recommended procedure for setting up a NetBSD CVSweb mirror:

  1. For setting up a mirror with this procedure, you'll need cvs, cvsup, and the cvsweb CGI script. NetBSD 1.6 and above ship with CVS, both cvsup and the cvsweb CGI script are available from the NetBSD packages collection (see devel/cvsup and www/cvsweb). Of course you'll also need a web server, such as www/apache.
  2. Retrieve a copy of the NetBSD CVS tree. See "Retrieving a copy of the NetBSD CVS tree" for instructions on how to do so with cvsup.
  3. Set things up to synchronize the contents of your mirror with the cvsup server. See "Syncing the NetBSD CVS tree contents" for an example of how to do so using a cron job.
  4. Send in the information about your mirror to . See "Contact information format for CVSweb Mirrors" for details on that.

How much disk space is needed for a CVSweb mirror? (top)

The amount of disk space needed is rather large due to the fact that you are providing an interface to the entire CVS repository. Currently (2005-11), the total disk space required for all sources in CVS is around 3.5 GB.

Retrieving a copy of the NetBSD CVS tree (top)

To retrieve a copy of the CVS tree via cvsup, you should do: 

# su - cvsweb
# mkdir -p /home/cvsweb/netbsd
# cd /home/cvsweb

Note: "/home/cvsweb/netbsd" should be where you want the mirror of the tree to reside. The directory should be owned by an e.g. "cvsweb" user. Create an "supfile" using your editor (For examples, see netbsd-supfile). Please take a look at the list of available public cvsup servers, you'll find the list here. 

# cvsup -g -L 2 netbsd-supfile

Syncing the NetBSD CVS tree contents (top)

The best way to do this is to set up a cron job for your e.g. cvsweb user to update the tree at a regular interval using cvsup: 

# crontab -u cvsweb -e
SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin
MAILTO=whoever@gets.the.mirror.mail
0 1 * * * cvsup -g /home/cvsweb/netbsd-supfile

Note: In the suggested contents, you'll want to change "MAILTO" to reflect who should get the mirror-related mail.

How to setup the cvsweb CGI script (top)

Take a look at the cvsweb.conf and be sure to edit the following options:

@CVSrepositories = (
        'netbsd'  => ['NetBSD',           '/home/cvsweb/netbsd'],
);

$mancgi = "http://netbsd.gw.com/cgi-bin/man-cgi?%s+%s+NetBSD-current";

Place cvsweb.cgi in your web servers' cgi-bin directory and be sure to edit it to point to your cvsweb.conf.

Ensuring HTTP/1.0 clients get the right page (top)

It is much preferred that NetBSD CVSweb mirror sites run a separate virtual web server for a CVSweb mirror on a separate IP address. This is so that clients using HTTP/1.0 still get to the correct web site.

If that's not possible in your set up, then the other solution for HTTP/1.0 clients would be putting the NetBSD mirror as the primary name-based virtual host.

Contact information format for CVSweb mirrors (top)

Contact information needs to be submitted to . This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD CVSweb mirrors should be sent in the following format:

mirror          *machine name*
country         *location (country) of host*
location        *more specific location (e.g, ``University of Foo'')*
cvsweb          *CVSweb URL*
cvsweb-host     *external CVSweb name*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          netbsd.foo-univ.ac.jp
country         Japan
location        Department of Computer Science, Foo University
cvsweb          http://netbsd.foo-univ.ac.jp/cgi-bin/cvsweb.cgi/
cvsweb-host     cvsweb.netbsd.foo-univ.ac.jp
organisation    Foo University
contactperson   Foo Bar
contactemail    Bar@foo-univ.ac.jp
contactkeyid	GA23FG12
contactphone    +81 3 xxxx yyyy
contactaddress  9876 Foo City, Tokyo 1234567 Japan

If the contact information for your mirror ever needs to be changed, please feel free to let us know at .

How to get cvsweb.country-code.NetBSD.org (top)

If you would like a cvsweb.country-code (e.g. cvsweb.jp) CNAME for your mirror, please ask . We do not normally do subdomain delegation, so you cannot run a nameserver for "country-code.NetBSD.org".

If you can no longer provide a CVSweb mirror (top)

Please let us know at so that we can remove your site from our list and redirect the DNS name to an alternative site.

Back to  NetBSD Documentation
[NetBSD flag] Home page

(contact us)   Generated from %NetBSD: mirror.xml,v 1.6 2006/03/04 20:27:54 jschauma Exp %
Copyright © 1994-2006 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.