I expected any announcement on Slashdot to generate a huge demand for bandwidth (the so called Slashdot effect) and decided to post an announcement to Slashdot last (ie., after posting announcements to all other applicable sites). While there have been some reports of the impact of the Slashdot effect diminishing in recently, I did not want to take any chances. I hoped that announcing the availability of the pdf file on a variety of forums over a two week period would take the sting out of the Slashdot bandwidth demand.
One annoying consequence of using the ntlworld site is that no log of file accesses or visitor information is provided. So I would have no way of knowing how many downloads of the pdf file, from this site, had occurred.
I posted a request to the ACCU general mailing list asking if anybody had any bandwidth to spare. Chris Wright came forward and offered the use of some of the bandwidth he had available to him.
The pdf file was also available from various other web sites. These sites had enough bandwidth to handle any downloads that were likely to occur (e.g., from people finding the site via a Google search).
Chris detected two people trying to hack into the server hosting the pdf file. Both of them downloaded the document, then started looking around at various directories and then tried a few port scans on the web server, as well as a few brute force attacks. They weren't too clever about hiding their IP address. This behavior was reported to their respective ISP.
Once I knew that my story had been accepted by the Slashdot editors I regularly checked to see whether it had been posted to the main web page (there can be a delay of several hours between a story being accepted and it appearing on the web site). I was lucky enough to catch the story before too many comments had been posted and was able to post a comment that included the link to the mirrordot mirror. This comment was soon moderated up to level 5 and appeared as the first comment after the story itself.
Jay Jacobson, one of the creators of mirrordot, reported that during the first 24 hours there had been 1,987 downloads of the pdf file and approximately 16 GB of bandwidth had been consumed.
The following shows how total demand for bandwidth, by mirrordot, varied over time (the book's pdf was the only file consuming significant amounts of bandwidth during this time).
The ntlworld address was given as the primary point of download. The address of an available mirror (supplied by Chris Wright appeared next to it). Experience suggests that Slashdot readers are very familiar with the Slashdot effect and may try to limit the load on the primary site by selecting one of the available mirrors.
Did the ntlworld address attract significantly more than the 11,500 visitors to Chris's site? If readers experienced problems or slow download speed with the primary site, are they likely to try one of the mirror sites? In the hours following the publication of the story on Slashdot I was able to download the pdf file at the full rate supported by my 2 Mbit link. But then I am an NTL customer connected directly to the NTL network, as are nearly 1.5 million broadband customers in the UK.
It is possible that the ntlworld site successfully processed twice as many visitors as Chris's site (ie., 23,000), and it is also possible that only half as many were successfully processed because of capacity problems (ie., 5,750).
The potential disadvantage of using BitTorrent is that downloads may take significantly longer (it depends on how many other people in the network have a copy of the requested file). When distributing very large files (e.g., iso images) BitTorrent may be the only method of download. I expected Slashdot readers, given the choice, to try non-BitTorrent methods of download first. I also believed there was sufficient bandwidth available to meet these demands and so did not provide a BitTorrent link.
Thirty-five minutes after the story was posted Michael Williamson set up and seeded a BitTorrent link. In the five days after the story appeared the .torrent file was accessed 274 times and there were 165 complete downloads. While this comment was quickly moderated to level 5, it was not listed close to the text of the original story.
The address of the book's web page was posted in a comment to the story and quickly moderated to level 5. However, this comment was not near the top of the list of posted comments. Traffic to the knosof.co.uk site did increase significantly. The following graph shows the dramatic increase in page hits that occurred. The increase in bandwidth was not so dramatic. The total amount of traffic did not come close to causing any maximum limits to be exceeded.
For some reason one Slashdot reader posted a comment listing my recently started blog as the book's web page. Although this comment was posted earlier than the book's current web page address at knosof.co.uk, it was not subsequently moderated to a high enough level to be prominently listed. The bandwidth used by the coding-guidelines.com site increased by approximately a factor of two throughout the week following the Slashdot story. Given that this web site only came into existence in June and was not yet widely known, this increase in bandwidth was not exceptional.
The time difference between stories that appeared immediately before the book pdf story was 2 hours 32 mins, 1 hour 14 mins, and 35 mins. The time difference between the book pdf story and the immediately subsequent stories was 47 minutes, 1 hour 20 mins, 2 hours 7 mins, 2 hours 51 mins, and 3 hours 16 mins. These figures are reasonably typical of the kinds of intervals that occur between stories appearing on Slashdot. The bandwidth and hit-count graphs appearing above show that traffic volume comes down from its peak within a few hours
The number of comments posted, at just over 400, was towards the upper end of the level of commenting usually seen.
The stories covered by the Inquirer cover a wide range of IT related subjects, with only a small percentage being directly related to software development. It is to be expected that visitors to the Inquirer site have a wide range of IT related backgrounds, with only a small percentage being software developers. The Inquirer displays a list of the current top five most popular stories down the left-hand side of every page. When it first appeared the C book story did not make it into this top five list (or if it did, the period of time was short enough that your author did not see it occur). However, a few hours after the C book story (which included a link to the Inquirer story) appeared on Slashdot, the original Inquirer book story appeared at number five on the top five list. The obvious conclusion to be drawn is that significantly more people read the Slashdot story, clicking on the link to read the Inquirer story, than read the original Inquirer story when it first appeared on that site. E-mail provides a possible method of estimating the number of downloads made by readers of the Inquirer story. Some of the people who had started to read the book sent me an e-mail (some thanking me for making it available, others pointing out typos). The Inquirer story resulted in one e-mail being sent to me by a person who had downloaded the pdf; I also received one e-mail from somebody who had seen one of the newsgroup announcements and downloaded the pdf; the Slashdot story generated 30 e-mails. Can we infer that the Inquirer story generated 1/30th of the downloads of the Slashdot story? A single e-mail does not have much statistical significance. Two e-mails would imply only a 1/15th difference.
Based on this analysis it is likely that the story in the Inquirer resulted in around about 1,000 downloads of the pdf file.
During the period June 28 to July 4 the logs provided by Chris Wright show 498 unique visitors and 3.6 GB of bandwidth used. The coding-guidelines.com logs show approximately 200 unique visitors during this period. Without a log of accesses to the ntlworld site it is difficult to estimate the total number of downloads occurring as a direct result of these posts. Numbers similar to those seen by Chris Wright would seem to be reasonable estimates.
While stories have appeared on other non-English web sites, accesses made via these stories have occurred as a steady stream over a period of time (a week or so).
Chris Wright 11,500 ntlworld 5,750 to 23,000 (estimated bounds) mirror dot 2,000 coding-guidelines.com 200 BitTorrent 165The following lists an estimate of the number of downloads attributed to announcements posted to news groups and entries in people blogs in the two weeks prior to the story appearing on Slashdot.
The Inquirer 500-1,000 News groups 1,200-1,600 Blogs 700-1,100
Leaving the posting of a story to Slashdot last is a dangerous plan that has little benefit. It is dangerous in that somebody else might read an announcement posted to another site and decide to post a story to Slashdot themselves. The evidence from this book's (i.e., a developer oriented book) series of announcements is that Slashdot is likely to generate over 90% of the downloads that occur in the first month or so of it being made available. By posting a story to Slashdot first an author is able to control when it appears (to within a few hours) and the primary links to where it can be downloaded.