Not messin with ya Dude! There's a lotta variables involved. You have issues when streaming stuff like video?
So....
Your experience isn't unique. A number of factors could be influencing download times:
1. The size of buffers/windows at the network protocol or application layer. When they fill, a download can suspend until more space is created/cleared.
2. Method employed to validate that no changes have occurred to the downloaded file(s) can be very CPU intensive.
3. CPU availability.
4. Network availability - your Airport may be exclusive to you but the pipe into it may be shared (e.g., DSL) or bounded by the provider.
5. Size of the files being downloaded plays into the other factors.
If you have some sort of monitor (e.g., windows task manager with resource monitors) you can watch your network activity as well as CPU utilzation (by process) to see if there are any resource hogs or "slow" players. Try downloads at different times (e.g., late at night when utilization is apt to be lower at the provider end - assuming they are located in the same or near time zone). Check if you have security apps (beyond the previously mentioned security/integrity checker) that might be inspecting the flows.
Heck, I have a windows machine running right now that has been choking on a 1.4 meg MS patch for over an hour. Depending on how routing/quality assurance is set up on the network, the slowest link in the path can gum up the works.