Due to ADSL being asymmetric you will notice quite clearly that the issue is upstream traffic. Many people who are knowledgeable in regard to networks will attempt to classify the traffic and attempt to use QoS to solve this issue.
The problem with this is that traffic coming from the router itself does not go through the router and thus misses being identified and prioritised. Traffic is this statement is somewhat different to the usual definition of traffic. This traffic is TCP acknowledgement packets necessary for the downstream traffic to not keep resubmitting. A server when not receiving an acknowledgement within a certain time interval (TTL) resubmits the same data again.
As the router is the device that sends the acknowledgements or replies, this traffic was being drowned out by my normal upstream traffic and therefore waiting in a queue where it would be delayed and eventually become irrelevant in the equation.
By simply lowering the upstream limit - as you did - you essentially limited normal upload traffic to a rate below the max upstream limit leaving enough headroom for the 'unseen' acknowledgements. This immediately decreases the number of resubmissions and repeated attempts to acknowledge the resubmissions, freeing up a huge amount of baggage on the network and immediately significantly improving performance including latency and therefore improving throughput and the quality of your connection.
I hope this explanation proves valuable :)