Many people may be suffering from "BufferBloat" see below:
"Bufferbloat is high latency in packet-switched networks caused by excess buffering of packets. Bufferbloat can also cause packet delay variation (also known as jitter), as well as reduce the overall network throughput. When a router or switch is configured to use excessively large buffers, even very high-speed networks can become practically unusable for many interactive applications like Voice over IP (VoIP), online gaming, and even ordinary web surfing.
Some communications equipment manufactures placed overly large buffers in some of their network products. In such equipment, bufferbloat occurs when a network link becomes congested, causing packets to become queued in buffers for too long. In a first-in first-out queuing system, overly large buffers result in longer queues and higher latency, but do not improve network throughput.
The bufferbloat phenomenon was initially described as far back as in 1985. It gained more widespread attention starting in 2009."
The above information was taken from Wikipedia (https://en.wikipedia.org/wiki/Bufferbloat).
So it isn't always about the ISP and certainly if you get them to investigate your link they tend to flush the buffers in their system as part of checking and after you have had this done your speed may increase for a period of time before it fills the buffers again. This Bufferbloat does also affect the buffers in your modem/router and networking equipment (such as your network card), newer modems are starting to adopt newer techniques to help avoid/mitigate Bufferbloat but not all modems are new
Other factors affecting speed can be numerous and include noisy lines, the multiple connections you have to make to get to the site you want to visit and time of day. ADSL in somewhat immune to Time of Day compared to cable but once you reach the ISP's network your bandwidth is shared and if more people are online using the Web through the same ISP you will get slower responses (and the speedtest will reflect this with slower speeds). With the copper network getting old noisy lines are having a big impact on speeds, this is because the information is corrupted on its way to whatever site you are visiting and that site and your computer have to re-request and re-resend the information. To overcome this modems will negotiate a slower and slower speed until a speed that has the least amount of errors is achieved. If you notice hissing on your telephone or crackling it is most likely bad copper, poor/dry solder joins, or maybe water in connections (and as long as your line meets old fax standards you won't get much joy in getting it fixed).
So before thinking that a change of ISP will fix your issue I would think of the other issues that could be leading to a low connection speed.