This is a possible solution to protect small coins against 51% miners who forward-stamp the time.
It appears any miner with 51% can cause difficulty to lower forever. Median and average do not affect this. 1/2 the time he will acquire the median, so half the time he will lower it to 1/6 D, averaging 1/3 D every 2N. So D=0 in 1 day. If you cut the high 6 out of 42, as well as the low 6, the result is the same. Cutting only the high 6 will cause difficulty to rise forever if miners are honest. Again, 51% = truth. 51% = owner of clock. There is no solution except for the coin to self-destruct until miners are honest.
If I am correct, big miners must already know this, so they must be doing it only some of the time because they do not want it to cause too much trouble and then cause a hard fork.
Timestamp fix idea: penalize all miners if some miners have a bad timestamp.
This will protect against a 20x attacker who is trying to control the clock. This numbers are approximate.It depends on how fast rest of difficulty algorithm brings difficulty back down. If a negative block timestamp appears, raise difficulty 10x and lower 11% per block for 20 blocks, then resume regular difficulty. If more than ~5% of network has a bad timestamp (either too far backward or too far forward that causes others to be backward), difficulty would rise indefinitely. Those not aware will stop mining. Those aware of this will remain until the miners with bad timestamps leave. It forces honesty and more stable difficulty. All miners are penalized if some miners keep a bad timestamp. 1% bad timestamps will cause 22% increase every 100 blocks.
Past coin holders are protected against coin dilution.
If miners have an error of either exactly +20 or -20 seconds then 25% of the time a forward stamp will be followed by a backward stamp. 1.24% of solves are within this 40 second window when T=240. 0.25 of 0.0125 = 0.3% of the time this will occur on accident. So everyone with ~20 second error is 1/3 as bad as 1% with 240 second error. These numbers are approximate since the statistics is complicated but it shows the things that have to be considered.
A problem with the above is that a malicious actor with only 1% hashrate can make your difficulty jump 10x once every 100 blocks.