Diminishing Returns (sometimes abbreviated as DR) refers to the decrease in a spell's effectiveness. Typically this refers to PvP play though there are some abilities that can be subject to DR in PvE play. It also typically refers to spells that can be considered crowd control or to stuns and slows. Note that DR only goes into effect if abilities are used on the same target within 15 seconds of the end of the duration of the effect.
The purpose of including DR is to try to balance classes with these abilities so they cannot simply lock down players permanently. In the 1.42 patch notes, Blizzard describes the selection of spell vulnerable to diminshing returns as those "spells as having durations and cooldowns sufficient to almost permanently slow or immobilize targets but without adequate controls or other limiting factors." Note, too, that some spells are grouped together in a category for the purposes of applying DR. What this means is that even if you don't cast the same spell, if it's in the same category as the first spell you cast, it will be subject to the DR effect.
For example if a rogue uses sap on a player and then uses gouge, the sap will have full effect but the gouge will be subjet to diminshing returns. However, due to the 15 second rule, if the rogue uses sap and lets the sap wear off (or be trinketed out of), waits 15 seconds and then uses gouge, the gouge will not be subject to DR and will have full effect.
The exact numbers for dimishing returns are (note this assumes that the below are cast within the 15 second window and are either the same ability being used or an ability in the same category):
In PvE, those abilities that are subject to diminishing returns have the DR effect applied to the target not to the player. What this means is that if one person in a party casts a spell that triggers DR, if a different player then casts an ability in the same category on the same target, that ability will be subject to DR and have a 50% reduced duration. This is so that two players cannot alternate to get around the DR effects in PvE.
Prior to the 2.4 patch Diminishing Returns used to also apply to honor in PvP. If Player A killed Player B more than once, each subsequent kill would be worth 10% less honor until eventually Player A would get no honor from any kill on Player B. (obviously the 15 second ruel didn't apply here). In 2.4, however, they removed diminishing returns on honor in PvP and now full honor is granted no matter how many times Player A kills Player B. Player B seriously hates Player A now. It should be noted, though, that in that same patch Blizzard also implemented the change that says that if a player dies more han 50 times in one battleground (no matter who killed them) that that player will no longer be worth honor for the rest of the battle.