Djurgården was the first to have the ball in goal when Kalle Holmberg was on the return from Niclas Bärkroth's post shot, but it was rejected due to offside.
Instead, it was the guests who took the lead when Alexander Johansson nodded in a nice corner.
Djurgården created more chances and in extra time in the first half they got a penalty after Haris Radetinac was sent off.
Midfielder Magnus Eriksson gained the confidence from eleven meters, but Philip Mårtensson saved his attempt.
The equalizer came early in the second half through a lively Edward Chilufya.
The stadium exploded, but that joy was short-lived.
Varberg's intruders saw to that.
In the 55th minute Ryan Moon made it 2-1 after a corner and just seven minutes later Albin Winbo extended the lead when he finished a nice attack.
Joel Asoro reduced to 2-3 in extra time but Djurgården did not come any closer than that.
The loss means that many results need to go Djurgården's path for them to be able to take gold.
They make up the lead with leading Malmö, AIK and Elfsborg.