It was a hard-earned win for Huddersfield who had to battle against the odds after having Danny Schofield sent off for a second yellow card after diving in Worcester's penalty area just before the interval.
Ten-man Town were often put out of their stride by a tenacious Worcester side that rarely held back in their bid for fresh Cup glory.
But Huddersfield broke the stalemate in the 61st minute with Brandon's out-of-the-blue strike to the disappointment of the Worcester side.
Brandon, who has recently signed a new two-year deal at the Galpharm Stadium, popped up to collect a long throw-in from Nathan Clarke. Before Worcester's defence could re-group, Brandon had whipped in a low shot which crept into the net to the right of goalkeeper Danny McDonnell.
It was a cruel blow as Worcester had more than held their own against the League One club. They survived some early pressure when McDonnell saved in quick succession from Schofield and Gary Taylor-Fletcher.
Late in the first half Taylor-Fletcher flicked a gilt-edged chance over the bar and then Schofield was sent off despite some strong protests.
Huddersfield's boss, Peter Jackson, was far from pleased with the decision and later expressed some strong feelings which could easily have cost the Yorkshire club their trip to Stamford Bridge.
To their credit Worcester played with courage and persistence but suffered a knock-out blow when Brandon pounced for his lucrative winner.
But Huddersfield could not rest on their laurels as a desperate Worcester side launched a series of late attacks which had the Terriers at panic stations.
Goalkeeper Paul Rachubka courageously saved at the feet of Leon Kelly and then kept out another effort from Adam Webster to enable Huddersfield to earn their high-profile game at Stamford Bridge.