Miller sent Town keeper Matt Glennon the wrong way from the spot in the 88th minute after Nathan Clarke was adjudged to have fouled Delroy Facey in the box.
Gillingham's victory ended a dreadful run of form that had seen them fail to score in three successive home games, with Mark Stimson's new regime threatening to send the Kent side into League Two.
It took until the 25th minute for either side to force a chance of note, Miller attempting a speculative effort from 30 yards that flashed wide of Glennon's left-hand post.
As a forgettable half drew to a close, both sides missed great chances to give the game the goal it so desperately needed.
Town striker Phil Jevons was the first to miss when he sliced his shot horribly wide from just inside the area following neat work down the left by Chris Brandon.
And his side should have fallen behind a minute later, only for Gillingham skipper Andrew Crofts to head a glorious chance wide from Sean Clohessy's perfect cross.
Chances remained at a premium in the second half as both managers made changes in an effort to break the deadlock.
Miller remained the chief threat for the home side and his low corner fashioned a chance for Simon King on 63 minutes that the defender side-footed narrowly over.
The visitors appeared content with a draw as Gillingham began to show greater signs of urgency in their play without creating anything clear cut.
But when Facey dramatically fell to the ground on 88 minutes under Clarke's challenge, referee Steven Cook surprisingly pointed to the spot and Miller kept his composure to score.
Huddersfield poured forward in response and both substitute Andy Booth and Andy Holdsworth missed late chances to snatch an equaliser.