Penicuik should have taken the lead in the ninth minute when a cross by Andy Forbes fell to Ross Montgomery. He volleyed the ball first time and keeper Kyle Marley made a top-class, one-handed save to deny Penicuik the lead.
Two minutes later, a Lough header was tipped over by Marley. Keith Lough then had a chance, but he dragged his shot a foot wide of the target.
At the other end, a long-range strike by Conrad Courts was well saved by Billy Bald. Then, a powerful strike by Chris Inglis had keeper Marley at full stretch to save low down to his left.
Just before half-time, the best chance of the game fell to Pencuik when Lough had only the keeper to beat – but he dragged his shot past the post.
Five minutes into the half and Penicuik broke quickly up the right wing. Forbes sent over a deep cross that was headed down by Lough to McCallum, but his volley was well saved by Marley.
In the 63rd minute, Wayne Sproule rounded the keeper and was just about to slot the ball home when Courts recovered to dispossess the winger.
Then, at the other end, Stuart Cargill had a great chance, but shot past the post.
In the 81st minute, Kelty had keeper Marley to thank once again when he stuck out a boot to deflect away a net-bound strike by substitute Aaron Somerville.
Immediately afterwards, Ross Montgomery had a chance but his half volley flashed past the post.
Then, in the 91st minute, Penicuik were hit with a sucker punch. Following a throw in to Penicuik, Neil Jancyzk was felled by Scott Dalziel but no foul was given.
He crossed the ball into the box and striker Scott Greig thumped the ball into the top corner of the net to seal all three points for Kelty.
Penicuik: Bald, Forbes, Inglis, Hume, Young, Yancyzk, Sproule, Barr, McCallum, Lough, Montgomery Subs: Somerville, K Scott, S Scott, Lister, Gay.