Three teenagers demanded cigarettes and cash before taking the card from the victim, in Three Bridges..
Detectives are appealing for information to help them prosecute three 16-year-olds who were arrested minutes after the attack.
A police spokesman said: "The incident happened at about 7.15pm on Friday in Haslett Avenue East, near to Pembroke Park.
"The victim, who lives locally, was walking along when a group approached him and asked him for cigarettes and money.
"The victim said he didn't have any and was threatened, so he admitted to having a bank card.
"The robbers ordered him to hand it over and give the pin number before letting him go."
Although no weapon was seen, the attackers told the victim they had a knife and left him shaken, though uninjured.
Police stopped three teenagers shortly after the incident and arrested them on suspicion of robbery.
The three - two from London and one from Surrey - were arrested in Southgate Avenue and later interviewed.
They have been released on bail until April 10 pending further enquiries.
Anyone with information is asked to call police on 0845 60 70 999 or contact Crimestoppers anonymously on 0800 555 111.