Jimmy James (Michael James) a US born lived most of his first 25 years in Jamaica, moving to London in 1965.
He already had 2 hits without the Vagabonds before he was asked by the them to become their lead singer. This song was his second hit. The other was Bewildered & Blue. The group on both songs is Hersang & His Combo the top band of that time on Jamaica
Come to Me Softly was redone many times. The US version is the best known. But there's also an other US version released on the Sundown label which to me comes second next to the Jamaican original.
I don't like the Atco version, maybe you'll understand after you've heard the original.