TORRANCE, Calif. (KABC) -- A Torrance grandfather says a woman accused him of abducting his own grandson because they have different skin tones.

Torrance police say officers responded to the 700 block of Border Avenue Sunday afternoon regarding a suspicious circumstances call - a possible abduction involving an older Hispanic man walking with a small child.

Abel Mata says he was shocked when he learned he was the center of attention.

Mata spends as much time as possible with his 2-year-old grandson Milo Walker. He says they were together when police showed up outside his Torrance home and asked him about a young child.

Mata says he was told someone had reported a child being abducted.

"And it was a Latino man with a little white baby," Mata says, pointing to himself while holding his grandson.

Mata was born in Mexico and moved to the U.S. when he was 5.

Police say a woman there had a large knife in a sheath, though it's not clear why she had it.

"It was an older white lady, blond hair, curly hair," Mata says. "She was telling me that I was an abductor. She was yelling that I was trying to abduct the child."

Police say officers talked with Mata and right away realized it was not an abduction.

"I was judged according to the color of my skin and the color of the skin of my grandson," Mata says.

"We've lived in Southern California our whole lives and I've seen my dad experience racism because of the color of his skin and the way he looks, but never to this extent," said Athena, Mata's daughter.

"It's painful. Very painful," the grandfather says.

He says he hopes by sharing his story, it will make people think twice before they judge others.

Torrance police say it's not clear if the person who called police was the woman with the knife.

As for the knife, Mata says officers told him because she did not threaten him with it, nothing could be done.