No ugly old man, that's a 10/10 for me.

He asked if she was enjoying herself doing something out of the ordinary, he asked if she was ok after spanking her, he held off on sex to spend time with her (you could argue the first time was edging her, but he did hold off at the end) and even during sex he held back on fondling and teasing her. Its clear he DOES care about her, unless we read two different Doujins.

It's pretty clear that he doesn't love her, she's just using her for her body.