They were saying it loud enough for Emily to hear.
Logan's publicly acknowledged daughter was still Sophia, not her. Sophia had an education and appeared respectable, whereas Emily hadn't attended college and had lived in the slums. So Logan chose Sophia to attend important occasions over her.
Logan went out...