Should i stop my ex having contact with my daughter?

my ex husband left me and our daughter 17 months ago as he ran off with a 16 year old girl he was 22 at the time, we got divorced and he now has access to our 2 year old daughter once a week on a wednesday for 1 hour and once a fortnight on a saturday for about 4 hours. this is only when he isnt making excuses not to see her. he is constantly late dropping her off and early to pick her up. he does pay child maintenance . when she comes back from her visits she always smells like an ashtray and always misbehaves quite badly and becomes lazy. i would love to cease all access as i feel he is a lousy father and doesnt care about her or her feelings but where do i stand legally?