Regardless, what normal conversation doesn't go off topic? You're out and you meet someone, begin talking about something, and when you finish, you end up talking about something totally different. It's normal, happens quite often I might say.
I'm not trying to be rude or anything, but to me, it really doesn't matter if you end up talking about something else in the end.
Just my