You say ‘Yes’ and the mysterious stranger says

’Where, where do you see him?  What is he doing?  Why is he late?’

Do you

 Say ‘Oh he’s just down there’ and point to the where the pirates are

 Say ‘Just joking, I’ve not seen him at all’