Whenever I want to know about a particular construction, I go to the "Books" section of Google, where thousands of books have been digitalized for us.
I have just gone there and typed (with the quotation marks) "my first time to visit." There were about 40 books that had that construction.
I then typed (with the quotation marks) "my first time visiting." There were at least 100 books with that construction and many more (I got tired counting).
I suggest to learners that they use "my first time visiting."