If he was danish, he would have attended the Danish Lutheran Church, and even if he wasn't a regular church goer, the parish priests were responsible for keeping the "vital statistics" of the danish inhabitants. So they would have recorded coming and goings into the parish area, births deaths and marriages. If you have reason to believe that anyone in your family married into an English or Spanish family, the same kinds of records were kept by the Catholic church on the islands.
Find the nearest Family History Center to you, through the Church of Jesus Christ of Latter-Day Saints, then you can order the parish records for the year(s) that you want. The parish registers will list when someone took communion, had birth, deaths, or marriages. Births will list the parents and godparents (usually). If you are not a member of the LDS church, you can still view the films through a microfilm reader at their genealogy library. It costs about fifty-cents to a dollar to get a film. Call the churches near you, and they will help you order the film, view it, and make copies of any pages you find that are relevant to your family.
I would also go through the census records of Copenhagen from http://www.ddd.dda.dk/
A lot of the families who lived on St. Croix or St. Thomas had some roots in Copenhagen, or went back and forth between the west indies and Copenhagen. If you find them on a Denmark census or any of the family, then that may help you find the parish they were from. Then, you'll be able to find a lot of information that you need farther back. I won't get more detailed about searching Denmark records that are online, though, until/unless you need help with that.
It was a common practice for the danes to send their children back to Denmark to be educated. This was especially true during and around the years of slave uprisings. So, I've sometimes found records by doing a blanket search for a part of Denmark (say Copenhagen) with only the birthplace as "dansk vesindien" or "vestindien" or "vest indien" . . . try all three. You can narrow it down with the last name if it gets to be over 250 records returned. For example: do a census search of Copenhagen for Andersen born in the vestindien.
Another strategy is to try the family search website. https://www.familysearch.org/
The LDS church has transcribed many of the danish birth records, and you may get a hit there. That also will give you the parish that you need to trace the family farther.
I've done a lot of research into the Danish West Indies since my own family was born there. I, also, don't travel to the USVI or Denmark. But, I got quite a bit done sitting right here in front of my computer . . . after I found them initially from the parish registers that I mentioned earlier from the danish west indies. If I can be of more specific help, please let me know.