I am trying to get iOS Universal Links to work for
This is expected behavior. There is nothing in the official documentation about supporting non-standard ports because non-standard ports aren't supported.
The only entitlement value allowed by iOS is a valid, bare domain (
applinks:my-domain.com), which iOS will then use to check for an AASA file over SSL using the standard HTTPS port. In other words, when iOS queries either
https://my-domain.com/.well-known/apple-app-site-association, it needs to find your AASA file without any redirects.