Is it possible to have multiple builds of the same application available on test flight, so that different user can download only builds for which they have been invited?
Also, right now, only build is what should be changed. If this is not possible, is there some way to accomplish this with a different setup ? (some setting in iTunesConnect or to setup an app differently)?
No you can't do it, but you have another way,
"Best way to distribute ipa to client is to upload ipa & provisioning profile to www.diawi.com and generate link. You have to send this link to client and client can install ipa in his device by opening generated link in his device's safari and then by clicking install application. Note that provisioning profile must include client's device UDID."