I am implementing the authentication for an app, and I am using a pluggable system with "authentication methods". This allows me to implement both HTTP Basic as well as HTML-based authentication.
With HTTP Basic/Digest auth the server sends a
The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource.
For HTML I think you should respond with a 400.
This may be true for non-HTML requests as well, since 401 is as far as I understand it more designed to respond to a request to content that requires authentication, not to respond to an authentication request.
HTML does not always allow for pure use of RESTful APIs, so it's ok to cut corners here and there imo, but maybe there is a better way I'm not seeing in this particular case.