I've been playing around with Bouncy Castle's implementation of RSA (Lightweight API) and got the basics figured out. Looking at their spec for JCE provider implementation I noticed that different padding schemes can be used with RSA. From what I understand, by default null padding is used. So I began exploring OAEP padding, particularly
RSABlindedEngine rsa = new RSABlindedEngine();
SHA512Diges sha512 = new SHA512Digest();
Digest arg3 = ???;
byte arg4 = ???;
AsymmetricBlockCipher cipher = new OAEPEncoding(rsa, sha512, arg3, arg4);
OAEP is specified by PKCS#1, section 7.1.
OAEP requires the following parameters:
There is only one defined mask generation function, called MGF1, and that function is built over a hash function. So your
arg3 is the hash function which MGF1 will use. It may be the same hash function than the first one (I am not sure it may be the same
Digest instance in the Bouncy Castle API; I am talking mathematically here). It may also be another hash function.
The label can be used as a kind of distinguishers between instances (e.g. you could encrypt data with an explicit "purpose" encoded in the label). It is handy in some mathematical proofs, but right now PKCS#1 recommends using an empty string and be done with it. For the purposes described in PKCS#1, an empty label is as good as any.
The decryption process must know those parameters to operate. It is customary to encode them in the structure which comes along with the encrypted message and says "this is encrypted with RSA/OAEP"; that's how it happens in CMS.
When in doubt, use the same hash function as first parameter and for MGF1, and use an empty label.