I found this script that find all manually installed packages in Fedora (every argument that I passed
If YUM works like APT (which I assume it does based on your description), it recognizes a difference between explicitly installed packages ("manually" as you say) and implicitly installed ones (i.e. as a dependency).
Rubygems has no such distinction, and thus this is not possible. As mentioned in the comments, Bundler can do this, but that's on a project level more than a system level.