To allow for "zero downtime" during updates and implementation there are two servers (configured with Apache2) with names a.org and b.org. These are domains not accessible via the public, you have to configure them in your local hosts files to access each of them. During development, the role of "live site" and "development site" switches between one and the other by commenting/uncommenting the ServerAlias tag of their server configuration. That ServerAlias is the actual true domain of the site, truedomain.org.
In order to speed up Drupal I want to do CSS aggregation and compression via the Advagg module. While working on the development site (this time b.org), it worked fine. There was no problem with font loads or anything like that. The problem happened when I made the switch for b.org to become live site by uncommenting the ServerAlias tag. When accessing the site, there were loads of font load errors caused because they were blocked due to CORS. Seems Advagg tried to load them from its b.org ServerName rather than through its ServerAlias tag.
The font imports are in a CSS file and they are relative because the font files are in the same directory. I can't use absolute paths because of my server setup.
The only way out I see is by allowing CORS just for fonts, such that:
Header set Access-Control-Allow-Origin "*"
@CBroe's comment delivered me to the answer (thank you). Thought I'd write it out here as well as give some extras and how to fix them.
admin/config/development/performance/advagg , in there you will find a collapsed menu under the title "Obscure options" , you have to check the following option: "Do not run CSS url() values through files_create_url()"
That fixed my CORS font load problem. The second problem arose when the fonts did not load at all and the text on my site reverted to some default font. This happened because on CSS Compression Drupal removes '' , and the font name had (aaag!) a gorgeous space on it. Of course, once the '' were removed from the url() in the font-face rule, the font didn't exist. So I had to change the font name and rules accessing that name to not have that space.
Once that was done, even after clearing caches I saw that Drupal was still sending old Advagg CSS compression. To fix that, I deleted everything in the folder sites/default/files/advagg_css to force new ones.
And with that, the problem was completed fixed, yay.