For the main style declarations, have a look in public_html/layout/nouveau/style.css around like 2546, you'll see:
PHP Formatted Code
/* start gl_mootip css */
background:url(layout/nouveau/images/gl_mootip_bg.png) top left;
padding:8px 8px 4px;
background:url(layout/nouveau/images/gl_mootip_bg.png) bottom right;
padding:4px 8px 8px;
Those are for regular mootips that you can use in your own code (stories, staticpages, in templates, etc.)
For forum specific mootips though, the settings were placed with the forum specific templates. Have a look in private/plugins/forum/css/style.css.
To save yourself from having to remember those changes on an upgrade, create public_html/layout/nouveau/plugins/forum/css/style.css and make your changes there, as the system will look there before going into the private/plugins/forum/css directory.