Apparently, Cowell didn't want her on board. Instead, he advised she join the rival singing competition.
Cowell explained to In Touch magazine, "Mariah called me about X Factor and I met with her, but my main concern was that she might have found it harder on X Factor than on American Idol.
"Idol is a more gentle show and Mariah is not someone who wants to say 'no' to people... I think she'll have an easier time on American Idol than this show, definitely."
It's not all bad news for Carey, who is reportedly making $18 million for her stint on "Idol."
Younger pop star Britney Spears and Demi Lovato are replacing "X Factor" season one judges Paula Abdul and Nicole Scherzinger. And Cowell is pleased with his decision.
He added, "I am delighted. We literally only approached two people seriously, and that was Britney and Demi."
"The X Factor" premieres tonight at 8 p.m. eastern on FOX.