As TMZ reported, Britney walked off the set last week during the first taping, leaving behind an empty judge's chair while 4 contestants did their thing.

Sources connected with the production tell us Britney was "overwhelmed" and needed to leave. People who were at the taping tell us she was upset by the audience reacting negatively to some of her comments.

Fact is ... "X Factor" wanted Britney so desperately, she was able to negotiate terms that are "unheard of" -- so say people with direct knowledge of her deal. As one source put it, "Simon is a smart and shrewd businessman who knew Britney could make his show and he was willing to give her just about anything to come on board -- and he did."