Most of the Normans ruled England from Normandy.  My little branch of the family actually did move to England, but many of us stayed.  We continued to speak our own little version of French, and to rule the island as we thought best.  The irony is that the greatest of Norman kings, forever known as Richard the Lionheart, never did learn to speak English.  You see, England was a secondary holding at the time, truly only important in that it gave us a claim to kingship.