The core differences between the games are primarily thematic in nature; in many ways Requiem is closer to Masquerade as it was originally conceived than Masquerade had become over the 13 years since its inception.
The intention of these changes were along two lines: mechanical (rules) alterations designed to make the game more playable, and cultural differences to alter the focus of the game from the action-horror game it had become to the personal-horror game the designers had originally intended.
As far as mechanical differences go, the first and most obvious divergence between the games is that the concept of "vampire generation" has been eliminated. A vampire's power is unrelated to that of the vampire who spawned him; instead, he starts off at the lowest level and steadily grows more powerful with age (though he can rise faster by absorbing the power of other vampires).
Vampires now have new abilities to help them uphold the Masquerade, including a blurred image on video cameras, as well as the previous games' device where a vampire may lick wounds he or she has inflicted to heal them.
Disciplines have broadly changed to make them less helpful in combat and more helpful in other ways. A number of original clans (some now bloodlines) have changed in theme and gained different Disciplines to reflect this; for example, the Nosferatu have changed from revolting outcasts to actual monsters, and the entirely new Nightmare Discipline helps them maintain this image.
The more obvious and far-reaching differences are cultural. Most noticeably, the number of clans has dropped from thirteen to five; each broadly represents a classic vampiric archetype from literature. In Masquerade, your clan informed most aspects of your social interactions with other clan members; in Requiem, the clan one is "born" into is less important than the political faction ("Covenant") one chooses to belong to. (This varies from game to game; sometimes neither has any bearing.) This has caused an overwhelming change to the way the politics of the game work.
The most important change between games is the emphasis on the sect (now called "Covenant") a vampire belongs to. In Vampire: The Masquerade, a vampire belonged to either the Camarilla (or their argumentative offshoots, the Anarchs), the Sabbat (and their secret, super-violent arm, the Black Hand), the Inconnu, or he or she was isolated and alone. In the Masquerade setting, perhaps 50 vampires were Inconnu, all exceptionally ancient and powerful (and rarely mentioned in supplements after 1st edition; was a notable exception). Most vampires belonged to either the Camarilla or the Sabbat. Given that the sects were locked in near-perpetual political, ideological, and physical conflict, this strongly discouraged involved political gaming between the two, and encouraged more direct conflict and espionage.
Requiem removes this perpetual war, in an effort to create a world more in keeping with the themes of ennui and politics (and to scale down the violent action scenes that war made possible). The Covenants are the replacement for sects, and no Covenant is at war with any other. Broad comparisons between previous sects and Covenants may be drawn between the Carthians and Anarchs, the Invictus and the Camarilla.
In many ways, the Circle of the Crone, Ordo Dracul, and Belial's Brood represent different faces of the Sabbat: the Ordo Dracul being the Sabbat drive to escape the vampiric state, the Circle of the Crone is what has replaced much of the Sabbat's more openly pagan-styled ceremonies and philosophies, and Belial's Brood is essentially how the openly violent and monstrous Sabbat was conceived back in 1st edition Vampire: The Masquerade, before revised edition made it more philosophical in tone. SeVIIen is clearly the replacement for The Black Hand, now with significantly less possibly confusing story elements.
Significantly, Requiem removes the creation myth involving Caine and the Antediluvians. This story is no longer the only origin story; vampires are uncertain of their origin. Due to Torpor, even the oldest Kindred have no clear memory of their past and are forced to guess at their races' beginnings. Though vampires still conspire against each other, they are no longer puppets of ancient all-fathers, nor does the doomsday prophecy of Gehenna loom over them. None of the Covenants even claim to know the origin of vampires, though the Lancea Sanctum and Ordo Dracul claim their founders were both uniquely cursed by God directly, instead of being Embraced.
This is partially tied into the addition of the Lancea Sanctum; in original Vampire: The Masquerade, Christianity was assumed to be the "default" truth behind the Classic World of Darkness. The introduction of Kindred of the East and the Laibon in later books caused huge problems for authors. The Christian faith has now been externalized, and is represented by its own notably Western Covenant.
Basic Mechanics Edit
Vampires no longer have a Generation. Blood Potency fulfills some of the same functions, but it is more in line with the CofD policy of a variable attribute that measures an abstract of 'power'. Embracing and ghouling are more expensive in VtR, costing a Willpower point (permanent for embrace, temporary for ghoul) every time the deed is done. This was designed to limit Ghoul Abuse and a variety of other tactics originally used in the canon of VtM, especially Mass Embrace, which proved difficult for inexperienced storytellers to manage.
There are fewer clans, and all VtR vampires are members of a clan. There are no Caitiff. Bloodlines are now formally defined, with a mechanism for joining them.
Vampires are ahistorical in VtR; unlike VtM with a definite beginning and Vampires who might conceivably remember it, the Torpor mechanism in VtR ensures that the true history of Vampirism is largely unknowable. VtR had a Camarilla, but it ended sometime around the fall of the Roman Empire.