Victor was born in London, the second son of William and Maud Lucia Cazalet, nee Heron-Maxwell, of Fairlawne, Shipbourne, Kent. The Cazalets had long connections with Russia where they were merchants in St Petersburg for several generations until 1917.
In 1901, Victor was living with his parents, brother and younger sister at Fairlawne House. They had twenty indoor staff. Victor was educated at Eton. During the First World War in which his brother, Edward, was killed, he served with the Queen’s Own West Kent Yeomanry and was awarded the Military Cross in 1917. He came up to Christ Church in 1919 and won Blues for Lawn Tennis and Racquets.
In 1924, he was elected Conservative M.P. for Chippenham and held the seat until his death. He was Amateur Squash Champion in 1925, 1927, 1929, and 1930. In the later 1920s and through the 1930s, he went, frequently, to the United States where he had many connections. He was godfather to Elizabeth Taylor, the actress and film star.
In 1939, at the outbreak of the war, he formed an ack-ack battery in Sevenoaks but because of his knowledge of Eastern European languages, the War Office employed him with the Polish forces led by General Wladyslaw Sikorski.
In July 1943, he toured the Middle East with the General. They were returning to the United Kingdom when their converted RAF Liberator bomber crashed in the sea whilst taking off from the short and difficult runway at Gibraltar. The pilot was the only one of the sixteen people on board who survived. Cazalet's body was found still strapped in his seat.
He is buried in the Gibraltar [North Front] Cemetery. Plot 1. Row A. Grave 5.