Rudd's first instrument was the French horn, which he studied from the age of 11. His father was an amateur drummer who introduced his son to jazz. In his teens, Rudd began teaching himself to play the trombone; Woody Herman's star trombonist, Bill Harris, was a particular favorite. He played Dixieland, while he attended Yale, with a band called Eli's Chosen Six. From 1960-1962 he worked with legendary pianist Herbie Nichols, who became something of a mentor to Rudd. From 1961-1963, Rudd played in a band with soprano saxophonist Steve Lacy and drummer Dennis Charles. The band would later be informally known as the School Days Quartet, after the 1963 Emanem album of that name. The group's repertoire consisted entirely of Thelonious Monk tunes. In 1962, he joined trumpeter Bill Dixon's free jazz group, which also included tenor saxophonist Archie Shepp and drummer Charles. In 1964, he founded and co-led the New York Art Quartet (with saxophonist John Tchicai) and participated in the October Revolution in Jazz, an early free jazz festival organized by Bill Dixon and held in a New York City café. Rudd spent the latter half of the '60s playing in Archie Shepp's band, Charlie Haden's Liberation Music Orchestra, and a group led by saxophonist Gato Barbieri. In 1968, he formed the Primordial Quartet (with saxophonists Lee Konitz and Robin Kenyatta and pianist/vibist Karl Berger). The group disbanded in 1970. Rudd's compositions for the Jazz Composer's Orchestra were recorded in 1973 on the album Numatik Swing Band (JCOA).

Although Rudd recorded occasionally in the '70s and '80s (notably under Lacy's leadership), he gradually became less visible, as matters of economic survival took precedence over creative concerns. He worked a variety of non-musical jobs and spent time teaching at the college level. After being denied tenure at the University of Maine in Augusta, Rudd moved to the Catskill region of New York state, where he worked steadily in a hotel resort band. The mid-'90s found him busy musically once again. Albums on the CIMP label, under his own name and as a sideman, helped reestablish him as a jazz player. At the turn of the millennium, Rudd performed with some frequency in Europe and New York, regaining his reputation as the father of free jazz trombone.

In 2000, Rudd and Lacy reunited to record (with Lacy's regular rhythm section, bassist Jean-Jacques Avenel and drummer John Betsch, and vocalist Irene Aebi) Monk's Dream for the Verve label; the band also toured in support of the album. Also during the the 2000s, Rudd began delving into various world music projects including 2001's Malicool album, featuring musicians from West Africa, and 2003's Blue Mongol, recorded with Mongolian throat singers. In 2007, Rudd kept the world vibe going with the Afro-Cuban and South American-flavored El Espiritu Jibaro. In 2008, Rudd featured vocalist Sunny Kim on Keep Your Heart Right. A year later, he released the trombone album Trombone Tribe, featuring a bevy of players including Josh Roseman, Wycliffe Gordon, and others. In 2011, Rudd celebrated his 75th birthday with the release of The Incredible Honk.