The most obvious way to get a muted sound is obviously by using a mute! A mute clamps onto the bridge and dampens down the vibrations from the strings, thus deadening and dulling the sound (mostly by stopping higher harmonics sounding). The term used is con sordino (or sord.).
Bartok String Quartet No. 5 Scherzo
Another way of getting a more muted sound is to ask for a line to be played on a lower string than normal. Playing high up on the lower strings produces a more mellow but also richer sound. The commonest example of this is asking violinists to play melodies on the G string (sul G). In this example the first violin stays on the G string whereas without the instruction they would move onto higher strings:
Sibelius Voces Intimae Fourth movement
A related effect can be gained by putting high melodic lines in the viola or cello, which produces a very different quality of sound. In this example the cello plays a tune that is in the violin register. The sound is slightly less bright and, at this volume a little forced (notice the fabulous slides or glissandi at the end of the recorded extract).
Bartok, String quartet No. 6, second movement
A glissando is when you slide rather than move cleanly between notes. In this example the effect is rather light-hearted but simultaneous glissandi (as in the second movement of Bartok’s sixth string quartet above) can be powerful and dramatic.
Shostakovich, Quartet op. 49, fourth movement