There's an exploitable buffer overflow in the current version of
MPlayer (v1.0rc1) which can be exploited with a maliciously crafted
video file. It's hidden in the function DMO_VideoDecoder() in the
file loader/dmo/DMO_VideoDecoder.c. The variable format->biSize gets
its value directly from the video file, and thus can have any value
up to LONG_MAX. In line 136 it is used without any further checks as
the length argument to the memcpy() call, which can overflow the
this->m_sVhdr->bmiHeader buffer with data directly from the video file.

117         unsigned int bihs;
118
119         bihs = (format->biSize < (int) sizeof(BITMAPINFOHEADER)) ?
120             sizeof(BITMAPINFOHEADER) : format->biSize;
121
122         this->iv.m_bh = malloc(bihs);
123         memcpy(this->iv.m_bh, format, bihs);
124
125         this->iv.m_State = STOP;
126         //this->iv.m_pFrame = 0;
127         this->iv.m_Mode = DIRECT;
128         this->iv.m_iDecpos = 0;
129         this->iv.m_iPlaypos = -1;
130         this->iv.m_fQuality = 0.0f;
131         this->iv.m_bCapable16b = true;
132
133         bihs += sizeof(VIDEOINFOHEADER) - sizeof(BITMAPINFOHEADER);
134         this->m_sVhdr = malloc(bihs);
135         memset(this->m_sVhdr, 0, bihs);
136         memcpy(&this->m_sVhdr->bmiHeader, this->iv.m_bh, this->iv.m_bh->biSize);

This got fixed [1] in trunk two weeks ago.

[1] http://svn.mplayerhq.hu/mplayer/trunk/loader/dmo/DMO_VideoDecoder.c?r1=22019&r2=22204

Best,
Moritz Jodeit