The library's failure to check if certain message types support a particular flag, the HDF5 1.8.16 library will cast the structure to an alternative structure and then assign to fields that aren't supported by the message type and the library will write outside the bounds of the heap buffer. This can lead to code execution under the context of the library.
Gentoo Linux Security Advisory 201701-13 - Multiple vulnerabilities have been found in HDF5 which could lead to the arbitrary execution of code. Versions less than 1.8.18 are affected.