I’m sure you’d adapt.

I do not think so. Your directional hearing depends on the shape of your face too. This means that everybody hears directions differently, because their face is different. I think people should be able to adapt to these smaller changes over time.

But I’m not sure what I’m saying is true. Games usually have binaural sound, but it may not be effective for some people, because the head-related transfer function (HRTF) that the game uses sounds different compared to what somebody usually hears irl.

According to the sources, elves have excellent directional hearing.

Just partially. It would likely create problems with pinna cues needed for vertical and front vs back sound localization.

Over the years, the work of Robert Butler and others has demonstrated that the directionality of the pinna not only provides the primary cues for vertical localization and for preventing front–back confusions, but also provides effective cues for localizing sound in the horizontal plane (e.g., Butler, R. A., 1975; Musicant, A. D. and Butler, R. A., 1985a,b). Indeed, as noted by Butler, R. A. (1999), “We live in a world where many sounds emanating from the side are not sufficiently intense to stimulate the opposite ear”, in which case a binaural comparison only indicates that the sound source is somewhere within the hemifield of the ear receiving the sound and, in the absence of pinna cues, may result in the perception of the sound being located within the ear itself. Thus, there are situations in which the pinnae provide the only locus cues as to the horizontal and vertical location of a sound source within a hemifield.

Source: Heffner, H. E. & Heffner, R. S. (2008). High-frequency hearing.