I had a lot of trouble with this. The video would convert without an issue, but I couldn't get the audio to encode with AAC. I searched deep into the VLC forums and found a solution for my particular case. To start with, here is the error from the messages log in VLC:
stream_out_transcode error: cannot find audio encoder (module:any fourcc:mp4a). Take a look few lines earlier to see possible reason.When searching for info on that error, here is the working solution I found:
I'm having this exact same problem; I have never been able to transcode to mp4a without some kind of difficulty since the update to v2 (this is why I still have v1.1.12, but it has compatibility issues of its own).After changing the "strict standards compliance" from the default of 0 to -2 I was able to convert the WMV screen recording from FastStone Capture into MP4 (H264 / AAC). Good stuff!
I've managed to get it to work sometimes by going into VLC's preferences, choosing show all/advanced and going into video codecs > ffmpeg, and setting strict standards compliance to -2. I'm not sure how you would do this with the command line but there must be an option somewhere, though setting it via the GUI may be good enough. But this only solves the problem for some files, not all, even though as far as I can tell all the files I've been trying to process are identical.
The ones that give me trouble are usually ones with AC3 audio (which I try to transcode out since AC3 audio codecs on OS X are a pain in the ass), but if I transcode only the video the audio won't be passed through as normal, it will simply be missing. I don't know why this happens, but it means that instead of having a nice folder of widely compatible .mp4 files I instead have a mix of files that worked and ones I had to copy at full-size because I can't seem to do anything with them, very annoying.