It is important to choose the appropriate editor type for your BookVars file. Generally the internal editor is fine, unless you have a very large number of variables or plan to have many variable groups. The internal editor is limited to 30K of data (this is about 1000 variable definitions).
The problem arises when high-ASCII characters (those with a decimal ASCII value above 127) are used in variable names or definitions. When writing to a text file, FrameMaker writes high-ASCII characters in a way that is not understood by external text editors. Therefore, when writing a BookVars file that is to be used by an external editor, a transformation is applied so the characters are written correctly. When writing to a BookVars file that is to be used by the internal editor, no transformation is applied.
Even if you don't think you're using high-ASCII characters, you may be. Specifically, the “para” (¶) symbol is a high-ASCII character. If you are using formatting in your variable definitions, you most likely are also using the "<Default ¶ Font>" building block. If you are, then you are using high-ASCII characters and should use caution.