kvisr
After reading your last post, I decided to download FSX and give it a "whack". In my testing I was able to successfully record and import 1280x960 FRAPS footage. Because of my 1920x1080 screen resolution I was unable to test the previous suggestion of 1436x1077. Yes, 1077 is a little less than 1080 but there's a little more involved than that.
The full screen options in FSX's settings dictate the full screen resolution. However, the resolution seems to carry over into the windowed mode but only as far as the aspect ratio is concerned. So what I did was set the full screen resolution to 1280x960x32. Then once the game is running, I wrote a script with AHK that resizes the window to 1296x1000. The increase in the numbers is to compensate for the titlebar at the top and border. From here the FRAPS captured footage shows up as "1280x960, FPS1" in Lightworks.
Now... in theory, you should be able to add 1436x1077x32 to the Mode entry in the FSX config file. Then run the game and once you are playing it, resize the window to 1452x1117. As you can see 1117 is bigger than 1080 and so I couldn't test it for you.
Attached you will find a ZIP containing two complied EXES. One for each resolution. The compiled scripts are there in case you don't have AHK or know how to use it. The source code (using that phrase loosely) is below.
They're only one line each
WinMove, Microsoft Flight Simulator X ahk_class FS98MAIN,,,, 1296, 1000 ; 1280x960
WinMove, Microsoft Flight Simulator X ahk_class FS98MAIN,,,, 1452, 1117 ; 1436x1077
I know for sure that the 1280x960 resolution works but I'm curious about the other one.
Good luck!