I already uninstalled it, but I think there was 'Project setting' if you right click on the tracks header.

according to th Wax user guide, at page 31:
"The quality options displayed in the toolbar above the preview window are
used for ALL rendering purposes, not just the preview display. So before
you render the full project in your host NLE remember to switch to best
quality in Wax (or have best quality enabled all the time). "
That's here:

But I can't find the resolution settings that I know are in there somewhere.

In the standalone app, at Project/Settings (hit F3) you have options to set the frame size and compression output options.
I think that when used as plugin it assumes your project settings.

That was my next thought.

must be that, as these options are not present in pluguin mode. Not even pressing F3