win: return HTCLIENT from WM_NCHITTEST to prevent modal resize freeze

[Shipi1]

• When hosted as a VST3 child window in DAWs (Ableton, etc.), moving the mouse to the bottom/right window border while clicking causes Windows to enter a modal resize loop that freezes the entire plugin window.
• Cause: When MouseDown is active and move the mouse outside the window's border, WM_NCHITTEST is triggered returning HTBOTTOM/HTRIGHT/HTBOTTOMRIGHT
• Fix: wnd_proc_inner always return HTCLIENT since plugin windows are embedded and should never be user-resizable via border dragging. this tells Windows the mouse is always in the client area.
• Tested in Ableton Live 12.3.1

[micahrj]

I'm very surprised that this would be necessary. I would think that the WM_NCHITTEST message would only ever be sent to the top-level window and would not be sent to the plugin's child window in the first place. I also can't find any similar logic in the JUCE or Visage Win32 windowing code. I could be wrong, but I'm somewhat skeptical that this is the correct diagnosis or fix for the problem you are running into.

[prokopyl]

I'm very surprised that this would be necessary. I would think that the WM_NCHITTEST message would only ever be sent to the top-level window and would not be sent to the plugin's child window in the first place. I also can't find any similar logic in the JUCE or Visage Win32 windowing code. I could be wrong, but I'm somewhat skeptical that this is the correct diagnosis or fix for the problem you are running into.

It is normal for WM_NCHITTEST to be sent to the window under the cursor, even if it is a child window.
This is how you implement things like custom titlebars or the custom bigger window resize corners.

What is not normal however, is for DefWindowProcW to return HTBOTTOM/HTRIGHT/HTBOTTOMRIGHT on a window that does not have a titlebar, border or size box.

I have tried reproducing it with the open_parented Baseview example, and I cannot get it to return anything other than HTCLIENT in child mode, regardless of the circumstances.

I'm going to close this PR, as the code here seems very much incorrect to me (if nothing else, this completely breaks the non-parented mode).

However, @Shipi1 if you can provide more information about which plugin you're running (and what baseview version it runs on), or even better if you could provide a minimal example to reproduce the issue, I'll gladly accept a new issue with more detailed reproduction steps, where we can also discuss the best approach to fix it. 🙂