WinHelp is one of Windows least documented features. As most Windows programmers are aware these days, many aspects of Windows do go undocumented by Microsoft, but for the rest of Windows, this tends to be less vital functionality. With WinHelp, many of the important aspects are kept undocumented. Whether this is the result of laziness or intentions, is unclear. WinHelp 4.0 is more fully documented than WinHelp 3.1 was, but its documentation still has holes.
Luckily, a few people have taken a lot of time to discover these undocumented features and made them available in one way or another. In particular, Ron Burk (editor of Windows/DOS Developer's Journal) has spent a lot of time working with WinHelp and he is arguably the most knowledgeable WinHelp expert outside of Microsoft (and maybe including Microsoft).
Below are a list of articles and other sources of undocumented WinHelp features listed in alphabetical order by author.
Ron Burk, "Automatic Help Topic Extraction", Windows/DOS Developer's Journal, February 1993
In this article Ron writes a .DLL that allows you to insert an "extract" button in your help file that will extract a WinHelp topic to a file. He also discusses the fact that WinHelp executes some macros asynchronously.
Ron Burk, "Automatic Help Topic Printing", Windows/DOS Developer's Journal, March 1993
In this article Ron writes a .DLL to allow you to print all topics in a WinHelp .HLP file.
Ron Burk, "Spying on WinHelp," Dr. Dobb's Journal, June 1993
In this article, Ron discusses the WM_WinHelp message used by the WinHelp() API call. The article describes some uses of this message and the parameters associated with it.
Ron Burk, "Controlling WinHelp's Always-on-Top State", Windows/DOS Developer's Journal, November 1993
Ron shows how to control the Always-on-Top State of WinHelp from your applications.
Pete Davis, "Documenting Documentation," Dr. Dobb's Journal, September 1993 and October 1993
In this two part article, a detailed, though not quite complete, description of the WinHelp file format is provided. It discusses WinHelp's internal file system and the structures of the internal files in a .HLP file.
Pete Davis, ".mrb and .shg File Formats", Windows/DOS Developer's Journal, February 1994
This article contains a fairly complete description of the .mrb (multi-resolution bitmap) and .shg (segmented hyper-graphic) file formats. These files are used by WinHelp to display bitmaps. The file formats are almost identical between the two files, though their purposes are quite different.
Jim Mischel, "Undocumented WinHelp Macros", Windows/DOS Developer's Journal, January 1994
Jim describes a lot of undocumented WinHelp 3.1 macros such as ExtAbleItem(), ExtInsertItem(), ExtInsertMenu(), ResetMenu(), JumpHash(), and PopupHash(). These have been documented in the WinHelp 4.0 help.
Undocumented Windows File Formats, by Pete Davis and Mike Wallace, R&D Books
This book is a good resource for programmer's seeking to understand the internal structure of WinHelp files, including .HLP, .SHG, .ANN, and .BMK file formats.