MakeSfxCA may produce dlls not following PE / COFF spec leading to Error 1723

[wixbot]

In rare but easily hand-crafted cases MakeSfxCA produces native dlls not following the PE / COFF spec. In practice this leads to very hard to find errors. When trying to run an affected custom action an error 1723, i.e. "A DLL required for this install to complete could not be run", pops up.

Recipe to reproduce the bug

1. Construct a CA, e.g. "HavocAction", having three exported entry points (i.e. marked with "CustomAction" attribute), named "HavocEntryPointa", "HavocEntryPointB", "HavocZappEntryPoint" (mind the exact spelling). The methods may just return "ActionResult.Success".
2. Come up with simple setups, a) invoking just "HavocEntryPointa", b) invoking just "HavocEntryPointB"
3. ==> Setup "a)" will work, Setup "b)" will fail
4. Uncomment the "CustomAction" attribute on "HavocZappEntryPoint" and the behaviour is inverted, i.e.
5. ==> Setup "a)" will fail, Setup "b)" will work

Further analysis

When you dump the wrapped CA.dll-file with

dumpbin /Exports HavocAction.CA.dll

you get something like (excerpt)

    125   7C 00003A36 
    126   7D 00003A4C HavocEntryPointa
    127   7E 00003A62 HavocEntryPointB
    128   7F 00003A78 HavocZappEntryPoint
    129   80 000042FC zzzEmbeddedUIHandler
    130   81 000043B8 zzzInitializeEmbeddedUI
    131   82 0000467F zzzShutdownEmbeddedUI
    132   83 00003AA5 zzzzInvokeManagedCustomActionOutOfProcW

This is wrong (search for "pecoff_v83.docx", cf. http://stackoverflow.com/questions/11107095). The entries are supposed to be sorted (by ASCII) in order to do a binary search
when loading methods from the dll (the entries "HavocEntryPointa" and "HavocEntryPointB" are
interchanged).

My educated guess is, when loading code from the dll the binary search fails, resulting in the error.
Due to the nature of a binary search, removing "HavocZappEntryPoint" inverts the effect.

Remark

1. This bug seems to be very old (at least Wix 3.5 up to Wix 3.8 is affected)
2. It is happening in practice and hard to analyze (cf. http://stackoverflow.com/questions/11974539)
3. If requested, I'll post (how?) an almost minimal, self-contained example demonstrating the bug

Originally opened by eike

[wixbot]

Release changed from v3.8 to v3.x

[wixbot]

If I had to guess, changing line 519 of src/DTF/Tools/MakeSfxCA/MakeSfxCA.cs to

Array.Sort<string>(slotSort, slotCount - entryPoints.Count, entryPoints.Count, String.CompareOrdinal)

would fix this. I don't have time to change and test that now but I encourage someone to try it.

Originally posted by wwahammy

[wixbot]

Originally changed by rseanhall
AssignedTo set to shall

[wixbot]

@eiki Thanks for the in depth analysis, it made fixing the bug very easy.

@wwahammy Almost :)

Pull request: wixtoolset/wix3#141

Originally posted by rseanhall

[wixbot]

Originally changed by rseanhall
Release changed from v3.x to v3.9
Resolution set to fixed
Status changed from Open to Resolved

[wixbot]

Either this is not fixed in 3.9, or there is another issue at play. I cannot seem to isolate it into a sample project, but consider the following outputs from DumpBin against working and non-working *.dll's:

Non-working:

     84   53 000036D0 BackupDataXml
     85   54 000036E6 CachedPropertiesCleanup
     86   55 000036FC CachedPropertiesRollback
     87   56 00003712 CheckDatabaseServerExists
     88   57 00003728 CheckPSR
     89   58 0000373E CreatePSRTask
     90   59 00003754 CreateSQLFlatList
     91   5A 0000376A DecodeProps
     92   5B 00003780 DownloadPsTools
     93   5C 00003796 DropDB
     94   5D 000037AC EncodeProps
     95   5E 000037C2 EncryptConfig
     96   5F 000037D8 ExtractPsTools
     97   60 000037EE GetSecurityGroups
     98   61 00003804 InsertGroupAccess
     99   62 0000381A ProcessingOnSharedFolderSettings
    100   63 00003830 QuerySQLList
    101   64 00003846 QueueCachedProperties
    102   65 0000385C ReadCachedProperties
    103   66 00003872 RemoveINSTALLDIR
    104   67 00003888 RestartWindowsService
    105   68 0000389E RestartWindowsServiceRollback
    106   69 000038B4 RestoreDataXml
    107   6A 000038CA SFDC_CheckClientID
    108   6B 000038E0 SharedFoldersCleanup
    109   6C 000038F6 SharedFoldersInit
    110   6D 0000390C SharedFoldersInstall
    111   6E 00003922 SharedFoldersRollback
    112   6F 00003938 ShowClientName
    113   70 0000394E SQLQueryDatabases
    114   71 00003964 SQLValidate
    115   72 0000397A StartAppPool
    116   73 00003990 StopWindowsService
    117   74 000039A6 TestError
    118   75 000039BC UpdateApplicationInformation
    119   76 000039D2 UpdatePSRConfiguration
    120   77 000039E8 ValidateAnonymousAuth
    121   78 000039FE ValidatePasswordCA
    122   79 00003A14 ValidateShares
    123   7A 00003A2A WriteCachedProperties
    124   7B 00003A40 XMLSearch
    125   7C 00003A56 ZipFilesCleanup
    126   7D 00003A6C ZipFilesInit
    127   7E 00003A82 ZipFilesInstall
    128   7F 00003A98 ZipFilesRollback
    129   80 00004327 zzzEmbeddedUIHandler
    130   81 000043E5 zzzInitializeEmbeddedUI
    131   82 000046AE zzzShutdownEmbeddedUI
    132   83 00003AC5 zzzzInvokeManagedCustomActionOutOfProcW

Working (Zip* actions removed):

     88   57 00003728 BackupDataXml
     89   58 0000373E CachedPropertiesCleanup
     90   59 00003754 CachedPropertiesRollback
     91   5A 0000376A CheckDatabaseServerExists
     92   5B 00003780 CheckPSR
     93   5C 00003796 CreatePSRTask
     94   5D 000037AC CreateSQLFlatList
     95   5E 000037C2 DecodeProps
     96   5F 000037D8 DownloadPsTools
     97   60 000037EE DropDB
     98   61 00003804 EncodeProps
     99   62 0000381A EncryptConfig
    100   63 00003830 ExtractPsTools
    101   64 00003846 GetSecurityGroups
    102   65 0000385C InsertGroupAccess
    103   66 00003872 ProcessingOnSharedFolderSettings
    104   67 00003888 QuerySQLList
    105   68 0000389E QueueCachedProperties
    106   69 000038B4 ReadCachedProperties
    107   6A 000038CA RemoveINSTALLDIR
    108   6B 000038E0 RestartWindowsService
    109   6C 000038F6 RestartWindowsServiceRollback
    110   6D 0000390C RestoreDataXml
    111   6E 00003922 SFDC_CheckClientID
    112   6F 00003938 SharedFoldersCleanup
    113   70 0000394E SharedFoldersInit
    114   71 00003964 SharedFoldersInstall
    115   72 0000397A SharedFoldersRollback
    116   73 00003990 ShowClientName
    117   74 000039A6 SQLQueryDatabases
    118   75 000039BC SQLValidate
    119   76 000039D2 StartAppPool
    120   77 000039E8 StopWindowsService
    121   78 000039FE TestError
    122   79 00003A14 UpdateApplicationInformation
    123   7A 00003A2A UpdatePSRConfiguration
    124   7B 00003A40 ValidateAnonymousAuth
    125   7C 00003A56 ValidatePasswordCA
    126   7D 00003A6C ValidateShares
    127   7E 00003A82 WriteCachedProperties
    128   7F 00003A98 XMLSearch
    129   80 00004327 zzzEmbeddedUIHandler
    130   81 000043E5 zzzInitializeEmbeddedUI
    131   82 000046AE zzzShutdownEmbeddedUI
    132   83 00003AC5 zzzzInvokeManagedCustomActionOutOfProcW

Originally posted by caryroys
Status changed from Resolved to Untriaged

[wixbot]

@caryroys:

1. Consensus was that your issue is probably not the same as the original bug as the strings shown by dumpbin are in the correct order.
2. Please open a new bug and post a verbose log demonstrating the problem.

It's possible there's a related problem but you're not hitting any known limits.

Status changed from Untriaged to Resolved

[urielka]

@wixbot I am having a similar issue using v3.10, I have three methods that start with Remove:

RemoveRegisteryKeys
RemoveC....
RemoveStartupShortcut

When I added RemoveStartup it started to report DLL missing error, changed the name to ZKK and it worked, after that changed to DeleteStartupShortcut and all is good.

[rseanhall]

@urielka Please create a separate issue with an MSI log showing the error.