This is gonna take some explaining but here it goes.
I need to author a multiple instance MSI which installs dynamic instances - i.e. instances defined when the user installs the package, not hard-coded in the MSI file. Now, I've already gone through the pains of creating a bootstrapper and using the MSI api to dynamically create a transform (MST) and apply it to the original MSI; after much tinkering, install and uninstall works fine (I'll post details as they're needed).
Basically, the MST contains transforms for ProductCode, ProductName, PackageCode (in Summary Info), changes GUIDs for all components (otherwise uninstall fails in silly ways) and the install location is protected from conflicts by the bootstrapper. Also, the bootstrapper starts the install with the command line parameter MSINEWINSTANCE=1, as detailed here.
However, I'd also like to upgrade an installed instance (via major upgrades), which is the main reason why the UpgradeCode is unique (or so I thought). However after I increment the MSI version and try to start it (again via the bootstrapper and passing in the desired instance's ProductCode via the MSIINSTANCEGUID property), it fails; the log says:
=== Verbose logging started: 12/13/2011 17:43:56 Build type: SHIP UNICODE 5.00.7601.00 Calling process: C:\Windows\SysWOW64\msiexec.exe ===
MSI (c) (5C:D0) [17:43:56:120]: Font created. Charset: Req=0, Ret=0, Font: Req=MS Shell Dlg, Ret=MS Shell Dlg
MSI (c) (5C:D0) [17:43:56:120]: Font created. Charset: Req=0, Ret=0, Font: Req=MS Shell Dlg, Ret=MS Shell Dlg
MSI (c) (5C:34) [17:43:56:120]: Resetting cached policy values
MSI (c) (5C:34) [17:43:56:120]: Machine policy value 'Debug' is 2
MSI (c) (5C:34) [17:43:56:120]: ******* RunEngine:
******* Product: D:\TestArea\AMLDC.msi
******* Action:
******* CommandLine: **********
MSI (c) (5C:34) [17:43:56:120]: Machine policy value 'DisableUserInstalls' is 0
MSI (c) (5C:34) [17:43:56:135]: MainEngineThread is returning 1625
=== Verbose logging stopped: 12/13/2011 17:43:56 ===
and a UI message pops up saying that 'the system administrator has set policies to prevent this installation'. Obviously that's not true (the policies would appear in the log and a rather more explicit message would be provided).
The 1625 error code seems to correspond to "ERROR_INSTALL_PACKAGE_REJECTED".
Any ideas to what I could try next? I'm thinking what the MSI engine should try to do in this case is examine the UpgradeCode, apply the original transform (which should be cached and reachable via the product code I give it via the MSIINSTANCEGUID parameter). However it's clear the engine never reaches that stage (it should be logged in the log file, right?)
Sigh, this has been much more painful than it should have been.
Edit: some time later...
Quick note on changing component GUIDs: it's only really necessary for non-file components (I have some registry entries I use to keep track of instances). If I don't change their GUID they aren't cleaned up correctly at uninstall, as detailed here. For files it works fine if the key paths are different, and I've verified that by only changing the registry component ones in my code.
So I've learned in the meantime that major upgrades probably won't work for me unless I change the UpgradeCode for each instance (because FindRelatedProducts only looks at the UpgradeCode, I think), and I tried something else before going there: minor upgrades.
Starting the installer for a new version with /fvamus together with MSIINSTANCEGUID={existing-instance-product-code} seemed to work, right up until I've tried adding a new file to the package (which I expect to happen in the future)... when of course it doesn't work (the component for the new file is not installed at reinstall, of course).
So I'll probably either have to change the UpgradeCode with the transform and see what that implies, or mess around with the output property of FindRelatedProducts through some custom actions and see if I can convince major upgrades to work that way. However the initial problem (the 1625 error) was precisely with major upgrades so not sure if I can do something about it without knowing the cause. To be entirely clear: what I pasted above is the entirety of the MSI verbose log, it doesn't seem to do anything before returning with error 1625. I also tried removing all the rows in the Upgrade table of the MSI and there was no change in behavior.
I also can't spend much more time on this silly issue so if nothing else works I'll be forced to do a silent uninstall followed by a regular install with the same settings. I cringe at the thought but if it can't be helped...
Edit: in all fairness, it probably would have gone faster if I didn't start on the MSI path altogether and coded my own installer from absolute scratch, with gzipped streams and simple xcopy. Even with a msbuild task that would have compressed the files from visual studio or something.