This is totally supported by MSBuild, although Visual Studio has no built-in support for it, so it requires some hand editing. We did this on the MvcSiteMapProvider project to support MVC versions 2-5.
<ItemGroup Condition=" $(DefineConstants.Contains('MVC2')) ">
<Reference Include="System.Web.Mvc, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" />
</ItemGroup>
<ItemGroup Condition=" $(DefineConstants.Contains('MVC3')) ">
<!-- Due to the windows update MS14-059, we need this hack to ensure we can build MVC3 both on machines that have the update and those that don't -->
<Reference Condition=" Exists('$(windir)\Microsoft.NET\assembly\GAC_MSIL\System.Web.Mvc\v4.0_3.0.0.0__31bf3856ad364e35\System.Web.Mvc.dll') " Include="System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" />
<Reference Condition=" !Exists('$(windir)\Microsoft.NET\assembly\GAC_MSIL\System.Web.Mvc\v4.0_3.0.0.0__31bf3856ad364e35\System.Web.Mvc.dll') " Include="System.Web.Mvc, Version=3.0.0.1, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.Mvc.3.0.20105.1\lib\net40\System.Web.Mvc.dll</HintPath>
</Reference>
<Reference Include="System.Web.Razor, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.Razor.1.0.20105.408\lib\net40\System.Web.Razor.dll</HintPath>
</Reference>
<Reference Include="System.Web.WebPages.Razor, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.WebPages.1.0.20105.408\lib\net40\System.Web.WebPages.Razor.dll</HintPath>
</Reference>
</ItemGroup>
<ItemGroup Condition=" $(DefineConstants.Contains('MVC4')) ">
<Reference Include="System.Web.Mvc, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.Mvc.4.0.20710.0\lib\net40\System.Web.Mvc.dll</HintPath>
</Reference>
<Reference Include="System.Web.Razor, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.Razor.4.0.20715.0\lib\net40\System.Web.Razor.dll</HintPath>
</Reference>
<Reference Include="System.Web.WebPages.Razor, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.WebPages.4.0.20710.0\lib\net40\System.Web.WebPages.Razor.dll</HintPath>
</Reference>
</ItemGroup>
<ItemGroup Condition=" $(DefineConstants.Contains('MVC5')) ">
<Reference Include="System.Web.Mvc, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.Mvc.5.0.0\lib\net45\System.Web.Mvc.dll</HintPath>
</Reference>
<Reference Include="System.Web.Razor, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.Razor.3.0.0\lib\net45\System.Web.Razor.dll</HintPath>
</Reference>
<Reference Include="System.Web.WebPages, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.WebPages.3.0.0\lib\net45\System.Web.WebPages.dll</HintPath>
</Reference>
<Reference Include="System.Web.WebPages.Razor, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<Private>True</Private>
<HintPath>..\packages\Microsoft.AspNet.WebPages.3.0.0\lib\net45\System.Web.WebPages.Razor.dll</HintPath>
</Reference>
</ItemGroup>
You can't use the NuGet package manager, either. You can create different dummy projects for each configuration you want to support, install the NuGet packages there and then copy over the packages files, the references in packages.config, and the references to them in the .csproj or .vbproj file.
The non-active files show up with a yellow icon, but the project will still compile and run this way.