As cool as it seems, I see issues with your optics. Who cares about a quick change assembly when it isn't sighted in when you assemble it? I think you could benefit greatly from having two designated upper assemblies. The ability to utilize multiple upper assemblies on a single lower is the true beauty of the AR platform. Your optics will be dialed for their upper, and they will not change zero upon removal. The real fun part here is the fact that this can be done in the field, with NO tools at all.
You can do what you like, but an entire upper is the route I would take.