Indeed. I know the past few Windows systems won't allow you to install to external drives, but others like Unix or the new ChromeOS can.
Windows from 95 Version B and forward support it via USB, however the system itself needs to support booting from USB, else you need a Linux-based OS to bypass that. (I had a similar issue with an old system that didn't support booting via CD, so I had to use MS-DOS to run the CD-based installer on the system).