Kick ass on a standardized test and get a scholarship?
Go to a lesser school and get a scholarship?
There's no reason you have to do 100k in debt. You've got the rest of your life to work on businesses. Going to school isn't going to "set you back". You'll get an enjoyable experience. In the end life isn't about the amount of money, its about your experiences. Don't forsake one because you're in a hurry.
Not to mention, if you find out you're NOT an entrepreneur(which many people aren't) it gives you a nice fallback!
Side note: You may find out you're interested in academia / intellectual pursuits. If you want to write papers on physics, you do need a PHD in it.
College may not be for everyone, but I wouldn't give up my years for anything.