Now, my own collection is something like that too. I don't own just one plaid, or 3 flannels, or only medium tone floral fabrics. I have a full selection of color and value in all types of quilting fabric so my buying may be a little different than yours. I am not building a stash as much as I'm filling in gaps.
If you are using a lot of different fabrics in a quilt, quarter yards are probably enough, but I still like half yards so I don't use it all up. If you want it all gone when you are done cutting for a project, then buy quarter yards. If I run out of a fabric I just look through my stash for a similar one to continue cutting. I would rather use 5 reds than just 1 if the quilt calls for red.
Think of the kind of quilts you like to make. I not only like a variety of color in my quilts, I also need a variety of values. I don't just buy my favorite color in one value....but every color.....every value (from light to dark).
The main thing when shopping is to know the selection you already own. Know what colors or values are missing and be on a mission to fill in with them. That means organizing your fabrics and having them where you can see them. Don't keep fabric you have purchased in the bag from the store and toss it on the pile. (yes, some people do that!).
Also, think about what type of quilts you like to make. If you are always adding white (or any other light) as a contrast to every piece you sew you might not need as many values. If you are doing a blend (colorwash, Trip Around the World) you need every step of value. Know thyself is the key to having a good usable stash, a stash that you can shop at midnight and be ready to start a quilt the next day.
I also addressed this subject back in 2007, the year I started my blog. Here is one post and another. The main thing that has changed since then is that my income is way down from what it was then so half yards and sale fabric are my 2 best choices now.