Distinguishing transient blocks from frequently used blocks enables servicing references to transient blocks from a small fully-associative auxiliary cache structure. By inserting only frequently used blocks into the main cache structure, we can reduce the number of conflict misses, thus achieving higher performance and allowing the use of direct mapped caches which offer lower power consumption and lower access latencies. In this paper we use a simple probabilistic filtering mechanism that uses random sampling to identify and select the frequently used blocks. Furthermore, by using a small direct-mapped lookup table to cache the most recently accessed blocks in the auxiliary cache, we eliminate the vast majority of the costly fully-associative lookups. Finally, we show that a 16K direct-mapped L1 cache, augmented with a fully-associative 4K filter, achieves on average 13\% more instructions per cycle than a regular 16K, 4-way set-associative cache, and even \tilde 7\% more IPC than a 32K, 4-way cache, while consuming 70\%-80\% less dynamic power than either of them.