Sekar.

I never said source based dedup is only theory or that it is only about checksum matching. My focus is on a much higher level. All methods are based on some kind of matching whether you call it checksum or signature or by some other name. Again my focus is not to get into details of fixed block length comparison or variable block length comparison etc. Those are details to make it more and more optimized and efficient.

What I said was that I am not fond of the source based dedup implementation because instinctively I felt what is good in theory may not be as good in practice. I may be wrong about that. When we implement dedup in StoreGrid we will consider all options and do what we think is practical and what works with least amount of headache.

The blog post is just to discuss some options at a high level rather than to get into details of the implementation.

Sekar

I agree, data deduplication is not a suit all technology. The type of data your customers deal with, it may or may-not be good for deduplication. You would have to check if the practical results are better than simple zip.

But, the source based deduplication for catching duplicates across the clients is not theory. Its been put to practice by some products. Avarmar, I checked recently also does it. But pure-disk is pure crap, you won’t find a single happy customer of pure-disk.

Deduplication now is much more than block checksum matching.

Various technology changes and optimizations have actually helped (in some cases) deliver 99% better bandwidth and storage utilization than traditional backups.

But, first you would have to check that how many duplicates does you data contain.

Second, ensure that the deduplicated data can be 100% recovered without any chance of failure, else the point of redundancy is gone.

Jaspreet