Answer by user147505 for Fast way to get checksum for all files within a huge nested directory

This may be a solved problem, but it's too specific to have any standard tool in the Unix/Linux world. Your question contains a large part of the answer. You need a database, or more precisely, you need an index of checksums. And also you need a component that will add, update and check new files against this index. I think you will have to implement it yourself and the natural place of implementation will be in the upload mechanism (eg. a web page).