Actually the shoes will cut the drums. If you have any groves or defects it will be cut into the drum.
The brake friction material although more brittle, is actually harder than the drums. That is why most premium machine shop cutting and finish tools are composites rather than steel. Also...