To change the ratio of items to pokemon spawned, open up /loot/rock_mash_loot.set, scroll to the bottom, and edit 'percentage.'

Prior to realizing this, I thought the way to increase the proportion was to go into all the pokemon spawning files for rocksmash and delete the pokemon spawning data and save it with an empty encounter table. Then no items ever spawned and I figured it was all terribly bugged.

I lifted the item chance to 100% once I found it, but no items ever spawned.

Turns out, you need to have at least one valid pokemon encounter table, even if there's a 100% chance of getting an item. Without some data in at least one, nothing will ever spawn.