This improves performance a lot compared to decoding the whole block. Eventually we may implement a cache, but threading makes that complex to make properly fast.