I've tried to setup auto-follow up messages that get the assignee slack ID so they can be tagged in follow-up messages.

It doesn't seem to be functioning as expected. Might be a bug that needs to be fixed.