How do you get permanent marker off of the mortar between the bricks on your fireplace?

My 3 year old colored the mortar space between the bricks on our formal fireplace. I want to clean it off but I am not sure how. Please help!