I thought the reason it didn't go away was that I equipped Brok's Smithy as well (How I got two relics equipped, I have no idea.) Neither went away but the Broks Smithy didn't get me anything, so it's your call as to if that one will be wandering off as well.
A child of five would understand this. Send someone to fetch a child of five. - Groucho Marx