problems transferring a call after announcing a call

I am one of two people answering the phones. We both randomly & repeatedly have problems transferring a call.  Upon answering the call, I will use the soft transfer button and dial the extension, announce the call to see if they would like that person or if I need to (*55 for voicemail).  When they accept the call and I hang up it has been bouncing back to me and ending up on my speaker (taking in office conversation).  It is very frustrating. There is no rhyme or reason as to when and why it will do this.