Is my knuckle broken?

Over the last three months, I have hit walls four different times (outside of a house wall). I realize this is obviously not the smartest thing to do but there's nothing I can do about it now. My last time was Thursday night (June 16). Normally, it's hard to bend my pinky finger the next day & it gets swollen but the two days after the incident it's fine. Today it's the second day and I cannot bend my finger without severe pain. The whole pinky knuckle is swollen solid, usually it's spongy. If I bend my pinky by using my other hand to push on it, it's bends easier but if I let off with my other hand, my pinky hurts really bad. I also get a pain down by my wrist after bending the pinky. It also feels like the knuckle is just too big to bend, it's very uncomfortable. I don't want to waste time going back to the hospital if it doesn't seem broken. Any advice?