What to do with your annoying brother-in-law?

There is only one problem in my happy relationship with my partner - its his younger brother. He is 31 and still lives with his dad (they lost their mom 2 years ago), never had a separate, grown-up life - his serious partners moved in with the family, together with his parents. My bf however, was absolute independent from the age of 18, that's one thing why he became my man I believe. But Little Brother (LB) always been mommy's little boy, their mum spoiled him very much... and now that she's gone of course he became vulnerable all of a sudden, and on the top of that last year his gf left him. My bf took full responsibility of him afterwards, and thats when I had to experience LB's character: he is an intellectual guy, really smart (wanna-be lawyer), but his personality is just unbearable. Extremely selfish, material, aggressive, and never realize if he is disturbing us with his presence. The last few weeks there was hardly any days when he didn't come over - you know he just pops in for "5 mins", then he stucks in our flat for hours, talking and talking and still talking. Me and my bf are both really calm and never talk too much - we love quiet and harmony when we are home, I kinda got used to it, so constant talking (and complaining about government, celebrities, basically about everything) is getting on my nerves... but my bf apparently loves him so much that he wouldn't tell him off, would do everything in his favour (maybe still cause of their mum's death? don't know), even if he might know that I would rather be only with him sometimes (only 1 or 2 evenings a week without friends or relatives, is it that much?). I cannot tell all this to my bf, because I don't wanna make him feel that he has to choose between two people he loves so much... I am dreading that he would hate me if I ever tell him all the offences and annoying things LB doing to me all the time. I can't see why my bf wouldn't try to help his brother with making him a bit more independent and grown-up, he is the elder one, and the only one who LB listens to... he has the power to change his ways, to help him, but he seems to treat him like a little boy, always protecting him, and never say no to him. Till what point do I have to bear this?? Please help if you have any experiences like this, or ideas what I could do...