The Dark Side is at work with Luke. The Dark Side will make you do things you would otherwise not.
Anakin- Slaughters the sand people, kills a bunch Jedi including little kids, tries to kill his wife and unborn children, tries and later kills his best friend.
Luke- arrogantly kills Jabba and henchmen, tries to kill a man who's asking him to kill him, tries to kill his father, briefly contemplates killing his nephew.
Ben- Kills his uncle's students and turns the rest, stabs his father through the chest.
So is Luke out of character when he comes into Ben's room? Yes, and no. The Dark Side will make you do things "out of character".
Also remember, Master Yoda says that the path to the Dark Side is "Fear. Fear leads to anger, anger to hatred, hatred to suffering" In the scene from TLJ, as in everytime Luke trips up, you see Luke go through fear, anger, hatred. In essence, if you are letting your emotions control you, then chances are the Dark Side is at work in you. But if you are calm and centered, and you are in charge of your emotions, then Light Side can freely flow through you.
