In these days of close corporate scrutiny, companies frequently investigate their own employees’ conduct.

But one highly litigious Silicon Valley company has taken the internal probe in a new direction: The board of embattled Rambus Inc. hired a pricey outside law firm to investigate its CEO’s wife.