The world of financial services is being upended by new technologies—from virtual currencies and blockchain to peer-to-peer lending and enhanced mobile banking—that are capturing customers, as well as the attention of Wall Street investors and industry regulators.