In this thesis we investigate how to improve the performance of MU-MIMO wireless system in terms of achieving Shannon capacity limit and efficient use of precious resource of radio spectrum in wireless communication.First a new suboptimal volume-based scheduling algorithm is presented, which can be applied in MU-MIMO downlink system to transmit signals concurrently to multiple users under the assumption of perfect channel information at transmitter and receiver. The volume-based scheduling algorithm utilises Block Diagonalisation precoding and Householder reduction procedure of QR factorisation. In comparison with capacity-based suboptimal scheduling algorithm, the volume-based algorithm has much reduced computational complexity with only a fraction of sum-rate capacity penalty from the upper bound of system capacity limit. In comparison with semi-orthogonal user selection suboptimal scheduling algorithm, the volume-based scheduling algorithm can be implemented with less computational complexity. Furthermore, the sum-rate capacity achieved via volume-based scheduling algorithm is higher than that achieved by SUS scheduling algorithm in the MIMO case.Then, a two-step scheduling algorithm is proposed, which can be used in the MU-MIMO system and under the assumption that channel state information is known to the receiver, but it is not known to the transmitter and the system under the feedback resource constraint. Assume that low bits codebook and high bits codebook are stored at the transmitter and receiver. The users are selected by using the low bits codebook; subsequently the BD precoding vectors for selected users are designed by employing high bits codebook. The first step of the algorithm can alleviate the load on feedback uplink channel in the MU-MIMO wireless system while the second step can aid precoding design to improve system sum-rate capacity.Next, a MU-MIMO cognitive radio (CR) wireless system has been studied. In such system, a primary wireless network and secondary wireless network coexist and the transmitters and receivers are equipped with multiple antennas. Spectrum sensing methods by which a portion of spectrum can be utilised by a secondary user when the spectrum is detected not in use by a primary user were investigated. A Free Probability Theory (FPT) spectrum sensing method that is a blind spectrum sensing method is proposed. By utilizing the asymptotic behaviour of random matrix based on FPT, the covariance matrix of transmitted signals can be estimated through a large number of observations of the received signals. The method performs better than traditional energy spectrum sensing method. We also consider cooperative spectrum sensing by using the FPT method in MU-MIMO CR system. Cooperative spectrum sensing can improve the performance of signal detection. Furthermore, with the selective cooperative spectrum sensing approach, high probability of detection can be achieved when the system is under false alarm constraint.Finally, spectrum sensing method based on the bispectrum of high-order statistics (HOS) and receive diversity in SIMO CR system is proposed. Multiple antennas on the receiver can improve received SNR value and therefore enhance spectrum sensing performance in terms of increase of system-level probability of detection. Discussions on cooperative spectrum sensing by using the spectrum sensing method based on HOS and receive diversity are presented.