The next-generation of radio facilities will produce huge volumes of data and the use of deep learning methods seems inevitable. However, in general these models produce overconfident predictions and provide no uncertainty estimates. In this work we use variational inference (VI) as an approximation to Bayesian inference and present the first application of a VI-based approach to morphological classification of radio galaxies, using a binary FRI/FRII classification. We show how posterior uncertainties on model predictions can be recovered and find that on average model uncertainty is correlated with the degree of belief of the human classifiers who curated the data set. Additionally, to reduce the computational and storage cost of these models at deployment, we test model pruning strategies and find that a Fisher information based metric allows for a higher proportion of the fully-connected layer weights of the network to be pruned, by up to 60%, compared to a SNR-based metric without compromising on model performance. We find that model uncertainty is reduced for both pruning methods. Finally, we show that our model experiences a cold posterior effect and examine whether this effect is due to model misspecification. We observe no difference in model performance on testing a hypothesis for mitigating this effect and conclude that model misspecification is not the major contributing factor to the cold posterior effect observed in our work.