It all depends on your preferences and how small of a board you have ever ridden. The thing when among to small wave boards, going too small will defeat the purpose of it when it comes to suit the low end of the small. Also by shortening the rail line some changes use be made in order to compensate.
In my case I'm 84Kg and like my boards on the 30lts range. When I got my first potato I was sckeptic of going too small since my smaller board at the time was a 5'8". So I ordered a 5'6" and after a few surfs I found it really over volumend. Fun on the really small stuff since I was catching anything and everything but when it came to performance it was just dragging a bit too much. That's why I changed for a 5'2" and could not be happier.Perfect for my day to day small stuff
It all depends what you want the board for and what conditions you regularly expect to surf it on.
What are your dims?.