I see what you mean, this option is not working properly when the Time column is a number. This should be something we can fix.
For now, change the column type for your Time column to a text column (use the gear menu on the column object or control-click on the column header) then you should have the option of sorting according to one of these columns. For example, use ’10 up arrow’.