Calculate proportions within subsets of a data frame

You can use function ddply() from library plyr to calculate proportions for each combination and then add new column to data frame.

 library(plyr)     
 DF<-ddply(DF,.(category1,category2),transform,prop=number/sum(number))
 DF
   category1 category2 animal number       prop
1          A         X    dog     17 0.44736842
2          A         X    cat      3 0.07894737
3          A         X  mouse     18 0.47368421
4          A         Y    dog      2 0.14285714

does this produce your desired output ?

 DF$proportion<-as.vector(unlist(tapply(DF$number,paste(DF$category1,DF$category2,sep="."),FUN=function(x){x/sum(x)})));