When you were born, your entire pancreas, if it had been taken out and smushed, would probably have fit inside a vial of the size that insulin comes in (most insulin vials hold 10mL of insulin and another 2 or so mL of air, and a baby's pancreas is about 9 mL- can be bigger or smaller). Almost half of your pancreas was made up of fat, with the other half being busy with the business of making digestive enzymes and hormones like insulin and glucagon.
Assuming you didn't develop type 1 diabetes or pancreatitis, your pancreas kept growing. The business part of your pancreas reached its maximum size sometime in your teens, twenties, or thirties. Then it began to slowly, slowly, slowly, atrophy.
The fat part of your pancreas grew too. Depending on how heavy you were, your pancreas may have put on a lot of weight. Unlike you, it didn't lose it. Your pancreas got more fatty as you got older, as the rest of it atrophied, and the fat part didn't.
If you were a normal or type 2 diabetic adult, your pancreas probably reached a maximum volume between 40 mL and 120 mL. After that... well, the average pancreas shrinks by about 20 mL between the ages of 30 years and 80 years old. Type 2 diabetes does not affect the size of the pancreas, although being overweight will make both the business part and the fat part of the pancreas larger, and being obese will make the fat part of the pancreas a lot larger.
If you are a type 1 diabetic... those antibodies likely attacked more than just your beta cells. About half of type 1 diabetics show mild signs of having exocrine pancreatic disease in terms of blood work (few show more than mild signs). An even higher portion have shrunken pancreata on imaging studies. The average pancreas size of type 1 diabetics is consistently small in the studies published, but the average size in different studies varies from 8 mL ( the size of a newborn baby's pancreas) to about 50 mL (the lower end of the normal range).
Source: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2680737 (I have also read through the studies linked to in this article, but the article I am linking to is my main source).