Every jiu jitsu gym has outliers when it comes to size and strength. I have seen places where most of the population weighs less than 170 pounds, and in those I was the outlier. I have seen places where most of the population was above 220 pounds. It is important to know how to roll with, and indeed to roll with, people who are not your size for many reasons.
Perhaps the most important reason to roll with people who are bigger or smaller than you is to test your techniques against those people. There are certain chokes that work much better against someone with a beefy neck, but those same chokes will not work against someone with a skinny little pencil neck.
On the other hand, that little guy with a pencil neck may be easier to arm bar because one of his arms is not stronger than your whole upper body (unlike the big beefy fellow who is easier to choke.) Different body types respond differently to different kinds of submissions and submission attempts.
Roll with everyone of every body type. If you weight 145 pounds and you duck guys who weigh 270 pounds simply because they are too big or too strong, you are only doing that to protect your ego. Granted there are acceptable reasons to duck someone much bigger than you, namely if they are risky to roll with because they lack self control, but that is something you can determine fairly quickly into a roll.
In a street fight, you do not get to pick the size of your opponent, in an absolute division you do not get to pick the size of your opponent. The more you roll with people, who are outside of what you are used to dealing with, the better you will be at dealing with people like that when the time comes to deal with them.
When you roll with someone who is bigger and stronger than you are, you are able to explore the realities of your technique. Does your technique REALLY work? Or do you have to make some adjustments so that they work against someone who is able to resist? It is easier to force a technique against someone 30 pounds lighter than you are than it is to force that same technique against someone 30 pounds heavier.
Being able to adjust how you roll is important as well. Just as you should not duck people who are much bigger than you unless they pose an increased risk of injuring you, when you roll with people who are much smaller you should make sure to adjust your intensity appropriately. If you weigh 230 pounds and are rolling with a training partner who weighs 160, maybe work on your defense, and if you do wind up in a dominant position and secure a submission remember that the weight disparity puts them at a greater risk of you injuring them.
The more opportunities you take to roll with people of different body types than you are accustomed to the more you will be able to test the limits of your techniques. When I roll with my 300+ pound training partners, I need to be far more aware of positional advantages and hierarchy, and I need to determine what strategies will and will NOT work.
If you find rolling with people much bigger than you difficult, you should roll more with people who are much bigger than you. Being smaller than another person is not necessarily a disadvantage if you are able to form an intelligent strategy. When it comes to grappling every body type has specific advantages and disadvantages. And sometimes those advantages show up when dealing with people of different body types.
Probably two of the finest examples of jiu jitsu practitioners who embrace their size as an advantage are the Miyao brothers, who as purple and brown belts often entered absolute divisions and absolutely crushed them. Just look at this video of Joao Miyao defeating superheavyweight stud Carlos Farias last year in the absolute division at the IBJJF Las Vegas Open.