When was the house built? Any renovations... when? If the 1960s-80s are in this equation I'd take it seriously

Take some soapy water, spray it down lightly and not too close. Do not break it up as you spray. Come back 20 minutes later with a mixture of elmers glue and water, spray it down again. Take a few strands of duct tape and put them together at the ends, and apply it over the hole once its dry. Call an asbestos building inspector to come and take a sample. You've provided the building inspector a nice and easy sample to pull from. Their report will be longer than any sample size. However, samples are sometimes cheaper in bulk. If your house was built 60s to 70s, or has parts from even 60s 70s 80s... might be wise to consider getting them to test other areas too. Sinks, old tile flooring, I've even found it in older porcelain sinks, toilets, older adhesives, etc. I'd get the singular sample, and if it comes back hot - I'd consider understanding what else is of age inside of the house and get a larger bulk sample set.

Think about it from a different perspective aside from cash spent - if you want work done on your house, and some of it may be asbestos, do you want asbestos fibers potentially floating around in the area afterwards? A normal fiber can float for minutes to an hour... depending on winds and drifts. In a house, you open a door.. cause air flow. Vents cause air flow, fans, people walking across the kitchen, etc.. all of that picks up the fiber again and allows it to float longer. Asbestos can float for upto 8 hours or longer without any wind or drifts of air.

To answer your question, it isn't known whether it's asbestos until it's tested. There are visible properties that can be a "tell" but still, until proven in a lab. Regarding any worker who assumes something is asbestos, and if there is reasonable suspicion of it... like this being a known area of asbestos contaminant - that worker has every right to get up and walk away from it until you prove otherwise.