That's weird. If it was a carbureted unit I would say float stuck open but with Fuel Injected unit it should shut fuel down when turned off. Definitely change oil, maybe fuel filter is a good place to start. If you got fine solids thru into injectors they might be passing gas thru. Really stumped me with that on though.