Just make sure it's around your neck at all times, and if not, it's in a case.
I don't have a really good camera but DH would still get mad if something happened to mine in the same scenario, but I have taken it with me. Just be really careful with it - unless you are really clumsy you should be OK!