The Sailfish xmpp client integrates perfect into your contacts, but doesn't work with your own jabber server as long you are using self-signed certificate. This is because SailfishOS even checks per default the server certificate, which is great! But there is no GUI to add you own CA or certificate, so you don't get any connection to an server which uses an unknown certificate. IMHO commercial Certificate Authority are broken by design, but that's another story :). The usual answer for a working jabber is disable the certificate check for your xmpp. You are disabling every feature of ssl (which is not much as long over hundrets of CA are in your chain). With checking you have a little bit more secury than nothing. As alternative you can add the certificates to the telepathy chain. The jabber client (telepathy-gabble) have an additional folder of certificates. telepathy add certificates from /home/nemo/.config/telepathy/certs/ before loading all certificates from /etc/pki/tls/certs/ca-bundle.crt. Just put it into that folder.
# mkdir -p /home/nemo/.config/telepathy/certs/
# cp /home/nemo/Downloads/jabber-ca_or_cert.crt /home/nemo/.config/telepathy/certs/
I've re-added my jabber account to make the certificate work. :Maybe a reboot will apply your certificate as well.